The BoilerIO Software Thermostat

It’s time to step up from command-line control of the heating system I’ve been working on to having a weekly schedule and temporary override function available through a UI, making the system practical on a day-to-day basis.

As always, code described here is available on github under an MIT license.

Overview

The scheduler chooses a target heating temperature throughout the day and needs to be usable by someone non-technical to show the state of the system, provide a means to enter and edit a schedule, and provide a means to change the target temperature for a fixed period of time.

system-overview

To achieve this a new daemon and web service have been added to the architecture.  Here is a summary of the responsibilities of those components:

  • The Scheduler reads the weekly schedule from the database, and sends commands to the Boiler Controller to change the target temperature at the appropriate times.
  • The SchedulerWeb REST service provides a REST API over HTTP to get a summary of system status, set or clear the temperature target override, and add and remove entries from the weekly schedule.  This is implemented using Flask in Python, with uWSGI and nginx to provide a robust service.
  • The scheduler configuration database is a PostegreSQL database holding the schedule, a cached copy of the current temperature, and the target override configuration.  It’s the primary store for this information: the REST service updates it and the scheduler uses it to control the temperature.
  • The SchedulerWeb web frontend is an HTML5/CSS3/Javascript single-page web UI that makes use of the REST service to interface with the system from the user’s web browser.

Experience and implementation notes

Rather than trying to document the entire implementation, I will instead talk about each area and some of the challenges or points of interest, and the major decisions that were taken and why.  My day job is a lot lower in the software stack than this, so there may better approaches than what I’m presenting.

The HTML frontend

alignment
1px alignment issue (red line illustrates the problem) without left padding on input boxes

The app is designed to largely follow Google’s Material Design guidelines; I hadn’t realised when I started that there are stylesheets provided by Google that can be used so I implemented the styles I needed myself.

Since the fashionable choice of web-framework seems to change frequently I was initially resistant to using anything at all fancy, and instead use traditional jQuery directly.  In retrospect, not having some of the features of a slightly higher-level framework probably made the code worse that it ought to be and in future I might look at whether a framework provides a better frontend implementation.

Modern web development has definitely come on a long way since I last attempted it: the developer tools inside Chrome are great, though there are still cross-browser issues even when only supporting latest-generation browsers (for example implementing modal dialogs is still done manually even though there’s the dialog tag now, and I had issues with alignment and appearance of certain input fields).

The REST service

The backend service uses Flask, a neat framework for web apps in Python.

When developing your service you should be aware that werkzeug, the built-in web server, is not suitable for production due to security and scalability issues.  However, if you do use it during test you’ll find it also makes it easy to accidentally keep global state within your app, which you shouldn’t be doing because it won’t work when you’re inside a production server.  For that reason, I suggest starting to use uWSGI relatively early in your development.  It’s not difficult to use for test.

I’ll come onto a step-by-step deployment guide in an upcoming post, but I recommend that the web service be deployed using uWSGI behind nginx or Apache to get a secure and scalable deployment.  However, on Raspbian at least there are a couple of pitfalls I found with uWSGI: I used pip to install a relatively modern version.  The installation used a PREFIX of /usr/local.  For some reason, even with /usr/local/bin on my PATH, uwsgi did not work correctly unless called with its full path (despite printing a message stating the full path it had detected, which was correct).  Perhaps this is a security measure, but the failure mode here and on other issues I experienced was somewhat opaque and better error messages would have been useful.

To use uwsgi in production, it is helpful to have it start to host relevant services on system boot.  On Raspbian, this requires a systemd unit to be created (on other systems an init script, upstart job, or just adding to rc.local would be needed).  That nothing was already in place could be a result of the way I installed uwsgi, but in any case I followed the Debian package’s convention of creating a configuration file in /etc/uwsgi/apps-available and linking to it in /etc/uwsgi/apps-enabled.  The config I used was this:

[uwsgi]
chdir = /var/www/app/boilerio
socket = /var/www/app/thermostat.sock
module = schedulerweb:app
logto = /var/log/uwsgi/thermostat.log
uid = boilerio
gid = www-data
chmod-socket = 664

Note here that, for better isolation, I’m using a system user specifically for this service, and sharing a group with the web-server so I can create the socket with permisions for both to be able to access.

nginx and the URL namespace

The web service natively places its REST endpoints at root level.  As I use nginx to also serve static content – the client HTML/JS/CSS files – I decided to map the service under /api.  Files for the web client get served from /, and they make API calls assuming the api prefix .  I use a simple nginx configuration file to achieve this:

server {
    listen      80;
    server_name hub;
    charset     utf-8;
    location / {
        root /var/www/app/boilerio/static;
    }
    location /api {
        try_files $uri @boilerio;
    }
    location @boilerio {
        include uwsgi_params;
        rewrite ^/api/(.*)$ /$1 break;
        uwsgi_pass unix:/var/www/app/thermostat.sock;
    }
}

The endpoints and the data they expect are currently hand-coded in the Python web application code, which is less than ideal.  Defining a clear API where constraints on input and validation can be consistently and mechanistically verified is a better approach and an area for improvement.  Swagger seems like one good option to implement this, has integration options with Flask, and has the side benefit of a nice web UI for making REST calls to the service too.

The scheduler

After modifying the maintaintemp script to listen for new target temperatures over MQTT rather than having a static target passed on the command-line, the scheduler is able to periodically update the target temperature.  The current, simple, implementation polls the database once per minute, or when a trigger message is received, to load the currently-active schedule and target override.  It then selects a target based on these inputs and sends a message to the boiler controller to update the target temperature.  At startup, and when the controller for a zone restarts, the target request is sent immediately to avoid having to wait a whole polling interval.

This is a likely area for innovation in future: enhancing how the target is chosen using additional inputs or policies, enabling features like pre-heating to reach an upcoming set-point, altering the target based on presence information, and intelligently dealing with installations with multiple zones or independent controls within a zone.

The database

A PostgreSQL database is used to store the schedule and configuration.  This might seem like overkill, but when developing a web app where multiple processes need read/write access to the data it seems sensible to use a tool that is designed for that kind of environment, even if the scale it is being deployed at is relatively tiny, for a few reasons:

  • It avoids designing scalability out now.  If we used another approach that was “simpler” but couldn’t be scaled if necessary it would be a potentially large undertaking to fix.
  • You get a lot of correctness for free.  If, for example, the schedule were stored in a plain text file (say, as JSON), then it is definitely possible to make everything atomic.  But the hassle of getting it exactly right does not seem worthwhile when the database can deal with it all more efficiently from the beginning.

Another approach would be to use a document store like MongoDB.  There are pros and cons to either way (this strongly-worded post is an interesting read concerning problems faced in a practical application), but I decided to go with something I was familiar and confident with.  While having a fixed schema seems to be considered by some to be “overhead” or to slow down development, it can also reduce problems by helping to identify programming or design errors earlier in the cycle and certainly did not seem to make development difficult.

What about AWS/PaaS/Serverless?

A somewhat different approach could be to use Amazon or Azure web services.  Amazon have several relevant offerings here (and I believe Microsoft have alternatives too):

  • AWS IoT: This is a service that maintains a ‘shadow device state’.  User agents can post new ‘desired state’ and devices can post the actual state of the device.  AWS publishes messages to indicate various conditions including when the target and current state are divergent so that the device can change its physical state to match the desired state.
  • AWS Lamda and API gateway: This provides a potentially simple way to implement the scheduler REST API without having to host the web service component yourself, potentially reducing the maintenance burden.  You can easily provide authenticated access regardless of where you are connecting from.  Zappa is a tool that lets you easily run a Flask applications within Lambda, so could be used to allow the BoilerIO code base to be used without modification.
  • S3 could host the static files, such as the CSS, JS, HTML, etc. for the client app.
  • AWS DynamoDB or Relational DB services.  The latter could be a drop-in replacement as it has PostgreSQL support, whereas changes to the app (relatively minor/contained within one module) would be required for DynamoDB although that option does have more attractive pricing

The first issue to consider with this approach is what the connectivity requirements are and what is the user’s expectation of behaviour when their Internet connection is unavailable.  In this case, the minimum requirements seem to be that (i) the schedule should continue to run without interruption regardless of Internet downtime, and (ii) the user should be able to supply at least an override even if the schedule is not editable.  Both are doable, the first trivially since the scheduler can (and should) be run locally to the installation.

The second issue is vendor lock-in.  These services are proprietary, and there’s no way to run local versions of them either for testing or deployments where using an online service is suitable.

In the end I decided to stick with a regular web service for now, which leaves the option open for either hosting it off-site, in a remote VM for example, or having a connector module that enables AWS IoT or similar to provide off-network access without hosting a public-facing service locally.

Next steps

This blog post covers part of one of the “next steps” I identified in the previous post.  Upcoming areas for further work are better documentation including a setup guide, and looking at additional features such as multiple zone support and pre-heating.

Advertisements

Boiler Control to Maintain a Set Temperature

I wanted to be able to control my home’s heating from a computer.  This post discusses the next phase in that project: a control layer that maintains a specified room temperature using a temperature sensor and the boiler control built in previous articles.

I published the boilerio repository on github that contains the code to do this.

Note that neither the code nor the article come with any warranty: please be careful if you’re using it as you could damage your heating system or create a safety hazard.

Heating system overview

Our heating system is fairly typical for the UK: an S-plan system using a gas-fired “system” boiler with a pressurised hot water storage vessel.  When the thermostats call for heat, water is heated by the boiler and pumped around a series of radiators.  As noted in the previous article, we have a Danfoss RX2 receiver whose control protocol I’ve reverse-engineered, so we will use that here to control the boiler.

System-diagram

This article looks at the common but relatively crude control method of simply turning the boiler on/off periodically to maintain a steady temperature.  Note that the boiler also has a manually-specified target flow temperature and will modulate the burners to achieve this when it is active.

More advanced controls, possibly the subject of future articles, could adjust boiler settings in response to flow temperature and other variables to ensure the boiler is within its most efficient operating parameters.  This requires integration with the boiler’s electronic systems not explored here.

The system has three main components:

  1. The thermostat transceiver.  Here we’re using simple on/off control by implementing an interface to the Danfoss RX2 receiver.  We’ll respond to MQTT messages to allow services to issue commands to the boiler and, for monitoring, publish to an MQTT topic when messages are received over RF.
  2. The heating controller.  This will be a Python daemon that works towards a temperature setpoint (for now this is a command-line parameter, but in future it will be hooked up to a scheduler) by monitoring the current temperature and deciding how to control the boiler to reach the target.
  3. The temperature sensor.  I’m using an EmonTH for this that measures and publishes room temperature to MQTT.  There are some software tweaks I will discuss below.

The heating controller

target_zone

There are three modes of operation in reaching and maintaining temperature: significantly below setpoint, significantly above setpoint, and near the setpoint.  The first two cases are easy: the boiler should either be on or off.  Within the target zone we can modulate the boiler on/off to produce an average heating input to the room of the desired level: this is a type of pulse-width modulation (PWM) with long pulse durations in the order of minutes.

To decide what the duty-cycle (the fraction of the time boiler is turned on for in the full cycle) should be we need to determine the required heat input for the room using a control mechanism that can ‘find’ the correct value, since it will differ based on various factors including the temperature difference to outside and outside weather, building materials and insulation type, effectiveness of the radiators, losses through pipes, etc.

After trying out a couple of approaches based around incrementally increasing or decreasing the PWM duty-cycle according to the current “error” (difference between target and actual temperature), I learned that using a PID controller is a common approach that can be effective given some tuning.  This computes a control variable (in our case the PWM duty cycle) given a process variable (the current temperature), and a setpoint (the target temperature).  The output u at time t is given by:

u(t) = K_pe(t) + K_i\int_0^t e(\tau) d\tau - K_d\frac{de(t)}{dt}

The PID output combines the error (difference between current value and setpoint), the total error over time (the integral component, which allows the controller to adjust to the current conditions), and the differential (to damp excessive corrections), in an amount that is application specific using the coefficients Kp, Ki, and Kd.  An initial implementation is in pid.py in the boilerio repository.  It is currently quite basic and could be further refined.

For a more detailed overview of PID controllers, the Wikipedia page is a good place to start, and then Brett Beauregard’s excellent Improving the Beginner’s PID article series and accompanying library for the Arduino provide a good explanation of some of the common issues and solutions with basic PID controllers.

Some things to note about this implementation:

  • The output is limited between 0.15 and 1.  Values below 0.15 are rounded down to 0 since such a small duty cycle doesn’t give the boiler chance to do anything useful.  Different limits may be suitable for different systems.
  • The integral component is limited between -1 and +1 to avoid it becoming excessively large in either direction (since it can’t influence the output beyond those limits anyway).
  • Unlike many applications of PID controllers where the process variable is actively moved in both directions, we can’t actively cool the room.  Therefore, we allow a negative integral that’s larger than one might in other systems, to accommodate the proportional term being too large.

The simulator

simulation

Choosing appropriate coefficients for the PID controller and the efficiency of test/dev cycles were both important challenges.  At this time of year there is little opportunity to do real-world testing where the temperature difference between inside and out is very high, and to do such a test is both time-consuming and potentially wasteful of energy.  Instead, I decided to write a simple simulator, sim.py, to help with the majority of the debugging and tuning.

There are various tools online for calculating heat loss in your home that take into account the building materials, insulation, windows, ventilation, etc.  To estimate heat loss through conduction they look at loss through each building element Q=UA(T_i-T_O); there is then the heat loss through ventilation to add in.

We use an extremely simple model that is sufficient to achieve the goals described above. Firstly we combine the U and A terms in the heat loss formula and assume an average across all building elements.  We assume that in each time increment, some fraction of the heat will be lost to the outside and some heat will be gained through transfer from the radiators, each with different efficiencies and therefore coefficients, without considering ventilation separately.  The radiator temperature itself is assumed to increase and decrease linearly over a ramp-up and ramp-down time when heating demand is indicated or ceases.

We inject a fake Boiler class that updates the model parameters rather than actually sending commands to the real system, allowing the model to interact with the controller.  The code is careful to only get the current time in one place and pass it as a parameter, to make mocking the passage of time easier.

To find a reasonable value for the heat-loss coefficient, I grabbed some real data from my temperature logs and used scipy to do a curve fit.  Then, keeping that value constant I did a similar exercise to determine the coefficient for heat transfer from the radiator in the room.  These values are obviously very rough; different time periods produced different results as the conditions at the time weren’t known (doors opened/closed, etc.).

Interfacing the boiler to MQTT

The boilerio repository includes a daemon, boiler_to_mqtt.py, that will interface with a serial port using the protocol implemented in the previous two articles.  This, like the other tools, uses a config file to specify the location of the MQTT broker and the topic names to use.

RF messages sent and received are published to the topic specified by info_basetopic in the config file.  The published payload contains a JSON message with keys “direction” which is ISSUE or RECV, and “cmd”, which is the command issued or received (ON, OFF, or LEARN).  An example payload might be:

{"thermostat": "0x1234", "cmd": "ON", "direction": "RECV"}

Clients can issue commands to the boiler by publishing to the topic specified as the heating_demand_topic in the configuration file.  The script expects a JSON payload consisting of an object containing two values: the command (“O” for On, “X” for Off, and “L” for Learn), and the thermostat ID as an integer.  A sample payload might be:

{"command": "X", "thermostat": 23123}

See the README.md for more information on using the boiler_to_mqtt.py script if you are using a Danfoss receiver.  Alternatively, you can still use the temperature management code but replace this script with something that can control whatever receiver you are using.

Temperature input

There are several options for temperature input: originally I had put together my own temperature ‘transmitter’ using an AVR microprocessor, a Dallas Instruments DS18B20 and an XBee radio (Sparkfun have a guide on the XBee).  If you go down that route be sure to get the right XBee hardware since v1 and v2 are not compatible.  I also had issues with an Arduino shield I bought, though breadboarding with Sparkfun’s XBee breakout worked fine.

I have since switched to using the excellent emonTH v2 from OpenEnergyMonitor.  These have a simpler RF69 radio, which is all that’s needed (and handily is the same one that supports interfacing with the Danfoss receiver), come pre-assembled, have a lower-power sensor that can also record humidity, and are battery powered.  The hardware design and software are open-source.

I did choose to make some modifications to the emonTH and emonhub software. For the emonTH I increased the resolution of the temperature readings, which required a number of updates across the stack:

emonth
— Programming the EmonTH
  • Support for setting the resolution in the library for the SI7021 sensor;
  • Setting the SI7021 resolution during emonTH startup and reporting hundredths rather than tenths of a degree over RF, which required reprogramming the emonTH using a USB-to-UART adapter;
  • Modifying the emonhub configuration to accommodate the change of packet format.

Increasing the resolution of the SI7021 sensor readings will also increase the time taken to acquire those readings, and therefore the overall power consumption, so expect batteries to run out quicker.  That being said, the OEM project estimate years of battery life from the default configuration so even at a quarter of that it would still be acceptable to me since I’m using rechargeable batteries anyway.

I also modified the format emonhub uses to post data to MQTT rather than using the pre-existing options of either a single message with a series of values whose order is significant in determining their meaning (the “rx” format), or one message per reading (e.g. to topics like emonth/temperature, emonth/humidity) where the grouping of the messages cannot be reconstructed.  My branch of emonhub posts a single MQTT message that has a JSON payload with the group of readings (temperature, humidity, battery voltage, etc.) that were taken simultaneously.  This is not strictly necessary but was helpful for other projects.

The modified emonhub, emonTH, and SI7021 code are available from github.

Real-world testing

reallife
Graph showing an example of real operation of the controller.  Highlighted areas show where the script called for heat.

I have used this code to control the real boiler a number of times, mostly with overnight tests.  With the weather getting warmer, I’ve not been able to get a feel for how it works when it’s really cold outside but, in the situations I’ve used it so far, it seems to have worked well.  Generally it maintains the temperature to within ±0.2ºC of the setpoint, which I consider to be a success.

Next steps

The upcoming good weather will surely slow progress but there is plenty that can still be done: three possible areas to investigate next are:

  1. Power measurement.  It would be useful to read gas usage automatically to better understand how efficiently gas is being used and what effect change have on this.
  2. Scheduling.  This isn’t really usable as control requires ssh and command-line knowledge.
  3. More advanced integration with the boiler.  Monitoring and setting parameters such as target, supply, and return water temperatures and burner on/off.

Hope you found this interesting and/or useful!

Danfoss Wireless Thermostat Hacking – Part Two

I’ve been trying to take over control of my home’s central heating using a combination of software and commodity hardware such as the Arduino and Raspberry Pi.  Part one of this series looked at how my existing RF thermostats worked and showed it should be possible to emulate them so that the receiver (which has relays that turn heating zones on/off) already connected to the boiler could be used by my own control system.  I currently have two Danfoss TP7000-RF wireless thermostats (one per zone) and a Danfoss RX2 receiver.

In this part, we look at programmatically receiving and transmitting packets from/to the Danfoss RX2 receiver in order to turn the boiler on and off, and start to look at how this could be integrated into a more complete system.

The RF69 radio module

In order to be able to transmit and receive thermostat messages, we need an FSK transceiver that can receive and transmit packets of the right format.  The RF69 family by HopeRF is a popular module used by enthusiasts; typical use cases include creating networks of home automation devices and sensors.  There are various libraries that make use of the packet format features of the module, or layer a packet format on top, to provide bi-directional communication.  However, in our case we need to integrate with the non-RF69 receivers/transmitters used by the existing installation.  This is possible: the RF69CW supports up to eight sync words, fixed- and variable-length packet formats that are flexible enough to receive packets in the format transmitted by the thermostats, and supports the 433MHz frequency.

One minor issue is that the data sheet claims that the minimum supported data-rate is 1.2kbps, however my experimentation shows that it can deal with the 1000bps rate used by the Danfoss thermostats.

img_1513
JeeLink v3c: an Atmega 328P and RF69CW on a USB stick

There are a variety of hardware options for incorporating the RF69 into your project:

  • Connect the RF69 directly to a Raspberry Pi: You could make up an interface board yourself or buy a PCB with the correct headers and pads for the RF69 and Pi (or facility to add them).  This has the downside of only working on a Pi.
  • OpenEnergyMonitor’s RFM69Pi module, which is an Arduino-compatible Pi “hat” including an AVR chip and the RF69 module on board.  You can easily upload new firmware to it for this project; I think it is well-suited though mine is currently busy in my energy-monitoring setup.  This approach shares the downside of requiring the Pi to operate it.
  • The JeeLink v3c by JeeLabs, which combines an Atmega 328P and RF69CW module into a USB form-factor that’s Arduino-compatible.  Be sure to purchase the 434MHz version.

I went with the JeeLink option as it’s a USB device so can be used easily both with the target Raspberry Pi as well as a traditional PC for development.

Firmware

The firmware used in this project is available on GitHub under an MIT license.

firmware

The first thing to deal with is interacting with the RF69 module.  There are a number of existing projects that implement libraries for RF69, though I decided to write my own because the others either didn’t quite fit my needs or had application logic embedded in the code. Both JeeLib and Mad Scientist Labs, whose work served as a useful reference here, deserve shout-outs.  DeKay’s posts at Mad Scientist Labs on reverse-engineering a Davis weather station are a fascinating read.

Some specific requirements we have for this project:

  • The sync words:  We’ll need to use the six encoded sync words that the thermostats transmit (0x6cb6 0xcb2c 0x92d9, which decodes to 0xdd46).  These come after the preamble; the RF69 normally uses a raw 10101... pattern as its preamble, but can be configured not to send one and seems to lock on to the transmission just fine even with the encoded version of that pattern being used by the thermostats.
  • The packet format: The RF69 supports whitening and Manchester encoding, checking and embedding CRCs, and variable-length packets (where the length is indicated in a byte contained within the packet).  We want to disable all these features: we use fixed-length packets, and receive the encoded packet into the Arduino firmware where we will decode them.

We want to provide a serial interface, emitting a line per received message with the thermostat ID and the command that was sent (on, off, or learn), as well taking commands as input to tell us to transmit packets with a particular thermostat ID and command.  I’ve tried to keep it machine- and human-parsable: the sketch I provide takes input of the form XTTTT\n, where X is the command (O for On, X for Off, and L for Learn), and TTTT is the thermostat ID in hex.  It prints lines like <RECV|ISSUE> TTTT CMD where RECV indicates that a packet was received or ISSUE is a command we just issued, TTTT is the thermostat ID, and CMD is either ON, OFF, or LEARN.

Encoding and decoding to/from the wire format

Encoding and decoding on the Arduino with the RF69 is simpler than in part one where we were using the wave file from the SDR because, once the RF module is programmed with the correct bit-rate etc., it does the data slicing and bit synchronisation for us. 

The representation of a bit in the encoded packet has a preceding 0 and trailing 1, and the middle bit is the unencoded value being transmitted (this is a simple technique to ensure the signal is constantly being modulated so that the gain on the receiver remains within usable bounds).

To decode, we set bit i of the output according to bit 1 + 3 * i of the input (counting from left to right in the binary representation, so bit 0 being the most-significant bit of the first byte of output). Similarly, on encoding we copy bit i from the input into bit 1 + 3 * i of the output, inserting the preceding 0 (at bit 3 * i) and trailing 1 (at bit 2 + 3 * i.  You can check out the sketch to see details of how this is done: the encode_3b and decode_3b functions are the relevant place to look.

Receiving packets

The receive code gets a packet of data from the RF69 and has to decode it, validate it, and extract the instruction and thermostat ID.  The thermostats retransmit the packet immediately, so the received packet has the sync word stripped off the first copy of the packet by the receiver but both it and the preamble are present in the second copy as passed to the micro-controller.

One annoying issue that there is a stray 0 bit in-between the first and second transmissions.  As a consequence the overall data is not a whole number of bytes, which is a problem because the packet length is specified in bytes to the RF69.  I experimented with programming the receiver to get the last byte, of which only the first bit is transmitted, but this causes problems such as the reported RSSI value being useless since the thermostats don’t transmit anything for 7 of the 8 bits in the last byte.  The sketch instead specifies a packet length that is the number of bytes rounded down and works around the missing bit at the end of the transmission.

To receive a packet we do the following:

  • Get the packet from the RF69’s FIFO into an array;
  • Shift the second copy of the received packet left by a bit so we can do direct comparisons between the two copies;
  • Decode the packet;
  • Validate the packet: check that the sync word is correct in the retransmission, and that the thermostat ID and command match in both copies (being sure to account for that missing bit);
  • Extract the thermostat ID and command.

If valid, the received data is then output to the serial console.

Transmitting a packet

Originally I’d hoped I could use the RF69’s preamble and sync word features for transmit also, but this would require the receiver to accept packets of a slightly different format than it sends.  Having tried this and found it not to work, the sketch instead closely emulates the thermostat’s packet structure.

During transmission we have to temporarily turn off the sync word feature of the RF69 in order to produce a packet with the custom preamble, followed by the sync words, the data, and then a repeat of the packet (the repeat doesn’t seem to be strictly necessary and therefore could potentially be handled more simply but I decided to maintain a close emulation anyway).  The RF69 library I wrote has support for temporarily disabling the sync words and using a different packet length than for receive.

Other than that, the transmit sequence is pretty simple: parse the command from serial, generate a thermostat packet (including preamble and sync words) with the appropriate values included, encode it to the line-encoding used by the receiver, put it in the RF69’s FIFO, and then transmit it.

The higher-level control system

So far we’ve provided a basic mechanism to turn on/off heating in a zone.  There are many options for how this can be used to achieve the features you’d expect from a heating control system.  Some characteristics of such a system might include:

  • It has inputs in the form of current temperature readings;
  • The available temperature data will be used to decide when to turn on/off heating in a zone;
  • There is a scheduling mechanism, choosing at what times specific temperatures should be targeted;
  • A way to see the current temperatures, the current state of the boiler, the target temperature, the schedule, etc.;
  • Safety features.  In particular, what happens if the control system, the devices providing temperatures (or those receiving them), the radio module, etc., fail?  What should the desired outcome and recovery be in these cases?
  • Being able to set a desired temperature and have the system automatically start heating earlier to reach the target temperature at the requested time;
  • Outside temperature and other factors (such as other heating sources interfering with the feedback mechanism) are inputs to the system to enable it to optimise central heating use.

Implementing all of the above is pretty large undertaking (not all of which I have yet done!) but would essentially provide an implementation of a domestic-grade heating control system.

By aiming to create decoupled components with clear interfaces we can enable substitution of alternatives suitable for the specific installation.  For example, users without Danfoss thermostats may wish to replace the component described in this and the previous post with their own system for turning on/off heating in a zone (e.g. using relays directly attached to an Arduino, or interfacing with a different RF receiver).

What’s next?

Future articles will examine the behaviour of the Danfoss system further and look at when it is turning heating on and off in response to input, and start to implement the higher level control mechanisms described above.

Code

The most up-to-date code including the sketch to implement the interfacing described in this article is available on GitHub.

Danfoss Wireless Thermostat Hacking – Part One

tp7000rf

I wanted to control my central heating system using a Raspberry Pi and Arduino micro-controllers to provide better control, flexibility, and a fun home automation project.

We originally chose wireless thermostats when we replaced the heating system in our home, but their user interface is not great and they are fiddly to use.  “Smart” thermostats were starting to come onto the market showing a glimpse of what could be done.

Having made some useful progress in my overall goal, I am documenting it here for the benefit of others.  My requirements were simple:

  • Easy to change the heating profile for a day, e.g. if we decided to light a fire and didn’t need heat from the central heating system;
  • The boiler should be used efficiently to reduce costs;
  • Changes should be minimally invasive to the existing setup (e.g. no major rewiring/plumbing).

This post talks about how I was able to control the boiler whilst being minimally invasive by using the existing thermostat receiver and reverse-engineering its protocol, thus avoiding any electrical modifications.

Setup

rx2

The system being ‘hacked’ is a Danfoss RX2 wireless receiver, with two TP7000 RF thermostats.  It’s plumbed to create a two-zone heating system, one zone for each floor of the house.

Options for controlling the boiler

Initially I planned to put my own relay into the system with a wireless module attached that I could control it with.  I chickened out of this approach mostly because I didn’t want my dodgy soldering interacting with always-on mains-voltage equipment.  This led me to the idea of a Z-Wave based relay.  Fibaro make a product (the FGS-222) that’s quite appropriate for this use case: it is a dual-relay unit (since my home has two heating zones) and has switched and permanent inputs so you can have the existing control system continue to operate, or override it with your own.  The problem here was that Z-Wave devices require a gateway (such as Domoticz) to get them working, which seemed a bit overkill, but I think in general this is a reasonable route to go down.

However, my goal is to be minimally invasive: by using the existing control mechanism (the Danfoss RX2 wireless receiver), no changes are needed to the boiler electrical circuits.  Of course, that is easier said that done since it requires emulation of the protocol used by the wireless thermostats.  In this post I talk about receiving and decoding the protocol; subsequent posts will talk about emulating it and taking over control of the system.

Signal acquisition

My starting point was to try to capture the signal being sent by the Danfoss wireless thermostats to the receiver unit, in order that I could at least replicate it bit-for-bit.  Ideally though, I’d also like to understand the contents of the payload of the messages being transmitted, and be able to capture them programatically in order to track when the existing system is calling for heat.

ask
Trying to decode the signal as amplitude-moduated

Having taken apart an RX1 receiver (a single-channel version of the RX2) that was given to me some time back, and photographed the circuit board in anticipation of this project, I can see it uses an Infineon TDA5210 chip for RF.  The datasheet indicates that this is a receiver only, which tells us that the protocol is one-way and could either be amplitude- or frequency-modulated.  Having looked at the circuit, I mistakenly thought that the signal was amplitude modulated and tried to use a basic RF receiver a friend gave me to receive the signal by having an Arduino dump its output over serial in variously increasingly complicated ways.  I quickly became frustrated on seeing a long “high” followed by silence as the gain circuit ramped back up to just amplifying noise in the receiver.  I initially thought I was missing the transmission, but was actually seeing it all along albeit unable to decode it because it was actually frequency modulated (and therefore seen by the ASK receiver as the long ‘high’ pulse).

nooelec
Nooelec SDR receiver

Unable to make progress I wondered if I was mistaken about the modulation, didn’t know much about the RF69 yet that we’ll use later, and needed to find a way of figuring out what was going on.  Software-defined radio seemed to provide the answer: enter the Nooelec USB software-defined radio receiver.  Note that this isn’t necessarily the best hardware to buy, but it was available quickly in the UK and seemed to be good enough.  The RTL-SDR blog sell a modified version of units like these that are optimised for use with SDR apps, but as they are shipped from China the shipping time can be quite long.

As a Mac/Linux user the software options for SDR are a bit limited.  The flagship option seems to be SDR# but this is only available on Windows.  You apparently can get it to work on macOS using Mono, but instead I decided to opt for gqrx using X11 installed via MacPorts.  Once installed, you can turn on the waterfall view and then try to trigger the signal.  From previous experimentation with the ASK decoders, I was pretty sure that just pressing a button (temperature up/down) would result in an RF transmission even if the boiler state wasn’t being changed, which is handy because it meant I could avoid cycling the boiler on/off without disconnecting it from the mains.

gqrx

On centring the receiver at 433.9MHz (chosen from looking at the TDA5210 datasheet) and triggering a transmission, it’s very clear that the signal is frequency modulated (the horizontal axis shows the frequency domain, the vertical axis shows time, and the colour show signal strength).  The waterfall display isn’t detailed enough to be able to see the signal content, but by experimenting with demodulation options in the software I found that the signal came out cleanly demodulated using the “FM (Stereo)” option:

  1. Choose the FM (Stereo) demodulation option
  2. Ensure the correct centre frequency, 433.9MHz, is chosen
  3. Press the Rec button in the bottom-right.
  4. Trigger the transmission.
  5. Press the Rec button again to stop the recording.

The signal is saved as a .wav file, which then takes us into similar territory documented by others of examining and trying to replay the signal ourselves.  You can use Audacity to view the waveform you saved from gqrx:

audacity

As a starting point, I captured the same signal multiple times with the target temperature being different (i.e. different set temperatures all of which result in no heating demand, and the room temperature not having been updated) and found each capture produced a signal that looked identical.  I then compared that with one where the boiler should be on and at that point it was looking good: the signal was pretty much the same apart from in one section where a couple of 0s become 1s and vice-versa.

Decoding the signal

Looking at the signal it seems like there is a clear pattern of 001 and 011 occurring; these likely correspond with 0s and 1s in the decoded signal.  Python has a handy library, wave, that you can use to easily read the values from a .wav file, so I used this first to dump the file to get an idea of how many frames the longer pulses lasted for.  I then used simple temporal and amplitude thresholding (detecting when a high or low has been seen for more than a fixed number of frames in the wave file) to find the encoded values: if we see two 0s together in the wire protocol we emit a 0, and if we see two 1s together a 1 is emitted.

This is the program that I used:

We can use xxd to dump this as hex so we can inspect it.  This is the decoded data for the ‘upstairs off’ signal:

andy@beta:~/rf$ python decode_danfoss.py -d up_off.wav | xxd
00000000: aadd 46c5 88cc 556e a362 c466            ..F...Un.b.f

Looking at this combined with other captured signals makes it pretty clear what’s going on:

  • 0xAA at the start is the preamble.  It is somewhat interesting that they transmit this as encoded data, I’m not sure if that is common practise.  The preamble is used by the receiver to set the gain correctly.
  • 0xDD and 0x46 are both part of a “sync word”, and are consistent across all messages from all thermostats.  This indicates to the receiver that the signal is of interest to them.
  • 0xC5 and 0x88 (together 0x88C5, also seen written in ink on the PCB of this particular thermostat) are the thermostat ID.  This is different for the other thermostats.
  • 0xCC is the instruction.  This is 0xCC for ‘off’, 0x77 for ‘learn’, and 0x33 for ‘on’.
  • The rest of the transmission is a repeat of the original message, and looks different at first glance in hex because there was a 0 bit between the two transmissions so the second one is offset by one bit.  (You can see this for yourself by running xxd with the -b option to dump the output as binary instead of hex.)

In part two

In the next part, we will use an RF69 module alongside an Arduino-compatible microprocessor to send messages to the receiver to turn on/off the boiler, as well as receive messages from the existing thermostats programmatically to observe their behaviour.

Continue to Part Two.