High Availability Domoticz Cluster

[Antori91]

Hello,

I've completed the first version of my own basic High Availability Domoticz Cluster. This cluster is based on two servers (active/passive nodes) synchronized via MQTT. Each server has its own Domoticz instance and is also an MQTT server.

Under normal condition:
- the service is delivered by the main server and its Domoticz instance,
- the backup server monitors the main server (by sending every minute a JSON request to Domoticz) and synchronizes its own Domoticz database.

Failover happens if:
- Domoticz in the main server fails. The Backup Domoticz instance is then used to continue to deliver the service. This is done by republishing the MQTT incoming/outcoming messages between the two servers,
- or the main MQTT server fails (or even the server goes down). In this situation, the backup server becomes the main server by taking the IP address previously used by the main server.

All the stuff is done by only one (nodejs) script running only in the backup server.

The two servers have not to be the same one or even don't have to have the same hardware architecture. Also, the Domoticz versions and the databases structure on the two servers can be different. There is a basic repository within the script where you can declare some attributes for each Domoticz device you want to synchronize under normal condition. These attributes are :
- the device id,
- if it's incoming or outcoming message,
- the device type from a Domoticz perspective. For incoming message, it can be any device type. For outcoming message, it's currently only Light/Switch, SelectorSwitch, Thermostat or Temperature.

In my environment, it's an End to end High Availability solution because all my sensors/actuators are ESP8266. They are polled by scripts from the backup server when they are web server or they are able to reconnect themself to MQTT after the failover is done when they are MQTT client.

[nigels0]

Are you going to put together a wiki?

[Antori91]

If it seems interesting to many people, I'm ok to write a Wiki. In the meantime, I have published the code of the script and an example of .sh file to launch it at Github : https://github.com/Antori91/Home_Automa ... %20Cluster

To try to run it, first you have to:
- install Domoticz and MQTT in each server. In my environnement the Backup server has Ethernet and WiFi interface (it's a Raspberry),
- install in each Domoticz the "MQTT Client Gateway with LAN interface" hardware with Remote address=localhost, port=1883 and Publish Topic=out,
- install node.js (with the mqtt module https://www.npmjs.com/package/mqtt ) in the Backup server,
- modify the setup of the main Domoticz, Local Networks (no username/password) parameter, in the Main server to accept connections from the Backup server without authentication (this for the Ethernet IP address and the WLAN0 WiFi IP address as well of the backup server). Also at least the Backup server must have "localhost;127.0.0.*" in this parameter to accept connection from this script without authentication,
- copy the two .js and the .sh files to your own installation directory in the Backup server,
- update the WiFi_DZ_MQTT_SecretKeys.js file with your own parameters,
- change line 18 of the script to tell it where to find your WiFi_DZ_MQTT_SecretKeys.js file (can be eventually another directory than your installation directory) : const MyJSecretKeys = require('/home/pi/iot_domoticz/WiFi_DZ_MQTT_SecretKeys.js');
- and of course update the basic repository in the script at line 75.

The repository looks like :

Code: Select all

// Here all Domoticz Idx to synchronize [$$SYNC_REPOSITORY]
var   myIDXtoSync      = [ new IDXtoSync( 50, "mqtt/out", "Light/Switch" ),  new IDXtoSync( 51, "mqtt/out", "Light/Switch" ),   new IDXtoSync( 34, "mqtt/in", ""),   // 50/51=Entree and Mezzanine lighting, 34=Hot Water Tank
                           new IDXtoSync( 27, "mqtt/in", "" ),  new IDXtoSync( 28, "mqtt/in", "" ),   new IDXtoSync( 29, "mqtt/in", ""), new IDXtoSync( 30, "mqtt/in", "" ),   new IDXtoSync( 31, "mqtt/in", ""),   // Ground Floor Heaters
                           new IDXtoSync( 35, "mqtt/in", "" ),  new IDXtoSync( 36, "mqtt/in", "" ),   new IDXtoSync( 37, "mqtt/in", ""), new IDXtoSync( 38, "mqtt/in", "" ),   // First Floor Heaters   
                           new IDXtoSync( 17, "mqtt/out", "SelectorSwitch" ), new IDXtoSync( 16, "mqtt/out", "Thermostat" ) ];   //  17=Main OFF/HORSGEL/ECO/CONFORT heating breaker, 16=Heating thermostat setpoint
                           // heating display/schedule to add to the repository in the future
// [$$SYNC_REPOSITORY]

For each device you would like to synchronize under normal condition, you have a declaration like new IDXtoSync( 50, "mqtt/out", "Light/Switch" ) or new IDXtoSync( 31, "mqtt/in", ""). For incoming MQTT message from a sensor/actuator, you just have to say new IDXtoSync( 31, "mqtt/in", "") where 31 is for example your device number. For outcoming MQTT message from Domoticz, you have to use IDXtoSync( 50, "mqtt/out", "Light/Switch") where 50 is for example your device number and "Light/Switch" your device type. For device type, you can use currently only "Light/Switch", "SelectorSwitch", "Thermostat" or "Temperature" (it is basic temperature device without humidity).

After a failover, before restarting the service of the main server, you must first restart this script (and eventually copy the backup Domoticz database to the main server). Currently, the failover happens after 15 mn of continuous failure. You can change this parameter at line 30: const TIMEOUT = 15; Also, in case of complete failover, I mean the Backup server becoming the main server, it is done by changing the WiFi IP address of the Backup server and not its Ethernet IP address. Can be changed but not tested at line 45 : commandLine = 'sudo ifconfig wlan0 ' + newIPaddress;

[Antori91]

Architecture schema here
https://github.com/Antori91/Home_Automa ... ecture.GIF

[schj]

Nice stuff! My setup with USB hardware (rflink) connected to my rpi does not scale so nicely. In case of a failover I would need to reconnect some wires....

[christophe2836]

Hello,
I am new and my English is not terribly good.
I do not understand or put both .sh and .js files
I do not have the directory iot_domoticz, so I created it and put the file WiFi_DZ_MQTT_SecretKeys.js in it.
in short, I am a little lost.
I want my Syno to be master and my RPI slave.
is it possible to detail a little more tutorials for beginners like me? (I know I'm abusing!)
(merci google traduction!)
Bonne Année à tous !!!!

[Antori91]

Hello Christophe,

These two files have to be copied to the backup server in a directory you choose. By default, this directory, you have to create, is /home/pi/iot_domoticz.

You didn't mention if all your sensors/actuators (or at least the critical ones) are devices communicating with Domoticz using Mqtt. If not, this high availability cluster is almost worthless.

[christophe2836]

Hello
Thank you for your reply, I did not understand that only the sensors/detectors communicating with MQTT were rescued by the slave.
Actually, I have very few devices that use MQTT, so more interest to me.
Thank you for your precisions.

[Antori91]

Hello,

A new version (0.20) is available (https://github.com/Antori91/Home_Automa ... %20Cluster) with three new main features :
- If the Main Domoticz instance fails, before starting the Backup DZ Failover, the Backup server tries first to restart the Main Domoticz instance using a SSH session to the Main server,
- After failure, when the Main Domoticz instance is back, the Backup DZ Failover is automatically stopped (don't have anymore to stop/restart this script in the backup server),
- SECPANEL synchronization between the Domoticz instances is now operational.

Also for easier server maintenance, the "Backup server not fully operational" warning/alert is raised only after LTIMEOUT heartbeat failures (to avoid false alerts during server maintenance).

To use this new version, Node.js version must be greater than 6. Also, you must have the Node.js SSH2 package installed. To install it, use the command:
npm install ssh2

To eventually upgrade your Node.js version, you have to enter the following commands:
sudo apt-get update
sudo apt-get dist-upgrade
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs

[Antori91]

A new feature ! I use a reverse proxy to access my home automation environment from Internet.
I now have three machines that were exposed in http to the Internet (the two Domoticz cluster servers and a dedicated Alarm server with motionEye and another custom Alarm software).
I just finished setting up a reverse proxy, the one installed by default in Synology: nginx.
What's cool:
- I have now only one machine in front of Internet (the one with the reverse proxy) which dispatches incoming requests to the corresponding local server. All connections from Internet are logged (new sessions / authentications only to have usable logs),
- all communications are https without having to change a single parameter of both Domoticz and motionEye configuration files. The reverse proxy makes on-the-fly https (Internet traffic) to http (local traffic) conversion. It's particularly nice with motionEye which its website doesn't support https,
- The reverse proxy is aware of my Domoticz cluster. By default, Domoticz users are routed to the Domoticz instance of the primary server. If this main Domoticz instance is down, the reverse proxy redirects automatically the Domoticz users to the standby server instance.

You can find here the nginx configuration file template : https://github.com/Antori91/Home_Automa ... uster.conf

Architecture update:

Landscape Architecture.GIF (167.11 KiB) Viewed 5437 times

[Antori91]

A short message to let you know I have reinstalled my Raspberry Domoticz cluster backup node using Raspbian Buster installed on USB hard-disk.
The Domoticz version is the last release version (4.10717).

No issue to install except the first USB case for hard-disk I've tried (This USB case wasn't 100% compatible with the Raspberry (or the opposite!), it allows to cold boot but not reboot (many topics about this in the Raspberry forums). The second one I've tried just works fine.

No issue to run. All works great:
- Buster installed on USB hard-disk (NO SDCARD installed anymore in my server),
- Domoticz with Buster.

[Antori91]

Hello,

I updated my program for managing a Domoticz cluster:
https://github.com/Antori91/Home_Automa ... Cluster.js

The new feature is the synchronization of the Main server from devices belonging to the backup server. In cluster management, I initially only needed the synchronization of the Domoticz database of the backup server. The need for a replication in the other direction (backup to main) appeared with two plugins that I run on Raspberry (my backup server) and not on Synology (my main server). This new feature is the equivalent of the native Domoticz feature for synchronization (Domoticz remote server) but in my case it also works for devices created and updated by a plugin.

[AllesVanZelf]

This is interesting. But not working if you depend on zwave hardware and 433 transceiver. Is it?
Maybe with an extra rfxcom433 attached to the backup, this functionality would work, But this is not the case with zwave. It is not possible to have an extra zwave stick and use this as backup. Or am I wrong here?

[Antori91]

This is interesting. But not working if you depend on zwave hardware and 433 transceiver. Is it?
Maybe with an extra rfxcom433 attached to the backup, this functionality would work, But this is not the case with zwave. It is not possible to have an extra zwave stick and use this as backup. Or am I wrong here?

Yes, you are right. The fail-over solution I've built and described here is mainly for sensors architecture using Ethernet (Lan and/or WiFi) and mqtt to communicate with Domoticz. Using my solution, you always have a standby Domoticz node synchronized with the primary one and ready to start if your primary server (Domoticz or mqtt or the full server) goes down. But, if you are a Zwave user, you still have to implement a fail-over solution for the Zwave controller attached to the primary server.