Using MQTT for comm between Domoticz instances

[roseen]

I have a Domoticz instance at home, on the local network.
I have also set up a Domoticz instance in the cloud, and I wanted the cloud instance to mirror some of the devices in the home instance. This could have been a piece of cake by configuring the home instance as a Remote Server on the cloud instance.

However, due to my ISP using CGNAT, it is difficult to connect from the internet to my home server. A port mapping in my firewall doesn't help, it is on the ISP level. I need my communication to be initiated on the inside, going out.

So now I started experimenting with MQTT. I set up a broker on the web (using Hive MQ) and got my home Domoticz to publish messages to it nicely (using an MQTT client gateway device).

Now I would like to set up the cloud Domoticz to subscribe to the messages on the broker and map them to devices.
I was hoping that an MQTT Autodiscovery client gateway would do that for me, but nothing shows up.
And I was thinking that the various mentions of "devices that support autodiscovery" that I have read here is a clue? Maybe the messages that the MQTT Client gateway sends is in the wrong format?

Is it possible to use MQTT for Domoticz-Domiticz communication?

[waltervl]

MQTT Autodiscovery is only for setting up a gateway to Domoticz. So the Hardware gateway should create autodiscovery topics. Domoticz cannot create those Autodiscovery topics so indeed this is not working out of the box.

Is dynamic DNS working on CGNAT? if so check for an external dynamic DNS service so you can map a domain (eg roseen.se) to your own dynamic IP. Then you can have your cloud Domoticz connect to eg https://domoticz.roseen.se and you can standard device sharing.
But then you also could just have a specific user login to your domoticz instance on https://domoticz.roseen.se and see the devices you want to have them see So you do not need device sharing at all.
I personally use a telegram bot to control my Domoticz devices from outside my home.

[roseen]

Unfortunately, DynDNS does not help when the ISP is using CGNAT. I used to have a DynDNS setup for a long time, but now, the ISP has added an additional NAT layer in front of my firewall. Apparently, it is very common nowadays due to the lack of IPv4 addresses.
However, I am investigating if they offer a public IP address for an extra fee.

The Telegram bot sound like an interesting option! I don't use Telegram, but it could still be worth looking into. Do you have some kind of plugin in Domoticz to set it up?

[roseen]

Never mind, I found a wealth of articles on the wiki about the Telegram bot...

[solarboy]

I use Google Home with Homebridge to control select chosen devices outside of my home network (also CGNAT) on my phone.
It's not perfect and so far I have been unable to use selector switches but for basic use it works well.

[waltervl]

Never mind, I found a wealth of articles on the wiki about the Telegram bot...

I use this one, python based, that is not fully described on wiki (which is lua based)
https://github.com/waltervl/dynamicTelegramBot

[FlyingDomotic]

One possible architecture:

On remote site:

• Setup a classical "MQTT Client Gateway with LAN interface" to get "domoticz/in" topic read

• Connect it on your remote MQTT instance

On local site:

• Setup a classical "MQTT Client Gateway with LAN interface" to get "domoticz/out" topic being written on each local change

• Connect it on your local MQTT instance

• Write a small piece of python that will listen to "domoticz/out" topic, translate local IDX to remote ones, and send modified message to "domoticz/in" on remote MQTT

I may help on that, if needed.

[HvdW]

https://stackoverflow.com/questions/329 ... qtt-broker

Read this.
It explains a lot and learns you how to test.
Like

Code: Select all

mosquitto_sub -h 192.168.1.21 -v -t "#"

which displays all the traffic coming from 192.168.1.21

[gizmocuz]

@roseen, does your provider support IPv6? Most providers do.
Do they charge a lot for a fixed IP address?
Another option is to host it on a cloud server like DigitalOcean

[DiaDomo]

Never mind, I found a wealth of articles on the wiki about the Telegram bot...

I use this one, python based, that is not fully described on wiki (which is lua based)
https://github.com/waltervl/dynamicTelegramBot

How to start this bot for the first time?
I read https://github.com/waltervl/dynamicTele ... /README.md

You need to start the bot manually for the first time because it will ask you for information to create an config:

but don't know how to start it?! What is the commandline command? /home/username/scripts/telegram/dynamicTelegramBot/squandorDynamicTelBot.py is not installed on my system.

[waltervl]

The command to run this as a python3 program should be something like

Code: Select all

python3 /home/username/scripts/telegram/dynamicTelegramBot/DynamicTelBot.py

[DiaDomo]

The command to run this as a python3 program should be something like

Code: Select all

python3 /home/username/scripts/telegram/dynamicTelegramBot/DynamicTelBot.py

Well, first I had to create the DynamicTelBot.py file (of course...), etc. but now it's working!
One question: I have webcams enabled but how do you get a screenshot from your cam in Telegram?

[waltervl]

I have them enabled in Domoticz (menu setup -other - camera) and then the script can forward them to the telegram Bot.

[FlyingDomotic]

I just wrote a new Python plugin synchronizing a master (local) instance of Domoticz to a slave (remote) one at https://github.com/FlyingDomotic/domoti ... ync-plugin

You just have to get list of IDX (or names) of devices you want to synchronize on master, and that's it. Everything is automatically created on slave, and updates are propagated from master to slave when they occurs. You can even propagate changes from slave to master, if you allow device to do so.

Connections are established by master to slave, avoiding opening something incoming at home.

Paranoiacs can even create a dedicated user to protect against accidentally configuring a device as being synchronized.

In addition to one master speaking to one slave, you can also have one master connected to multiple slaves, one slave connected to multiple masters, multiple masters connected to multiple slaves. Cascades also supported.

This may be useful if Telegram disappears one day, after its creator being arrested in France, for example

[DiaDomo]

Hi, nice plugin!
I want to use it because I have 2 domoticz servers running in my LAN.
One question:

Code: Select all

masterMqttHost": "masterHost",
		"masterMqttPort": "1883",
		"masterMqttUser": "masterMqttUser",
		"masterMqttPassword": "masterMqttPassword",
		"masterDomoticzOutTopic": "domoticz/out",
        "masterDomoticzUrl": "http://masterHost:8080",
		"slaveName": "slave",
		"slaveMqttHost": "slaveHost",

Can masterMqttHost & slaveMqtt the same MQTT-server? In my case I have one mosquitto server running on 192.168.x.y and both domoticz servers are using this.

[FlyingDomotic]

For sure. You may even run both on the same host (that's the way I do initial tests, using slaveDevicePrefix to differentiate original and sync devices).

[DiaDomo]

Hi, I used it to sync 2 devices from master to slave.
Both are domoticz instances running on 2 seperate raspberry's.
Both are sharing the same MQTT broker.

Unfortunately, after installing nothing happened....
What could be the case?

My MqttSync.json file:

Code: Select all

{
        "settings": {
                "configVersion": "V1.0.2",
                "masterName": "master",
                "masterMqttHost": "192.168.2.51",
                "masterMqttPort": "1883",
                "masterMqttUser": "",
                "masterMqttPassword": "",
                "masterDomoticzOutTopic": "domoticz/out2",
        "masterDomoticzUrl": "http://192.168.2.51:9090",
                "slaveName": "slave",
                "slaveMqttHost": "192.168.2.51",
                "slaveMqttPort": "1883",
                "slaveMqttUser": "",
                "slaveMqttPassword": "",
        "slaveDomoticzUrl": "http://192.168.2.209:9090",
        "slaveDevicePrefix": "MqttSync - "
        },
        "mapping": [
                {"idx": 85},
                {"idx": 87}
        ]
}

BTW: I'm running domoticz on port 9090.
MQTT explorer dump shows that domoticz/out2 is correct and there is also a mttSync topic.

[FlyingDomotic]

Did you checked Domoticz log on both machines?

If something is wrong, plugin will certainly write something in log.

However, reading configuration file, it seems:
- There's no user/password on your MQTT instances. Is this the case?
- Domoticz out topic seems not standard (domoticz/out2 instead of domoticz/out). You tell Mqtt Explorer shows it. Did you also have a Domoticz/out? When staying in MqttExplorer, does domotiz/out2 change?
- Both instances are using the same MQTT server at 192.168.2.51. Are the 2 Domoticz instances connected to this server?
- Domoticz url doesn't include username/password. Is master IP range in trusted networks (127.0.0.1 will do it).

But first, as said, have a look to Domoticz log on both instances. If there's an error, plugin will write a line with details. If this is not sufficient, setting debug mode to "Debug" will trace every action (and grow significantly the log

[DiaDomo]

Hi, answering your questions: everything correct (no passwords, domoticz/out & domoticz/out2, trusted networks, using the same mqtt server...)

I enabled the debug option:
1) on the slave: no errors
2) on the master: several errors:

Code: Select all

2024-09-01 08:16:51.820 MQTT Sync: 'HomeFolder':'/home/xxx/domoticz/plugins/MqttSync/'
2024-09-01 08:16:51.821 MQTT Sync: 'StartupFolder':'/home/xxx/domoticz/'
2024-09-01 08:16:51.821 MQTT Sync: 'UserDataFolder':'/home/xxx/domoticz/'
2024-09-01 08:16:51.821 MQTT Sync: 'Database':'/home/xxx/domoticz/domoticz.db'
2024-09-01 08:16:51.821 MQTT Sync: 'Language':'en'
2024-09-01 08:16:51.821 MQTT Sync: 'Version':'1.0.0'
2024-09-01 08:16:51.821 MQTT Sync: 'Author':'Flying Domotic'
2024-09-01 08:16:51.821 MQTT Sync: 'Name':'MQTT Sync'
2024-09-01 08:16:51.821 MQTT Sync: 'Port':'0'
2024-09-01 08:16:51.821 MQTT Sync: 'Key':'domoticz-mqtt-sync'
2024-09-01 08:16:51.821 MQTT Sync: 'Mode1':'MqttSync.json'
2024-09-01 08:16:51.821 MQTT Sync: 'Mode5':'Master'
2024-09-01 08:16:51.821 MQTT Sync: 'Mode6':'Debug'
2024-09-01 08:16:51.821 MQTT Sync: 'DomoticzVersion':'2024.7'
2024-09-01 08:16:51.821 MQTT Sync: 'DomoticzHash':'b317cfab1'
2024-09-01 08:16:51.821 MQTT Sync: 'DomoticzBuildTime':'2024-07-13 16:59:31'
2024-09-01 08:16:51.821 MQTT Sync: Device count: 0
2024-09-01 08:16:51.821 MQTT Sync: Debug logging mask set to: PYTHON
2024-09-01 08:16:51.822 MQTT Sync: requestName2IdxData:
2024-09-01 08:16:51.822 MQTT Sync: HttpClient/name2idx::__init__: 192.168.2.51, 9090, False
2024-09-01 08:16:51.822 MQTT Sync: HttpClient/name2idx::Open: 192.168.2.51:9090
2024-09-01 08:16:51.873 MQTT Sync: name2idx::onConnect: Status 0, Description Success
2024-09-01 08:16:51.873 MQTT Sync: name2idx::onHttpConnected:
2024-09-01 08:16:51.873 MQTT Sync: askForDeviceList:
2024-09-01 08:16:51.873 Error: MQTT Sync: Call to function 'onConnect' failed, exception details:
2024-09-01 08:16:51.892 Error: MQTT Sync: Traceback (most recent call last):
2024-09-01 08:16:51.892 Error: MQTT Sync: File "/home/xxx/domoticz/plugins/MqttSync/plugin.py", line 1097, in onConnect
2024-09-01 08:16:51.892 Error: MQTT Sync: onHttpConnected(Connection)
2024-09-01 08:16:51.892 Error: MQTT Sync: File "/home/xxx/domoticz/plugins/MqttSync/plugin.py", line 796, in onHttpConnected
2024-09-01 08:16:51.892 Error: MQTT Sync: askForDeviceList(Connection)
2024-09-01 08:16:51.892 Error: MQTT Sync: File "/home/xxx/domoticz/plugins/MqttSync/plugin.py", line 668, in askForDeviceList
2024-09-01 08:16:51.892 Error: MQTT Sync: authorizationText += ":" + variables.domoticzPassword
2024-09-01 08:16:51.892 Error: MQTT Sync: TypeError: can only concatenate str (not "NoneType") to str
2024-09-01 08:17:11.828 MQTT Sync: name2idx::onDisconnect:

Can you advice?

[DiaDomo]

When I add my domoticz username & password for the master in the URL, I get the following error in the domoticz log:

Code: Select all

2024-09-01 09:34:00.624 MQTT Sync: 'DomoticzBuildTime':'2024-07-13 16:59:31'
2024-09-01 09:34:00.624 MQTT Sync: Device count: 0
2024-09-01 09:34:00.624 MQTT Sync: Debug logging mask set to: PYTHON
2024-09-01 09:34:00.625 MQTT Sync: requestName2IdxData:
2024-09-01 09:34:00.625 MQTT Sync: HttpClient/name2idx::__init__: 192.168.2.51, 9090, False
2024-09-01 09:34:00.625 MQTT Sync: HttpClient/name2idx::Open: 192.168.2.51:9090
2024-09-01 09:34:00.626 MQTT Sync: name2idx::onConnect: Status 0, Description Success
2024-09-01 09:34:00.626 MQTT Sync: name2idx::onHttpConnected:
2024-09-01 09:34:00.626 MQTT Sync: askForDeviceList:
2024-09-01 09:34:00.626 MQTT Sync: askForDeviceList: Send {'Verb': 'GET', 'URL': '/json.htm?type=command&param=getdevices&used=true', 'Headers': {'Content-Type': 'application/json; charset=utf-8', 'Connection': 'keep-alive', 'Accept': 'Content-Type: text/html; charset=UTF-8', 'Host': '192.168.2.51:9090', 'Authorization': 'Basic YWRtaW46ZG9tb3RpY3o=', 'User-Agent': 'Domoticz/1.0'}}
2024-09-01 09:34:00.677 MQTT Sync: name2idx::onMessage:
2024-09-01 09:34:00.677 Error: MQTT Sync: name2idx::onMessage: Error 401 returned by HTTP
2024-09-01 09:34:20.662 MQTT Sync: name2idx::onDisconnect:

Apparently something with 'name2idx'.
IDX 85 & 87 = are both Temp + Humidity sensors
Their names includes spaces like: "Windmeter Temp"