Domoticz

Open source Home Automation System
https://forum.domoticz.com/

Using MQTT for comm between Domoticz instances

https://forum.domoticz.com/viewtopic.php?t=42599

Page 1 of 2

Using MQTT for comm between Domoticz instances

Posted: Wednesday 14 August 2024 13:39
by roseen
I have a Domoticz instance at home, on the local network.
I have also set up a Domoticz instance in the cloud, and I wanted the cloud instance to mirror some of the devices in the home instance. This could have been a piece of cake by configuring the home instance as a Remote Server on the cloud instance.

However, due to my ISP using CGNAT, it is difficult to connect from the internet to my home server. A port mapping in my firewall doesn't help, it is on the ISP level. I need my communication to be initiated on the inside, going out.

So now I started experimenting with MQTT. I set up a broker on the web (using Hive MQ) and got my home Domoticz to publish messages to it nicely (using an MQTT client gateway device).

Now I would like to set up the cloud Domoticz to subscribe to the messages on the broker and map them to devices.
I was hoping that an MQTT Autodiscovery client gateway would do that for me, but nothing shows up.
And I was thinking that the various mentions of "devices that support autodiscovery" that I have read here is a clue? Maybe the messages that the MQTT Client gateway sends is in the wrong format?

Is it possible to use MQTT for Domoticz-Domiticz communication?

Re: Using MQTT for comm between Domoticz instances

Posted: Wednesday 14 August 2024 14:16
by waltervl
MQTT Autodiscovery is only for setting up a gateway to Domoticz. So the Hardware gateway should create autodiscovery topics. Domoticz cannot create those Autodiscovery topics so indeed this is not working out of the box.

Is dynamic DNS working on CGNAT? if so check for an external dynamic DNS service so you can map a domain (eg roseen.se) to your own dynamic IP. Then you can have your cloud Domoticz connect to eg https://domoticz.roseen.se and you can standard device sharing.
But then you also could just have a specific user login to your domoticz instance on https://domoticz.roseen.se and see the devices you want to have them see So you do not need device sharing at all.
I personally use a telegram bot to control my Domoticz devices from outside my home.

Re: Using MQTT for comm between Domoticz instances

Posted: Wednesday 14 August 2024 14:52
by roseen
Unfortunately, DynDNS does not help when the ISP is using CGNAT. I used to have a DynDNS setup for a long time, but now, the ISP has added an additional NAT layer in front of my firewall. Apparently, it is very common nowadays due to the lack of IPv4 addresses.
However, I am investigating if they offer a public IP address for an extra fee.

The Telegram bot sound like an interesting option! I don't use Telegram, but it could still be worth looking into. Do you have some kind of plugin in Domoticz to set it up?

Re: Using MQTT for comm between Domoticz instances

Posted: Wednesday 14 August 2024 15:00
by roseen
Never mind, I found a wealth of articles on the wiki about the Telegram bot... :-)

Re: Using MQTT for comm between Domoticz instances

Posted: Wednesday 14 August 2024 16:01
by solarboy
I use Google Home with Homebridge to control select chosen devices outside of my home network (also CGNAT) on my phone.
It's not perfect and so far I have been unable to use selector switches but for basic use it works well.

Re: Using MQTT for comm between Domoticz instances

Posted: Wednesday 14 August 2024 17:09
by waltervl
roseen wrote: ↑Wednesday 14 August 2024 15:00 Never mind, I found a wealth of articles on the wiki about the Telegram bot... :-)
I use this one, python based, that is not fully described on wiki (which is lua based)
https://github.com/waltervl/dynamicTelegramBot

Re: Using MQTT for comm between Domoticz instances

Posted: Friday 16 August 2024 19:10
by FlyingDomotic
One possible architecture:

On remote site:
  • Setup a classical "MQTT Client Gateway with LAN interface" to get "domoticz/in" topic read
  • Connect it on your remote MQTT instance
On local site:
  • Setup a classical "MQTT Client Gateway with LAN interface" to get "domoticz/out" topic being written on each local change
  • Connect it on your local MQTT instance
  • Write a small piece of python that will listen to "domoticz/out" topic, translate local IDX to remote ones, and send modified message to "domoticz/in" on remote MQTT
I may help on that, if needed.

Re: Using MQTT for comm between Domoticz instances

Posted: Saturday 17 August 2024 0:39
by HvdW
https://stackoverflow.com/questions/329 ... qtt-broker

Read this.
It explains a lot and learns you how to test.
Like

Code: Select all

mosquitto_sub -h 192.168.1.21 -v -t "#"
which displays all the traffic coming from 192.168.1.21

Re: Using MQTT for comm between Domoticz instances

Posted: Saturday 17 August 2024 8:17
by gizmocuz
@roseen, does your provider support IPv6? Most providers do.
Do they charge a lot for a fixed IP address?
Another option is to host it on a cloud server like DigitalOcean

Re: Using MQTT for comm between Domoticz instances

Posted: Sunday 18 August 2024 22:08
by DiaDomo
waltervl wrote: ↑Wednesday 14 August 2024 17:09
roseen wrote: ↑Wednesday 14 August 2024 15:00 Never mind, I found a wealth of articles on the wiki about the Telegram bot... :-)
I use this one, python based, that is not fully described on wiki (which is lua based)
https://github.com/waltervl/dynamicTelegramBot
How to start this bot for the first time?
I read https://github.com/waltervl/dynamicTele ... /README.md
You need to start the bot manually for the first time because it will ask you for information to create an config:
but don't know how to start it?! What is the commandline command? /home/username/scripts/telegram/dynamicTelegramBot/squandorDynamicTelBot.py is not installed on my system.

Re: Using MQTT for comm between Domoticz instances

Posted: Monday 19 August 2024 9:18
by waltervl
The command to run this as a python3 program should be something like

Code: Select all

python3 /home/username/scripts/telegram/dynamicTelegramBot/DynamicTelBot.py

Re: Using MQTT for comm between Domoticz instances

Posted: Monday 19 August 2024 11:40
by DiaDomo
waltervl wrote: ↑Monday 19 August 2024 9:18 The command to run this as a python3 program should be something like

Code: Select all

python3 /home/username/scripts/telegram/dynamicTelegramBot/DynamicTelBot.py
Well, first I had to create the DynamicTelBot.py file (of course...), etc. but now it's working!
One question: I have webcams enabled but how do you get a screenshot from your cam in Telegram?

Re: Using MQTT for comm between Domoticz instances

Posted: Tuesday 20 August 2024 11:05
by waltervl
I have them enabled in Domoticz (menu setup -other - camera) and then the script can forward them to the telegram Bot.

Re: Using MQTT for comm between Domoticz instances

Posted: Wednesday 28 August 2024 10:44
by FlyingDomotic
I just wrote a new Python plugin synchronizing a master (local) instance of Domoticz to a slave (remote) one at https://github.com/FlyingDomotic/domoti ... ync-plugin

You just have to get list of IDX (or names) of devices you want to synchronize on master, and that's it. Everything is automatically created on slave, and updates are propagated from master to slave when they occurs. You can even propagate changes from slave to master, if you allow device to do so.

Connections are established by master to slave, avoiding opening something incoming at home.

Paranoiacs can even create a dedicated user to protect against accidentally configuring a device as being synchronized.

In addition to one master speaking to one slave, you can also have one master connected to multiple slaves, one slave connected to multiple masters, multiple masters connected to multiple slaves. Cascades also supported.

This may be useful if Telegram disappears one day, after its creator being arrested in France, for example ;-)

Re: Using MQTT for comm between Domoticz instances

Posted: Thursday 29 August 2024 9:34
by DiaDomo
Hi, nice plugin!
I want to use it because I have 2 domoticz servers running in my LAN.
One question:

Code: Select all

masterMqttHost": "masterHost",
		"masterMqttPort": "1883",
		"masterMqttUser": "masterMqttUser",
		"masterMqttPassword": "masterMqttPassword",
		"masterDomoticzOutTopic": "domoticz/out",
        "masterDomoticzUrl": "http://masterHost:8080",
		"slaveName": "slave",
		"slaveMqttHost": "slaveHost",
Can masterMqttHost & slaveMqtt the same MQTT-server? In my case I have one mosquitto server running on 192.168.x.y and both domoticz servers are using this.

Re: Using MQTT for comm between Domoticz instances

Posted: Thursday 29 August 2024 10:32
by FlyingDomotic
For sure. You may even run both on the same host (that's the way I do initial tests, using slaveDevicePrefix to differentiate original and sync devices).

Re: Using MQTT for comm between Domoticz instances

Posted: Saturday 31 August 2024 18:58
by DiaDomo
Hi, I used it to sync 2 devices from master to slave.
Both are domoticz instances running on 2 seperate raspberry's.
Both are sharing the same MQTT broker.

Unfortunately, after installing nothing happened....
What could be the case?

My MqttSync.json file:

Code: Select all

{
        "settings": {
                "configVersion": "V1.0.2",
                "masterName": "master",
                "masterMqttHost": "192.168.2.51",
                "masterMqttPort": "1883",
                "masterMqttUser": "",
                "masterMqttPassword": "",
                "masterDomoticzOutTopic": "domoticz/out2",
        "masterDomoticzUrl": "http://192.168.2.51:9090",
                "slaveName": "slave",
                "slaveMqttHost": "192.168.2.51",
                "slaveMqttPort": "1883",
                "slaveMqttUser": "",
                "slaveMqttPassword": "",
        "slaveDomoticzUrl": "http://192.168.2.209:9090",
        "slaveDevicePrefix": "MqttSync - "
        },
        "mapping": [
                {"idx": 85},
                {"idx": 87}
        ]
}
BTW: I'm running domoticz on port 9090.
MQTT explorer dump shows that domoticz/out2 is correct and there is also a mttSync topic.

Re: Using MQTT for comm between Domoticz instances

Posted: Saturday 31 August 2024 21:39
by FlyingDomotic
Did you checked Domoticz log on both machines?

If something is wrong, plugin will certainly write something in log.

However, reading configuration file, it seems:
- There's no user/password on your MQTT instances. Is this the case?
- Domoticz out topic seems not standard (domoticz/out2 instead of domoticz/out). You tell Mqtt Explorer shows it. Did you also have a Domoticz/out? When staying in MqttExplorer, does domotiz/out2 change?
- Both instances are using the same MQTT server at 192.168.2.51. Are the 2 Domoticz instances connected to this server?
- Domoticz url doesn't include username/password. Is master IP range in trusted networks (127.0.0.1 will do it).

But first, as said, have a look to Domoticz log on both instances. If there's an error, plugin will write a line with details. If this is not sufficient, setting debug mode to "Debug" will trace every action (and grow significantly the log ;-)

Re: Using MQTT for comm between Domoticz instances

Posted: Sunday 01 September 2024 8:21
by DiaDomo
Hi, answering your questions: everything correct (no passwords, domoticz/out & domoticz/out2, trusted networks, using the same mqtt server...)

I enabled the debug option:
1) on the slave: no errors
2) on the master: several errors:

Code: Select all

2024-09-01 08:16:51.820 MQTT Sync: 'HomeFolder':'/home/xxx/domoticz/plugins/MqttSync/'
2024-09-01 08:16:51.821 MQTT Sync: 'StartupFolder':'/home/xxx/domoticz/'
2024-09-01 08:16:51.821 MQTT Sync: 'UserDataFolder':'/home/xxx/domoticz/'
2024-09-01 08:16:51.821 MQTT Sync: 'Database':'/home/xxx/domoticz/domoticz.db'
2024-09-01 08:16:51.821 MQTT Sync: 'Language':'en'
2024-09-01 08:16:51.821 MQTT Sync: 'Version':'1.0.0'
2024-09-01 08:16:51.821 MQTT Sync: 'Author':'Flying Domotic'
2024-09-01 08:16:51.821 MQTT Sync: 'Name':'MQTT Sync'
2024-09-01 08:16:51.821 MQTT Sync: 'Port':'0'
2024-09-01 08:16:51.821 MQTT Sync: 'Key':'domoticz-mqtt-sync'
2024-09-01 08:16:51.821 MQTT Sync: 'Mode1':'MqttSync.json'
2024-09-01 08:16:51.821 MQTT Sync: 'Mode5':'Master'
2024-09-01 08:16:51.821 MQTT Sync: 'Mode6':'Debug'
2024-09-01 08:16:51.821 MQTT Sync: 'DomoticzVersion':'2024.7'
2024-09-01 08:16:51.821 MQTT Sync: 'DomoticzHash':'b317cfab1'
2024-09-01 08:16:51.821 MQTT Sync: 'DomoticzBuildTime':'2024-07-13 16:59:31'
2024-09-01 08:16:51.821 MQTT Sync: Device count: 0
2024-09-01 08:16:51.821 MQTT Sync: Debug logging mask set to: PYTHON
2024-09-01 08:16:51.822 MQTT Sync: requestName2IdxData:
2024-09-01 08:16:51.822 MQTT Sync: HttpClient/name2idx::__init__: 192.168.2.51, 9090, False
2024-09-01 08:16:51.822 MQTT Sync: HttpClient/name2idx::Open: 192.168.2.51:9090
2024-09-01 08:16:51.873 MQTT Sync: name2idx::onConnect: Status 0, Description Success
2024-09-01 08:16:51.873 MQTT Sync: name2idx::onHttpConnected:
2024-09-01 08:16:51.873 MQTT Sync: askForDeviceList:
2024-09-01 08:16:51.873 Error: MQTT Sync: Call to function 'onConnect' failed, exception details:
2024-09-01 08:16:51.892 Error: MQTT Sync: Traceback (most recent call last):
2024-09-01 08:16:51.892 Error: MQTT Sync: File "/home/xxx/domoticz/plugins/MqttSync/plugin.py", line 1097, in onConnect
2024-09-01 08:16:51.892 Error: MQTT Sync: onHttpConnected(Connection)
2024-09-01 08:16:51.892 Error: MQTT Sync: File "/home/xxx/domoticz/plugins/MqttSync/plugin.py", line 796, in onHttpConnected
2024-09-01 08:16:51.892 Error: MQTT Sync: askForDeviceList(Connection)
2024-09-01 08:16:51.892 Error: MQTT Sync: File "/home/xxx/domoticz/plugins/MqttSync/plugin.py", line 668, in askForDeviceList
2024-09-01 08:16:51.892 Error: MQTT Sync: authorizationText += ":" + variables.domoticzPassword
2024-09-01 08:16:51.892 Error: MQTT Sync: TypeError: can only concatenate str (not "NoneType") to str
2024-09-01 08:17:11.828 MQTT Sync: name2idx::onDisconnect:
Can you advice?

Re: Using MQTT for comm between Domoticz instances

Posted: Sunday 01 September 2024 9:36
by DiaDomo
When I add my domoticz username & password for the master in the URL, I get the following error in the domoticz log:

Code: Select all

2024-09-01 09:34:00.624 MQTT Sync: 'DomoticzBuildTime':'2024-07-13 16:59:31'
2024-09-01 09:34:00.624 MQTT Sync: Device count: 0
2024-09-01 09:34:00.624 MQTT Sync: Debug logging mask set to: PYTHON
2024-09-01 09:34:00.625 MQTT Sync: requestName2IdxData:
2024-09-01 09:34:00.625 MQTT Sync: HttpClient/name2idx::__init__: 192.168.2.51, 9090, False
2024-09-01 09:34:00.625 MQTT Sync: HttpClient/name2idx::Open: 192.168.2.51:9090
2024-09-01 09:34:00.626 MQTT Sync: name2idx::onConnect: Status 0, Description Success
2024-09-01 09:34:00.626 MQTT Sync: name2idx::onHttpConnected:
2024-09-01 09:34:00.626 MQTT Sync: askForDeviceList:
2024-09-01 09:34:00.626 MQTT Sync: askForDeviceList: Send {'Verb': 'GET', 'URL': '/json.htm?type=command&param=getdevices&used=true', 'Headers': {'Content-Type': 'application/json; charset=utf-8', 'Connection': 'keep-alive', 'Accept': 'Content-Type: text/html; charset=UTF-8', 'Host': '192.168.2.51:9090', 'Authorization': 'Basic YWRtaW46ZG9tb3RpY3o=', 'User-Agent': 'Domoticz/1.0'}}
2024-09-01 09:34:00.677 MQTT Sync: name2idx::onMessage:
2024-09-01 09:34:00.677 Error: MQTT Sync: name2idx::onMessage: Error 401 returned by HTTP
2024-09-01 09:34:20.662 MQTT Sync: name2idx::onDisconnect:
Apparently something with 'name2idx'.
IDX 85 & 87 = are both Temp + Humidity sensors
Their names includes spaces like: "Windmeter Temp"
All times are UTC+01:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited