[90] MQTT Support

[wblock]

I have built a bunch of sensors based on ESP8266/NodeMCU hardware. They report to a Mosquitto MQTT server with topics like office/temperature and office/humidity, or shed/temperature and shed/batteryvoltage. This all works, and I can see the published data with mosquitto_sub. (I looked for existing protocols for sensors, and MQTT seemed to be the thing to use.) For example:

Code: Select all

office/temperature 65
office/humidity 39
shed/temperature 33
shed/humidity 64
shed/batteryvoltage 12.75

The next step is to provide web graphs and, hopefully, alarms. Domoticz looked good because it does not rely on PHP or Java, but the MQTT graphic in the wiki (https://www.domoticz.com/wiki/MQTT) has me wondering if it is the right tool for the job.

I can add the broker (running on the same machine as Domoticz), but then don't see a way to add sensors attached to that broker. It appears that setting this up might involve Lua, Node.js, JSON, and reprogramming all the sensors. I was hoping to just add the topics in a GUI, though.

Maybe there is a plugin or add-on that simplifies this?

Thanks!

[wildfire]

wblock,
at this point you domoticz only listens on domoticz/in and you have to use Json.
also you have to add the idx of the device and you have to create it first, at least thats how i managed to get this working.
so you make a dummy hardware then a virtual device, then check the idx and then send to that idx.
example for temperature
{
"idx" : 8,
"nvalue" : 0,
"svalue" : "26.2"
}
example for humi
{
"idx" : 14,
"nvalue" : 50,
"svalue" : ""
}

i added a node-red flow to change this.
also i noticed it was buggy when you post some sensors a the same time, i changed to the json post method for now.

[shkaff]

Dear community, let me please ask you to confirm my understanding that domoticz isn't actually an MQTT client and has no settings to specify to which MQTT topic to subscribe.
The platform Domoticz only allows external applications like Node-RED script to manipulate with MQTT messages and send / receive them to / from Domoticz, right?

I looked thru settings of Domoticz and found no way to specify an address of Mosquito or any other MQTT broker to subscribe

[Kouseri]

Dear community, let me please ask you to confirm my understanding that domoticz isn't actually an MQTT client and has no settings to specify to which MQTT topic to subscribe.
The platform Domoticz only allows external applications like Node-RED script to manipulate with MQTT messages and send / receive them to / from Domoticz, right?

I looked thru settings of Domoticz and found no way to specify an address of Mosquito or any other MQTT broker to subscribe

Have you created a MQTT Client Gateway with LAN Interface device in Setup/Hardware?

If you haven't, please take a look available settings of the MQTT Client Gateway with LAN Interface device. According to my current understanding it's not possible
to subscribe to any other topic than "domoticz/in" but there are some options to modify publish topics though...

[Nautilus]

Hi,

In my Raspberry Pi setup there has been some inconsistency with the MQTT hardware in Domoticz (not receiving anything although everything should be in order with both Domo & Mosquitto / Node-red). Therefore I set the "Data Timeout" to 30 minutes in hopes of the connection would reset itself if no data received through this hardware for 30 minutes. The result, however, is that it resets the connection every 30 minutes like there's no data received ever (it is receiving messages every 5 minutes in domoticz/in and publishing as well to domoticz/out). Is this because I don't have any devices created where MQTT would be set as the hardware? I just have devices that are updated through messages coming through this channel...

Also, the inconsistency with this channel continues, e.g. this morning there was in the log:

Code: Select all

2015-12-14 06:42:31.028  Error: Restarting: MQTT

and after this one update and then two hours of "down-time" (until restart of Domoticz). So it did not even try to restart it after that anymore even though now clearly there was a timeout in receiving data through this channel for more than 30 minutes. Have others had problems with this and any ideas what I could do to monitor that MQTT channel is working?

[shkaff]

...MQTT Client Gateway with LAN Interface device...

Aha! That's the way to go! Thanks, Kouseri!
I would agree with the topic starter of How to use "MQTT Client Gateway with LAN interface" that the description of MQTT at WIKI isn't clear.
Will be digging further.

[Nautilus]

Code: Select all

2015-12-14 06:42:31.028  Error: Restarting: MQTT

and after this one update and then two hours of "down-time" (until restart of Domoticz).

Same thing as above happening again, except start was at around 3:30AM this time and restart happened at 9:30. I noticed now that after this also Wunderground and Weather IO stop updating until the next restart (other services still update). Also, the restart seems to happen at the time of logging in to Domoticz (which I believe is because the service freezes - well at least the page keeps loading and loading - and then Monit kicks in).

So maybe it is MQTT, maybe Weather IO or Wunderground or some combination of these that is causing the issue. I've now disabled the weather hardware, let's see if this improves the situation

[Toni]

My MQTT-based sensors have also randomly stopped working in the latest stable version (I think it was V2.3530). Before this I have been running on MQTT for months without problems.

Yesterday, in hope of new fixes, I switched to the beta channel and updated to V2.3852, but the same problem continues. For example, I restarted Domoticz yesterday, and this morning all sensors had stopped updating at around 7AM (EET, it's now 9AM). The 'domoticz/in' topic does get messages, I verified this with 'mosquitto_sub -t domoticz/in', and the sensors will again start working when I restarted Domoticz.

[Toni]

I made a Lua script to watch when my outdoor sensor has been last updated, and keep track of the number of Lua script executions with a user variable. If this is not updated frequently enough (should be updated every minute), the script will restart Domoticz. It also increments another user variable, so I have an idea on how frequently this happens.

[Toni]

11 restarts due to MQTT stopped responding, since yesterday. It's definately broken... Any ideas on where to start looking for the root cause?

[Toni]

Anybody? I have updated to the latest beta a number of times, and I still face this problem daily. Looking at 'mosquitto_sub' output, I can see that MQTT messages are coming as they should, and there doesn't seem to be any issues with anything like corrupted data etc. Still Domoticz stops responding to the MQTT messages, and only a restart of Domoticz helps.

[wildfire]

Tony,
i also had problems with input from mqtt and moved away from updating sensors using mqtt, it keeps crashing domoticz.
i use node-red to get the mqtt input and then post it using HTTP requests.
I have 10 sensors up and running, posting every 10 seconds, i do notice domoticz slows down of all the http posts, i realy wish mqtt would get fixed.
if you need help using node-red, ill help if i can.

[Toni]

I can actually switch into the HTTP JSON api fairly easily, I'd just like to use MQTT interface instead Bummer...

There's absolutely nothing in the Domoticz log, so I don't even know where to start looking... Perhaps I should start by writing some small breakpoint script with gdb. First I would just need to learn gdb. It almost sounds like the MQTT handling thread would exit or something. Mosquitto and lsof both show that Domoticz is still connected to the tcp socket, it just doesn't seem to process anything.

[wildfire]

yup, thats what i get to, and there are no logs or noting to find what happens

[Toni]

I would be happy to produce logs etc. if somebody could help me in getting started. This happens several times a day on my Raspberry. Currently I'm just logging the domoticz/in topic into a file (including timestamps), as well as the domoticz.log. I think a gdb breakpoint script would be useful, I'd just need to find time (or preferably get help from somebody) to get started with in. I.e. what function calls to break, and what to run in the breakpoint script etc.

[kylegordon]

Ok so this is the same behaviour that I've been living with for months. I thought it was just my configuration, but apparently not. I'll have another search around to see what's going on. I'm tempted to try starting domoticz under strace, and logging everything...

Sent from my A0001 using Tapatalk

[kylegordon]

Came home to a dark house with the heating on. This morning I woke up to cold dark house... The past few iterations of the master branch have been getting worse and worse, but I put it down to something in my configuration. I've been kill -9'ing it it at least twice a day this year.

Anyway, it's now stracing to a log file. It's not quite gdb, but it might show something.

[nayr]

Might go create an issue on github so the devs notice this issue, with all these reports of crashes its unlikely to be caused by user error.

There is enough people here indicating a severe bug in the code, so I think creating an issue report at this point is justified and unlikely to annoy any of the devs.

Sounds like its deadlocking the MQTT interface, any traces you can get would be a big help.. might force a core when its not responding and run a gdb backtrace on it and report the output.

Add:

Code: Select all

ulimit -c unlimited

to your init script so Domoticz dont have any limit on core sizes, then kill -3 to force a core on the proccess after MQTT is hung, finally run a backtrace

Code: Select all

gdb ~/domoticz/domoticz corefile
thread apply all bt full

It'll give alot of output, but with any luck we'll see what function the MQTT thread is hung on.

[kylegordon]

Cool, I'll do that too. I'll look for, or open, an issue when I have some more output from strace. Will also dump it using gdb.

At this rate... 12 hours and I should have something

[nayr]

added a small example script to the Wiki to help people get started: https://www.domoticz.com/wiki/MQTT#Publish
It has functions for all basic publishing needs, nothing yet for parsing subscriptions.

I am thinking of writing a node-domoticz-mqtt module, that exports functions for watching incoming json for specific devices and firing off calls, and other functions for easy device control.

It shouldn't be all that difficult, put it on NPM and I think MQTT will become alot more friendly of an interface.. I just havent written anything for combing through /domoticz/out yet, but I need to for projects I am working on, anyone have anything already?