[90] MQTT Support

[RvRijsselt]

Having Domoticz subscribe to a single topic (domoticzIn/) means that all sensors and gateways must be modified or configured to know about Domoticz. This is a bad idea imho. You would be forcing all device developers to learn about Domoticz and its interface (not realistic)... or alternatively to make the user responsible for configuring it correctly (very cumbersome).

Personally when I install something then I want to do the least amount of configuration. So when I install for example Owntracks then it will publish to owntracks/. If I want to make Domoticz aware of the published data then I would expect to only have to configure Domoticz. In other words, I would expect Domoticz to subscribe to owntracks/#/# and process any data that is published to it.

[gizmocuz]

Thats also for the other way around.

Sensors can send 'anything' to the topic, and domoticz should understand this ?

working on the MySensors MQTT implementation at the moment.
(But with an own sketch), messages are send to the topic, the body will contain the original mysensor messages

But it depends if you want to use an existing broker, if its only for use between domoticz instances, then the message can be
Idx/nvalue/svalue

[RvRijsselt]

MySensors looks like a cool project. Myself, I bought a couple of Spark Cores which basically have the same functionality.

When I was building a remote control for my ventilation system (based on RS232 and the Spark Core) I didn't want it to be dependent on my Domoticz system. So I chose to publish to /device/wtw/temperatures (data is for example "18.5, 19.5, 20.0, 18.0") and /device/wtw/state (data is 1,2 or 3). Subscription and data parsing is currently done by some nodejs scripts and passed to Domoticz via the rest api. I'm also working a html5 website which connects to these topics via mqtt websockets, so I thought it would be strange to subscribe to for example domoticzIn/<some number>/.

It is true that sensors can send anything (even the format of my own temperatures message is very confusing, (in my own defense I tried Json first but thought it was a bit overkill to have a machine print readable text for another machine)). I don't really have an answer on how to solve this. I can imagine some kind of 'device driver' which can be enabled per type of mqtt enabled device, much like the current hardware configuration.

PS. I miss a tri-state button type in Domoticz for my ventilation system. I want to switch it between low, normal and high...ahh wrong topic

[gskout]

I think mqtt was created with simplicity in mind.
All parties should be configurable at some point, so you finally make them talk together.
Should be a single connection between any subscriber-publisher with the broker, and topics are the channels.
Someone should have basic mqtt knowledge to make things work, but involving node.js or other stuff means you have deeper knowledge of your system and beyond the 'simple user'.
It's not about domoticz dependence, but a way to make things work together, provided that hardware can be configured for basic staff.
Maybe the proposed messaging is not basic, but if someone is aware of a standard way, why don't let others know?
It's not easy to make software understand the 'whole world' different messages without having some rules.
Probably configuration files per hardware that would map topics and messages to internal domoticz structures, could be a food idea to start.

[maxtrash]

I agree that it would make sense if Domoticz would both subscribe and publish to topics.

the incoming MQTT interface (Domoticz subscribing to a topic) could be similar to the current JSON-solution. This makes for the easiest implementation and yes it would be specific for Domoticz. The JSON-interface supports creating devices, so the same would apply to the MQTT-interface.
In my current experiments i'm publishing all changed-device events to MQTT using LUA (so that includes newly created devices as soon as they exist). The subscribers to this topic can decided themselves which topics/messages to act upon. Here I can also see a format similar to how Domoticz is storing information. E.g. idx/nvalue/svalue/ etc.

in my perception any sensor or whatever publishing mqtt-messages will probably have it's own format and any subscriber to an mqtt topic would like to receive messages in another format. There is no general standard here. So I think in ALL cases some sort of republishing (or gateway aas gskout calls it) is needed. I'm experimenting with node-red which I think could be very nice for this purpose.
of course, later on Domoticz could add it's own functionalities for this. I can imagine that the LUA engine in Domoticz could be used to transform incoming MQTT-messages to change them to the Domoticz native format, thereby eliminating the need for external republishing. But that would be optional in my perception.

I'll try and share some of my node-red stuff but I'm kind of lacking time currently...

[maxtrash]

Like I've explained before I'm looking at Node-Red. http://nodered.org/ is a "A visual tool for wiring the Internet of Things".

As a proof of concept this is what I've done:

1. Made a script called script_device_nodered.lua for Domoticz

It looks like this:

Code: Select all

--[[
Node-red device script

devicechanges are pushed to localhost port 1880 if there's any value to publish
node-red will be listening and will respond by calling back Domoticz in order to retrieve the status using the JSON interface

NB. Domoticz will add _ and property to the name of certain sensors (e.g. _temperature, _humidity). These are all ignored by filtering on value = empty
--]]

NodeRedUrl = "127.0.0.1:1880/domoticz"
commandArray = {}

logging = false
counter = 1

for i, v in pairs(devicechanged) do
	if (logging) then print ("i="..i, "v="..v) end
	value = v .. ""		-- make sure value is a string
	if (#value == 0) then
		commandArray[counter]={['OpenURL']=string.gsub(NodeRedUrl..'?device='..i, "%s+", '%%20')}
		counter = counter +1
	end
end

return commandArray

2. In Node-Red I made a publish flow



The first flow, which I call "Domoticz out" consists of a "http in" node which is listening to the messages which are sent by the LUA script. The LUA script pushes the names of changed devices to Node-red. Node-red will first do a http-request back to Domoticz to get the status of all devices. Then it will filter out the device which was sent from LUA and finally it will publish it to MQTT in two different ways. First as a message to topic /domoticz/out/json containing the full json-status message. Secondly it will post the contents of the Data-field to topic /domoticz/out/device/[devicename]. Obviously this can be expanded with publishing to all sorts of MQTT-topics.

Code: Select all

[{"id":"86f0c28c.790f4","type":"mqtt-broker","broker":"localhost","port":"1883","clientid":""},{"id":"b5e13be0.4a1ec8","type":"http in","name":"domoticz","url":"/domoticz","method":"get","x":90,"y":28,"z":"d8820d48.277df","wires":[["f0adae54.0f525","3eb9cdbe.c14632"]]},{"id":"f0adae54.0f525","type":"http response","name":"","x":268,"y":28,"z":"d8820d48.277df","wires":[]},{"id":"f83a1546.07c5e8","type":"mqtt out","name":"","topic":"/domoticz/out/json","qos":"","retain":"","broker":"86f0c28c.790f4","x":743,"y":243,"z":"d8820d48.277df","wires":[]},{"id":"5bdbbd55.a42444","type":"inject","name":"","topic":"","payload":"{\"device\":\"Elektriciteit\"}","payloadType":"string","repeat":"","crontab":"","once":true,"x":88,"y":138,"z":"d8820d48.277df","wires":[["8e8dc74e.717238"]]},{"id":"8e8dc74e.717238","type":"http request","name":"","method":"GET","ret":"txt","url":"http://192.168.1.24:8080/json.htm?type=devices","x":309,"y":138,"z":"d8820d48.277df","wires":[["6b8cff2c.9473","150db3be.eaf24c"]]},{"id":"e7db09cb.1824f8","type":"function","name":"FilterResult","func":"var data = msg.payload.result;\n\nfunction FilterResultsByName(Name) {\n  return data.filter(\n      function(data){return data.Name == Name}\n  );\n}\n\narray = FilterResultsByName(msg.device);\nif (array.length > 0) { \n\tmsg.payload.result = array[0];\n\tmsg.idx = msg.payload.result.idx;\n}\n\nreturn msg;","outputs":1,"x":481,"y":249,"z":"d8820d48.277df","wires":[["a951a4b1.56ae58","f83a1546.07c5e8","6696a892.996958","7e40be52.81bf4"]]},{"id":"a951a4b1.56ae58","type":"debug","name":"","active":false,"console":"false","complete":"idx","x":624,"y":122,"z":"d8820d48.277df","wires":[]},{"id":"3eb9cdbe.c14632","type":"function","name":"SaveDevice","func":"msg.device = msg.payload.device;\nreturn msg;","outputs":1,"x":201,"y":85,"z":"d8820d48.277df","wires":[["8e8dc74e.717238"]]},{"id":"150db3be.eaf24c","type":"debug","name":"","active":false,"console":"false","complete":"false","x":598,"y":70,"z":"d8820d48.277df","wires":[]},{"id":"6696a892.996958","type":"debug","name":"","active":false,"console":"false","complete":"device","x":677,"y":177,"z":"d8820d48.277df","wires":[]},{"id":"6b8cff2c.9473","type":"json","name":"","x":434,"y":184,"z":"d8820d48.277df","wires":[["e7db09cb.1824f8"]]},{"id":"3b97f57b.c4680a","type":"debug","name":"","active":true,"console":"false","complete":"payload","x":754,"y":315,"z":"d8820d48.277df","wires":[]},{"id":"507bd6e6.af8428","type":"mqtt out","name":"","topic":"","qos":"","retain":"","broker":"86f0c28c.790f4","x":745,"y":379,"z":"d8820d48.277df","wires":[]},{"id":"7e40be52.81bf4","type":"function","name":"Repub ()","func":"var newmsg = {};\nnewmsg.payload = msg.payload.result.Data;\nnewmsg.topic = \"/domoticz/out/device/\"+msg.payload.result.Name;\n\nreturn newmsg;","outputs":1,"x":589,"y":319,"z":"d8820d48.277df","wires":[["3b97f57b.c4680a","507bd6e6.af8428"]]}]

this is wat the message look like on the device topic hierarchy:



3. A flow to send back commands to Domoticz

For the opposite direction I've created another flow. This is a very simple example which will post messages from MQTT to Domoticz.

Code: Select all

[{"id":"86f0c28c.790f4","type":"mqtt-broker","broker":"localhost","port":"1883","clientid":""},{"id":"339707e6.cc68f8","type":"http request","name":"","method":"GET","ret":"txt","url":"","x":523,"y":285,"z":"8b1c06b3.74e3f8","wires":[["8f3947d6.70c6b8"]]},{"id":"c8a94fa4.3756b","type":"mqtt in","name":"","topic":"/domoticz/in/command/udevice/#","broker":"86f0c28c.790f4","x":162,"y":384,"z":"8b1c06b3.74e3f8","wires":[["57b203b3.a84dfc"]]},{"id":"8f3947d6.70c6b8","type":"debug","name":"","active":true,"console":"false","complete":"true","x":720,"y":381,"z":"8b1c06b3.74e3f8","wires":[]},{"id":"57b203b3.a84dfc","type":"function","name":"ConstructURL","func":"var parts = msg.topic.split(\"/\");\nmsg.idx = parts[parts.length-1];\nmsg.svalue = msg.payload;\nmsg.url = 'http://192.168.1.24:8080/json.htm?type=command&param=udevice&idx='+msg.idx+'&svalue='+msg.svalue;\nreturn msg;","outputs":1,"x":405,"y":362,"z":"8b1c06b3.74e3f8","wires":[["8f3947d6.70c6b8","339707e6.cc68f8"]]}]

Now the message mosquitto_pub -t /domoticz/in/command/udevice/211 -m 'This is a text message'
will update the device in domoticz:



4. Connecting the internet of things

Now that we've got the MQTT-stuff in place let's do some interfacing.

In this example I created a flow which subscribes to MQTT topic /domoticz/out/device/Gas



In the flow Node-red will check if the time is within 10 minutes past midnight and if so post the current Gas-meter value to mindergas.nl which is a dutch site for keeping track of gas-usage.

Code: Select all

[{"id":"86f0c28c.790f4","type":"mqtt-broker","broker":"localhost","port":"1883","clientid":""},{"id":"ce6aad20.31955","type":"mqtt in","name":"","topic":"/domoticz/out/device/Gas","broker":"86f0c28c.790f4","x":121,"y":97,"z":"2697221f.d968de","wires":[["d54ab5d6.2ab548"]]},{"id":"dc6d59eb.2392a8","type":"http request","name":"Mindergas.nl","method":"POST","ret":"txt","url":"https://www.mindergas.nl/api/gas_meter_readings","x":521,"y":186,"z":"2697221f.d968de","wires":[["30f622a5.cf09de"]]},{"id":"bd9f730c.42609","type":"inject","name":"","topic":"","payload":"{\"result\": {\"Data\":\"6014.32\"}}","payloadType":"string","repeat":"","crontab":"","once":false,"x":134,"y":38,"z":"2697221f.d968de","wires":[["d54ab5d6.2ab548"]]},{"id":"30f622a5.cf09de","type":"debug","name":"","active":false,"console":"false","complete":"true","x":688,"y":147,"z":"2697221f.d968de","wires":[]},{"id":"81df4693.7e20b8","type":"function","name":"Set values","func":"var newmsg = {};\nvar yesterday = new Date(new Date().setDate(new Date().getDate()-1));\nvar day = (\"0\" + yesterday.getDate()).slice(-2);\nvar month = (\"0\" + (yesterday.getMonth() + 1)).slice(-2);\nvar senddate = yesterday.getFullYear() + \"-\" + (month) + \"-\" + (day);\n\nif ((yesterday.getMinutes() < 10) && (yesterday.getHours() == 0)) {\t\t\t\n\tnewmsg.headers = { \"Content-Type\":\"application/json\" };\n\tnewmsg.headers[\"AUTH-TOKEN\"] = \"put your token here\";\n\t\n\tnewmsg.payload = {};\n    newmsg.payload.date = senddate;\n    newmsg.payload.reading = msg.payload;\n\treturn newmsg;\n}\n        \n","outputs":1,"x":389,"y":134,"z":"2697221f.d968de","wires":[["dc6d59eb.2392a8","30f622a5.cf09de"]]},{"id":"d54ab5d6.2ab548","type":"json","name":"","x":286,"y":78,"z":"2697221f.d968de","wires":[["81df4693.7e20b8","30f622a5.cf09de"]]}]

In Node-red lot's of other little programs could be made which listen to a MQTT-topic and then do whatever is needed. Lot's of nodes are available, like nodes to send message to twitter, send e-mail, store data in a database, convert to XML/HTML etc.
By the way: technically MQTT is not really needed since the flows could also be linked directly instead of through MQTT. But this is a proof of concept so that's why I've done it like this.

Hope to hear what you guys think of all this!

[sj3fk3]

awesome

[gizmocuz]

new version #2524

- now sends all device updates to mqtt (json format)
- now possible to send updates to domoticz via mqtt

output topic: domoticz/out
input topic: domoticz/in

example received from output:

Code: Select all

{ "dtype" : "Temp + Humidity", "id" : "47118", "idx" : 2433, "name" : "Unknown", "nvalue" : 0, "stype" : "Cresta, TFA TS34C", "svalue1" : "7.6", "svalue2" : "75", "svalue3" : "3", "unit" : 4 }

example to send to domoticz/in

Code: Select all

{"idx": 123, "nvalue": 1, "svalue": "123;456" }

i will make a new raspberry sdcard image for the next stable release including nodejs/node-red (auto start)

[maxtrash]

Excellent, ik do you want me to do anything on the node red side?

[gizmocuz]

Not at the moment, maybe you can test the new behaviour (we still need to have user defined out/in topics, but thats for later)
If its working, maybe you can adjust the mqtt wiki

i dont think we need any special scripts anymore now

[maxtrash]

Yes, the lua script should be obsolete now . I'll have a look at the wiki and stuff whenever I have some time.

[gizmocuz]

once we have a new stable sdcard image... i think that everything is possible.... maybe we should create a forum for this.

i am quite new to node-red, maybe its possible to store values, and after 5 minutes we can push data to for instance pvoutput, or mindergas etc...

some people want to log to a mysql database, should also be possible

[SweetPants]

new version #2524

- now sends all device updates to mqtt (json format)
- now possible to send updates to domoticz via mqtt

output topic: domoticz/out
input topic: domoticz/in

This is great, now I can make my MQTT devices 'talk/listen' to domoticz.
Many thanks

[maxtrash]

I'm looking at the new MQTT functionality. The messages are coming in fine!
The stuff I tried before in this topic is done using the results of a http /json.htm?type=devices call.
Comparing that to the new MQTT messages they look like this:

Code: Select all

topic: domoticz/out (sent by Domoticz itself)
{
   "dtype" : "Temp + Humidity",
   "id" : "17422",
   "idx" : 17,
   "name" : "TX320 Buiten",
   "nvalue" : 0,
   "stype" : "Cresta, TFA TS34C",
   "svalue1" : "16.9",
   "svalue2" : "66",
   "svalue3" : "1",
   "unit" : 2
}

domoticz/out/json (information as received from http call)
{"ActTime":1433580536,"ServerTime":"Jun 06 2015 10:48:56","Sunrise":"05:25:00","Sunset":"21:51:00","result":{"AddjMulti":1,"AddjMulti2":1,"AddjValue":0,"AddjValue2":0,"BatteryLevel":100,"CustomImage":0,"Data":"16.9 C, 66 %","DewPoint":"10.51","Favorite":1,"HardwareID":1,"HardwareName":"Rfxcom","HaveTimeout":false,"Humidity":66,"HumidityStatus":"Comfortable","ID":"440E","LastUpdate":"2015-06-06 10:48:55","Name":"TX320 Buiten","Notifications":"true","PlanID":"0","Protected":false,"ShowNotifications":true,"SignalLevel":5,"SubType":"Cresta, TFA TS34C","Temp":16.9,"Timers":"false","Type":"Temp + Humidity","TypeImg":"temperature","Unit":2,"Used":1,"XOffset":"0","YOffset":"0","idx":"17"},"status":"OK","title":"Devices"}

The http-call gives back more information. Some of that might be interesting (battery level, "comfortable")

same is true for some AC-switch

Code: Select all

MQTT output from domoticz
{"dtype":"Lighting 2","id":"03A6D2A","idx":8,"name":"Kubusjes","nvalue":0,"stype":"AC","svalue1":"0","unit":1}

output from JSON
{
   "ActTime" : 1433583674,
   "ServerTime" : "Jun 06 2015 11:41:14",
   "Sunrise" : "05:25:00",
   "Sunset" : "21:51:00",
   "result" : [
      {
         "AddjMulti" : 1.0,
         "AddjMulti2" : 1.0,
         "AddjValue" : 0.0,
         "AddjValue2" : 0.0,
         "BatteryLevel" : 255,
         "CustomImage" : 0,
         "Data" : "Off",
         "Favorite" : 1,
         "HardwareID" : 1,
         "HardwareName" : "Rfxcom",
         "HaveDimmer" : true,
         "HaveGroupCmd" : true,
         "HaveTimeout" : false,
         "ID" : "03A6D2A",
         "Image" : "Light",
         "IsSubDevice" : false,
         "LastUpdate" : "2015-06-06 11:41:09",
         "Level" : 0,
         "LevelInt" : 0,
         "MaxDimLevel" : 15,
         "Name" : "Kubusjes",
         "Notifications" : "false",
         "PlanID" : "0",
         "Protected" : false,
         "ShowNotifications" : true,
         "SignalLevel" : 7,
         "Status" : "Off",
         "StrParam1" : "",
         "StrParam2" : "",
         "SubType" : "AC",
         "SwitchType" : "On/Off",
         "SwitchTypeVal" : 0,
         "Timers" : "false",
         "Type" : "Lighting 2",
         "TypeImg" : "lightbulb",
         "Unit" : 1,
         "Used" : 1,
         "UsedByCamera" : false,
         "XOffset" : "0",
         "YOffset" : "0",
         "idx" : "8"
      }
   ],
   "status" : "OK",
   "title" : "Devices"
}

what do you think?

[gizmocuz]

That is to much information and not needed for MQTT

i agree on the signal/battery level, what else is valid data to give back ?

[maxtrash]

That is to much information and not needed for MQTT

i agree on the signal/battery level, what else is valid data to give back ?

Will have a look later, but whatever you pick is fine by me! I think I'll create a little proof of concept flow which can be used for the small minority of people who would like additional info from Domoticz which is not in the MQTT-interface. I've started editting the wiki (http://www.domoticz.com/wiki/Setup_MQTT ... al_devices) and created a new architecture drawing

[gizmocuz]

Yes! this took me some time....

i now have a flow that logs everything from mqtt into InfluxDB !
(later i will try to use grafana to plot it!)

Code: Select all

[{"id":"b8bfeb9b.474018","type":"mqtt-broker","broker":"127.0.0.1","port":"1883","clientid":""},{"id":"3a5610dc.c5a9f","type":"mqtt in","name":"Domotic Out","topic":"domoticz/out","broker":"b8bfeb9b.474018","x":102,"y":51,"z":"821a73e2.7de59","wires":[["602f86f6.9fd078","de998181.7b26a8"]]},{"id":"ac1edc15.53e12","type":"debug","name":"Original Input","active":false,"console":"false","complete":"payload","x":528,"y":94,"z":"821a73e2.7de59","wires":[]},{"id":"602f86f6.9fd078","type":"function","name":"Message Splitter","func":"//original message will be published on PIN1\n\n//console.log(msg);\n\nvar date = new Date(msg.payload);\n\ndata=JSON.parse(msg.payload);\n\n\nvar newMsg = { payload: data.dtype };\nreturn [msg, newMsg];","outputs":"2","valid":true,"x":293,"y":138,"z":"821a73e2.7de59","wires":[["ac1edc15.53e12"],[]]},{"id":"de998181.7b26a8","type":"function","name":"ToInfluxDB","func":"var date = new Date(msg.payload);\ndata=JSON.parse(msg.payload);\n\nmsg.payload = {}; //null msg obj and create new properties\n\nvar influx_obj = \"[{\";\n\n\ninflux_obj += '\"name\": \"' + data.dtype.split(\" + \").join(\"\") + '\",';\nvar columns='\"columns\": [';\n\ncolumns+='\"idx\", \"nvalue\"';\n\nvar points='\"points\": [[';\n\npoints+= data.idx + ', ' + data.nvalue;\n\nfor (ii=0; ii<10; ii++)\n{\n    var vidx = (ii+1).toString();\n    var vname = \"svalue\" + vidx;\n    if (typeof data[vname] != 'undefined') {\n        columns+=', \"' + vname + '\"';\n        var vdata=data[vname];\n        if (!isNaN(vdata)) {\n            points+=', ' + parseFloat(vdata);\n        }\n        else {\n            points+=', \"' + vdata + '\"';\n        }\n    }\n}\n\n\ncolumns+=']';\npoints+=']]';\n\ninflux_obj+=columns+\",\" + points;\n\ninflux_obj+=\"}]\";\n\nmsg.payload=influx_obj;\n\nreturn msg;","outputs":1,"valid":true,"x":283,"y":203,"z":"821a73e2.7de59","wires":[["137700f1.3aacb7","8ccc317c.b1fc48"]]},{"id":"137700f1.3aacb7","type":"debug","name":"","active":true,"console":false,"complete":false,"x":545.0833129882812,"y":164.08331298828125,"z":"821a73e2.7de59","wires":[]},{"id":"8ccc317c.b1fc48","type":"http request","name":"","method":"POST","ret":"txt","url":"http://192.168.0.148:8086/db/domoticz/series?u=root&p=root","x":488.08331298828125,"y":231.08331298828125,"z":"821a73e2.7de59","wires":[["3ce5eece.ff517a"]]},{"id":"3ce5eece.ff517a","type":"debug","name":"","active":false,"console":false,"complete":false,"x":651,"y":229,"z":"821a73e2.7de59","wires":[]}]

for example the Energy sensor with idx=123 is logged.

In influxdb we can query it with:
SELECT svalue1 from Energy WHERE idx=2440

where svalue2 = counter

[gizmocuz]

Okey, got grafana kind of working (need to read the manual)

grafana.PNG (48.5 KiB) Viewed 8314 times

[maxtrash]

Nice!

[chrille]

Hi,
I am a new domoticz user and would really like to try the MQTT support, as my current setup is based on MQTT. However, I don't see anywhere to configure username/password for the broker connection. Is there no way to set login credentials? I tried to write the hostname URL style like mqtt://username:password@broker but it doesn't seem to work

2015-06-08 16:22:00.263 MQTT: Connecting to mqtt://<user>:<password>@<broker>:1883
2015-06-08 16:22:03.326 Error: MQTT: connection failed, restarting