[90] MQTT Support

[andriej]

if you want to use the latest domoticz.js you should try publishing to /actions/domoticz/hac
in the latest version I simplified the payload so it's not an array anymore.
Therefore you could try {"idx": 58, "svalue": "1"}

I'd love to try, but I didn't setup the nodejs yet, as I don't know how to do this on RPI.

Just got no-array values implemented:

Code: Select all

/actions/domoticz/hac {"hid": 3, "did": "105", "dunit": "1", "dsubtype": "7", "dtype": "82", "svalue": "26.10;39.50;0"}
/actions/domoticz/hac {"hid": 3, "did": "102", "dunit": "1", "dsubtype": "7", "dtype": "82", "svalue": "25.1;39;0"}
/actions/domoticz/hac {"hid": 3, "did": "106", "dunit": "1", "dsubtype": "7", "dtype": "82", "svalue": "26.20;35.80;0"}
/actions/domoticz/hac {"hid": 3, "did": "105", "dunit": "1", "dsubtype": "7", "dtype": "82", "svalue": "26.10;39.40;0"}
/actions/domoticz/hac {"hid": 3, "did": "101", "dunit": "1", "dsubtype": "7", "dtype": "82", "svalue": "8.2;74;0"}
/actions/domoticz/hac {"hid": 3, "did": "102", "dunit": "1", "dsubtype": "7", "dtype": "82", "svalue": "25.2;39;0"}

Also, I'd like to implement the callback already on MEGA.
What topic should it subscribe to and how the statuses will be sent?

[andriej]

--edit:
Doesn't matter, got it to work.
I've switched my void() to url-query to MQTT for tests now.
Can't wait to see Wiki updated tho.

[sebas]

if you want to use the latest domoticz.js you should try publishing to /actions/domoticz/hac
in the latest version I simplified the payload so it's not an array anymore.
Therefore you could try {"idx": 58, "svalue": "1"}

of course you can also make your own script in nodejs

@sebas: I think you started with the array in Json? I thought this is easier and if you want to send more than one json-action to domoticz, why not use two mqtt messages? What's your opinion?

I used the array so I could put data from multiple sensors in one message. Publishing multiple messages also works and probably takes up less memory in the Arduino. If Node.JS can keep up with the number of messages that will come in no problem. If processing a message takes more than 1 second and you're sending 10 sensor readings every 10 seconds you will end up with a queue of messages.

Nice one, creating the wiki-page and github repository! I'll add stuff to later on. No experience yet with github. How should I upload code?

this is the LUA-script I made for Domoticz. I called it script_device_mqtt.lua and put it in domoticz\scripts\lua

Code: Select all

--[[
MQTT demo device script
 
devicechanges are pushed to localhost port 5001 if there's any value to publish
a node.js script is waiting and publishes using mqtt

NB. Domoticz will add _ and property to the name of certain sensors (e.g. _temperature, _humidity). This is passed as lowest level of message in mqtt
--]]

commandArray = {}

logging = true
d = otherdevices
device = ""
for i, v in pairs(devicechanged) do
  if (#device == 0 or #i < #device) then device = i end
  text = v..''
  if #text > 0 then
	text = string.gsub(i, "_", '/')
	text = '127.0.0.1:5001/'..text..'?'..v
	text = string.gsub(text, "%s+", '%%20')
	commandArray['OpenURL']=text
	if (logging) then print(text) end
  end
end

if (logging) then print('--- LUA Device script MQTT --- Triggered by ' .. device .. ' now ' .. d[device]) end

return commandArray

Thanks for the script.

Working with Github is not very complicated. You will need to get a free account at Github.com. Then setup Git on your own computer. There are some nice help pages here. On Mac I work with SourceTree and on Windows with TortoiseGit. The Git client Github developed themselves for both Windows and Mac also work. The graphical clients make life mucht easier than working on the command line.

Basically what you do is work on your local version of the software. After you're done you commit your work and enter a message with it what you've done. You then push your changes to the central server. Your work can than be combined (merged) with the work of others. I use it for all the projects I work on without others as well as it also serves as a backup of my source code and keeps it in sync between my desktop and macbook.

When you have a Github account please PM me your e-mail address you've signed-up with or your username. I'll add you to the project.

[andriej]

Strange thing goes off, as the only sensor that is getting updated is the one with fixed 'idx' and value '1;1' or '0;0' sent.
Those who I used to update via webclient doesn't work and they're showing in 'not used' devices.
Electricity counter doesn't work with dtype/subtype/etc. at all, I'll try update some code and see what happens.

The other thing is, for the approach of receiving messages.
Wouldn't it be better if LUA would update /events/domoticz/{idx}?
I probably will totally move to fixed-IDX as it's the only way to keep those sensors in place.
Sensor's name are hard to parse and somewhat unusable.

At current moment I'm receiving messages on MEGA via the WebServer.

[camillo777]

Hi, nice work on MQTT, however, to understand better what You are doing and maybe help You with some testing or developing, can You draw a picture of how this MQTT works in the Domoticz environment and how is the hardware/software architecture?

How the remote Arduinos talk with the MEGA? Via ethernet or via 433MHz?

How the MEGA talks then to the RaspberryPi and MQTT server? Via Ethernet?

As I understand, the MEGA publish messages to the MQTT server queue on the RPi and from there Domoticz takes the messages from the queue?

Please correct me if I am wrong.

Thank you!
Camillo

[andriej]

Consider MQTT as a standalone service, to which arduino, your phone, domoticz and any other system can connect to.
Devices can publish and subscribe to topics on it.
My MEGA connects to RPI MQTT via Ethernet.

I tried to fully switch my code to "one sendVoid which sends them all" but well, I think I failed.
Combining strings, floats and int's in one query kills me.

I think I'll wait for inspiration from your code guys..

[maxtrash]

Strange thing goes off, as the only sensor that is getting updated is the one with fixed 'idx' and value '1;1' or '0;0' sent.
Those who I used to update via webclient doesn't work and they're showing in 'not used' devices.
Electricity counter doesn't work with dtype/subtype/etc. at all, I'll try update some code and see what happens.

The other thing is, for the approach of receiving messages.
Wouldn't it be better if LUA would update /events/domoticz/{idx}?
I probably will totally move to fixed-IDX as it's the only way to keep those sensors in place.
Sensor's name are hard to parse and somewhat unusable.

At current moment I'm receiving messages on MEGA via the WebServer.

Not sure what's going wrong but the code is still in alpha state I guess.
Your suggestion publishing to idx topic is a good one but for now the idx value is not a variable I can access from the LUA script so we would need dev support to make that happen

[andriej]

I've just moved everything to direct IDX report with svalue - 100% working, great thing.
I wish there's more support from domoticz threating MQTT as a hub, so:
1) faster messages/statuses
2) lua IDX report

Just a demo of MEGA interface (Serial.prints - DEBUG off radio traffic):

Code: Select all

>EC: 300W/h (total: 1146.0311 kWh) T: 34.5'C
MQ>: T: /actions/domoticz/ec, M: {"idx": 40, "svalue": "300;1146031.10"}
MQ>: T: /actions/domoticz/ec, M: {"idx": 34, "svalue": "34.5;34.5"}
>EC: Ping!
>BH: T: 25.9'C, H: 32.8%, PIR: 0
MQ>: T: /actions/domoticz/bh, M: {"idx": 39, "svalue": "25.90;32.80;0"}
MQ>: T: /actions/domoticz/ec, M: {"idx": 43, "svalue": "1;1"}
MQ>: T: /actions/domoticz/bh, M: {"idx": 44, "svalue": "1;1"}
MQ>: T: /actions/domoticz/ir, M: {"idx": 42, "svalue": "1;1"}
MQ>: T: /actions/domoticz/kc, M: {"idx": 41, "svalue": "1;1"}
---: Mesh: EC:1, IR:1, BH:1, KC:1 | PIR: BH: 0, KC: 0
---: Uptime: 12 min | Free RAM: 5.13kb | LDR: 85.39lx
MQ>: T: /actions/domoticz/mega, M: {"idx": 84, "svalue": "85.4;85.4"}
>KC: T: 25.0'C, H: 30.6%, PIR: 0
MQ>: T: /actions/domoticz/bh, M: {"idx": 55, "svalue": "25.00;30.60;0"}
MQ>: T: /actions/domoticz/mega, M: {"idx": 37, "svalue": "24.4;31;0"}
>HC: T: 24.4'C, H: 30.6%
>BH: T: 25.9'C, H: 32.7%, PIR: 0
MQ>: T: /actions/domoticz/bh, M: {"idx": 39, "svalue": "25.90;32.70;0"}
>IR: Ping!

--edit:
After running my code with debug on (and with correction of JSON query bugs ), I can clearly see that the only way for me that works flawless is using "idx" update with "svalue" for sensors and "nvalue" for devices/switches.
I'm glad I got MQTT running and I'd like to see it more implemented in Domoticz.

[gizmocuz]

As stated by other users above, also i am not known with MQTT

To get this good supported, please draw a picture of how this works, also what tools do i need to test/monitor

With kind regards,
Rob

[andriej]

To get this good supported, please draw a picture of how this works, also what tools do i need to test/monitor.

Rob, I'll try even as I don't have much experience with it totally, but year 2014 is year of "Internet of Things" and MQTT is totally connected with that. It could bring Domoticz a lot of users/attention.
It's a simple protocol, which works something like between IRC/hub. There's a server (broker), which gets messages "published" by it's clients.
There are publishers, there are subscribers, there are topics.

Topic may be something like:
/home/device/aircondition
/home/device/light1
/home/device/light2
etc.

Now, Domoticz subscribes to a thread as a publisher and as a listener, so if I click on website (domoticz) "light1" ON, Domoticz publishes to broker (Mosquitto on same machine):
/home/device/aircondition, message: ON (or it can be /home/device/$idx, message: on, or whatever - there's no one standard for protocol).

Now, my Arduino (or any other client on the wolrd, even cellphone) subscribes to thread:
/home/device/# (# - so it gets every device ever posted to topic)
I write a simple rule "(if $device == aircondition) { if "on" -.... } etc.
Also, my Arduino can publish a message (if I use physical remote), same as Domoticz.

As it was mentioned in thread, Domoticz is (as heard) easy to integrade with MQTT as it uses nodejs. Maybe sebas and maxtrash can point out more how to do it easily.
Currently the nodejs hangs up from time to time and my messages are not delivered.

And, as a example - output from my arduino:

Code: Select all

-->: Turning DEBUG on!
MQ>: T: /actions/domoticz/mega, M: {"idx": 85, "nvalue": "1"}
>HC: T: 25.8'C, H: 36.7%
MQ>: T: /actions/domoticz/mega, M: {"idx": 37, "svalue": "25.8;36.7;0"}
>BH: Ping!
>EC: Ping!
>BH: T: 26.4'C, H: 33.9%, PIR: 0
MQ>: T: /actions/domoticz/bh, M: {"idx": 55, "svalue": "26.4;33.9;0"}
>IR: Ping!
>EC: 480W/h (total: 1191.8145 kWh) T: 33.5'C
MQ>: T: /actions/domoticz/ec, M: {"idx": 40, "svalue": "480;1191814.60"}
MQ>: T: /actions/domoticz/ec, M: {"idx": 34, "svalue": "33.5"}

That's what seen on mosquitto (broker):

Code: Select all

 $ mosquitto_sub -t /# -v
/devices/hac Connected!
/actions/domoticz/mega {"idx": 85, "nvalue": "0"}
/actions/domoticz/bh {"idx": 39, "svalue": "25.9;43.3;0"}
/actions/domoticz/ec {"idx": 40, "svalue": "480;1191896.60"}
/actions/domoticz/ec {"idx": 34, "svalue": "34.0"}
/actions/domoticz/mega {"idx": 37, "svalue": "25.8;37.2;0"}
/actions/domoticz/bh {"idx": 55, "svalue": "26.3;31.7;0"}

More info can be found:
http://en.wikipedia.org/wiki/MQ_Telemetry_Transport
http://mqtt.org/
and in this thread

[maxtrash]

As stated by other users above, also i am not known with MQTT

To get this good supported, please draw a picture of how this works, also what tools do i need to test/monitor

With kind regards,
Rob

I have added some sketch to the wiki page:



To be frank, most of the things that can be done with MQTT could be done in other ways too. And most of the things needed for MQTT-support are already provided by Domoticz as is. But if you ask me how you could support, in sequence of importance (see also numbers in above sketch):
(1) Devices, devices, devices. The more (generic) devices (Multi-state, text etc.) the better. MQTT could connect a vast amount of different devices and we would like to see them and interact with them in the GUI. Even without MQTT this is the main thing for many users I guess (e.g. by using the LUA-scripting, JSON or whatever)
And let's not to forget: it would be nice to have a list of all devicetypes/sub devices and what they need (svalue, nvalue, etc.).

(2) In the LUA scripts it is not yet possible to know the IDX of the device that changed. Although it is possible to work around that (by using name to read idx directly from the database), it think it would be a lot cleaner if it is available.
.
.
.
(3a) It would make for a cleaner solution if some http-url (OpenUrl) could be configured from within the GUI instead of having to rely on LUA. For instance a global setting to have every event happening in Domoticz posted as a Json-object to a configurable ip/port. (that could also be generic functionality that people without an interest in MQTT would benefit from). The domoticz.js will then relay the json objects to the broker and LUA would not be needed anymore.
(3b) a native interface with MQTT-brokers would also be nice. Like 3a, but instead of http/json it would be tcpip with MQTT-flavour (I'm sure libraries exist)

ps. as seen in the picture it is also possible to access sqlite from node.js. Although I feel this might be practical (e.g. reading some settings, info from Domoticz), I'm not sure it's a great thing from an architectural point of view to have an interface like that.

I'm very curious what ideas are for the future, with regards to Domoticz. Can you share some of them? (I mean in general, I don't want to push anything, or MQTT in particular. Just interested)

[kylegordon]

This is amazing! Thank you everyone for all your ideas I'm just a lowly Python amateur that uses MQTT for a lot of stuff, and it's lovely to see it being expanded upon like this!

Just as an aside, I believe / isn't required at the root of the topic hierarchy, as it's a topic separator in its own right. Although the specification is vague on the subject, it's possible for a topic structure to start without a leading /, although many do

I really should update my code for that...

[maxtrash]

This is amazing! Thank you everyone for all your ideas I'm just a lowly Python amateur that uses MQTT for a lot of stuff, and it's lovely to see it being expanded upon like this!

Just as an aside, I believe / isn't required at the root of the topic hierarchy, as it's a topic separator in its own right. Although the specification is vague on the subject, it's possible for a topic structure to start without a leading /, although many do

I really should update my code for that...

I believe you're right, however mosquitto_sub -t /# works but mosquitto_sub -t # doesn't so for practical reasons at least it makes sense to start with a /

what do you use MQTT for?

[sebas]

As stated by other users above, also i am not known with MQTT

To get this good supported, please draw a picture of how this works, also what tools do i need to test/monitor

With kind regards,
Rob

I have added some sketch to the wiki page:

[...]


ps. as seen in the picture it is also possible to access sqlite from node.js. Although I feel this might be practical (e.g. reading some settings, info from Domoticz), I'm not sure it's a great thing from an architectural point of view to have an interface like that.

I'm very curious what ideas are for the future, with regards to Domoticz. Can you share some of them? (I mean in general, I don't want to push anything, or MQTT in particular. Just interested)

Great sketch on how you can use MQTT.

Quick note on the sql thing. SQLLite is really terrible at dealing met multiple applications accessing the db file. It's lock mechanism isn't well suited for that. Would be better to use Domoticz for that and have that as the single user of the database.

[andriej]

Great sketch on how you can use MQTT.

Quick note on the sql thing. SQLLite is really terrible at dealing met multiple applications accessing the db file. It's lock mechanism isn't well suited for that. Would be better to use Domoticz for that and have that as the single user of the database.

Well, currently I'll be updating my sketch to let me easily switch between Arduino -{json}-> Domoticz requests and Arduino -{mqtt-> nodejs -> {json} Domoticz.
Why? Just because it's a little slower in some conditions like PIR sensors. Currently Arduino 'sends' message via 2 channels and for nodejs it takes a little to update query from mqtt.
I'll probably reverse to MQTT as soon as it gets support in Domoticz, so one step (or "point of failure") can be skipped.
Can't wait for that.

[maxtrash]

very cool Rob, I saw some option appearing "on action" and "off action". I'll check it tonight!

edit:
putting http://127.0.0.1:5001/pathname?value in the on-action of a particular virtual dimmer device will result in this messages to mqtt:

/events/domoticz/Slider On
/events/domoticz/pathname value

1st is generated by the LUA-script which publishes everything to domoticz.js/mqtt.
2nd is because of the on action in domoticz.

[bek]

I have setup my rpi with domoticz, mosquitto & nodejs and my iPhone with mqttitude.

In server.js is only two rows:
var domoticz = require('./domoticz.js');
var mqttitude = require('./mqttitude.js');

I am using the mqttitude.js from http://www.domoticz.com/forum/viewtopic ... t=20#p7872, and the command mosquitto_sub -h localhost -p 1883 -v -t '/actions/#' shows
/actions/domoticz/mqttitude {"SensorType":"counter","svalue":"22.35km"}.

Now I need a domoticz.js file that can send this to domoticz. Have tried to modify several ones that I have found but no one works with mqttitude. They would maybe work with temperatures or something else but not with mqttitude.

So if someone could help me with a short mqttitude (no temps, hum or other stuff) domoticz.js file maybe others too could start using this.

[sebas]

I have setup my rpi with domoticz, mosquitto & nodejs and my iPhone with mqttitude.

In server.js is only two rows:
var domoticz = require('./domoticz.js');
var mqttitude = require('./mqttitude.js');

I am using the mqttitude.js from http://www.domoticz.com/forum/viewtopic ... t=20#p7872, and the command mosquitto_sub -h localhost -p 1883 -v -t '/actions/#' shows
/actions/domoticz/mqttitude {"SensorType":"counter","svalue":"22.35km"}.

Now I need a domoticz.js file that can send this to domoticz. Have tried to modify several ones that I have found but no one works with mqttitude. They would maybe work with temperatures or something else but not with mqttitude.

So if someone could help me with a short mqttitude (no temps, hum or other stuff) domoticz.js file maybe others too could start using this.

What's special about mqttitude? Don't have an iphone so I can't test it.
I can post my js files if you want. Everything is in there just replace the payload with what you want to send. I use them for connecting an Arduino.

[bek]

That would be nice. Could you publish the file here?
Well there is maybe nothing special with mqttitude, I just can't get it added as device into Domoticz.

[maxtrash]

attached are two files which work fine.
I will pick up the mqtt stuff a bit more, was kind of busy with other experiments last weeks