[90] MQTT Support

[gizmocuz]

14 June 2015:

This forum topic has been cleanup.

MQTT is implemented, and additional info can be found on the wiki:
http://www.domoticz.com/wiki/MQTT

Please note that the next SDCard image of the stable release will include native NodeJS and Node Red.

[maxtrash]

I can see (imagine) some of the power that MQTT could bring us, but how many devices already "talk" MQTT that Domoticz would benefit from?

[kylegordon]

I can see (imagine) some of the power that MQTT could bring us, but how many devices already "talk" MQTT that Domoticz would benefit from?

Finding MQTT enabled hardware can be tricky as it's often rebranded. A badly formatted list follows...

Firstly, there's a list at https://xively.com/dev/hardware/
Android apps to monitor the environment, location, hardware stats, etc.
Anything that connects to an Arduino, Netduino or Launchpad microprocessor can also be MQTT enabled (so, in my house, that's PIRs, LED strips, temperature sensors, magnetic switches, Lux sensors, Bluetooth proximity).
Ford has about 80 cars in their range that can be sensor enabled (even when 'parked') using the OpenXC platform (think Geofencing, windshield rain sensors, engine air temperature and pressure sensors, humidity, etc - your car is a mobile weather station)
Sprint are using it for their in-vehicle systems

It's not so much about the 'what hardware uses it right now?', but more 'what hardware will it enable?' - it's the same argument that stunted IPv6 deployment for over a decade.

A useful article about home automation with MQTT is by the inventor of it here http://electronicdesign.com/embedded/mq ... net-things

[kylegordon]

If anyone else is using MQTT at the moment, there's a great plugin for XBMC that will display MQTT notifications on your big screen. Great for watching out for those updates

[maxtrash]

I'm experimenting a little, and trying to learn a bit about javascript, node.js, sockets, mqtt etc. in the process. I managed to install node.js on my raspberry and got basic MQTT functionality working. The nodejs script server.js will subscribe to a particular topic and show the temperature payload that is published to it.
I will continue my experimenting, this is just basic prototyping I guess.

Code: Select all

var h2d = function(h) {
	return parseInt(h,16);
};

var	HID = '3'; 											//Hardware ID of dummy in Domoticz
	IP = '127.0.0.1'; 									//IP address of Domoticz (127.0.0.1 for same machine)
	Port = '8080';										//Port of Domoticz

// for types please refer to http://sourceforge.net/p/domoticz/code/HEAD/tree/domoticz/main/RFXtrx.h
// for Maxbuddy API please refer to http://bugs.maxbuddy.de/projects/maxbuddy/wiki/Buddy_API
var	WMT_dtype1 = '80', WMT_subtype1 = '9'; 				//WallMountedThermostat Domoticz types for current temperature (temperature , RUBiCSON)
var	HT_dtype1 = h2d('0x50'), HT_subtype1 = '9'; 		//HeatingThermostat Domoticz types for current temperature (temperature , RUBiCSON)
var	HT_dtype2 = h2d('0x52'), HT_subtype2 = h2d('0xA'); 	//HeatingThermostat Domoticz types for current temperature (temperature , RUBiCSON)
var	SC_dtype1 = h2d('0x20'), SC_subtype1 = '01';		//ShutterContact 

var sys = require('sys');
var net = require('net');
var mqtt = require('./MQTTClient.js');
var request = require('request');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

var toString = function(text) {
	return text + '';
};

 
io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    client.subscribe(data.topic);
  });
});
 
client.addListener('mqttData', function(topic, payload){
  var DeriveUrl;
  
  sys.puts(topic+'='+payload);
  console.log('listening');
  io.sockets.emit('mqtt',{'topic':String(topic),
    'payload':String(payload)});
  
  DeriveUrl = 	'http://'+IP+':'+Port+'/json.htm?type=command&param=udevice&hid='+HID+'&did='+topic+'&dunit='+'1'+'&dsubtype='+HT_subtype1+
				'&dtype='+HT_dtype1+'&nvalue='+'0'+'&svalue='+payload;
  console.log( DeriveUrl);	
  request( DeriveUrl, function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body) // Print the web page.
  }
  })
});

[maxtrash]

not sure if anyone is interested but let's just document my progress here.
the previous example was about domoticz "subscribing" to mqtt and showing a value in it's GUI.
The following LUA-script will publish every event in Domoticz to MQTT. The accompanying node.js script is part of the comments.

Code: Select all

--[[
MQTT demo device script
 
devicechanges are pushed to localhost port 5001 if there's any value to publish
a node.js script like this is waiting and publishes using mqtt

var url = require("url");
var http = require('http');
var pubclient = new mqtt.MQTTClient(1883, '127.0.0.1', 'domoticz');

http.createServer(function (req, res) {
	res.writeHead(200, {'Content-Type': 'text/plain'});
	res.end('Response from Node.js \n');
	pubclient.publish('/events/domoticz'+url.parse(req.url).pathname, url.parse(req.url).query);	
}).listen(5001, '127.0.0.1');

NB. Domoticz will add _ and property to the name of certain sensors (e.g. _temperature, _humidity). This is passed as lowest level of message in mqtt
--]]

commandArray = {}

logging = true
d = otherdevices
device = ""
for i, v in pairs(devicechanged) do
  if (#device == 0 or #i < #device) then device = i end
  text = v..''
  if #text > 0 then
	text = string.gsub(i, "_", '/')
	text = '127.0.0.1:5001/'..text..'?'..v
	text = string.gsub(text, "%s+", '%%20')
	commandArray['OpenURL']=text
	if (logging) then print(text) end
  end
end

if (logging) then print('--- LUA Device script MQTT --- Triggered by ' .. device .. ' now ' .. d[device]) end

return commandArray

[alfred_j_kwak]

I'm interested, but I haven't find any MQTT for dummies documentation and starting feels quite difficult. So it's nice to read your progress especially with Domoticz.

-Jussi-

[gizmocuz]

Yep, thats my thoughts too

What is the difference compared to NodeJS and would the later not be better to get implemented?

[sebas]

This looks as a great solution to connect an Arduino to Domoticz. I've got an Arduino setup that measures temperature and humidity and controls lighting that I'd like to connect to Domoticz. I've found a library for Arduino that can push and receive data to/from a MQTT broker.

@Maxtrash: your script, if I understand it correctly, subscribes to a broker and that pushes the data to virtual devices in Domoticz via the JSON API. Do you also run a MQTT broker (server) on your Pi or do you use an externally hosted one? I've found m2m.io as an externally hosted broker.

I'll try to get the combo with an Arduino up and running the coming days.

[maxtrash]

I did struggle a little, but thats was mainly because I was clueless about it all. Node.js is a runtime environment which allows running javascripts on the server. I think I more or less followed this guide to get it running.
http://blog.rueedlinger.ch/2013/03/rasp ... sic-setup/

with the help of nodejs it's relatively easy to create some sort of "star topology". E.g. I could have a file server.js running within nodejs looking like this:

Code: Select all

// https://github.com/maartendamen/DomoMQTT/wiki
// actions/device_id/property
// events/device_type/device_id/property
// config/id
// logging/id

var domoticz = require('./domoticz.js');
var pusher = require('./pusher.js');
var elvmax = require('./elvmax.js');
var x10 = require('./x10.js');
var plugwise = require('./plugwise.js')
var eventghost = require('./eventghost.js');
var usbuirt = require('./usbuirt.js');

This way domoticz only needs to have one interface. As long as all drivers use the same protocol (structures)

For all of this we don't need MQTT, node.js alone would be enough. However instead of just sending messages point to point (e.g. x10 to domoticz), it would make sense to publish them to a bus. That way anyone/anything interested in the messages coming from the x10-driver could subscribe to it.

@gizmocuz: you don't need to implement node.js because you've already done it using the http json-api node.js can send information to domoticz and with lua/openurl domoticz can send info back to nodejs. Implementing a native mqtt interface could be nice in the future, but frankly there's not that much practical use for MQTT at the moment. But who knows, for instance this is also interesting stuff: https://play.google.com/store/apps/deta ... .mqttitude

on thing though: it would be nice to have more information available in the lua-scripts. Now I can publish messages using the name of devices and it's state, but it would also be very practical to have the devicetype and room available in LUA

[maxtrash]

This looks as a great solution to connect an Arduino to Domoticz. I've got an Arduino setup that measures temperature and humidity and controls lighting that I'd like to connect to Domoticz. I've found a library for Arduino that can push and receive data to/from a MQTT broker.

@Maxtrash: your script, if I understand it correctly, subscribes to a broker and that pushes the data to virtual devices in Domoticz via the JSON API. Do you also run a MQTT broker (server) on your Pi or do you use an externallay hosted one? I've found m2m.io as an externally hosted broker.

I'll try to get the combo with an Arduino up and running the coming days.

correct and I'm using the broker Mosquitto on the PI.

[maxtrash]

Maybe to avoid confusion: the script subscribes to a broker. Then I can publish messages to the topic and the nodejs script then pushes them to Domoticz

[sebas]

Maybe to avoid confusion: the script subscribes to a broker. Then I can publish messages to the topic and the nodejs script then pushes them to Domoticz

Thanks for the info and the great work in the script. I've got node.js and mosquitto up and running and I'm able to send messages from my Android phone to my Macbook via Mosquitto and back.

I'm now figuring out how I can get all the values for the device id's etc. to push things to Domoticz. I've created a virtual device and added a Temp + Humidity sensor to that.

[sebas]

I've managed to update Domoticz using the post of maxtrash as a guide. I did change the server.js file a bit. Wanted to make it more "general" without the need for configuration per sensor upfront. It expects a json string with all the data as an aray in it.

It's still a work in progress.

Code: Select all

/*Notes
Removed Socket.io as it is not needed for subscribing to MQTT and pushing to Domoticz
Installed MQTT via NPM as node.js module. npm install mqtt
Installed Request via NPM to do HTTP requests to Domoticz. npm install request

Input is JSON
Example data:
{
   "data":[{
      "idx":247,
      "svalue":"21.1;70%"
   }]
}

*/

var IP = '127.0.0.1';                            //IP address of Domoticz (127.0.0.1 for same machine)
var Port = '8080';                              //Port of Domoticz

var sys = require('sys');
var net = require('net');
var request = require('request');
var mqtt = require('mqtt')

client = mqtt.createClient(1883, 'localhost');

var toString = function(text) {
   return text + '';
};

client.subscribe('test');
client.publish('test', 'This is the Arduino');

client.on('message', function (topic, message) {
  console.log(message);
  
  try {
    var payload = JSON.parse(message);  
    console.log(payload);
    var url = 'http://'+IP+':'+Port;
    var svalue = 0;
    var idx = 0;
    var nvalue = 0;
    console.log(payload.data);

    for (var i = payload.data.length - 1; i >= 0; i--) {
      var data = payload.data[i];
      if(data.idx) {
        idx = data.idx;
      }
      if(data.nvalue) {
        nvalue = data.nvalue;
      }
      if(data.svalue) {
        svalue = data.svalue;
      }
      url = url + "/json.htm?type=command&param=udevice&idx="+idx+"&nvalue="+nvalue+"&svalue="+svalue;
      console.log(url);
      request(url, function(error, response, body){
        console.log("Sending requesd");
        console.log(url);
        console.log(error);
        console.log(response);
        console.log(body);
      });
    };
  }
  catch(e) {
    console.log("Could not parse JSON.");
    console.log(e);
  }

});

[andriej]

I guess it looks just too complicated for me ATM to report everything from my mega with this protocol.
Any sketches out there to see example of Arduino-Raspberry-Domoticz cooperation?
Everything sounds great and I'd really like to transfer data via RF24 module from nodes (nanos) to MEGA and then via LAN to Domoticz.

--edit:
I've done some research and MQTT sounds perfect for using around Domoticz and arduino.
Can't wait to see progress in this.

[maxtrash]

nice work Sebas. You seem to be more experienced in java script?
I like to use the standard NPM MQTT library as well, thanks for pointing me to it. I'm not too sure about using json for payloads, it might complicate things a bit but that might change once I get the hang of it.
Personally in the json to domoticz I like to use hid/did/dunit/dsubtype instead of idx because that way you can push new devices to domoticz that would automaticly show up in the unused devices tab. But it should be possible to support both ways. I'll come up with some code suggestions later on!

[sebas]

I guess it looks just too complicated for me ATM to report everything from my mega with this protocol.
Any sketches out there to see example of Arduino-Raspberry-Domoticz cooperation?
Everything sounds great and I'd really like to transfer data via RF24 module from nodes (nanos) to MEGA and then via LAN to Domoticz.

--edit:
I've done some research and MQTT sounds perfect for using around Domoticz and arduino.
Can't wait to see progress in this.

The example sketches I've found for the Arduino seemed easy to use. There's a library that takes care of all the communication over MQTT. You'll only need to create the payload message for the data you want to send. It can also receive messages so it would be possible to control an Arduino using Domoticz.

Once I've got the server setup and figured out how to update sensors I start on the Arduino side of things. I'll publish the sketch code when I get it to work.

nice work Sebas. You seem to be more experienced in java script?
I like to use the standard NPM MQTT library as well, thanks for pointing me to it. I'm not too sure about using json for payloads, it might complicate things a bit but that might change once I get the hang of it.
Personally in the json to domoticz I like to use hid/did/dunit/dsubtype instead of idx because that way you can push new devices to domoticz that would automaticly show up in the unused devices tab. But it should be possible to support both ways. I'll come up with some code suggestions later on!

I've done some web development that involved javascript, that helps I'm new to Node.js so it's partly copy-paste from tutorials I've found.

JSON is nice to use as it is easy to create. Allmost all languages have support for it and in the end it's just a string of text that's easy to construct on an Arduino. When it's parsed in JS it's an object that you can loop through. That way it's possible to work with an unlimted number of sensors without the need to do any coding in the server.js file.

Is there any documentation around the hid/did/dunit/dsubtype? The only thing I've found is a small description about idx, nvalue and svalue in the Wiki and that also leaves me with more questions. I can update the values in Domoticz but somehow my temp + humidity sensor expects 3 values where I would expect to have 2 and also it's storing the temp and showing it in the graph it's not showing it on the temperature screen in the web front-end. Code examples will help.

[maxtrash]

In my max buddy script I'm using temp + humidity sensors. I'll have a look tonight. Back to work again...
Do you happen to know an easy way to parse a topic string in parts (eg. Split a/b/c in a, b and c?)

[sebas]

In my max buddy script I'm using temp + humidity sensors. I'll have a look tonight. Back to work again...
Do you happen to know an easy way to parse a topic string in parts (eg. Split a/b/c in a, b and c?)

var parts = topic.split("/");
Where topic is the string of topics. The variable parts will be an array of the topic parts. parts[0] will return a, parts[1] returns b etc.

[andriej]

The best way I see the cooperation between Domoticz and multiple Arduino sensors is MQTT on i.e. Arduino MEGA and nodes (i.e. Nanos).
That's the setup I'm currently using, so Arduino Mega is connected to LAN via Ethernet and it uses RF24Network library (2.4GHz rx/tx) to get values from sensors (up to >2000 possible nodes).

The only problem I see is that RF24Network can send/receive only 24bytes of data in one packet, so I guess it's not enough for longer commands/data for MQTT?
I can't wait to see your examples/sketches. I'll try to help in this thread, as it's the only way I can get my remotes, sensors and weather station connected to Domoticz (RFXCom currently doesn't support it, so I'm left with unusable device...)

Regards

--edit:
Did some googe'ing for RF24 and MQTT to implement.
Found this library (for Raspberry): https://github.com/mannkind/Ripcord
May be a hint.
But still - better use would come from "RF24Network" library on MEGA with proper rx/tx of MQTT.