Get sensor values with MQTT into Domoticz [Solved]

[McMelloW]

Hello to all.

I have a ESP8266 device with a temperature, a tilt and a gravity sensor on it. Every 5 minutes a MQTT message is send to the MQTT-broker.
With MQTT-Explorer, I can see the messages on the MQTT-broker.

How can I get these values into Domoticz?

[waaren]

..How can I get these values into Domoticz?

By adding MQTT to your hardware [setup] [hardware] add type MQTT Client Gateway with LAN interface
and publish to domoticz/in. Described in detail on this wiki page

[FireWizard]

Hi,

Just to add to waaren's post.

You have to "translate/convert" the MQTT messages received from your ESP8266 device to a format that Domoticz "understands". See that wiki
A good description also at: https://piandmore.wordpress.com/2019/02 ... -domoticz/
The easiest tool for that is Node Red.

If you have not installed Node Red, yet, you can do it as follows:
This will install the latest LTS version of NodeJS.

1. wget https://nodejs.org/dist/latest-v12.x/no ... v7l.tar.gz
2. tar -xzf node-v12.14.0-linux-armv7l.tar.gz
3. cd node-v12.14.0-linux-armv7l
4. sudo cp -R * /usr/local/

Check If Everything Is Installed Ok

1. node -v
2. npm -v

Now you can install Node Red with npm

1. sudo npm install -g --unsafe-perm node-red

You will see that it has been installed if you see at the end something similar as:
+ [email protected]
added 332 packages from 341 contributors in 18.494s
found 0 vulnerabilities

This will update your Node Red and NodeJS to the latest available version for arm v7 (12.14.0) and Node Red to version 1.0.3.

If you want to install NodeJS and Node Red on a Rasberry Pi 1 or Zero (arm v6) see my post:
viewtopic.php?f=28&t=30567&p=231293#p231293

If you need help with the translation/conversion, please post the output of your MQTT for the ESP8266 devices.

Regards

[McMelloW]

Thanks for your reactions Waaren and FireWizard

On my RPi 3B Node Red and MQTT are installed and working.
The Domoticz hardware MQTT Client Gateway with LAN interface is also installed and working.
I am using 5 Sonoff devices with Tasmota firmware. Tasmota device can be configured for MQTT with a Domoticz device idx.
This works perfect. The commands are working and the status is returned to the devices
So I am beyond this already.

My ESP8266 (an iSpindel) can configured for MQTT, but not for Domoticz. How do I get the MQTT-message from the broker into Domoticz.
Can I read the broker with a dzVents script or so? Once the values are a script it is easy to update them to a device.

If you can put me in the right direction it would be very helpful

[FireWizard]

Hi McMelloW,

Probably @waaren can tell you, if it is possible to do the conversion with dzVents.

As you said you have MQTT and Node Red already running, I would take that route.
If you can publish the messages that your ESP8266 produce, I can send you a flow.

Regards

[waaren]

My ESP8266 (an iSpindel) can configured for MQTT, but not for Domoticz. How do I get the MQTT-message from the broker into Domoticz.
Can I read the broker with a dzVents script or so? Once the values are a script it is easy to update them to a device.

If you can put me in the right direction it would be very helpful

dzVents can send to a broker using os.execute but cannot receive from a broker. You need a process that can translate the MQTT messages to the syntax domoticz understand and send them to the domoticz/in topic. nodered is an excellent candidate for that.

[McMelloW]

dzVents can send to a broker using os.execute but cannot receive from a broker. You need a process that can translate the MQTT messages to the syntax domoticz understand and send them to the domoticz/in topic. nodered is an excellent candidate for that.

Thanks a lot. I think I am going to got it. Do you know a place with some node Red examples?

[FireWizard]

Hi McMelloW,

Have a look: http://noderedguide.com/examples/

Google is your friend

But if you post the output from your ESP8266 here, we can easily create a flow.

Regards

[McMelloW]

Hi FireWizard,

Thanks for your reply. I know google is your friend, but if you know the way (link) it saves me a lot of searching
Here is the output of my ESP8266. I think it is in JSON, but not sure.
MQTT-Explore might have its own build in translator and it is displayed on a graphic window

Code: Select all

iSpindel01
  tilt = 77.84726
  temperature = 10.8125
  temp_units = C
  battery = 4.244004
  gravity = 1146.871
  interval = 300
  RSSI = -55

I Hope this make some sense

[FireWizard]

Hi McMelloW,

It could be JSON or Javascript. I assume, that you receive something like this:

Code: Select all

{
iSpindel01
  tilt = 77.84726
  temperature = 10.8125
  temp_units = C
  battery = 4.244004
  gravity = 1146.871
  interval = 300
  RSSI = -55
}

However , if I use Google, I see only output, similar to:

Code: Select all

{
iSpindel01
  tilt : 77.84726
  temperature : 10.8125
  temp_units : C
  battery : 4.244004
  gravity : 1146.871
  interval : 300
  RSSI : -55
}

I think, the following data is interesting to have in Domoticz.

1. Tilt ( in degrees or ^o); This will be a "Custom Sensor" and will be displayed under the "Utility" tab.
2. Temperature(in ^oC); This will be a "Temperature Sensor" and will be displayed under the "Temperature" tab.
3. Battery Level (in V); This will be a "Voltage Sensor" and will be displayed under the "Utility" tab.
4. Gravity (in N?, but what is the value?); This will be a "Custom Sensor" and will be displayed under the "Utility" tab.

I don't think that the RSSI (Received Signal Strength Indicator) is of interest, but if so create a "Custom Sensor for it in dBm

You can start by creating 4 (or 5 in case RSSI) virtual sensors. Note down their IDX nmbers.You will find these under "Setup" => "Devices".

If you have not done already, you can configure in "Setup" => "Hardware". your MQTT interface over LAN to your MQTT broker.

MQTT Domoticz.png (134.46 KiB) Viewed 18673 times

I will try to make a test flow.

Regards

[FireWizard]

Hi McMelloW,

Your flow will probably be a rather simple flow, something like:

Screenshot_MQTT-in MQTT-out flow.png (9.84 KiB) Viewed 18664 times

The first node is a MQTT input node (found in the network section) and you have to configure this one to communicate with your ESP 8266 (iSpindel) device.
I assume that you can already communicate with your device. If not, do not hesitate to ask.
The topic, you have to subscribe to, depends on the topic that your iSpindel device publishes to.

The second node is a default Javascript to JSON node (found in the parser section). It will convert the received payload to a JSON object.

The third node we will do later.

The fourth node is a MQTT output node (found in the network section) and you have to configure this one to communicate with your Domoticz.
I assume that you can already communicate with Domoticz. You have to publish to domoticz/in.

The third node is a function node (found in the function section), which converts your received payload, converted to JSON, from your device, to something that Domoticz understands. I assume that we are going to publish to 4 sensors (Tilt,Temperature, Battery level and Gravity)

Can you connect a debug node (found in the common section) to the output of the JSON node and check in the debug pane that the response is identical to the screenshot.
If not, can you publish the screenshot, how it looks like?

Screenshot_MQTT test.png (14.96 KiB) Viewed 18664 times

Screenshot_MQTT test output.png (13.31 KiB) Viewed 18664 times

Regards

[McMelloW]

Hi FireWizard,

This looks great. I am going to look or I can get this to work. I will keep you updated on the results.

[FireWizard]

Hello McMelloW,

Just check that the output after the JSON node matches with the screenshot.

If so, you can insert the following in the "Function" node:

Code: Select all

var msg1 ={};
var msg2 ={};
var msg3 ={};
var msg4 ={};
msg1.payload = {"command":"udevice","idx":1001,"nvalue":0,"svalue": msg.payload.tilt};
msg2.payload = {"command":"udevice","idx":1002,"nvalue":0,"svalue": msg.payload.temperature};
msg3.payload = {"command":"udevice","idx":1003,"nvalue":0,"svalue": msg.payload.battery};
msg4.payload = {"command":"udevice","idx":1004,"nvalue":0,"svalue": msg.payload.gravity};
return [[msg1,msg2,msg3,msg4]];

Replace the idx values (1001,1002,1003 and 1004) for your idx numbers of the sensors tilt, temperature, battery and gravety)

Connect another "debug" node to the output of the "Function" node, so that it exactly matches with the next screenshot:

Screenshot_MQTToutput.png (33.33 KiB) Viewed 18652 times

If this is correct, then you can connect the "MQTT out" node and, if this one is properly configured, you will see that your sensors are updated.

Regards

[McMelloW]

Thanks for this addition, FireWizard.

At the moment I am struggling to get something visible from the json node and the debug node. Changed the iSpindel interval to 10 sec. So every 10 seconds a message is sent to the broker.

A pitty Domoticz does not have a proper MQTT connection. Now you have study and configure Nod-Red.

[FireWizard]

Hello McMelloW,

You wrote:

A pitty Domoticz does not have a proper MQTT connection. Now you have study and configure Nod-Red.

I do not have the problems you obviously encounter.

Have you installed a hardware device under "Setup" => "Hardware", called MQTT Client Gateway with LAN interface?
If so, configure the IP address of the MQTT server, if that server (broker) is installed on a different piece of hardware.
If it is installed on the same hardware, use localhost.
Leave the port (1883) as default.
If you have not secured your MQTT server with a username/password leave those fields blank.
However, I recommend to do so, but for now you can leave it.
Leave the "Publish Topic" as default (out)

See the screenshot above.

For this moment your MQTT interface is configured.

Now it is time to configure Node Red with MQTT.

From the "Network" section (Node Red, version 1.0.3) drag and drop the node "mqtt in" to your work space.
Double click on the node and the following will appear:

Screenshot_VoC_12.png (18.58 KiB) Viewed 18631 times

Feel free to give the "Server" a name, to your liking.
Configure "Topic" as domoticz/out.
QoS and Output, you can leave as default.
Feel free to give the node an "Name", e.g. Domoticz Out

Next click on the pencil, right of server, and the properties will appear:

Screenshot_VoC_3.png (28.45 KiB) Viewed 18631 times

In the Tab "Connection", configure the IP Address of your MQTT server.
If Node Red and the MQTT server are installed on the same hardware, you can use localhost
Leave the port as default (1883)
The rest you can leave, as shown in the screenshot.
In the "Security" tab you can configure the username/password, if you have the MQTT server secured with it.
"Messages" you can leave untouched for the moment.

Do not forget to "Update"

This view is closed.
Click on "Done" and on "Deploy" in the main work space.
You should now see een green square, indicating "connected"

You have now configured your Node Red interface to your MQTT server Broker. You need to do this only once.
Every time you use a mqtt in or mqtt out, your interface is configured.

Now, if you connect a "debug" node ( from the common section) to it, you should see all the data published by Domoticz (default to domoticz/out)

You wrote:

At the moment I am struggling to get something visible from the json node and the debug node.

If above works for you, you can change the topic to what your iSpindel publishes to.
Then you should see the data in the debug pane.

Now you can add the "json" node from the "parser" section. Next move your "debug" node to the output of the "json" node.
You should see the JSON output, like the screenshot above.

Can you publish that output?

Regards

[McMelloW]

Hi Firewizard,

Little by little we getting a bit further.
Below the output from the command mosquitto_sub at a Putty session.

Code: Select all

pi@mcmellow-pi:~ $ mosquitto_sub -h 192.168.123.22 -v -t ispindel/#

ispindel/ispindel01/tilt 5.684503
ispindel/ispindel01/temperature 20.8125
ispindel/ispindel01/temp_units C
ispindel/ispindel01/battery 4.139729
ispindel/ispindel01/gravity 893.3198
ispindel/ispindel01/interval 10
ispindel/ispindel01/RSSI -63

And this is the output from the Node-Red debug window

Code: Select all

24-12-2019 23:56:11node: is-msg
ispindel/ispindel01/tilt : msg.payload : string[8]
"5.687649"
24-12-2019 23:56:11node: is-msg
ispindel/ispindel01/temperature : msg.payload : string[2]
"21"
24-12-2019 23:56:11node: is-msg
ispindel/ispindel01/temp_units : msg.payload : string[1]
"C"
24-12-2019 23:56:11node: is-msg
ispindel/ispindel01/battery : msg.payload : string[8]
"4.139729"
24-12-2019 23:56:11node: is-msg
ispindel/ispindel01/gravity : msg.payload : string[8]
"893.3358"
24-12-2019 23:56:11node: is-msg
ispindel/ispindel01/interval : msg.payload : string[2]
"10"
24-12-2019 23:56:11node: is-msg
ispindel/ispindel01/RSSI : msg.payload : string[3]
"-62"

The output is not much of a json format. So I have to figure out this.

[FireWizard]

Merry Christmas to all,

Hi McMelloW,

This looks good.

Your device publishes all available data to the topic ispindel/#
In the MQTT input node fill in as topic ispindel/#.
Check with a debug node, both after the MQTT input node and after the JSON node
Publish your result.

Regards

[McMelloW]

Hi FireWizard.

Somehow I can not get the JSON node to work,

The output from the JSON node is the same as the input (=output from MQTT node)
Here is a paste & copy

Code: Select all

25-12-2019 12:24:59node: Debug-msg
ispindel/ispindel01/tilt : msg.payload : string[7]
"5.44848"
25-12-2019 12:24:59node: Debug-msg
ispindel/ispindel01/temperature : msg.payload : string[5]
"22.25"
25-12-2019 12:24:59node: Debug-msg
ispindel/ispindel01/temp_units : msg.payload : string[1]
"C"
25-12-2019 12:24:59node: Debug-msg
ispindel/ispindel01/battery : msg.payload : string[8]
"4.108446"
25-12-2019 12:24:59node: Debug-msg
ispindel/ispindel01/gravity : msg.payload : string[8]
"892.1183"
25-12-2019 12:24:59node: Debug-msg
ispindel/ispindel01/interval : msg.payload : string[2]
"10"
25-12-2019 12:24:59node: Debug-msg
ispindel/ispindel01/RSSI : msg.payload : string[3]
"-63"

It seems there are 7 sperated messages send from the iSpindel
The following command: mosquitto_sub -h MQTT-Broker -d -v -C 7 -t ispindel/# > ispindel.txt with -C 7 (close after 7 msg
Results with this output:

Code: Select all

Client mosqsub|31313-mcmellow- sending CONNECT
Client mosqsub|31313-mcmellow- received CONNACK
Client mosqsub|31313-mcmellow- sending SUBSCRIBE (Mid: 1, Topic: ispindel/#, QoS: 0)
Client mosqsub|31313-mcmellow- received SUBACK
Subscribed (mid: 1): 0
Client mosqsub|31313-mcmellow- received PUBLISH (d0, q0, r0, m0, 'ispindel/ispindel01/tilt', ... (8 bytes))
ispindel/ispindel01/tilt 5.459947
Client mosqsub|31313-mcmellow- received PUBLISH (d0, q0, r0, m0, 'ispindel/ispindel01/temperature', ... (6 bytes))
ispindel/ispindel01/temperature 22.125
Client mosqsub|31313-mcmellow- received PUBLISH (d0, q0, r0, m0, 'ispindel/ispindel01/temp_units', ... (1 bytes))
ispindel/ispindel01/temp_units C
Client mosqsub|31313-mcmellow- received PUBLISH (d0, q0, r0, m0, 'ispindel/ispindel01/battery', ... (8 bytes))
ispindel/ispindel01/battery 4.108446
Client mosqsub|31313-mcmellow- received PUBLISH (d0, q0, r0, m0, 'ispindel/ispindel01/gravity', ... (8 bytes))
ispindel/ispindel01/gravity 892.1768
Client mosqsub|31313-mcmellow- received PUBLISH (d0, q0, r0, m0, 'ispindel/ispindel01/interval', ... (2 bytes))
ispindel/ispindel01/interval 10
Client mosqsub|31313-mcmellow- received PUBLISH (d0, q0, r0, m0, 'ispindel/ispindel01/RSSI', ... (3 bytes))
ispindel/ispindel01/RSSI -67
Client mosqsub|31313-mcmellow- sending DISCONNECT

Is there a way to combine this to 1 JSON msg?

BTW, where do you put your screenshots for publishing?

[FireWizard]

Hi McMelloW.

You asked:

BTW, where do you put your screenshots for publishing?

This is a simple answer. It is a program, that takes just screenshots.I save them and attach them to the post.
The screenshot utility, I use is Spectacle (a KDE utility for Linux) See: https://kde.org/applications/utilities/ ... .spectacle.
If you use Windows or a Mac you may shop here: https://alternativeto.net/software/kde-spectacle/

You wrote:

Somehow I can not get the JSON node to work,

The output from the JSON node is the same as the input (=output from MQTT node)

That the output of the JSON node is the same as the input, does not mean, that the JSON node does not work.
If it really didn't work, there would be no output at all.

It seems there are 7 sperated messages send from the iSpindel

Indeed there are. See the screenshot "Screenshot_MQTT test output.png" above. It are:
1. Temperature (a value)
2. Temperature Units (temp_units) in ^oC
3. Tilt (value in degrees)
4. Battery Voltage (value in V)
5. Gravity (value, but I'm not sure, what the unit is. N(ewton)? It is not a g value, which is 9.81 m/s²
Is it gal from Galileo, also cm/s²? See: https://en.wikipedia.org/wiki/Standard_gravity
6. Interval (this value, you have set yourself, in this case a message is published every 10 seconds.)
7. RSSI value (See: https://www.metageek.com/training/resou ... -rssi.html. -67 is not too bad)

Is there a way to combine this to 1 JSON msg?

It is already combined in ispindel/ispindel01/
If you use in your MQTT input topic (Screenshot_VoC_12.png) as Topic ispindel/ispindel01/#. What do you see in the debug node?

Regards

[McMelloW]

The screenshot_MQTT shows a perfect JSON message. I never get this. If I get this it is perfect. I have written dzVents scripts parsing such json messages.
But the function node to Domoticz-in will do this also.

5. Gravity Has to do with the specific gravity)
Actually an iSpindel is an tilt-hydrometer use to follow fermenting of beer. At the start the beer contains a lot of sugar and zero alcohol. During the fermenting the yeast is eating the sugar and producing alcohol. The changes the gravity of the liquid and also the angel of the iSpindel in the liquid. With the iSpindel you can follow the fermenting without opening the fermenting vessel an d without getting samples of the beer.
See also www.ispindel.de

Is there a way to combine this to 1 JSON msg?
It is already combined in ispindel/ispindel01/
If you use in your MQTT input topic (Screenshot_VoC_12.png) as Topic ispindel/ispindel01/#. What do you see in the debug node?

Done that, The results stays the same.
[attachment=0]Debug_ispindel01.png[/attachment]