MQTT topic hierarchy suggestion.

[xAPPO]

I am the developer of an MQTT integration into Hubitat (HE) and some other devices. I have been asked to assist a user trying to link Domoticz with Hubitat in a bidirectional real time implementation.

What I believe happens is that Domoticx publishes messages in a couple of selectable formats but always publishes all devices under one umbrella topic with the particular device identified by an idx key value within the json payload. What this results in as a consumer of such payloads is that I have to subscribe to this topic which updates for every Domoticz device and I have to examine and extract the json payloads and within that the idx key value to determine if I am even interested in 'this' device. This is totally inpractical as it consumes resources when I may not be even interested in the device update.

What I think should be considered is to include the idx within one of the topic hierarchies such that many different topics now become generated by Domoticz but a consumer can just choose to subscribe to specific ones of interest.

For example an existing topic might be

Code: Select all

domoticz/out/mqtt/mqttdevices = { json..., "idx"; 32, ...json }

and I'm suggesting something along the lines of

Code: Select all

domoticz/out/mqtt/mqttdevices/32 =  { json..., "idx"; 32, ...json } 

the idx json value becomes redundant but does no harm

Now instead of a listener subscribing to domoticz/out/mqtt/mqttdevices/# I can subscribe to
domoticz/out/mqtt/mqttdevices/32 and I will only receive updates for that specific device.
I can create multiple individual subscriptions for the devices I am interested in and now not get flooded by irrelevant messages

This will be backwards compatible with your domticz/in/.. handling as you can just subscribe to

Code: Select all

domoticz/in/* 

or

Code: Select all

domoticz/in/#

or even I a remote user just publishes to

Code: Select all

domoticz/in

as it is implemented currently.

Kevin

[waaren]

I am the developer of an MQTT integration into Hubitat (HE) and some other devices. I have been asked to assist a user trying to link Domoticz with Hubitat in a bidirectional real time implementation.

I also think there is room for improvement in the MQTT area of domoticz and I already played a bit with the C++ code to test (with some- but limited succes) and created some dzVents scripts to send various kinds of domoticz data (devices / hardware / variables / groups / etc) to any required topic. (works but not very flexible)

As far as I understand your suggestion for using domoticz/out/<idx> I see a potential problem for existing users. Do I miss something or should they all change their receiving code ?

Can part of your suggestion not already be done by using Hierarchical - publish outgoing messages on topic domoticz/out/${floorplan name}/${plan name}. which is configurable when setting up the MQTT hardware module in domoticz?

What I would like to see in domoticz is the option to configure it to listen to multiple topics. Not only domoticz/in or domoticz/in/# but also to
anyTopic/# + anotherTopic/# + ...

[xAPPO]

As far as I understand your suggestion for using domoticz/out/<idx> I see a potential problem for existing users. Do I miss something or should they all change their receiving code ?

I envisage enabling this inclusion of some 'device identifier' in the topic hierarchy as an additional format selection - in the same way Hierarchical is a possibility now - a user isn't obliged to use it (Although I would recommend they do).. The device identifier could be idx but if device names are enforced to be unique in Domoticz that is an alternative and more readable.

Can part of your suggestion not already be done by using Hierarchical - publish outgoing messages on topic domoticz/out/${floorplan name}/${plan name}. which is configurable when setting up the MQTT hardware module in domoticz?

Well I did look briefly at this but I don't understand enough about Domoticz and it's floorplan and plan names to know. If this could be used to say 'I know you want this device so look up it's 'idx' . i.e as a tag then fine, but can you allocate devices to multiple floor plans and multiple plan names in Domoticx ? if not this encumbers / overloads an existing feature.

Kevin

[FireWizard]

HI

@waaren

I also think there is room for improvement in the MQTT area of domoticz

I fully agree with that, but if the Domoticz developers are prepared to modify the topic structure of Domoticz, we should tackle also some other issues.

1. Many users use MQTT in parallel with the Remote Server solution, because especially virtual sensors and/or text sensors are not supported.
As both the Domoticz Master and the Domoticz Slave are publishing to domoticz/out and subscribing to domoticz/in, you need 2 MQTT servers, otherwise you have conflicts between each other. This is also a waste of resources.

2. There has already been a request/discussion to implement the Homie Convention.
See: https://www.domoticz.com/forum/viewtopi ... ie#p235826
A lot of info, you can find at: https://homieiot.github.io/

As you can see @xAPPO has been involved in this thread.

As you said, it is important that backward compatibility is maintained, so that existing users should not be forced to redesign their solutions.
I believe a lot of them use MQTT together with Node Red (not only) and a node for Homie already exists.
See: https://flows.nodered.org/node/node-red ... convention

If, and only if, the developers are prepared to implement a new topic structure, they should be involved in the Homie discussion on github

Regards

[waaren]

If the Domoticz developers are prepared to modify the topic structure of Domoticz

domoticz is open source and essential we are all Domoticz developers. If you look at the commits you will see that a lot of forum members are already helping to fix bugs / -add functionality.
In my opinion the best and quickest way forward is to create pull requests for the requested functionality. @gizmocuz will comment where needed and
merge into beta when ok.

[FireWizard]

@waaren

Thanks for your response.

domoticz is open source and essential we are all Domoticz developers.

I agree, however I'm not a C++ programmer and in this case the PR should come from someone else.
If I can contribute in another way, I'm happy to do so.

Regards

[rrozema]

Why would you modify the existing structure if we can introduce a new hardware type that has it's own mqtt format off it's own root name? That way existing users can continue to use the existing format, whereas developers of the new hardware have all the freedom they need.

I would suggest to have a good look at best practices and at other existing mqtt implementations, or even adopt the format of one that already exists. Last-Will-And-Testament messages for example are an essential tool to create a reliable system. Domoticz' current mqtt implementation presents domoticz as one big monolith to it's subscribers, being one big controller for all devices we have installed. As xAPPO rightfully states, most mqtt subscribers will however only be interested in one or a few devices. This should be reflected in the mqtt paths used. On top of the mqtt implementation other features should be addressed as well. Domoticz new mqtt should not only be able to operate as the controller to devices in the mqtt space, but also be able to have it's devices controlled by another controller for full interoperability. Device discovery for example is a feature that should be included from the very start, so that it becomes an integral part of the commonly used api. I know Home-Assistant and Domoticz are sort of each other's competitors, but what if Domoticz could work together with HA, both by having HA discover and control Domoticz's devices, but also by Domoticz being able to control the same devices HA sees & controls. We have to admit HA's strong suite is mqtt, whereas Domoticz is still in the lead for ease of use in configuration and user interface. If the user can have both via Domoticz, we are all winners, aren't we? A new MQTT hardware could be the connecting factor.

For the topic names I would also suggest not to use idx values as the idx values can not be changed by the end user. f.e. the end user would not be able to replace a -for whatever reason- malfunctioning device by a new one without re-configuring their mqtt clients. I would instead suggest to use the name of the device or -to avoid potential duplicates- introduce some sort of a device-code in Domoticz: a user defined, unique value for each device. Giving 3 identifiers for each device in Domoticz: idx, description and device-code. idx would be for internal Domoticz use only (for backward compatibility it can still be used by scripts, existing plugins, etc), description would be the device's pretty name for use in any user interface. But this new device-code would be the preferred way to identify a device from scripts and other external references like for example mqtt. To make it easy to use and to make an easy transition the device-code can for a new device be initialized to be the same value as the idx value, but the user must be able to modify the code. Or Domoticz should enable the user to edit the idx value for an existing device, but I think that will be harder to implement.

For an example of a "better" mqtt topic implementation other that HA's, have a look at homeautio. If we could have domoticz represent it's devices (plus maybe it's own state) in the format described there, we could have out-of-the-box support for local google assistant support and a lot more devices would be able to communicate with domoticz's devices directly.

[xAPPO]

I just popped back to check if there was any progress. I haven’t come across homeautio so I will have a peruse of that too. MQTT discovery protocols are not fully thought through yet.

kevin

[gizmocuz]

Just added some new code in beta 12464

Added two publishing schemes::
Index - publish outgoing messagen on topic {domoticz/out}/$idx
Name - publish outgoing messagen on topic {domoticz/out}/$name

And added the possibility to have custom input/output topics

[megamarco83]

Just added some new code in beta 12464

Added two publishing schemes::
Index - publish outgoing messagen on topic {domoticz/out}/$idx
Name - publish outgoing messagen on topic {domoticz/out}/$name

And added the possibility to have custom input/output topics

hi, thanks first of all for support.
i'm interested on mqtt of domoticz, because right now mqtt is the most diffused way of comunication.
do you think that is possible to integrate also a broker inside domoticz that can listen to a topic and publish something customize when a domoticz idx is triggered?

i try to explain better with an example

to control my blinds i use ESP8266 where i flash tasmota and with GPIO i connect switches to move UP or DOWN

in tasmota the topic and messages for blinds are:
topic is:

Code: Select all

cmnd/tasmota_blind/shutterposition1

mesage payload is:

Code: Select all

43

this will move shutter1 at 43%

cmnd/tasmota_blind/shutterposition3 11
this will move shutter3 at 11%

so, if i set inside domoticz the idx conresponded to shutter1 (dummy device with percentage) at 43%
domoticz should publish:

Code: Select all

cmnd/tasmota_blind/shutterposition1 43

but this is possible only if we could have the freedom to set publish topic and payload configuration specific for every idx device inside domoticz

same story to update the idx device when the blinds are moved by wall switch (so mqtt is sended by external device tasmota and domoticz should listen) also here we should have the freedom to set for every device idx a specific topic and message to listen

example:

Code: Select all

stat/tasmota_blind/RESULT = {"Shutter1":{"Position":24,"Direction":0,"Target":32},"Shutter2":{"Position":51,"Direction":0,"Target":21},"Shutter3":{"Position":12,"Direction":0,"Target":41},"Shutter4":{"Position":11,"Direction":0,"Target":25}}

so looking at stat "Target" domoticz should update the percentage position of blind if they are moved in tasmota.
looking at blind1 on the example:
shutter1 is positioned at 24 and tasmota will set a position (target) 32
so domoticz should have the freedom to set, for the conresponded device idx, a specific "listen topic" and so in this ay recognize that tasmota(or every different mqtt sender) has send out an update and so modify accordingly the position percentage of that idx.

this is only an example for blind, but many devices "talking" with mqtt....we can garantee an easy integration of plenty components if we can set for every idx device a topic and message specific to publish and topic and message specific to listen

do you think that it could be possible?

right now my shutter blind are controlled by dzvents that send out commandOS of mosquitto . Waaren put many effort on this using dzvents, but it's not easy and it is specific.
i know that other domotic software like homeassistant or openHAB, can manage with internal mqtt in a very easy way different publish/listen topic and so they have a good integration with tasmota/espeasy/every mqtt device.

THANKS

[FearNaBoinne]

@firewizard, what is the progress on this? I'm looking at the Hubitat C8, but taking away the Domoticz interface now, after 10-ish years, is going to lower the WAF to below zero...