Autodiscovery with lots of MQTT devices not working

[JanVlietland]

Lately I'm experiencing a troubling MQTT autodiscovery feature in Domoticz.

Only 147 devices are recognized at once, which is far from the 1089 published in MQTT (99% are Zigbee devices). The consequence is that most MQTT connected devices are not accessible by Domoticz.

The 147 devices (= entities in HA) can be expanded afterwards to manually leave and reenter the Zigbee devices. Can anybody help me with explain the implementation of autodiscovery in Domoticz. Is there a initial cap? Is there a time limit for autodiscovery? Is it possible to enforce a complete autodiscovery from Domoticz, or manually complete it by entering the domoticz database, or whatever feasible solution.

The alternative I thought of is entering all devices as virtual devices and implement an adapter between Domoticz and MQTT to translate the domoticz/out - domoticz/in messages to the pub/sub of MQTT. For instance like this: https://github.com/FlyingDomotic/domoti ... me-ov-file

Hopefully this is simpler alternative to creating a mapper file for approx 200 specific devices that needs to be used.

ps: The issue emerged since I needed to reconfigure Domoticz due to this issue:
https://www.domoticz.com/forum/viewtopi ... 66#p314766

[waltervl]

It seems you are looking in the wrong overview.

The first screenshot seems to be from Mqtt hardware setup button (menu Setup -Hardware - yourMqtt-setup) There only setup devices to change the behavior of a zigbee device are shown. https://www.domoticz.com/wiki/MQTT#MQTT ... very_Setup

The real devices are shown in menu Setup - Devices. https://www.domoticz.com/wiki/Setup_Devices

[JanVlietland]

Hi Walter, can you help me explain this on the page of the first link.
'After creation of the MQTT Autodiscover hardware in menu Setup - Hardware you can click on the setup button to configure the discovered devices if they have that option using Autodiscovery Number devices..'

What I read is: If I click on setup I'm able to configure the devices only that have been discovered by Domoticz autodiscovery. So if Domoticz cannot discover anything, the user is unable to configure anything?

Or is this my misintepretation?

[JanVlietland]

After some testing, another example. In this case with a Mysensors gateway.

On the hardware/setup page nothing is shown, in other tests most devices are shown, with or without mixed up texts.

[JanVlietland]

Again, am I the only one experiencing this with autodiscovery?

[waltervl]

Hi Walter, can you help me explain this on the page of the first link.
'After creation of the MQTT Autodiscover hardware in menu Setup - Hardware you can click on the setup button to configure the discovered devices if they have that option using Autodiscovery Number devices..'

What I read is: If I click on setup I'm able to configure the devices only that have been discovered by Domoticz autodiscovery. So if Domoticz cannot discover anything, the user is unable to configure anything?

Or is this my misintepretation?

What you should read is: if Zigbee2MQTT is sending a number device for the zigbee node you can find it in this configuration device list. This list could well be empty ( in your case it is not).

But please ignore this setup list, please go to menu setup - Devices (second wiki link) and look for Zigbee2MQTT devices (with the filter option on the left side)

My sensors is a completely different hardware type and not comparable with the setup of normal MQTT Auto Discover for Zigbee2MQTT.

[JanVlietland]

Ok, I'll ignore the setup list. Yes, it's another hardware/interface type. Do you imply that the autodiscovery feature in Domoticz for each interface type has been developed with different (source) code? Sorry haven't digged too deep into the code base (yet).

Anyway, let's focus on MQTT. By connecting to MQTT (enabling the hardware connector in Domoticz) only 135 devices were added. Approx 20% of the total number. Why does Domoticz stop at 20% adding devices? I waited and refreshed several times.
Also after restarting the Domoticz process (terminal > ssh > service domoticz.sh restart) no more devices. But after rebooting the rpi lots of devices were added, approx 10 per second.

Why do I need to restart the rpi to get more devices added? Is this indended practice? How can I be sure that all devices have been added, without checking each of the 1000+ devices? What do you advice?

Hereafter a few of the added MQTT/Zigbee devices cannot be controlled by Domoticz.

However via the pub/sub interface and in the Zigbee2MQTT portal I can control the device perfectly.
-> mosquitto_pub -h localhost -t "zigbee/LivingSpotsEast/set" -m '{"brightness": 60}' -q 1

Furthermore other exact same dimmer type can be controlled by Domoticz. How should I fix this error? I'm afraid that after deleting the device in Setup/Devices it will not be added again (had this before).

[waltervl]

If the config topics in MQTT are set as retained in Zigbee2MQTT your MQTT broker mosquitto should remember them and if Domoticz restarts (or you press update in the mqtt AD hardware) Domoticz will read those config topics again and create/update the related Domoticz devices.

[JanVlietland]

I see the same error occurs with mysensor devices. What is your advice?

[gizmocuz]

Don't know what the saying goes there Jan, but as this is an opensource project, maybe you should not wait and contribute with some code/fixes/improvements?
And yes, there is 1 MQTT-AD implementation, and we do support most devices
Sure, it is possible that some devices are not working, but the focus here is Zigbee2MQTT/ZWaveJS-UI
Maybe MySensors implemented it differently.

So, your best bet is to have an actual look in the code, debug, contribute...
Or start with one MySensor temperature sensor, if that is not working, read the wiki how to report MQTT-AD issues and open a Github issue
And maybe, someone in his/hers free time, is willing to help you

[JanVlietland]

The update in september....

I pinpointed the problem to the zigbee dimmers and zigbee setpoint devices. Autodiscovery often failed when restarting the rpi. Probably three zigbee networks with approx 80 zigbee devices (500 zigbee entities) is too much for auto discovery when restarting. Guess it's a combination of these devices and autodiscovery.

To be completely honest: I experienced insufficient supporting culture on the forum during my previous posts and considered the learning curve too steep for me in the codebase to bugfix the code.

Therefore, I decided in April not to dig into the codebase to bugfix (sorry for that). Instead it was quite easy to solve the matter by the following:
1. Configure a manual MQTT interface in domoticz
2. Create the 28 virtual dimmers and 8 setpoint devices in domoticz and add them to the manual interface
3. Craft python adapters to translate between three zigbee2mqtt networks and domoticz (zigbee2domoticz.py and domoticz2zigbee.py)
4. Create two linux services to autostart the adapters

It's now runing for almost 6 months without any problem. If auto discovery is not working with other devices I will add them to the adapters.

With that my domoticz implementation has been completed. Finally!

[FlyingDomotic]

Another possibility is to use multiple MqttMapper instances (each of them can manage 255 devices). This makes only 2 or 3 for 500 devices.

But you have to configure them manually... so your implementation is probably better.