Autodiscovery with TheengsGateway

[StephenR]

Hi,

I am trying to setup MQTT autodiscovery to find sensors which are created by a TheengsGateway.

All looks like it works on the surface and I see entries like this in my log :
2023-01-08 19:05:54 .224 Status: BBS2_MQTT: discovered: ATC_2C87D4/LYWSD03MMC_PVVX-tempc (unique_id: A4C1382C87D4-tempc)

but no devices get created that I can find.

I think the basic autodiscovery is working because Zwave do get created ok and I can control them. The log seems to show them being found in a similar way to the above:
2023-01-08 19:05:55 .639 Status: BBS2_MQTT: discovered: nodeID_6/nodeID_6_switch (unique_id: zwavejs2mqtt_0x14ce319_6-37-0-currentValue)

I am using a new installation of Domoticz : Version: 2022 . 2

Any help would be brilliant!

Thanks

Stephen

[waltervl]

If I look at instructions here https://gateway.theengs.io/use/use.html#details-options

Did you enable hass-discovery? Edit: It should not.
Did you create in Domoticz a MQTT Autodiscover gateway with homeassistant as topic (autodiscover prefix)? And did you set that also as discovery topic in theengs?

[StephenR]

Thanks very much walter.

I have got hass-discovery turned on (I think) by leaving it to its default. Were you suggesting that I should not? Will give that a go.

It seems to be creating the nodes in MQTT (but I don't 100% know what they should look like)
I added an attachment with a screenshot but when try to add it inline I get a post error about it being an external image. (User error I am sure but can't see quite how...)

I have created a Domoticz MQTT auto discovery with homeassistant as the prefix and that seems to be working (I think) because it is picking up the ZWave nodes.

The homeassistant message being posted is:

{
"stat_t": "+/+/BTtoMQTT/A4C1382C87D4 ",
"dev_cla": "humidity",
"unit_of_meas": "%",
"name": "LYWSD03MMC_PVVX-hum",
"uniq_id": "A4C1382C87D4-hum",
"val_tpl": "{{ value_ json. hum | is_defined }}",
"state_class": "measurement",
"device": {
"identifiers": [
"A4C1382C87D4"
],
"connections": [
[
"mac",
"A4C1382C87D4"
]
],
"manufacturer": "Xiaomi",
"model": "LYWSD03MMC_PVVX",
"name": "ATC_2C87D4",
"via_device": "TheengsGateway"
}
}

(I had to put a couple of spaces in the above because the forum posting didn't like the text and thought it was a URL to an external link)

As I posted this, I do notice that the path to the status topic used by TheengsGateway seems to use a 'wildcard' in the path but the ZWave topic details the specific path. I wonder if I can use the -pt option in TheengsGateway to force it to use a full path. Will give that a go too.

Thanks very much for your help! Very appreciated.

Stephen

[StephenR]

Looking at the source code for autodiscovery, I am a bit more worried about the use of the wild cards in the status topic.

Caveat : This is the first time I have used MQTT and the first time I have looked at Domoticz source code so I am not so confident but :

in MQTTAutoDiscover::handle_auto_discovery_sensor_message(const struct mosquitto_message* message)
@Line 1225 it checks if
pSensor->state_topic == topic

From what I can see,
pSensor->state_topic will be the wildcard topic path from the homeassistant node definition (+/+/BTtoMQTT/A4C1382C87D4 in my example)
but I think that
topic will be the actual topic that the message was received on? (home/TheengsGateway/BTtoMQTT/A4C1382C87D4)

I currently have Domoticz installed from packages but possibly the next best move is to build from source so I can add some debug logging.

Thanks

Stephen

[StephenR]

It is the wild cards.

I have put in a horrible hack which replaces any message topic which starts with home/TheengsGateway/ with +/+/ (so it matches the template) and everything starts to work.

Obviously the right way to do this is to go the other way and to change the compare code with a wildcard aware version. I will see if I can remember enough cpp to have a go at that...

I also posted a question at Theengs Gateway to ask if there was a way of disabling wildcards (i.e. just to hide the problem!)

I think that Theengs Gateway is basically a copy of OpenMQTTGateway so it maybe is hitting some other people using that?

[waltervl]

Good you have some c++ knowledge to debug this.
As MQTT Autodiscover is relatively new in Domoticz I don't think it is used a lot for OpenMQTTGateway.

So indeed it could need some improvements, the developers would be happy for your contributions.

[waltervl]

When looking at the code in OpenMQTTGateway it seems that those wildcards are added when the entity is not belonging to the gateway. I do not understand what that could mean but possible solution would be to add the entity to the gateway.....

https://github.com/1technophile/OpenMQT ... y.ino#L240

Code: Select all

    // If not an entity belonging to the gateway we put wild card for the location and gateway name
    // allowing to have the entity detected by several gateways and a consistent discovery topic among the gateways
    if (gateway_entity) {
      strcpy(state_topic, mqtt_topic);
      strcat(state_topic, gateway_name);
    } else {
      strcpy(state_topic, "+/+");
    }
    strcat(state_topic, st_topic);
    sensor["stat_t"] = state_topic;
  }

[waltervl]

It seems you can set gateway nam and location manually
https://github.com/1technophile/OpenMQT ... -767885707

Is there a commands config topic for renaming the gateway? I had trouble entering all the config in wifimanager before it kicks me, and i have to start all over with wifimanager
>
You can set the gateway name in your .ini file or user_config.h file before building the code or into wifimanager portal. But you cannot modify it by MQTT currently.

[Doler]

When looking at the code in OpenMQTTGateway it seems that those wildcards are added when the entity is not belonging to the gateway. I do not understand what that could mean but possible solution would be to add the entity to the gateway.....

https://github.com/1technophile/OpenMQT ... y.ino#L240

Code: Select all

    // If not an entity belonging to the gateway we put wild card for the location and gateway name
    // allowing to have the entity detected by several gateways and a consistent discovery topic among the gateways
    if (gateway_entity) {
      strcpy(state_topic, mqtt_topic);
      strcat(state_topic, gateway_name);
    } else {
      strcpy(state_topic, "+/+");
    }
    strcat(state_topic, st_topic);
    sensor["stat_t"] = state_topic;
  }

For the sake of experiment I added 'gateway_entity = true' before the test. The result is that my LYWSD03MMC sensors are discovered by Domoticz and indeed devices are created. But the end result is a bit disappointing:

Spoiler: show

2023-01-13 13:40:13.251 Status: OMG-AD: discovered: OMG_BLE/SYS: Uptime (unique_id: 30C6F705A22C-uptime)
2023-01-13 13:40:13.353 Status: OMG-AD: discovered: OMG_BLE/SYS: Free memory (unique_id: 30C6F705A22C-freemem)
2023-01-13 13:40:13.454 Status: OMG-AD: discovered: OMG_BLE/SYS: IP (unique_id: 30C6F705A22C-ip)
2023-01-13 13:40:13.555 Status: OMG-AD: discovered: OMG_BLE/SYS: Rssi (unique_id: 30C6F705A22C-rssi)
2023-01-13 13:40:13.656 Status: OMG-AD: discovered: OMG_BLE/SYS: Low Power Mode (unique_id: 30C6F705A22C-lowpowermode)
2023-01-13 13:40:13.757 Status: OMG-AD: discovered: OMG_BLE/SYS: Restart gateway (unique_id: 30C6F705A22C-restart)
2023-01-13 13:40:13.860 Status: OMG-AD: discovered: OMG_BLE/SYS: Erase credentials (unique_id: 30C6F705A22C-erase)
2023-01-13 13:40:13.965 Status: OMG-AD: discovered: OMG_BLE/SYS: Auto discovery (unique_id: 30C6F705A22C-discovery)
2023-01-13 13:40:14.078 Status: OMG-AD: discovered: OMG_BLE/BT: Interval between scans (unique_id: 30C6F705A22C-interval)
2023-01-13 13:40:14.179 Status: OMG-AD: discovered: OMG_BLE/BT: Connnect every X scan(s) (unique_id: 30C6F705A22C-scanbcnct)
2023-01-13 13:40:14.280 Status: OMG-AD: discovered: OMG_BLE/BT: Force scan (unique_id: 30C6F705A22C-force_scan)
2023-01-13 13:40:14.387 Status: OMG-AD: discovered: OMG_BLE/BT: Publish only sensors (unique_id: 30C6F705A22C-only_sensors)
2023-01-13 13:40:14.490 Status: OMG-AD: discovered: OMG_BLE/BT: Publish HASS presence (unique_id: 30C6F705A22C-hasspresence)
2023-01-13 13:40:14.593 Status: OMG-AD: discovered: OMG_BLE/SYS: Low Power Mode command (unique_id: 30C6F705A22C-lowpowermode)
2023-01-13 13:40:15.681 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-tempc (unique_id: A4C1385DBC50-tempc)
2023-01-13 13:40:15.782 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-hum (unique_id: A4C1385DBC50-hum)
2023-01-13 13:40:15.885 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-batt (unique_id: A4C1385DBC50-batt)
2023-01-13 13:40:15.986 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-volt (unique_id: A4C1385DBC50-volt)
2023-01-13 13:40:24.067 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-tempc (unique_id: A4C138516C6F-tempc)
2023-01-13 13:40:24.168 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-hum (unique_id: A4C138516C6F-hum)
2023-01-13 13:40:24.270 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-batt (unique_id: A4C138516C6F-batt)
2023-01-13 13:40:24.371 Status: OMG-AD: discovered: OMG_BLE/LYWSD03MMC_ATC-volt (unique_id: A4C138516C6F-volt)

Resulting devices (sensors):
42 OMG-AD 30C6F705A22C-rssi 1 SYS: Rssi General Sound Level -50 dB
38 OMG-AD A4C138516C6F-volt 1 LYWSD03MMC_ATC-volt General Voltage 2.829 V
37 OMG-AD 30C6F705A22C 1 OMG_BLE Temp + THGN122/123/132, 19.5 C, 52 %
Humidity THGR122/228/238/268
36 OMG-AD A4C138516C6F-batt 1 LYWSD03MMC_ATC-batt General Percentage 68%
35 OMG-AD A4C1385DBC50-volt 1 LYWSD03MMC_ATC-volt General Voltage 2.933 V
34 OMG-AD A4C1385DBC50-batt 1 LYWSD03MMC_ATC-batt General Percentage 81%

Battery percentage and voltage are correct for both sensors but temperature, humidity and rssi are represented at 'system level'.

I used OMG as Base_topic and OG_BLE as Gateway_name.

[waltervl]

I do not understand what you mean with 'system level'. Domoticz has combined sensors devices temperature + humidity
Perhaps also post some related MQTT messages.

[Doler]

I do not understand what you mean with 'system level'. Domoticz has combined sensors devices temperature + humidity
Perhaps also post some related MQTT messages.

This device (battery percentage) is reporting at LYWSD03MMC_ATC:
36 OMG-AD A4C138516C6F-batt 1 LYWSD03MMC_ATC-batt General Percentage 68%

This device (combined temp/humi) is reporting at 'systemlevel' OMG_BLE:
37 OMG-AD 30C6F705A22C 1 OMG_BLE Temp + THGN122/123/132, 19.5 C, 52 % Humidity THGR122/228/238/268

and is using the gateway_name OMG_BLE iso LYWSD03MMC. And it uses (arbitrary) the report of one of the sensors, so the report of the other sensor is not present.

The messages in MQTT Explorer are ok so I don't understand what is going on: MQTT config is ok after the 'change', MQTT messages are ok but mis-interpreted by MQTT-AD in Domoticz.

[waltervl]

Is there a is_defined statement in the config messages? Try to set OpenHABAutoDiscovery to true into config_mqttDiscovery.h

#define OpenHABDiscovery true

[Doler]

Is there a is_defined statement in the config messages? Try to set OpenHABAutoDiscovery to true into config_mqttDiscovery.h

#define OpenHABDiscovery true

There is a is_defined in the config message:

Code: Select all

omg-ad/sensor/A4C1385DBC50-tempc/config (OpenHABDiscovery = false)

{
  "stat_t": "+/+/BTtoMQTT/A4C1385DBC50",
  "dev_cla": "temperature",
  "unit_of_meas": "°C",
  "name": "LYWSD03MMC_ATC-tempc",
  "uniq_id": "A4C1385DBC50-tempc",
  "val_tpl": "{{ value_json.tempc | is_defined }}",
  "state_class": "measurement",
  "device": {
    "identifiers": [
      "A4C1385DBC50"
    ],
    "connections": [
      [
        "mac",
        "A4C1385DBC50"
      ]
    ],
    "manufacturer": "Xiaomi",
    "model": "LYWSD03MMC_ATC",
    "name": "LYWSD03MMC-5DBC50",
    "via_device": "OMG_BLE"
  }
}

With OpenHABDiscovery turned on the config message looks like this:

Code: Select all

omg-ad/sensor/A4C1385DBC50-tempc/config (OpenHABDiscovery = true)

{
  "stat_t": "+/+/BTtoMQTT/A4C1385DBC50",
  "dev_cla": "temperature",
  "unit_of_meas": "°C",
  "name": "LYWSD03MMC_ATC-tempc",
  "uniq_id": "A4C1385DBC50-tempc",
  "val_tpl": "{{ value_json.tempc}}",
  "state_class": "measurement",
  "device": {
    "identifiers": [
      "A4C1385DBC50"
    ],
    "connections": [
      [
        "mac",
        "A4C1385DBC50"
      ]
    ],
    "manufacturer": "Xiaomi",
    "model": "LYWSD03MMC_ATC",
    "name": "LYWSD03MMC-5DBC50",
    "via_device": "OMG_BLE"
  }
}

The is_defined is gone but the problem is not solved: sensors are discovered in Domoticz but there are no sensor devices created. The value messages look absolutely normal in both cases:

Code: Select all

{
  "id": "A4:C1:38:5D:BC:50",
  "mac_type": 0,
  "name": "ATC_5DBC50",
  "rssi": -84,
  "brand": "Xiaomi",
  "model": "LYWSD03MMC",
  "model_id": "LYWSD03MMC_ATC",
  "tempc": 23.7,
  "tempf": 74.66,
  "hum": 50,
  "batt": 81,
  "volt": 2.936
}

Problem must be the wildcards in the config message.

[StephenR]

I wrote a fix this morning which I am testing out. It seems to be working ok at the moment.

Assuming it looks good, I will read up on how to submit the change as a potential update to the system.

It isn't too lovely a fix because I wanted to keep the changes to a minimum and the system currently relies quite heavily on the subscription topic as a key for linking up sensors to incoming messages but I don't think it is too 'ugly'. It might not pass muster with the moderators though! (which I would understand!)

[StephenR]

Hi,

I have got a proposed fix. I will try to trigger a 'pull request' but this is the first time I have submitted a patch anywhere so I am very much on a learning curve!

If anyone wants to test it out before it is available in the main tree (if accepted), you should be able to build it from source just as per the instructions in the wiki.
The only difference is the git path:
git clone https://github.com/StephenR1/domoticz.git sr-domoticz
(obviously you can use any directory name)

and also you need to switch to the 'master' branch. (I think all patches are asked to target that). You can do that with :
git checkout --track remotes/origin/master

Slightly more complex, if you usually use the development branch and you have a newish version, then you will probably find that the db version is too far forward. I didn't hit any problems for my testing by setting the version back from 158 to 156 to make it compatible with the 'master' branch but you would definitely want to keep a backup of your old .db file!

Thanks

Stephen

[waltervl]

I cannot test but just 1 remark: Normally these fixes are build to the development branch of domoticz. Not to the Master (stable).
So in the near future if you want to submit a PR to the domoticz environment you will ahve to do that on the development branche.

[StephenR]

Hi waltervl,

I was surprised (and probably got it wrong) but the notes on how to contribute code in the github repository say :

Please base your bug fixes against the master branch. The master branch is considered the stable and is used for our releases.

(from https://github.com/domoticz/domoticz/bl ... IBUTING.md)

?

Thanks

Stephen

[StephenR]

Am still learning this!

If I should move changes to dev branch and re-trigger the pull request, please do say!

[waltervl]

Ah yes, this seems completely wrong in the contributing description. This should be the development branch. You can see a lot more actions on that branch than on the Master branch.

[waltervl]

Hi waltervl,

I was surprised (and probably got it wrong) but the notes on how to contribute code in the github repository say :

Please base your bug fixes against the master branch. The master branch is considered the stable and is used for our releases.

(from https://github.com/domoticz/domoticz/bl ... IBUTING.md)

?

Thanks

Stephen

Thanks tou your comment document https://github.com/domoticz/domoticz/bl ... IBUTING.md has bee updated.