TTNMQTT module processes decoded_payload even in the absence of a frm_payload Topic is solved

[hferinga]

Using build 14.166 on raspberry pi.

Every now and then one or more sensors publish a message without a frm_payload. However the decoded_payload is present as probably processed by the provided payload formatter in TTN. These decoded payload values are all based on 0(null) values and have no meaningful values.

The calculation on the temperature results in a -32 degrees centigrade. This is the part of the payload formatter used in TTN that creates that value:

Code: Select all

          temperature: (bytes[2] & 0x7f) - 32,

This gives a significant negative spike in the graphs on Domoticz.

Is it possible to ignore messages with missing frm_payload?

This is an example of the message:

Code: Select all

{"tst":"2022-02-08T20:27:11.905264+0100","topic":"v3/first-lorawan-app@ttn/devices/eui-e8e1e100010910c9/up","qos":0,"retain":0,"payloadlen":1525,"payload":{"end_device_ids":{"device_id":"eui-e8e1e100010910c9","application_ids":{"application_id":"first-lorawan-app"},"dev_eui":"E8E1E100010910C9","join_eui":"58A0CB0001500000","dev_addr":"260B690D"},"correlation_ids":["as:up:01FVDCVQGBEVNKFYG0ETY5B6A5","gs:conn:01FVBZ1886KVF4MCNFK0A7R5CY","gs:up:host:01FVBZ188PQVP85GGX5N14VQCZ","gs:uplink:01FVDCVQ95TDN885MXPF3GT3TA","ns:uplink:01FVDCVQ97RV6Z60B644FAQAKP","rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01FVDCVQ972JNXTHVTK1YXGHAB","rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01FVDCVQGANA7JARYXDW74VC05"],"received_at":"2022-02-08T19:27:11.884713022Z","uplink_message":{"session_key_id":"AX6cPSqpNS9in/+WtPtRaA==","f_port":103,"f_cnt":368,"decoded_payload":{"battery":2.5,"humidity":0,"status":0,"temperature":-32},"rx_metadata":[{"gateway_ids":{"gateway_id":"hf-lorawan-gw01","eui":"58A0CBFFFE804796"},"time":"2022-02-08T19:27:11.590468883Z","timestamp":807436691,"rssi":-60,"channel_rssi":-60,"snr":8.5,"uplink_token":"Ch0KGwoPaGYtbG9yYXdhbi1ndzAxEghYoMv//oBHlhCTg4KBAxoMCI+Ii5AGEKvnz7cCILjM6Pe/9goqDAiPiIuQBhCTrseZAg=="}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"coding_rate":"4/5","frequency":"867100000","timestamp":807436691,"time":"2022-02-08T19:27:11.590468883Z"},"received_at":"2022-02-08T19:27:11.655431697Z","consumed_airtime":"0.051456s","version_ids":{"brand_id":"browan","model_id":"tbhh100","hardware_version":"1.0","firmware_version":"1.0.1","band_id":"EU_863_870"},"network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"ttn-eu1"}}}}

[hferinga]

I built domoticz from source (branch development) with this change to fix it for my situation:

Code: Select all

diff --git a/hardware/TTNMQTT.cpp b/hardware/TTNMQTT.cpp
index cd11b9873..078e64e8a 100644
--- a/hardware/TTNMQTT.cpp
+++ b/hardware/TTNMQTT.cpp
@@ -732,6 +732,11 @@ void CTTNMQTT::on_message(const struct mosquitto_message *message)
 
                if (!Decoded)
                {
+                       if (uplinkMessage["frm_payload"].isNull()) {
+                                // do not process the decoded_payload as it might not be meaningful
+                               Log(LOG_STATUS, "Not processing as the decoded payload might not be meaningful in the absence of a frm_payload!");
+                                return;
+                        }
                        if (!uplinkMessage["decoded_payload"].empty())
                        {
                                // Maybe we received a pre-decoded message? TTN has the option for custom decoders where the decoding is done by TTN already

If the frm_payload does not exist, it stops processing.

I don't actually know how this would impact other user's situation though. I am also not familiar with cpp, so it could be that there is another possible (better) solution.

[kiddigital]

Thx for reporting and I can probably create a small fix for the module.

What I am not sure of is the assumption that a missing frm_payload always means that the decoded_payload should be ignored. Are there valid scenarios where there is a decoded_payload without an frm_payload?

I would assume not as the frm_payload is the actual LoRa payload which could be decoded by a payload_formatter.

Could you also maybe raise an issue with TTN as I think it might be an issue there? Somehow the payload_formatter gets executed generating a message without having received an actual message or the message is bad/gone.

Thx

[hferinga]

I will look into it and see if I can raise an issue with TTN. From the documentation frm_payload is optional.

Looked at the payload formatter and made a small change in the provided one from the repository and changed it to javascript with the following code:

Code: Select all

function decodeUplink(input) {
  var bytes = input.bytes;

  if ( bytes.length < 4 ) {
    return {
      errors: ['undefined frm_payload'],
    };
  }
  
  switch (input.fPort) {
    case 103:
      return {
        data: {
          status: (bytes[0] >> 3), 
          battery: (25 + (bytes[1] & 0x0f)) / 10,
          temperature: (bytes[2] & 0x7f) - 32,
          humidity: bytes[3] & 0x7f
        }
    };
  default: 
    return {
      errors: ['unknown FPort'],
    };
  }
}

If the length of bytes is less than 4, then it returns an errors object. This results in a mqtt message that is both lacking the frm_payload and the decoded_payload.

In domoticz this will result in the following error:

Code: Select all

Error: TheThingsNetwork: Invalid data received! The payload_raw does not contain a payload that could be decoded (Port 103)!

The data is ignored and thus not recording these meaningless values. Which is another way of solving it, but then on the TTN side.

BTW every 24 hours (almost to the minute) this uplink message is sent with the missing frm_payload, so it looks like an intended message.
It happens only with the two Browan sensors I have. I have two dragino sensors that do not show this behavior.

[hferinga]

I posted a topic on the TTN forum. It seems that the uplinkMessage without application data is valid, it’s a useful way of sending a very short message - like “I’m alive”. That explains that it is sent with a 24h interval.

[kiddigital]

Ok so frm_payload is optional.

Such a ‘ping’ message scenario is something the payload formatter should cope with as it is the function of the payload formatter to do the decoding logic instead of something else downstream (for example the TTNMQTT module of Domoticz).

A downstream module can only try to process an already decoded payload without applying (payload formatter or other) logic to it.

There is no easy way (at least I can think of at the moment) to ‘build’ something generic into the module that could somehow filter things when receiving a decoded payload.

Maybe only a generic setting to ‘ignore decoded payload when frm payload is missing’.

[kiddigital]

Did some digging in the LoRa/TTN docs and came to the conclusion that the module should handle situation with no frm_payload better and not rely on an external decoder only.

I will make a small patch for this.

[hferinga]

Tested. Works. Thanks.

[kiddigital]

Great

Then this topic can be closed