Getting data from LoRa - TTN network

[costo]

I'm trying to extract data from a node that is communicating with TheThingsNetwork over LoRaWan.
In a terminal window I can sent this command:

curl -X GET --header 'Accept: application/json' --header 'Authorization: MyKey' 'MyURL'

where MyKey is a long private key for this particular node and MyURL is on .data.thethingsnetwork.org/etc
I get a response between [ ] brackets which contain several strings separated by comma's:
each line is like this:

{"device_id":"costonode0","hum":55,"raw":"3x4gGw==","temp":21.9375,"time":"2018-03-16T13:50:23.154294105Z","volt":3227},

So I want to write a script either in BASH or LUA which analyses the strings and with JSON returns the time, hum(idity) , temp(erature) and voltage of my battery (in mV). Ofcourse this node can generate other data so the script needs to be a little configurable.
I am not a very good programmer, I have a little experience with LUA so I ask here for some help and clues how to write this script, either in LUA or another language.

[costo]

I made this LUA script but apparently there is something wrong in the formatting of the curl string , i get this error:

-- 2018-03-17 11:32:00.391 Error: EventSystem: in /home/pi/domoticz/scripts/lua/script_time_lora.lua: /home/pi/domoticz/scripts/lua/script_time_lora.lua:14: ')' expected near 'Accept'

Code: Select all

    local idxHum  = 644  
    local idxTemp = 645  
    local idxVolt = 643  
    local hum = ' '
    local temp = ' '
    local volt = ' '

    commandArray = {}

    time = os.date("*t")
    if  ((time.min % 5)==0)  then         -- Run every 5 minutes.

        json = (loadfile "/home/pi/domoticz/scripts/lua/JSON.lua")()
        local jsondata = assert(io.popen('curl -X GET --header 'Accept: application/json' --header 'Authorization: key ttn-account-v2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' 'https://costonode100.data.thethingsnetwork.org/api/v2/query/costonode0?last=5m'))

        local jsondevices = jsondata:read('*all')
        jsondata:close()

        local jsonLoRa = json:decode(jsondevices)
        hum = jsonLoRa.hum
        temp = jsonLoRa.temp
        volt = jsonLoRa.volt

        commandArray[1] = {['UpdateDevice'] = idxHum .. '|0|' .. hum}
        commandArray[2] = {['UpdateDevice'] = idxTemp .. '|0|' .. temp}
        commandArray[3] = {['UpdateDevice'] = idxVolt .. '|0|' .. volt}
    end
    
return commandArray

[costo]

Does someone see the error ?
I cannot figure it out what is going wrong.
I get error in line 23: attempt to index local 'jsonLoRa' (a nil value)
So maybe the json does not execute well or maybe there is an error in the Curl string.
I am out of options.
Can someone who understands LUA better than me help me here a bit please. ?

Code: Select all

    local idxHum  = 644  
    local idxTemp = 645  
    local idxVolt = 643  
    local hum = ''
    local temp = ''
    local volt = ''
    local accessKey = 'ttn-account-v2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
    local urlTTN    = 'https://costonode100.data.thethingsnetwork.org/api/v2/query/costonode0?last=5m'
	
commandArray = {}

   -- time = os.date("*t")
   -- if  ((time.min % 5)==0)  then         -- Run every 5 minutes.

        json = (loadfile "/home/pi/domoticz/scripts/lua/JSON.lua")()

	local config = assert(io.popen("curl -X GET -H \'Accept: application/json\' -H \'Authorization: ..'accessKey'..\' \'..'urlTTN'..\'"))
        local devices = config:read('*all')
        config:close()
        print(output)
        local jsonLoRa = json:decode(devices)
	
        hum = jsonLoRa.hum
        temp = jsonLoRa.temp
        volt = jsonLoRa.volt

        commandArray[1] = {['UpdateDevice'] = idxHum .. '|0|' .. hum}
        commandArray[2] = {['UpdateDevice'] = idxTemp .. '|0|' .. temp}
        commandArray[3] = {['UpdateDevice'] = idxVolt .. '|0|' .. volt}
   -- end

return commandArray

[costo]

I found out that the problem is in the square brackets.
ttn server returns something like this:

[{"device_id":"costonode0","hum":50,"raw":"3RogEg==","temp":21.8125,"time":"2018-03-17T23:52:14.248591862Z","volt":3218}]
or
[{"device_id":"costonode0","hum":50,"raw":"3RogEg==","temp":21.8125,"time":"2018-03-17T23:52:14.248591862Z","volt":3218},{"device_id":"costonode0","hum":50,"raw":"3hogEg==","temp":21.875,"time":"2018-03-17T23:55:56.381884587Z","volt":3218}]

data for the last 5 minutes usually returns one json object and sometimes two.
look like local jsonLoRa = json:decode(devices) gives an error (nil value) because of the square brackets .

tried to remove the square brackets with this:
x = string.gsub(devices,"[{","{")
devices = string.gsub(x,"}]","}")
that was not working.

someone has a clue for me ?

[alexsh1]

someone has a clue for me ?

I am afraid you are on your own.
I have just started to looking into LoraWAN/TTN

[gizmocuz]

Hi,

I just finished the first implementation of a hardware class (available in current beta, 9946 and up) that connects to the TTN MQTT broker and is able to work with devices that use CayenneLPP

"The Things Network (MQTT/CayenneLPP) with LAN interface"

First try to get yourself connected via a commandline MQTT client to test if everything works, then you can add this hardware
(Do not forget to enable 'accept new hardware' in the Domoticz settings)

Hope all works well , tested here with a temp/hum/baro sensor (BME 280), but separate should also work, as well as the other sensors.
Currently it is not possible to 'send' data (like send a switch ON/OFF command)

If you use the same channel for your temp,humidity,baro sensor, domoticz will combine them into 1 sensor, same for temp+hum, or temp+baro

In the Domoticz github repo you can find a class to encode to CayenneLPP in C++ (folder hardware/CayenneLPP)

[Toulon7559]

Could this software be useful?
https://pypi.org/project/cayenneLPP/

[gizmocuz]

Sure... thats what i implemented... the CayenneLPP protocol

[Toulon7559]

Any hints for application with KPN LoRa?

[QSKONE]

Hi , Can you make tutorial?

My log:

Code: Select all

Status: TTN_MQTT: Connecting to 192.168.1.5:1883
Error: TTN_MQTT: Failed enabling TLS mode, return code: 3 (CA certificate: 'ttn-account-xxx')
2018-12-27 19:58:28.404 Status: TTN_MQTT: Worker stopped...

[gizmocuz]

You need to use the same privilege settings as you would when using MQTT to connect to the TTN network.
Did you try using a MQTT client ?

[costo]

I have been puzzling for some time but I cannot make this plugin working. It is not clear to me what data i need to put in the fields of the TTN plugin.
If I look at my 'MQTT Client Gateway with LAN Interface' plugin that works with a IP in the 192.168.178.xx range, and port 1883. The Username and Password are the ones to connect to the RasPi server where Domoticz also resides.

I imaging that I need to use the TTN credentials for accessing my TTN-node . That access key looks like 'ttn-account-vs.ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%&*'. I do not know of any Username/Password/CA_Filename combination to access the TTN network.

[jeroenkl]

I have been puzzling for some time but I cannot make this plugin working. It is not clear to me what data i need to put in the fields of the TTN plugin.
If I look at my 'MQTT Client Gateway with LAN Interface' plugin that works with a IP in the 192.168.178.xx range, and port 1883. The Username and Password are the ones to connect to the RasPi server where Domoticz also resides.

I imaging that I need to use the TTN credentials for accessing my TTN-node . That access key looks like 'ttn-account-vs.ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%&*'. I do not know of any Username/Password/CA_Filename combination to access the TTN network.

Hi Costo,

I'm dealing with the same issue, do you have already the solution?

BR, Jeroen

[QSKONE]

Integration TTN to Domoticz it's possible ,but nobody knows how do it.

[jeroenkl]

I have been puzzling for some time but I cannot make this plugin working. It is not clear to me what data i need to put in the fields of the TTN plugin.
If I look at my 'MQTT Client Gateway with LAN Interface' plugin that works with a IP in the 192.168.178.xx range, and port 1883. The Username and Password are the ones to connect to the RasPi server where Domoticz also resides.

I imaging that I need to use the TTN credentials for accessing my TTN-node . That access key looks like 'ttn-account-vs.ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789!@#$%&*'. I do not know of any Username/Password/CA_Filename combination to access the TTN network.

I'm able to connect:
2019-01-13 19:49:27.925 Status: TTN_MQTT: Connecting to eu.thethings.network:1883
2019-01-13 19:49:28.053 Status: TTN_MQTT: connected to: eu.thethings.network:1883
2019-01-13 19:49:28.158 Status: TTN_MQTT: Subscribed

2019-01-13 19:57:07.336 TTN_MQTT: Topic: xxxxx/devices/xxxxxx/up
2019-01-13 19:57:07.337 Error: TTN_MQTT: Invalid data received!

with this:
mosquitto_sub -h <Region>.thethings.network -t '+/devices/+/up' -u '<AppID>' -P '<AppKey>' -v

I've got the data from my devices, so something wrong with the domoticz plugin?

pi@raspberrypi:~ $ mosquitto_sub -h eu.thethings.network -t '+/devices/+/up' -u 'xxxxxxx' -P 'ttn-account-v2.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-v

xxxxxx/devices/xxxxxxxxxxx/up"app_id":"xxxxx","dev_id":"xxxxxxxxxxxxx","hardware_serial":"xxxxxxxxxxxxxxxxxx","port":1,"counter":1124,"payload_raw":"A+gDDA==","payload_fields":{"humidity":10,"temperature":7.8},"metadata":{"time":"2019-01-13T19:19:59.379375846Z","frequency":867.7,"modulation":"LORA","data_rate":"SF7BW125","airtime":51456000,"coding_rate":"4/5","gateways":[{"gtw_id":"xxxxxxxxx","gtw_trusted":true,"timestamp":2412230724,"time":"2019-01-13T19:20:02Z","channel":6,"rssi":-106,"snr":2,"rf_chain":0,"latitude":52.26572,"longitude":5.183096,"altitude":12,"location_source":"registry"}]}}
^C
pi@raspberrypi:~ $

[jeroenkl]

You need to use the same privilege settings as you would when using MQTT to connect to the TTN network.
Did you try using a MQTT client ?

Pfff finally all is working. I'll provide the info later this week.

[Toulon7559]

As start of my LoRaWAN learning curve, now experimenting with a Marvin Development Board, with related, temporary 'Free' subscriptions to the KPN LoRa-Developer Portal (6 months) and to the Mendix Dashboard (70 days).

'New Land' and therefore looking around at Internet, to sites like this one with related experiences.
So far got my Marvin-board with connected SHT31-sensor periodically uploading T/H-data over the KPN LoRa-Network towards the Mendix Dashboard:
uplink is quick, but storage and read-out is not very frequent, because the free subscription limits the handling to 6messages/hour.

For more direct & quicker read-out, the next hurdle/challenge is how to get online hold of the JSON-file which is told to be 'there', containing the uploaded data.
Also trying data collection by means of other Endpoint_URLs like Hookbin.com, but not yet with results.
The example scripts at the home page of hookbin.com probably are good starting-material for an extraction script, but first have to know which link&key-information to insert and how .....

For 'data grabbing' from the KPN Network somebody having a tested, (relatively) simple example setup for the applicable (C)URL-call, etc.?

[Toulon7559]

@Gizmocuz

Reading your remark that Domoticz now supports Cayenne linking to TTN's MQTT broker, looked around and found a contribution by Hans Boksem for a Meteo-System linked through TTN and NodeRed to Domoticz, later extended[?] with a variant linking to Cayenne.

Question related to last line of your statement

I just finished the first implementation of a hardware class (available in current beta, 9946 and up) that connects to the TTN MQTT broker and is able to work with devices that use CayenneLPP

"The Things Network (MQTT/CayenneLPP) with LAN interface"

First try to get yourself connected via a commandline MQTT client to test if everything works, then you can add this hardware

;-( Possible to give example-for-dummies for

connect via a commandline MQTT client

?

Just wondering how to fit my Marvin in that picture, presently operating at the KPN LoRa-Network:
apparently have to mimick the TTN-'scheme' created by Hans van Boksem and Gizmocuz, with a KPN-replacement for TTN's MTTQ-broker.

[robert13b]

You need to use the same privilege settings as you would when using MQTT to connect to the TTN network.
Did you try using a MQTT client ?

Pfff finally all is working. I'll provide the info later this week.

Looking forward to it. I get the same kind of data from the command line with de mosquito client. Bout how do i get it into Domoticz referring to an IDX?

[jeroenkl]

1; First ensure you got the Lora required application data in cayenne visibel (tutorial on the web, in case you need help let me know)
2: Next add new hardware in domoticz: The Things Network (MQTT/CayenneLPP) with LAN interface, eu.thethings.network:1883, login: Lora application and passwd
3: Discovered devices are automatic added in domoticz

Need help? Let me know.

BR, Jeroen