With hints from friends at Stackoverflow achieved the conversion from the basic file-layout from Luftdaten.info into a format with unique fieldnames.
Also the related script running, which then extracts the information and transfers towards Domoticz.
2-Step approach:
Step1 = Python-script
running under cronjob, which
- calls the local measuring setup,
- changes the format of the json-file,
- makes a 2nd set of json-file + xml-file for upload, and
- uploads those 2 files to a server
[
Reason for upload of json+xml: in that way the data is accessible for multiple applications also by other users]
Step2 = lua-script which
- reads the 2nd xml-file with the name given in step1 [therefore do no change filenames in Step1 unless you know what you do!],
- dissects that file and extracts temperature, humidity, PM2.5 and PM10, and
- uploads the extracted information to Domoticz
[upload of temperature+humidity is towards a virtual sensor which mimicks a DHT22, and towards 2 Custum sensors for PM2.5 respectively PM10]
Python-script for Step1
Code: Select all
#!/usr/bin/python
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2017 script compiled by Toulon7559 from various material from forums, version 00
# (c)2018 extended by extra step to extract info from json-file from Luftdaten.info
# --------------------------------------------------
# Line006 = PREPARATION & SETTING
# --------------------------------------------------
# Imports for script-operation
import json
import urllib
import dicttoxml
# SDS011 is dust-sensor at IP=192.168.0.141
# IP-address obviously to be tuned to YOUR configuration
# --------------------------------------------------
# Line015 Read data and compile json-file + xml-file
# --------------------------------------------------
page = urllib.urlopen('http://192.168.0.141/data.json')
content_test = page.read()
obj_test = json.loads(content_test)
print(obj_test)
with open('json_sds011_upload1.json', 'w') as outfile:
json.dump(obj_test, outfile)
xml_test = dicttoxml.dicttoxml(obj_test)
print(xml_test)
xml_output = open("xml_sds011_upload1.xml",'w')
xml_output.write(xml_test)
xml_output.close()
# --------------------------------------------------
# Line030 = Extract Field-names and values from json
# --------------------------------------------------
import json
import pprint
with open('json_sds011_upload1.json') as j:
data = json.load(j)
for sdv in data.pop('sensordatavalues'):
data[sdv['value_type']] = sdv['value']
print(data)
pprint.pprint(data)
# --------------------------------------------------
# Line043 Compile 2nd set of json-/xml-file
# --------------------------------------------------
with open('json_sds011_upload2.json', 'w') as outfile:
json.dump(data, outfile)
xml_test2 = dicttoxml.dicttoxml(data)
print(xml_test2)
xml_output = open("xml_sds011_upload2.xml",'w')
xml_output.write(xml_test2)
xml_output.close()
# --------------------------------------------------
# Line054 = FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
ext = os.path.splitext(file)[1]
if ext in (".txt", ".htm", ".html"):
ftp.storlines("STOR " + file, open(file))
else:
ftp.storbinary("STOR " + file, open(file, "rb"), 1024)
# --------------------------------------------------
# Line068 = Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("<ftp-server>")
ftp.login("<ftp-username>", "<ftp-password>")
upload(ftp, "json_sds011_upload2.json")
upload(ftp, "xml_sds011_upload2.xml")
Lua-script for Step2
Code: Select all
------------------------------------------------------------------------------------------
-- Version 00 20180226
--
-- Domoticz lua script to convert XML output for SDS011-sensor plus related DHT22-sensor
-- Reads the status based on the unique web-address and passes values
-- to virtual sensor(s) in Domoticz
--
--------------------------------------------------------------------------------------------------
-- Original script by RATA1 at Domoticz-forum, adapted by Toulon7559 (c)2016-2018
--------------------------------------------------------------------------------------------------
-- Line 11: Start of script
commandArray = {}
function XML_Capture(cmd,flatten)
local f = assert(io.popen(cmd, 'r'))
local s = assert(f:read('*a'))
f:close()
if flatten then
s = string.gsub(s, '^%s+', '')
s = string.gsub(s, '%s+$', '')
s = string.gsub(s, '[\n\r]+', ' ')
end
return s
end
-- Line 26: Set debug to true to print to log file, set to false is not.
debug = true
-- Line 29: Define the idx of YOUR virtual sensor(s) in Domoticz
-- => what you have called your sensor(s) and the related IDX number(s)
PT_SDS011M = 1582 -- Virtual device for DHT22, temp + baro
PT_SDS011A = 1583 -- Virtual device for SDS PM2.5 [= SDS_P2 in file], custom sensor
PT_SDS011B = 1584 -- Virtual device for SDS PM10 [= SDS_P1 in file], custom sensor
-- Line 35: Define your device IP-address: the xml-filename was set in Step1!
-- Lines 37 ~ 45 define where the XML-file can be read
Device_IP = "xxxx.nl" -- if you read from a remote website xxxx.nl
-- Device_IP = "192.168.1.x/home/pi" --if you read from your local Domoticz@Raspberry
if debug == true then
print("Reading values from: 'http://"..Device_IP.."/xml_sds011_upload2.xml'")
end
-- Line 43: Read the XML data from your device and from your XML-string
XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/xml_sds011_upload2.xml'",1)
valid = string.find(XML_string, "<root>") -- check that we are looking in the right place
if debug == true then
print("SDS011-string = "..XML_string)
end
if valid == nil then
print ("Bad XML status read - info NOT updated")
else
-- Line 57: Find in the XML_string the info-fields based on their labels
-- Line 58: read position of SDS_strings
s1 = string.find(XML_string,"<SDS_P1") -- start of header SDS_P1
print(s1)
t1 = string.find(XML_string,"</SDS_P1") -- start of tail SDS_P1
print(t1)
s2 = string.find(XML_string,"<SDS_P2") -- start of header SDS_P2
print(s2)
t2 = string.find(XML_string,"</SDS_P2") -- start of tail SDS_P2
print(t2)
-- Line 67: read position of temperature & humidity strings
s3 = string.find(XML_string,"<temperature") -- start of header temperature
print(s3)
t3 = string.find(XML_string,"</temperature") -- start of tail temperature
print(t3)
s4 = string.find(XML_string,"<humidity") -- start of header humidity
print(s4)
t4 = string.find(XML_string,"</humidity") -- start of tail humidity
print(t4)
-- Line 76: Manually determine/set start/end-positions of the values within the substrings
p1 = s1+19 -- number = characters of header-string SDS_P1
print(p1)
p2 = s2+19 -- number = characters of header-string SDS_P2
print(p2)
p3 = s3+24 -- number = characters of header-string temperature
print(p3)
p4 = s4+21 -- number = characters of header-string humidity
print(p4)
-- Line 85: Extract the values and process for upload to Domoticz
-- Line 86: read SDS_P1 string
SDSP1 = string.sub(XML_string,p1,t1-1)
SDS_P1 = tonumber(SDSP1)
print(SDS_P1)
-- Line 90: read SDS_P2 string
SDSP2 = string.sub(XML_string,p2,t2-1)
SDS_P2 = tonumber(SDSP2)
print(SDS_P2)
-- Line 94: read temperature string
tmp1 = string.sub(XML_string,p3,t3-1)
DHT_T = tonumber(tmp1)
print(DHT_T)
-- Line 98: read humidity string
hm1 = string.sub(XML_string,p4,t4-1)
DHT_H = tonumber(hm1)
print(DHT_H)
-- Line 103: Upload to Domoticz
commandArray[1] = {['UpdateDevice'] = PT_SDS011M.."|0|"..DHT_T..";"..DHT_H..";1"} -- send updated values to Domoticz
-- with the virtual device set in line 31 the above { layout } as result mimicks a DHT22, except for the 3rd (static) value
commandArray[2] = {['UpdateDevice'] = PT_SDS011A.."|0|"..SDS_P2} -- send updated value for PM2.5 to Domoticz
commandArray[3] = {['UpdateDevice'] = PT_SDS011B.."|0|"..SDS_P1} -- send updated value for PM10 to Domoticz
if debug == true then
print("Temp/Hum = ".."'"..DHT_T.."/"..DHT_H.."'")
print("PM2.5 = ".."'"..SDS_P2.."'")
print("PM10 = ".."'"..SDS_P1.."'")
end
end
return commandArray
For testing & checking the scripts have plenty of print-lines, which obviously can be removed to get leaner scripts (with simultaneous adaptation or removal of the line-numbers)
Different versions of Python sometimes cause trouble ....
