Generate & Export XML-file from Domoticz

[Toulon7559]

To feed another program with information, looking for a small script (lua or Python) with the following functions:
1) take 1 (or more) specific value(s) from Domoticz:
1a) direct calling either an svalue or another data-sort
1b) make a json-call covering the targeted value
2) from the call-result generate an xml-file including the selected value(s)
3) export this xml-file to another computer at the LAN and/or at internet

Somebody using such software, or seen a usable pieces of software as starter for development?

[Toulon7559]

A rainy/cold day is a good reason for some experiments .......

Started to make a lua-script (borrowing ideas and scripts from internet), and found a solution for the indicated step 1b), but now stuck at step 2):
the conversion by the borrowed script from JSON to XML produces a file which (euphemistically) does not meet the definitions for an XML-file.
The script is below, together with 2 related extracts from Domoticz's log.

Any ideas how to proceed?
Applicable, working examples?

Code: Select all

-- =Lua script= Transfer of Info from Domoticz to another computer
-- Original script from StackOverflow, adapted/extended by Toulon7559, version 0.2 for Domoticz@RPI3
-- ===============================================================================================
-- Script-objectives:
-- 1) Make a JSON-call to a source computer at LAN
-- 2) Extract contents and put in JSON-file
-- 3) Convert JSON-file to XML-file
-- 4) Upload JSON-file and/or XML-file to destination computer at LAN or at Internet

-- Line 10 Make JSON-Call to source computer, Extract contents and make JSON-file
J='curl "http://192.168.1.4:8080/json.htm?type=devices&rid=89"'
print('Curl&JSON-call: '.. J)
local JSON_Light1=assert(io.popen(J))
local Contents1 = JSON_Light1:read('*all')
JSON_Light1:close()
print('JSON-file:'..Contents1) -- Check contents of JSON-file

-- Line 18 Convert JSON-file to XML-file
X=Contents1:gsub('{','<ITEM>\n'):gsub('},?','</ITEM>\n')
X=X:gsub('"(%w-)"%s*:%s*(%b[])',
    function (k,v)
        return string.format("<%s>%s</%s>\n",k,v:sub(2,-2),k)
    end)
X=X:gsub('"(%w-)"%s*:%s*"(.-)",?',
    function (k,v)
        return string.format("<%s>%s</%s>\n",k,v,k)
    end)
print ('XML-file: '.. X) -- Check contents of XML-file

-- Line 30 Upload JSON-/XML-file to destination computer by URL-call

-- Upload_URL = to be defined/inserted

commandArray = {}

--remove -- before the line below to actually upload
--commandArray['OpenURL']= Upload_URL
    
return commandArray

Print of JSON-file

screenshot-Lua_JSONCall_File.png (26.41 KiB) Viewed 3686 times

The second screenshot proceeds where the first one ends, and shows the result of the XML-conversion according to the script:
the elements can be recognized, but the required brackets and all headings are missing.
How to add those brackets and headers to the XML-output?

Print of 'XML-file'

screenshot-Lua_JSON_to_XML.png (20.71 KiB) Viewed 3686 times

And then obviously the final section of the script needs to be filled with some small & reliable scriptlines ....

[Toulon7559]

For that final section/ aspect 3), for upload of a well-formed file from the Raspberry to the websiteserver perhaps looking too far?
In Python the FTP-function is ready-for-use according to this description

Looking around at internet also Python-module dicttoxml seems quite useful to make much of the desired functionality.
Other information at StackOverflow leads to the following Python-script, which performs the required calls, conversions as well as upload.
This example is for RID=87. When you copy the script, obviously to be adapted to your own application and for your own configuration.

Code: Select all

#!/usr/bin/python
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2016 script compiled by Toulon7559 from various material from forums, version 2a
# --------------------------------------------------
# Line005 = PREPARATION & SETTING
# --------------------------------------------------
# Imports for script-operation
import json
import urllib
import dicttoxml
page = urllib.urlopen('http://<source-IP-address>:8080/json.htm?type=devices&rid=87')
content_test = page.read()
obj_test = json.loads(content_test)
print(obj_test)
with open('json87_upload.json', 'w') as outfile:
    json.dump(obj_test, outfile)
xml_test = dicttoxml.dicttoxml(obj_test)
print(xml_test)
xml_output = open("xml87_upload.xml",'w')
xml_output.write(xml_test)
xml_output.close()
# --------------------------------------------------
# Line024 = FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
    ext = os.path.splitext(file)[1]
    if ext in (".txt", ".htm", ".html"):
        ftp.storlines("STOR " + file, open(file))
    else:
        ftp.storbinary("STOR " + file, open(file, "rb"), 1024)

# --------------------------------------------------
# Line038 = Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("<ftp-server>")
ftp.login("<ftp_username>", "<ftp_password>")

upload(ftp, "json87_upload.json")
upload(ftp, "xml87_upload.xml")

[Toulon7559]

Once your are able to extract from the Domoticz' RID the data by JSON-call and can upload the resulting JSON-file and XML-file to your webserver, at the receiving end at the website you can extract & apply the data in those files for display.
Example for an html-script for extraction, and subsequent display by means of gauge(s) could can be found in the PVLoggers from FP4All for their 'local' web-picture.

[Was accessible on internet at https://github.com/openremote/Documenta ... ta-sources ,but unfortunately only the example XML-files remain and not the 'dissecting' script]

[Toulon7559]

The script from the previous messages makes JSON-calls, and from the responses makes directly related JSON-files and XML-files.
Usually for filling the website you will only need 1 or 2 values from each uploaded file.
If you want to collect various values from Domoticz, it means that you make a series of JSON-calls, and at the website's side you have to 'dissect' the various related JSON/XML-files to get hold of the values, disregarding the other elements.

Clarity, and significantly less effort at the side of the website if the upload could be limited to just 1 file with a 'simple' structure, with only those values you need at the website. That will require that the sifting=>extraction=>compilation has to be done before upload.
The resulting upload might be an XML-file like below, with value1 till value5 selected & filled by the user's script.

Code: Select all

<response>
  <value1>0.0</value1>
  <value2>10.0</value2>
  <value3>10.0</value3>
  <value4>0.0</value4>
  <value5>0.00</value5>
  <time_stamp>20161215 20:44</time_stamp>
</response>

Who has (seen) a script which from selected internal Domoticz-info can compile an XML-file? Or a comparable JSON-file?

[Toulon7559]

Further study/search seems to indicate that (for making an XML-file) the starting point in Python is a dictionary, because from a dictionary by means of dicttoxml you can make an XML-file.

Initial question then is 'reduced' to: how to make a dictionary which contains selected dynamic values from Domoticz, and readable by dicttoxml?
In my opinion it means:
1. make a default list-structure for the dictionary
2. values in that structure represented by parameters
3. those parameters to be linked to selected values in Domoticz
4. conversion of the dictionary to XML-file

Addition 20th of January 2017
A cold afternoon sometimes breeds ideas.
Found some useful hints on the Python Dictionary, which after some experiments resulted in the script below.
As can be seen in the comments and names, these lines of script have been tested as Annex to the Python-script developed for read-out of the kWh-meter DDS238-1ZN. Obviously the print-lines are optional.

Code: Select all

# --------------------------------------------------
# Make XML-file for DDS238-1ZN Read-out
# --------------------------------------------------
# Take care of extra dependencies
import dicttoxml
# Create the default dictionary
DDS238_dict = {'gauge_power': 1.2, 'gauge_temp': 20.4, 'gauge_vpv': 150, 'gauge_iac': 1.5, 'energy_today': 2.5, 'energy_total': 45.00, 'time_stamp': 0}
print(DDS238_dict)
# Link the directory to dynamic values in the Python-script (or fill-in placeholders)
DDS238_dict['gauge_power'] = ActivPower
DDS238_dict['gauge_temp'] = 20.1
DDS238_dict['gauge_vpv'] = VOLTAGE
DDS238_dict['gauge_iac'] = CURRENT
DDS238_dict['energy_today'] = Wh_today
DDS238_dict['energy_total'] = Wh_life
DDS238_dict['time_stamp'] = now
print(DDS238_dict)
# Convert dictionary to XML-file
DDS238_xml = dicttoxml.dicttoxml(DDS238_dict)
print(DDS238_xml)

Next 'exercise' is making a script taking a collection of svalues from Domoticz and combining them in a single XML-file for periodic upload to my website for dissect&extract by AJAX/Javascript.

[Toulon7559]

By means of JSON-calls also in a Python-script the values can be extracted from the IDXes/RIDs made&maintained by Domoticz.
The Python-script below performs that function and recompiles a selection of extracted values into a new json-file and a new xml-file, which are uploaded/pushed by ftp to a remote server.
The compilation is input for the 'next stage' at the ftp-server which reads the json-file or the xml-file.
Therefore both json-file and xml-file, because not yet sure of the further application, and therefore 'freedom' required.
Consider this script as conceptual/example material, to be tuned & improved & expanded, especially looking at an application by the 'next stage'!!!

Code: Select all

#!/usr/bin/python
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2017 script compiled by Toulon7559 from various material from forums, version 1.0
# 
# Plenty of optional print-lines as aid during debugging, when running this script from Putty's command line.
# For YOUR configuration YOU need to select the RIDs from which you want to extract info
# Dependent on the number of RIDs, YOU have to replicatie  the section "Read RIDxx & Extract Values"
# Tune & align the "Build ..." section for the values you want to include in the dictionary
# Do not forget to insert YOUR information in the lines setting the ftp-transfer!!! 
# --------------------------------------------------
# PREPARATION & SETTING
# --------------------------------------------------
# Imports for script-operation
import json
import urllib
import dicttoxml
# Tested at Raspberry3
# RID=87 = DHT22, RID=88 = DS18B20, RID=xx = RV_analoog
HUM2 = 90.1 # Placeholder info for RIDxx
print
print ('Start of script')
print
# --------------------------------------------------
# Read RID87 & Extract Values
# --------------------------------------------------
print ('Info RID87')
page = urllib.urlopen('http://192.168.1.6:8080/json.htm?type=devices&rid=87')
content_rid = page.read()
# Check1 contents of read JSON-call
#print(content_rid)
#print
obj_test1 = json.loads(content_rid)
# Check2 contents of extracted JSON-info
#print(obj_test1)
#print
# Check contents of some str's
print(obj_test1['ServerTime'])
print(obj_test1['Sunrise'])
print(obj_test1['Sunset'])
# Extract & Check numeric values
HUM1 = obj_test1['result'][0]['Humidity']
print(HUM1)
TEMP1 = obj_test1['result'][0]['Temp']
print(TEMP1)
DEW1 = float(obj_test1['result'][0]['DewPoint'])
print(DEW1)

# TO BE ADDED: extraction of text and extraction/conversion of numeric values which are now str or utc (e.g. Sunrise, Sunset, ServerTime, ActTime, status)
print
# --------------------------------------------------
# Read RID88 & Extract Values
# --------------------------------------------------
print ('Info RID88')
page = urllib.urlopen('http://192.168.1.6:8080/json.htm?type=devices&rid=88')
content_rid = page.read()
#print(content_rid)
#print
obj_test2 = json.loads(content_rid)
#print(obj_test2)
#print
print(obj_test2['ServerTime'])
print(obj_test2['Sunrise'])
print(obj_test2['Sunset'])
TEMP2 = obj_test2['result'][0]['Temp']
print(TEMP2)
print
# --------------------------------------------------
# Build JSON-file 'Collection1'
# --------------------------------------------------
print ('File-building')
# Setup default dictionary
pythonDict = { 'HaveTimeout': False, 'Temp1': 12.5, 'Temp22': 11.5, 'Hum1': 90.5, 'Hum2': 91, 'Dew1': 9.0 }
# Check Dictionary
print('Default Dict : ', pythonDict)
# Link values of dictionary to dynamic values
pythonDict['Temp1'] = TEMP1
pythonDict['Temp2'] = TEMP2
pythonDict['Hum1'] = HUM1
pythonDict['Hum2'] = HUM2
pythonDict['Dew1'] = DEW1
pythonDict['timestamp'] = obj_test2['ActTime']
# Check Dictionary
print('Dynamic Dict : ', pythonDict)
# Make json-file
with open ('Json_coll1_upload.json', 'w') as outfile:
    json.dump(pythonDict, outfile)
# Make xml-file
xml_coll = dicttoxml.dicttoxml(pythonDict)
# print(xml_coll)
xml_output = open("Xml_coll1_upload.xml",'w')
xml_output.write(xml_coll)
xml_output.close()
print 
# --------------------------------------------------
# Definition of FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
    ext = os.path.splitext(file)[1]
    if ext in (".txt", ".htm", ".html"):
        ftp.storlines("STOR " + file, open(file))
    else:
        ftp.storbinary("STOR " + file, open(file, "rb"), 1024)

# --------------------------------------------------
# Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("<ftp-servername>")
ftp.login("<ftp_username>", "<ftp-password>")

upload(ftp, "Json_coll1_upload.json")
upload(ftp, "Xml_coll1_upload.xml")
print ('End of script')

Question:
how to extract & convert numeric values which are now 'hidden' in a textstring?
[see in the script the remark TO BE ADDED]

[Toulon7559]

The Python-script in my previous message in this thread is a solution to push info from Domoticz to a webserver, but it is an indirect way in a number of steps to get hold of values and to recompile them for their next application.

Looking for a shortcut, direct way to read Domoticz' svalues in Python.
"Search & Google" do not give much response:
anybody with a workable, simple & short solution in Python-script?