Reading XML with LUA?

[RATA1]

Hello,
I have a KMTronics web temp sensor and I want to read the values from to update some virtual sensors it but can figure it out.

Basically I browse to http://x.x.x.x/status.xml and it displays this:

<response>
<sensor>
<id>28006D2307000061</id>
<name>HW_Flow</name>
<temp>20.50</temp>
</sensor>
<sensor>
<id>28AC0924070000C8</id>
<name>HW_Return</name>
<temp>19.56</temp>
</sensor>
<sensor>
<id>28FA1B24070000AB</id>
<name>HW_Tank_Top</name>
<temp>22.00</temp>
</sensor>
<sensor>
<id>28B30924070000B7</id>
<name>HW_Tank_Bottom</name>
<temp>21.12</temp>
</sensor>
<version>
<model>KMtronic DS1820 Temperature Monitor</model>
<ver>1.0</ver>
</version>
</response>

I (not being very good at all with programming) can't figure out how to open it in LUA before I even attempt to parse it stick it in the appropriate "otherdevices_svalues".
Can anyone give some pointers?

Thanks

[georgesattali]

Hello,
I suggest you 2 possibilities :
1) explore lua xml lib : https://github.com/manoelcampos/LuaXML and download file "xml.lua"
and follow example here https://github.com/manoelcampos/LuaXML/ ... er/example
maybe this is the good thing to do.

2) a function I wrote to extract meteo from wunderground xml :

Code: Select all

function Mcapture(cmd, flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten then 
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end


if now.min%30 == 5 then
   -- returns the number of rain mm that's gonna rain in max 24 next hours
   print("curl -s 'http://api.wunderground.com/api/"..apikey.."/hourly/lang:FR/q/"..country.."/"..town..".xml'")
   forecastString=Mcapture("curl -s 'http://api.wunderground.com/api/"..apikey.."/hourly/lang:FR/q/"..country.."/"..town..".xml'")
   debug("forecastString(1,20)="..forecastString:sub(1, 20))
   i=0
   sValue=""
   sumQpf=0
   maxPop=0
   for h=1,24 do
      i = string.find(forecastString, "<qpf>", i+1)    -- find 'next' newline
      if i == nil then break end
      qpf=string.match(forecastString, "<metric>(%d+)</metric>", i+1)
      pop=string.match(forecastString, "<pop>(%d+)</pop>", i+1)
      sumQpf = sumQpf+qpf
      maxPop = math.max(maxPop, pop)
      hh=(h+now.hour)
      if hh>24 then hh=hh-24; end
      if hh<10 then hh=" "..hh end
      sValue=hh.."h : Proba "..maxPop.."% "..sumQpf.."mm "

      if h==1 then
	 ModifierCapteurTexte(101, sValue)
      end
      
      if h==6 then
	 ModifierCapteurTexte(100, sValue)	 
      end

      if h==12 then
	 ModifierCapteurTexte(99, sValue)
      end

      if h==24 then
 	 ModifierCapteurTexte(98, sValue)
      end

      debug("..IsGonnaRain "..qpf.." mm dans "..h.." ("..hh..") heure avec une probabilité de "..pop.."%")
   end
end

[RATA1]

Hello.
Thanks for the reply - I didn't see it until just now.
I will have a look at the link and what you have done and see what I come up with. Will take some time I imagine.

I did look at LuaXML too.

By the time I get it working someone smarter will have included the module in the Domoticz hardware library (hopefully)!

Thanks

[Westcott]

Thanks for the links, Georgesattali.
One also really needs Handler.lua
This code fragment reads one of the Domoticz config files -

Code: Select all

function ReadXML(filename)
local xmltext = ""
local f, e = io.open(filename, "r")
if f then
	XML = (loadfile "/home/pi/domoticz/scripts/lua/XML.lua")()
	XMLhandler = (loadfile "/home/pi/domoticz/scripts/lua/Handler.lua")()
	local xmlhandler = simpleTreeHandler()
	--Instantiate the object that parses the XML to a Lua table
	local xmlparser = xmlParser(xmlhandler)
  --Gets the entire file content and stores into a string
	xmltext = f:read("*a")
	xmlparser:parse(xmltext)
--Recursivelly print the table
	printable(xmlhandler.root)
else
  error(e)
end
end

function printable(tb, level)
  level = level or 1
  local spaces = string.rep(' ', level*2)
  for k,v in pairs(tb) do
      if type(v) ~= "table" then
         print(spaces .. k..'='..v)
      else
         print(spaces .. k)
         level = level + 1
         printable(v, level)
      end
  end  
end

ReadXML("/home/pi/domoticz/Config/options.xml")

[RATA1]

Thanks to georgesattali and others on the forum I have written a little LUA script to read from the device and pop it into virtual sensors - result!

The KMTronic LAN DS1820 box is pretty good IMO and can be got on ebay at a good price.

See here http://www.domoticz.com/forum/viewtopic ... 096#p83096 for the script - it's my first attempt..

[leplan73]

Hi,

No need to do XML parsing by hand in LUA, domoticz already support XML parsing with the domoticz_applyXPath lua fonction.

See https://www.domoticz.com/wiki/Inserting ... ua_parsers

bye
Seb

[Toulon7559]

Using the script-concept by RATA1, I managed to get the status.xml from my PVLogger, comprising the components shown below in the actual (example) layout at this moment

<response>
<gauge_power>0</gauge_power>
<gauge_temp>0.0</gauge_temp>
<gauge_vpv>0.0</gauge_vpv>
<gauge_iac>0.0</gauge_iac>
<energy_today>0.000</energy_today>
<energy_total>5796.2</energy_total>
<hours_total>14866</hours_total>
<time_stamp>20160417 23:14</time_stamp>
</response>

All lines of the script by RATA1 can be 'recycled' till the line
HWTemp_Flow_Temperature=string.sub(XML_string,p+6,p+10) --temperature is the 5 characters after <temp> - i.e. 6-10

For my status.xml the extraction of the value per component is a bit more difficult than the fixed-format temperature layout xx.xx coming from the KMtronics-device. Simple & robust solution to be found.
Reason: the reported info per component is variable in magnitude and therefore with a variable number of characters before the decimal dot, in combination with per component a fixed, different amount of decimals after the decimal dot. See the red markings

[Toulon7559]

Looking at LuaXML: what is a simple & reliable method of installation for this add-on to a Raspberry?

[Westcott]

LuaXML seems to use an add-on C module.
For a pure Lua reader, please see my earlier post above.

[Toulon7559]

@Westcott

Seems to align with my experience related to LuaRocks as loader for LuaXML
Luarocks installs on Raspberry without problem, using command-line
sudo apt-get install luarocks
but the next step with command-line
sudo luarocks install luaxml
is without success.
And then questions pop up .....

[Toulon7559]

Using the script made by RATA1, in the meantime I 'rudely' further derived a similar lua-script to read & process the XML-outputfile from the FP4ALL-PVLogger which reads my STECA-inverters. Relative to the script by RATA1 the code below has been expanded to cover the values for power, temperature, voltage, current, energy and running-time. Also a very simple mechanism has been included to cope with the variable length of the values within the XML-string: the offsets required for parameters p1 till el2 have been calculated by hand looking at the layout of the XML-outputfile.
Reading the messages earlier in this thread, it must be possible to automate that calculation with a 'leaner & clever' script, but this basic setup also works .......

If you are only interested in Power & Energy to 'feed' a Power-meter in Domoticz, and not interested in Temperature, Voltage, Current and Energy_Today, you can delete the related segment between Line 89 and Line 115.
On the other hand, if you want to upload to Domoticz the values for Temperature, Voltage, Current and Energy_Today, you must additionally repeat&align the 'command-array-line' at script-line 129.
Obviously the IDX and IP-address are for my configuration: for your application you have to replace by info valid for your configuration.

Code: Select all

------------------------------------------------------------------------------------------
-- Version 2b 20160430
--
-- Domoticz lua script to convert XML output from FP4All_PVLogger
-- Reads the status based on the unique IP of the PVLogger and passes values
-- to virtual sensor(s) in Domoticz
-- Complete for Power & Energy, prepared for Temperature, Voltage and Current
--------------------------------------------------------------------------------------------------
-- Original script by RATA1 at Domoticz-forum, adapted by Toulon7559 (c)2016
--------------------------------------------------------------------------------------------------

commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file.
debug = true

-- Line 29: Define the idx of your virtual sensors - i.e. what you have called your sensors and their IDX numbers

PE_STECA = 218
-- VPV_STECA =
-- IAC_STECA =
-- Temp_STECA =
-- Hours_STECA = 

-- Line 37: Define your device IP-address

Device_IP = "192.168.0.110"

if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/status.xml'")
end

-- Line 45: Read the XML data from the device

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/status.xml'",1)

valid = string.find(XML_string, "<response>")    -- check we are looking in the right place

    if debug == true then
        print(XML_string)
    end
   
    if valid == nil then
        print ("Bad XML status read - info NOT updated")
    else
 
-- Line 59: Find in the XML_string the info-fields based on their labels
-- Determine start/stop-positions of the values within the substrings

    p = string.find(XML_string,"<gauge_power>")    -- read position of power string
    t = string.find(XML_string,"<gauge_temp>")     -- read position of temperature string
    v = string.find(XML_string,"<gauge_vpv>")      -- read voltage string
    i = string.find(XML_string,"<gauge_iac>")      -- read position of current string
    ed = string.find(XML_string,"<energy_today>")  -- read position of day-energy string
    el = string.find(XML_string,"<energy_total>")  -- read position of life-energy string
    h = string.find(XML_string,"<hours_total>")    -- read position of hours_total string
    p1 = p+13
    p2 = t-16
    t1 = t+12
    t2 = v-15
    v1 = v+11
    v2 = i-14
    i1 = i+11
    i2 = ed-14
    ed1 = ed+14
    ed2 = el-17
    el1 = el+14
    el2 = h-17

-- Line 82: Extract the values and process for upload to Domoticz

    p = string.find(XML_string,"<gauge_power>")  -- read power string
    print(p)
    power_actual = string.sub(XML_string,p1,p2)  
    print(power_actual)
    power_actual1 =tonumber(power_actual)
-- Line 89
    t = string.find(XML_string,"<gauge_temp>")  -- read temperature string
    print(t)
    temp = string.sub(XML_string,t1,t2)
    print(temp)
    temp1 =tonumber(temp)
    
    v = string.find(XML_string,"<gauge_vpv>")  -- read voltage string
    print(v)
    vpv= string.sub(XML_string,v1,v2)
    print(vpv)
    vpv1 =tonumber(vpv)

    i = string.find(XML_string,"<gauge_iac>")  -- read current string
    print(i)
    iac= string.sub(XML_string,i1,i2)
    print(iac)
    iac1 =tonumber(iac)

    ed = string.find(XML_string,"<energy_today>")  -- read day-energy string
    print(ed)
    energy_day= string.sub(XML_string,ed1,ed2)
    print(energy_day)
    energy_day1=tonumber(energy_day)
    energy_day2= energy_day1 * 1000 -- energy in Wh as required in Domoticz
    print(energy_day2)
-- Line 115
    el = string.find(XML_string,"<energy_total>")  -- read life-energy string
    print(el)
    energy_life= string.sub(XML_string,el1,el2)
    print(energy_life)
    energy_life1=tonumber(energy_life)
    energy_life2= energy_life1 * 1000 -- energy in Wh as required in Domoticz
    print(energy_life2)

    h = string.find(XML_string,"<hours_total>")    -- read position of hours_total string
    print(h)

-- Line 127: Upload to Domoticz; to be expanded for upload of other values to related IDXes
     
    commandArray[1] = {['UpdateDevice'] = PE_STECA.."|0|"..power_actual1..";"..energy_life2}  -- send updated values to Domoticz
       
        if debug == true then
            print("power/energy returned = ".."'"..power_actual1.."/"..energy_life2.."'")
        end   

    end 
 
return commandArray

This screenshot from Domoticz' log shows an example output of the print-commands

Example printout

screenshot-Domoticz_logSTECA_XML.png (6.11 KiB) Viewed 14603 times

[Westcott]

sudo luarocks install luaxml

This did work for me, but it took a day to get luarocks working properly.
These were some of the things I had to do, if I remember correctly (IIRC) -

sudo apt-get install lua5.2
sudo apt-get install lua-socket
sudo apt-get install luarocks
sudo luarocks install luasec
sudo luarocks install luasocket
sudo luarocks install luaxml

Documentation at -
http://viremo.eludi.net/LuaXML

[bradnywells]

Check this one to know more about....Parse XML

Wells

[Toulon7559]

Sometimes the answer to a question is just laying in front ....
In my previous message with script, I remarked that it would be nice to have an automatic method to determine the length of a value-string.
The Lua-Wiki contains interesting descriptions on this subject.
The method to determine the start postion of the value-string, is also applicable to find the end position of the value-string:
you just have to change the textstring to be checked (instead of the 'opening' label, take the 'closing' label after the value).
And then empirically check whether in the read-instruction an offset is required.

The adapted script below (= version 3) is using that method.

Code: Select all

------------------------------------------------------------------------------
-- Version 03 20170310
--
-- Domoticz lua script to convert XML output from FP4All_PVLogger
-- Reads the status based on the unique IP of the PVLogger and passes it
-- to virtual sensor(s) in Domoticz
--
------------------------------------------------------------------------------
-- Source-script from Domoticz-forum, adapted by Toulon7559 (c)2017
------------------------------------------------------------------------------

commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file.
debug = true

-- Line 29: Define the idx of your virtual sensors - i.e. what you have called your sensors and their IDX numbers

PE_STECA = 589
-- VPV_STECA =
-- IAC_STECA =
-- Temp_STECA =
-- Hours_STECA = 

-- Line 37: Define your device IP@

Device_IP = "192.168.0.110"

if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/status.xml'")
end

-- Line 45: Read the XML data from the device

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/status.xml'",1)

valid = string.find(XML_string, "<response>")    -- check we are looking in the right place

    if debug == true then
        print(XML_string)
    end
   
    if valid == nil then
        print ("Bad XML status read - info NOT updated")
    else
 
-- Line 59: Choose in the XML_string the info-fields (=substrings) based on their labels
-- Extract the start/end-positions of the values within the substrings
-- Extract the values from the substrings and (if required) scale for upload to Domoticz

-- Line 63: read power string
    p1 = string.find(XML_string,"<gauge_power>")    -- read start position of power substring
    p2 = string.find(XML_string,"</gauge_power>")  -- read end position of power substring
    print(p1, p2)
    power_actual = string.sub(XML_string,p1,p2-1)   -- read content from the substring
    print(power_actual)
    power_actual1 =tonumber(power_actual)            -- convert content to number

-- Line 71: read temperature string
    t1 = string.find(XML_string,"<gauge_temp>")
    t2 = string.find(XML_string,"</gauge_temp>")
    print(t1, t2)
    temp = string.sub(XML_string,t1,t2-1)
    print(temp)
    temp1 =tonumber(temp)
    
-- Line 79: read voltage string
    v1 = string.find(XML_string,"<gauge_vpv>")
    v2 = string.find(XML_string,"</gauge_vpv>")
    print(v1, v2)
    vpv= string.sub(XML_string,v1,v2-1)
    print(vpv)
    vpv1 =tonumber(vpv)

-- Line 87: read current string
    i1 = string.find(XML_string,"<gauge_iac>")
    i2 = string.find(XML_string,"</gauge_iac>")
    print(i1, i2)
    iac= string.sub(XML_string,i1,i2-1)
    print(iac)
    iac1 =tonumber(iac)

-- Line 95: read day-energy string
    ed1 = string.find(XML_string,"<energy_today>")
    ed2 = string.find(XML_string,"</energy_today>")
    print(ed1, ed2)
    energy_day= string.sub(XML_string,ed1,ed2-1)
    print(energy_day)
    energy_day1=tonumber(energy_day)
    energy_day2= energy_day1 * 1000 -- correction for energy in Wh as required in Domoticz
    print(energy_day2)

-- Line 105: read life-energy string
    el1 = string.find(XML_string,"<energy_total>")
    el2 = string.find(XML_string,"</energy_total>")
    print(el1, el2)
    energy_life= string.sub(XML_string,el1,el2-1)
    print(energy_life)
    energy_life1=tonumber(energy_life)
    energy_life2= energy_life1 * 1000 -- correction for energy in Wh as required in Domoticz
    print(energy_life2)

-- Line 115: read hours_total string
    h1 = string.find(XML_string,"<hours_total>")
    h2 = string.find(XML_string,"</hours_total>")
    print(h1, h2)

-- Line 120: Upload Power & Energy to Virtual Device at Domoticz
     
    commandArray[1] = {['UpdateDevice'] = PE_STECA.."|0|"..power_actual1..";"..energy_life2}  -- send updated values to Domoticz
       
        if debug == true then
            print("power/energy returned = ".."'"..power_actual1.."/"..energy_life2.."'")
        end   

    end 
 
return commandArray

Unfortunately, not 100% complete, because I now get an error report on an unchanged segment of the script.

Code: Select all

2017-03-12 17:01:01.237 Error: EventSystem: in /home/pi/domoticz/scripts/lua/script_time_XML_Domoticz_STECA_RPI3A.lua: ...icz/scripts/lua/script_time_XML_Domoticz_STECA_RPI3A.lua:102: attempt to perform arithmetic on global 'energy_day1' (a nil value)

Have been looking at that line 102, but don't see the difference with the previous version, nor why the error would be applicable, because the structure is same for all, and energy_day1 certainly is not a nil value.
Anybody an idea for cause & correction?

Pragmatic solution is not use the script in this message, but the script in a next message

[geertvercamer]

Looks like

Code: Select all

energy_day1=tonumber(energy_day)

doesn't store a number in 'energy_day1'. Print that variable to see its value

[DennisD]

Sometimes the answer to a question is just laying in front ....
In my previous message with script, I remarked that it would be nice to have an automatic method to determine the length of a value-string.
The Lua-Wiki contains interesting descriptions on this subject.
The method to determine the start postion of the value-string, is also applicable to find the end position of the value-string:
you just have to change the textstring to be checked (instead of the 'opening' label, take the 'closing' label after the value).
And then empirically check whether in the read-instruction an offset is required.

The adapted script below (= version 3) is using that method.

Code: Select all

------------------------------------------------------------------------------
-- Version 03 20170310
--
-- Domoticz lua script to convert XML output from FP4All_PVLogger
-- Reads the status based on the unique IP of the PVLogger and passes it
-- to virtual sensor(s) in Domoticz
--
------------------------------------------------------------------------------
-- Source-script from Domoticz-forum, adapted by Toulon7559 (c)2017
------------------------------------------------------------------------------

commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file.
debug = true

-- Line 29: Define the idx of your virtual sensors - i.e. what you have called your sensors and their IDX numbers

PE_STECA = 589
-- VPV_STECA =
-- IAC_STECA =
-- Temp_STECA =
-- Hours_STECA = 

-- Line 37: Define your device IP@

Device_IP = "192.168.0.110"

if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/status.xml'")
end

-- Line 45: Read the XML data from the device

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/status.xml'",1)

valid = string.find(XML_string, "<response>")    -- check we are looking in the right place

    if debug == true then
        print(XML_string)
    end
   
    if valid == nil then
        print ("Bad XML status read - info NOT updated")
    else
 
-- Line 59: Choose in the XML_string the info-fields (=substrings) based on their labels
-- Extract the start/end-positions of the values within the substrings
-- Extract the values from the substrings and (if required) scale for upload to Domoticz

-- Line 63: read power string
    p1 = string.find(XML_string,"<gauge_power>")    -- read start position of power substring
    p2 = string.find(XML_string,"</gauge_power>")  -- read end position of power substring
    print(p1, p2)
    power_actual = string.sub(XML_string,p1,p2-1)   -- read content from the substring
    print(power_actual)
    power_actual1 =tonumber(power_actual)            -- convert content to number

-- Line 71: read temperature string
    t1 = string.find(XML_string,"<gauge_temp>")
    t2 = string.find(XML_string,"</gauge_temp>")
    print(t1, t2)
    temp = string.sub(XML_string,t1,t2-1)
    print(temp)
    temp1 =tonumber(temp)
    
-- Line 79: read voltage string
    v1 = string.find(XML_string,"<gauge_vpv>")
    v2 = string.find(XML_string,"</gauge_vpv>")
    print(v1, v2)
    vpv= string.sub(XML_string,v1,v2-1)
    print(vpv)
    vpv1 =tonumber(vpv)

-- Line 87: read current string
    i1 = string.find(XML_string,"<gauge_iac>")
    i2 = string.find(XML_string,"</gauge_iac>")
    print(i1, i2)
    iac= string.sub(XML_string,i1,i2-1)
    print(iac)
    iac1 =tonumber(iac)

-- Line 95: read day-energy string
    ed1 = string.find(XML_string,"<energy_today>")
    ed2 = string.find(XML_string,"</energy_today>")
    print(ed1, ed2)
    energy_day= string.sub(XML_string,ed1,ed2-1)
    print(energy_day)
    energy_day1=tonumber(energy_day)
    energy_day2= energy_day1 * 1000 -- correction for energy in Wh as required in Domoticz
    print(energy_day2)

-- Line 105: read life-energy string
    el1 = string.find(XML_string,"<energy_total>")
    el2 = string.find(XML_string,"</energy_total>")
    print(el1, el2)
    energy_life= string.sub(XML_string,el1,el2-1)
    print(energy_life)
    energy_life1=tonumber(energy_life)
    energy_life2= energy_life1 * 1000 -- correction for energy in Wh as required in Domoticz
    print(energy_life2)

-- Line 115: read hours_total string
    h1 = string.find(XML_string,"<hours_total>")
    h2 = string.find(XML_string,"</hours_total>")
    print(h1, h2)

-- Line 120: Upload Power & Energy to Virtual Device at Domoticz
     
    commandArray[1] = {['UpdateDevice'] = PE_STECA.."|0|"..power_actual1..";"..energy_life2}  -- send updated values to Domoticz
       
        if debug == true then
            print("power/energy returned = ".."'"..power_actual1.."/"..energy_life2.."'")
        end   

    end 
 
return commandArray

Unfortunately, not 100% complete, because I now get an error report on an unchanged segment of the script.

Code: Select all

2017-03-12 17:01:01.237 Error: EventSystem: in /home/pi/domoticz/scripts/lua/script_time_XML_Domoticz_STECA_RPI3A.lua: ...icz/scripts/lua/script_time_XML_Domoticz_STECA_RPI3A.lua:102: attempt to perform arithmetic on global 'energy_day1' (a nil value)

Have been looking at that line 102, but don't see the difference with the previous version, nor why the error would be applicable, because the structure is same for all, and energy_day1 certainly is not a nil value.
Anybody an idea for cause & correction?

Did you found a fix for the error? Playing with it too!

[Toulon7559]

No fix found yet for the error in version 3.
Because older version 2b is OK for my application with the STECA-read-out, not further investigated.
But version 2b relies on manual counting of positions in the XML-file:
not nice & easy

Obviously quite frustrating that the same script flawlessly operates for XML-files coming from other FP4All-loggers:
must be something very subtle, like a hidden character sneaked in somewhere while copying & adapting.
Have retyped the 'offending' scriptline and the related ones preceeding it, but no progress ....

The script below is operational to get the information from a BMP180 which is part from a remote Raspberry-setup, from which is the below screenshot from the page with current states.

screenshot_currentstate_bmp180

screenshot-192.168.1.5 8080-2017-10-25-21-45-53.png (2.71 KiB) Viewed 10253 times

That remote Raspberry uploads the BMP180-information to my website as an XML-file.
This script on another Raspberry then reads that XML-file and provides the 'local' Domoticz with info like from a BMP180-sensor.

The printcommand at line 48 in the log shows what rubbish is coming in.
In this script between lines 55 and 69 you see how to find the start and stop positions for extraction of values from the substrings.
Unfortunately the headers of substrings are sometimes rather long.
Instead of fully automatic or 100% manual, this script is semi-automatic: it's search finds the start of the 'header' and the start of the 'tail' of the substring, and then you have to count the numbers of characters till (and including) the end of the 'header'.
As indicated, the command at line 81 results in a device-content as for a BMP180-sensor, but not complete:
the 4th value from the screenshot is missing. Can somebody explain that value?

Code: Select all

------------------------------------------------------------------------------------------
-- Version 00 20171025
--
-- Domoticz lua script to convert XML output for BMP180-sensor as uploaded to remote website
-- Reads the status based on the unique web-address and passes values
-- to virtual sensor(s) in Domoticz
-- Complete for Pressure & Temperature
--------------------------------------------------------------------------------------------------
-- Original script by RATA1 at Domoticz-forum, adapted by Toulon7559 (c)2016
--------------------------------------------------------------------------------------------------
-- Line 11: Start of script
commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file.
debug = true

-- Line 29: Define the idx of your virtual sensor(s) - i.e. what you have called your sensor(s) and the related IDX number(s)

PT_BMP180 = 1138 -- Virtual device for temp + baro

-- Line 33: Define your device IP-address

Device_IP = "vannwnhzn.nl"
  -- Device_IP = "192.168.1.x/home/pi" -- for local readout
if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/xml130_upload.xml'")
end

-- Line 41: Read the XML data from the device

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/xml130_upload.xml'",1)

valid = string.find(XML_string, "<root>")    -- check we are looking in the right place

    if debug == true then
        print("XML130-string = "..XML_string)
    end
   
    if valid == nil then
        print ("Bad XML status read - info NOT updated")
    else
 
-- Line 55: Find in the XML_string the info-fields based on their labels
-- Line 56: read position of pressure string
    p0 = string.find(XML_string,"<Barometer ")  -- start of 'header'
print(p0)
    p2 = string.find(XML_string,"</Barometer>") -- start of 'tail'
print(p2)
-- Line 59: read position of temperature string
    t0 = string.find(XML_string,"<Temp ")       -- start of 'header'
print(t0)
    t2 = string.find(XML_string,"</Temp>")      -- start of 'tail'
print(t2)

-- Line 65: Manually determine/set start/stop-positions of the values within the substrings
    p1 = p0+24 -- number = characters of header-string <Barometer ... >
print(p1)
    t1 = t0+19 -- number = characters of header-string <Temp ... >
print(t1)
-- Line 69: Extract the values and process for upload to Domoticz
-- Line 70: read pressure string
    pressure = string.sub(XML_string,p1,p2-1) 
    pressure1 = tonumber(pressure)
print(pressure1)
-- Line 74: read temperature string
    temp = string.sub(XML_string,t1,t2-1) 
    temp1 = tonumber(temp)
print(temp1)

-- Line 79: Upload to Domoticz
     
    commandArray[1] = {['UpdateDevice'] = PT_BMP180.."|0|"..temp1..";"..pressure1..";1"}  -- send updated values to Domoticz
--  for the virtual device set in line 31 the above { layout } as result mimicks a BMP180, except for the 4th value
       
        if debug == true then
            print("pressure/temperature = ".."'"..pressure1.."/"..temp1.."'")
        end   

    end 
 
return commandArray

Because this script makes a call to a public website, probably a testrun with the unchanged script will have no errors till line 80:
certainly line 81 will cry for help, because your virtual device will have another IDX than listed in line 31.

[xtenz]

Sometimes the answer to a question is just laying in front ....
In my previous message with script, I remarked that it would be nice to have an automatic method to determine the length of a value-string.
The Lua-Wiki contains interesting descriptions on this subject.
The method to determine the start postion of the value-string, is also applicable to find the end position of the value-string:
you just have to change the textstring to be checked (instead of the 'opening' label, take the 'closing' label after the value).
And then empirically check whether in the read-instruction an offset is required.

The adapted script below (= version 3) is using that method.

Code: Select all

------------------------------------------------------------------------------
-- Version 03 20170310
--
-- Domoticz lua script to convert XML output from FP4All_PVLogger
-- Reads the status based on the unique IP of the PVLogger and passes it
-- to virtual sensor(s) in Domoticz
--
------------------------------------------------------------------------------
-- Source-script from Domoticz-forum, adapted by Toulon7559 (c)2017
------------------------------------------------------------------------------

commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file.
debug = true

-- Line 29: Define the idx of your virtual sensors - i.e. what you have called your sensors and their IDX numbers

PE_STECA = 589
-- VPV_STECA =
-- IAC_STECA =
-- Temp_STECA =
-- Hours_STECA = 

-- Line 37: Define your device IP@

Device_IP = "192.168.0.110"

if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/status.xml'")
end

-- Line 45: Read the XML data from the device

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/status.xml'",1)

valid = string.find(XML_string, "<response>")    -- check we are looking in the right place

    if debug == true then
        print(XML_string)
    end
   
    if valid == nil then
        print ("Bad XML status read - info NOT updated")
    else
 
-- Line 59: Choose in the XML_string the info-fields (=substrings) based on their labels
-- Extract the start/end-positions of the values within the substrings
-- Extract the values from the substrings and (if required) scale for upload to Domoticz

-- Line 63: read power string
    p1 = string.find(XML_string,"<gauge_power>")    -- read start position of power substring
    p2 = string.find(XML_string,"</gauge_power>")  -- read end position of power substring
    print(p1, p2)
    power_actual = string.sub(XML_string,p1,p2-1)   -- read content from the substring
    print(power_actual)
    power_actual1 =tonumber(power_actual)            -- convert content to number

-- Line 71: read temperature string
    t1 = string.find(XML_string,"<gauge_temp>")
    t2 = string.find(XML_string,"</gauge_temp>")
    print(t1, t2)
    temp = string.sub(XML_string,t1,t2-1)
    print(temp)
    temp1 =tonumber(temp)
    
-- Line 79: read voltage string
    v1 = string.find(XML_string,"<gauge_vpv>")
    v2 = string.find(XML_string,"</gauge_vpv>")
    print(v1, v2)
    vpv= string.sub(XML_string,v1,v2-1)
    print(vpv)
    vpv1 =tonumber(vpv)

-- Line 87: read current string
    i1 = string.find(XML_string,"<gauge_iac>")
    i2 = string.find(XML_string,"</gauge_iac>")
    print(i1, i2)
    iac= string.sub(XML_string,i1,i2-1)
    print(iac)
    iac1 =tonumber(iac)

-- Line 95: read day-energy string
    ed1 = string.find(XML_string,"<energy_today>")
    ed2 = string.find(XML_string,"</energy_today>")
    print(ed1, ed2)
    energy_day= string.sub(XML_string,ed1,ed2-1)
    print(energy_day)
    energy_day1=tonumber(energy_day)
    energy_day2= energy_day1 * 1000 -- correction for energy in Wh as required in Domoticz
    print(energy_day2)

-- Line 105: read life-energy string
    el1 = string.find(XML_string,"<energy_total>")
    el2 = string.find(XML_string,"</energy_total>")
    print(el1, el2)
    energy_life= string.sub(XML_string,el1,el2-1)
    print(energy_life)
    energy_life1=tonumber(energy_life)
    energy_life2= energy_life1 * 1000 -- correction for energy in Wh as required in Domoticz
    print(energy_life2)

-- Line 115: read hours_total string
    h1 = string.find(XML_string,"<hours_total>")
    h2 = string.find(XML_string,"</hours_total>")
    print(h1, h2)

-- Line 120: Upload Power & Energy to Virtual Device at Domoticz
     
    commandArray[1] = {['UpdateDevice'] = PE_STECA.."|0|"..power_actual1..";"..energy_life2}  -- send updated values to Domoticz
       
        if debug == true then
            print("power/energy returned = ".."'"..power_actual1.."/"..energy_life2.."'")
        end   

    end 
 
return commandArray

Unfortunately, not 100% complete, because I now get an error report on an unchanged segment of the script.

Code: Select all

2017-03-12 17:01:01.237 Error: EventSystem: in /home/pi/domoticz/scripts/lua/script_time_XML_Domoticz_STECA_RPI3A.lua: ...icz/scripts/lua/script_time_XML_Domoticz_STECA_RPI3A.lua:102: attempt to perform arithmetic on global 'energy_day1' (a nil value)

Have been looking at that line 102, but don't see the difference with the previous version, nor why the error would be applicable, because the structure is same for all, and energy_day1 certainly is not a nil value.
Anybody an idea for cause & correction?

Pragmatic solution is not use the script in this message, but the script in a next message

It looks like that the start position of the values is not determined correctly.
E.g.

Code: Select all

p1 = string.find(XML_string,"<gauge_power>")

should be:

Code: Select all

p1 = string.find(XML_string,"<gauge_power>")+13

I have fixed all start positions and the script is now running without errors.

Code: Select all

------------------------------------------------------------------------------
-- Version 03 20170310
--
-- Domoticz lua script to convert XML output from FP4All_PVLogger
-- Reads the status based on the unique IP of the PVLogger and passes it
-- to virtual sensor(s) in Domoticz
--
------------------------------------------------------------------------------
-- Source-script from Domoticz-forum, adapted by Toulon7559 (c)2017
------------------------------------------------------------------------------

commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file.
debug = true

-- Line 29: Define the idx of your virtual sensors - i.e. what you have called your sensors and their IDX numbers

PE_STECA = 338
-- VPV_STECA =
-- IAC_STECA =
-- Temp_STECA =
-- Hours_STECA =

-- Line 37: Define your device IP@

Device_IP = "192.168.1.7"

if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/status.xml'")
end

-- Line 45: Read the XML data from the device

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/status.xml'",1)

valid = string.find(XML_string, "<response>")    -- check we are looking in the right place

    if debug == true then
        print(XML_string)
    end

    if valid == nil then
        print ("Bad XML status read - info NOT updated")
    else

-- Line 59: Choose in the XML_string the info-fields (=substrings) based on their labels
-- Extract the start/end-positions of the values within the substrings
-- Extract the values from the substrings and (if required) scale for upload to Domoticz

-- Line 63: read power string
    p1 = string.find(XML_string,"<gauge_power>")+13    -- read start position of power substring
    p2 = string.find(XML_string,"</gauge_power>")  -- read end position of power substring
    -- print(p1, p2)
    power_actual = string.sub(XML_string,p1,p2-1)   -- read content from the substring
    print("power_actual=" .. power_actual)
    power_actual1 =tonumber(power_actual)            -- convert content to number

-- Line 71: read temperature string
    t1 = string.find(XML_string,"<gauge_temp>")+12
    t2 = string.find(XML_string,"</gauge_temp>")
    -- print(t1, t2)
    temp = string.sub(XML_string,t1,t2-1)
    print("temp=" .. temp)
    temp1 =tonumber(temp)

-- Line 79: read voltage string
    v1 = string.find(XML_string,"<gauge_vpv>")+11
    v2 = string.find(XML_string,"</gauge_vpv>")
    -- print(v1, v2)
    vpv= string.sub(XML_string,v1,v2-1)
    print("vpv=" .. vpv)
    vpv1 =tonumber(vpv)

-- Line 87: read current string
    i1 = string.find(XML_string,"<gauge_iac>")+11
    i2 = string.find(XML_string,"</gauge_iac>")
    -- print(i1, i2)
    iac= string.sub(XML_string,i1,i2-1)
    print("iac=" .. iac)
    iac1 =tonumber(iac)

-- Line 95: read day-energy string
    ed1 = string.find(XML_string,"<energy_today>")+14
    ed2 = string.find(XML_string,"</energy_today>")
    -- print(ed1, ed2)
    energy_day= string.sub(XML_string,ed1,ed2-1)
    print("energy_day=" .. energy_day)
    energy_day1=tonumber(energy_day)
    energy_day2= energy_day1 * 1000 -- correction for energy in Wh as required in Domoticz
    print("energy_day2=" .. energy_day2)

-- Line 105: read life-energy string
    el1 = string.find(XML_string,"<energy_total>")+14
    el2 = string.find(XML_string,"</energy_total>")
    -- print(el1, el2)
    energy_life= string.sub(XML_string,el1,el2-1)
    print("energy_life=" .. energy_life)
    energy_life1=tonumber(energy_life)
    energy_life2= energy_life1 * 1000 -- correction for energy in Wh as required in Domoticz
    print("energy_life2=" .. energy_life2)

-- Line 115: read hours_total string
    h1 = string.find(XML_string,"<hours_total>")+13
    h2 = string.find(XML_string,"</hours_total>")
    hours_total= string.sub(XML_string,h1,h2-1)
    -- print(h1, h2)
    print("hours_total=" .. hours_total)

-- Line 120: Upload Power & Energy to Virtual Device at Domoticz

    commandArray[1] = {['UpdateDevice'] = PE_STECA.."|0|"..power_actual1..";"..energy_life2}  -- send updated values to Domoticz

        if debug == true then
            print("power/energy returned = ".."'"..power_actual1.."/"..energy_life2.."'")
        end

    end

return commandArray

[Toulon7559]

A script that runs without errors is always nice!
But your addition of the variable numbers +11 ~ +14 at the end of the lines, is exactly the aspect I tried to avoid, because it still means manual counting of positions. I tried to achieve 'full automation', which seemed successful in 2 out of 3 cases.
One more step to go ......

Addition February 02, 2019
With this script-version no completely automatic solution found, which always correctly, automatically takes care of the number of characters in the header-textstring.
;-( Still wondering why it worked without problems in the 2 other cases:
must be something very subtle ..........

[xtenz]

Full automation is indeed better.
Second attempt :

Code: Select all

------------------------------------------------------------------------------
-- Version 04 20171125
--
-- Domoticz lua script to convert XML output from FP4All_PVLogger
-- Reads the status based on the unique IP of the PVLogger and passes it
-- to virtual sensor(s) in Domoticz
--
------------------------------------------------------------------------------
-- Source-script from Domoticz-forum, adapted by Xtenz (c)2017
------------------------------------------------------------------------------

commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

function GetValueFromXML(XMLvalue)
   -- Extract the start/end-positions of the values within the substrings
   local XMLvalueLength=string.len(XMLvalue)+2
   local XMLvalueStartPos=string.find(XML_string,"<" .. XMLvalue .. ">")+XMLvalueLength
   local XMLvalueEndPos=string.find(XML_string,"</" .. XMLvalue .. ">")
   -- Get the value substring and convert it to a number
   local ReturnValue=tonumber(string.sub(XML_string,XMLvalueStartPos,XMLvalueEndPos-1))
   print(XMLvalue .. "=" .. ReturnValue)
   return ReturnValue
end

-- Line 37: Set debug to true to print to log file.
debug = true

-- Line 40: Define the idx of your virtual sensors - i.e. what you have called your sensors and their IDX numbers

PE_IDX = 338
-- VPV_IDX =
-- IAC_IDX =
-- Temp_IDX =
-- Hours_IDX =

-- Line 48: Define your FP4all logger device IP@

Device_IP = "192.168.1.7"

if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/status.xml'")
end

-- Line 56: Read the XML data from the device

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/status.xml'",1)

valid = string.find(XML_string, "<response>")    -- check we are looking in the right place

    if debug == true then
        print(XML_string)
    end

    if valid == nil then
        print ("Bad XML status read - info NOT updated")
    else

-- Line 70: Choose in the XML_string the info-fields (substrings) based on their labels
-- Extract the values from the substrings and (if required) scale for upload to Domoticz

-- Line 73: read power string
    power_actual = GetValueFromXML("gauge_power")

-- Line 76: read temperature string
    temp = GetValueFromXML("gauge_temp")

-- Line 79: read voltage string
    vpv =  GetValueFromXML("gauge_vpv")

-- Line 82: read current string
    iac =  GetValueFromXML("gauge_iac")

-- Line 85: read day-energy string
    energy_day= GetValueFromXML("energy_today") * 1000 -- correction for energy in Wh as required in Domoticz
    print("energy_day=" .. energy_day)

-- Line 89: read life-energy string
    energy_life= GetValueFromXML("energy_total") * 1000 -- correction for energy in Wh as required in Domoticz
    print("energy_life=" .. energy_life)

-- Line 93: read hours_total string
    hours_total = GetValueFromXML("hours_total")

-- Line 96: Upload Power & Energy to Virtual Device at Domoticz

    commandArray[1] = {['UpdateDevice'] = PE_IDX.."|0|"..power_actual..";"..energy_life}  -- send updated values to Domoticz

        if debug == true then
            print("power/energy returned = ".."'"..power_actual.."/"..energy_life.."'")
        end

    end

return commandArray