Domoticz

Open source Home Automation System
https://forum.domoticz.com/

Script LUA to read value on a webpage ?

https://forum.domoticz.com/viewtopic.php?t=25321

Page 1 of 2

Script LUA to read value on a webpage ?

Posted: Thursday 18 October 2018 15:29
by cycloneseb
Hello all Domoticz's Experts !

I'm back to the Domoticz Forum to explain my dream :lol:

As I'm living near a lake in Italy, I was a few years ago almost to be flooded ...

As the value of the lake level can be monitored through internet, I'm wondering if it's possible to extract the data from the webpage using by example an LUA script, and then insert it into a dummy device in Domoticz ?

Webpage : http://www.girovaghi.it/METEO/livelli_i ... ticino.php
Data te be removed : after "quota invaso", by example today : 192,81

Is someone already did it, would it be possible to have some tips to extract this data ? or better, a tutorial !

many thanks !

Regards

Re: Script LUA to read value on a webpage ?

Posted: Thursday 18 October 2018 18:03
by jvdz
It is actually pretty strait forward to do this.
This script requires LUA socket.http to be installed:

Code: Select all

DEVIDX=123
weburl='http://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'
domurl='http://192.168.0.xx:8080/json.htm?type=command&param=udevice&idx=' .. DEVIDX .. '&nvalue=0&svalue='
-- retrieve web info
http = require "socket.http";
-- Get info from website
response, status = http.request(weburl)
-- Get level from info
response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')
print(response)
-- update text device IDX
response, status = http.request(domurl..response)
print(status,response)
I leave it to you to either schedule it outside of domoticz ( which I would do) or incorporate it in an event script and only do the update x times per day and add the required error checking.
Let me know when you have questions.

Jos

Re: Script LUA to read value on a webpage ?

Posted: Saturday 27 October 2018 18:39
by cycloneseb
Hello Jos;

many thanks for your answer !

I'm back home and trying your code, but up to now, I got some errors due to the wrong location of the LUA socket.

Thanks again !

Seb.

Re: Script LUA to read value on a webpage ?

Posted: Saturday 27 October 2018 18:51
by jvdz
Look at DTGBOT installation in the wiki, which has a download for it and tells you where it needs to be located. :)

Edit: this is the forum post with links: http://domoticz.com/forum/viewtopic.php ... lit=Dtgbot
Wiki link: https://www.domoticz.com/wiki/Remote_Co ... legram_Bot
Jos

Re: Script LUA to read value on a webpage ?

Posted: Sunday 28 October 2018 18:39
by cycloneseb
Hi Jos;

many thanks for your help, I succeeded to install the sockett and now receive the right information.

but, if I can again abuse of your knowledge....

When I'm trying to send the information to update the device using the LUA script, (http.request ('http://192.168.....), no update is done on the device, but surprisingly, inserting the direct adress into my webbrowser, the update is done without any problem !!

do you've an idea of the trouble ? and how to solve it ?

Many thanks again

Seb

Re: Script LUA to read value on a webpage ?

Posted: Sunday 28 October 2018 18:47
by jvdz
Abuse huh? :)

You really will have to provide the script with the code you tried to be able to see what could be wrong.
Also tell me what type of dummy device you try to update? (text?)

Jos

Re: Script LUA to read value on a webpage ?

Posted: Sunday 28 October 2018 19:02
by cycloneseb
not abuse, let say .. benefit too much from your time :lol:

please find the code from the LUA script

Code: Select all

DEVIDX=7180
weburl='http://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'

domurl='http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=' .. DEVIDX .. '&nvalue=0&svalue='

-- retrieve web info

local http = require "socket.http";

-- Get info from website

commandArray={}

domurl='http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue='

response, status = http.request(weburl)

-- Get level from info

response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')
print(response)

-- update text device IDX

-- Trial to insert a predefined value(500) :

response, status = http.request('http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue=500')

print(status,response)


return commandArray

from the log, I have this comment :
LUA: <html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html
but, when I' typing directly into my web browser "http://192.168.1.102:8080/json.htm?type ... svalue=500", the right value (500) is displayed on the device !

The device to be updated is a custom sensor.

Many thanks for your help

Seb

Re: Script LUA to read value on a webpage ?

Posted: Sunday 28 October 2018 19:51
by jvdz
Are you using Userid/Password security for Domoticz?
When the script is running on the same server you could make the url:

Code: Select all

response, status = http.request('http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue=500')
And make sure the 127.0.0.1 is in the Setup/Settings page defined in
Local Networks (no username/password): Networks:

Jos

Re: Script LUA to read value on a webpage ?

Posted: Tuesday 30 October 2018 22:53
by cycloneseb
Hi Jos;

I just come back to you to thank you for your precious help, now the lua script is working fine .. and I can manage the level of the lake every 15' (still increasing those days).

Many many thanks for your time and for sharing your knowledge !

for interested people, I join the code :

Code: Select all

-- Website
weburl='http://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'

-- domurl='http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=' .. DEVIDX .. '&nvalue=0&svalue='

-- LUA socket http
local http = require "socket.http"

-- Current time
t1=os.time()

-- Last upadte of the domoticz display
s = otherdevices_lastupdate['Niveau Lac Majeur']
year = string.sub(s, 1, 4)
month = string.sub(s, 6, 7)
day = string.sub(s, 9, 10)
hour = string.sub(s, 12, 13)
minutes = string.sub(s, 15, 16)
seconds = string.sub(s, 18, 19)


commandArray={}

print('Script LUA - Lake level')

-- 1° time difference

t2 = os.time{year=year, month=month, day=day, hour=hour, min=minutes, sec=seconds}
difference = (os.difftime (t1, t2))

if (difference > 900 and difference < 3600) then

        -- 2° website information
        response, status = http.request(weburl)

        response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')

        -- send info to the virtual device
        commandArray['UpdateDevice']='7180|0|'..response
end

return commandArray

Re: Script LUA to read value on a webpage ?

Posted: Wednesday 11 September 2019 14:06
by lewy0111
Hi,
there is any similar method for https pages? I tried to use local https = require "ssl.https", but it ends with error due to tlsv1 is not supported by this webpage. Thank you all for answer
regards
Pawel

Re: Script LUA to read value on a webpage ?

Posted: Wednesday 11 September 2019 16:12
by elmortero
lewy0111 wrote: ↑Wednesday 11 September 2019 14:06 Hi,
there is any similar method for https pages? I tried to use local https = require "ssl.https", but it ends with error due to tlsv1 is not supported by this webpage. Thank you all for answer
regards
Pawel
What page are you trying to get data from? Maybe they have an API available, which would make it a lot easier

Re: Script LUA to read value on a webpage ?

Posted: Thursday 12 September 2019 7:42
by lewy0111
There was API, but was restricted due to goverment train company in Poland has very strange politic... btw i would to get push notification and notification on my domoticz display about delay of my morning train. All of trains have gps and this data is avaiable and fresh, but it's reachable only by phone app or website, on website it's so hard to parse, but possible. If it's to hard to do using lua scripts I'll do it in python (which will update my custom 'train delay' sensor) and it will be runned by crontab, but embadding it in domoticz seems be more elegant

Re: Script LUA to read value on a webpage ?

Posted: Friday 01 November 2019 23:57
by cycloneseb
Hello all;

I'm back with my request to parse a value from a webpage, and more precisely the level of the lake near my house.

The LUA script developed by Jos was working perfectly since a few months. Now, the script cannot recover the data, and in the domoticz log I've this error : "attempt to concatenate global 'response' (a nil value)".

I'm wondering if this problem can come from the fact that the webpage is now https ????

If someone from the domoticz's expert team can give me a few advise to solve this problem, and recover this value, it'll be very helpful !

many thanks

Cycloneseb

Re: Script LUA to read value on a webpage ?

Posted: Saturday 02 November 2019 1:05
by jvdz
Have you seen my post with a specific question about your reported issue?

Jos

Re: Script LUA to read value on a webpage ?

Posted: Saturday 02 November 2019 12:32
by cycloneseb
Ciao Jos;

nice to read again !

No,sorry, I haven't seen it. When it was published ?

Seb.

Re: Script LUA to read value on a webpage ?

Posted: Saturday 02 November 2019 13:54
by jvdz
This one as you mention you changed to https. ;)
Else I would need more info which line causes the concatenate error and you exact script that is used, as something is returning a nil value.

Jos

jvdz wrote: ↑Sunday 28 October 2018 19:51 Are you using Userid/Password security for Domoticz?
When the script is running on the same server you could make the url:

Code: Select all

response, status = http.request('http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue=500')
And make sure the 127.0.0.1 is in the Setup/Settings page defined in
Local Networks (no username/password): Networks:

Jos

Re: Script LUA to read value on a webpage ?

Posted: Sunday 03 November 2019 21:20
by cycloneseb
Hi;

the code I used to perform test is the following ;

Code: Select all

-- Website
weburl='https://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'

-- retrieve web info
local http = require ("socket.http")

-- FONCTION

commandArray={}

-- 2° website information
        response, status = http.request(weburl)

print ("page web :"..status)

response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')
print("LAKE LEVEL : "..response)

return commandArray
According to the log of Domoticz :" Error: EventSystem: in /home/pi/domoticz/scripts/lua/script_time_lac2.lua: /home/pi/domoticz/scripts/lua/script_time_lac2.lua:17: attempt to concatenate global 'response' (a nil value)", so on this line "print("LAKE LEVEL : "..response)", this is strange for me...

Re: Script LUA to read value on a webpage ?

Posted: Saturday 10 October 2020 16:58
by tinus
Hello i'am new to lua scripting and try to get a value from a website as well.
I found the value i'am loking for but can't get it as a usable value.

the value i'am trying to get is the "36.8" at the end of the following code:

<div id="ui-id-3" data-filteredid="ui-id-3" data-filtervalueid="907442" data-group="00_0010_2251_0100_006_2_0-ui-id-3-value" data-toggle="context" data-target="none" data-angle="0" data-type="value" data-fieldid="00_0010_2251_0100_006_2_0" data-symbol="false" data-src="false" class="font" style="position: absolute; top: 385.736px; left: 475.137px; z-index: 1; transform: rotate(0deg); font-size: 10.0191px; font-weight: bold; font-style: normal; font-family: sans-serif; color: rgb(51, 51, 51); text-decoration: none;" data-newfilter="yes">36.8&nbsp;°C</div>

Thank for your help.

Re: Script LUA to read value on a webpage ?

Posted: Saturday 10 October 2020 17:15
by jvdz
One way it could be done:

Code: Select all

Web_Data='<div id="ui-id-3" data-filteredid="ui-id-3" data-filtervalueid="907442" data-group="00_0010_2251_0100_006_2_0-ui-id-3-value" data-toggle="context" data-target="none" data-angle="0" data-type="value" data-fieldid="00_0010_2251_0100_006_2_0" data-symbol="false" data-src="false" class="font" style="position: absolute; top: 385.736px; left: 475.137px; z-index: 1; transform: rotate(0deg); font-size: 10.0191px; font-weight: bold; font-style: normal; font-family: sans-serif; color: rgb(51, 51, 51); text-decoration: none;" data-newfilter="yes">36.8&nbsp;°C</div>'
tmp = Web_Data:match('data.newfilter="yes">(.-)°C</div>')
tmp = tmp:gsub("&nbsp;","")
print("tmp:"..tmp)
Jos

Re: Script LUA to read value on a webpage ?

Posted: Saturday 10 October 2020 17:29
by waaren
tinus wrote: ↑Saturday 10 October 2020 16:58 the value i'am trying to get is the "36.8" at the end of the following code:
[Edit] Sorry, did not see the solution from Jos.. But what about when newfilter="No" ?

This should do it.

Code: Select all

local a = '<div id="ui-id-3" data-filteredid="ui-id-3" data-filtervalueid="907442" data-group="00_0010_2251_0100_006_2_0-ui-id-3-value" data-toggle="context" data-target="none" data-angle="0" data-type="value" data-fieldid="00_0010_2251_0100_006_2_0" data-symbol="false" data-src="false" class="font" style="position: absolute; top: 385.736px; left: 475.137px; z-index: 1; transform: rotate(0deg); font-size: 10.0191px; font-weight: bold; font-style: normal; font-family: sans-serif; color: rgb(51, 51, 51); text-decoration: none;" data-newfilter="yes">36.8&nbsp;°C</div>'

local b =  a:match("%b><") -- get everything between > and <
print('Chopped a to: ' .. b)
local c = b:match("%d+%.*%d*") -- get the first float (or integer) of the remaining string 
print('Found ' .. c .. ' as first float or integer in the chopped string')
All times are UTC+02:00
Page 1 of 2

Powered by phpBB® Forum Software © phpBB Limited