Script LUA to read value on a webpage ?

cycloneseb · Post by **cycloneseb** » Thursday 18 October 2018 15:29

Hello all Domoticz's Experts !

I'm back to the Domoticz Forum to explain my dream

As I'm living near a lake in Italy, I was a few years ago almost to be flooded ...

As the value of the lake level can be monitored through internet, I'm wondering if it's possible to extract the data from the webpage using by example an LUA script, and then insert it into a dummy device in Domoticz ?

Webpage : http://www.girovaghi.it/METEO/livelli_i ... ticino.php
Data te be removed : after "quota invaso", by example today : 192,81

Is someone already did it, would it be possible to have some tips to extract this data ? or better, a tutorial !

many thanks !

Regards

jvdz · Post by **jvdz** » Thursday 18 October 2018 18:03

It is actually pretty strait forward to do this.
This script requires LUA socket.http to be installed:

Code: Select all

DEVIDX=123
weburl='http://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'
domurl='http://192.168.0.xx:8080/json.htm?type=command&param=udevice&idx=' .. DEVIDX .. '&nvalue=0&svalue='
-- retrieve web info
http = require "socket.http";
-- Get info from website
response, status = http.request(weburl)
-- Get level from info
response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')
print(response)
-- update text device IDX
response, status = http.request(domurl..response)
print(status,response)

I leave it to you to either schedule it outside of domoticz ( which I would do) or incorporate it in an event script and only do the update x times per day and add the required error checking.
Let me know when you have questions.

Jos

cycloneseb · Post by **cycloneseb** » Saturday 27 October 2018 18:39

Hello Jos;

many thanks for your answer !

I'm back home and trying your code, but up to now, I got some errors due to the wrong location of the LUA socket.

Thanks again !

Seb.

jvdz · Post by **jvdz** » Saturday 27 October 2018 18:51

Look at DTGBOT installation in the wiki, which has a download for it and tells you where it needs to be located.

Edit: this is the forum post with links: http://domoticz.com/forum/viewtopic.php ... lit=Dtgbot
Wiki link: https://www.domoticz.com/wiki/Remote_Co ... legram_Bot
Jos

cycloneseb · Post by **cycloneseb** » Sunday 28 October 2018 18:39

Hi Jos;

many thanks for your help, I succeeded to install the sockett and now receive the right information.

but, if I can again abuse of your knowledge....

When I'm trying to send the information to update the device using the LUA script, (http.request ('http://192.168.....), no update is done on the device, but surprisingly, inserting the direct adress into my webbrowser, the update is done without any problem !!

do you've an idea of the trouble ? and how to solve it ?

Many thanks again

Seb

jvdz · Post by **jvdz** » Sunday 28 October 2018 18:47

Abuse huh?

You really will have to provide the script with the code you tried to be able to see what could be wrong.
Also tell me what type of dummy device you try to update? (text?)

Jos

cycloneseb · Post by **cycloneseb** » Sunday 28 October 2018 19:02

not abuse, let say .. benefit too much from your time

please find the code from the LUA script

Code: Select all

DEVIDX=7180
weburl='http://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'

domurl='http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=' .. DEVIDX .. '&nvalue=0&svalue='

-- retrieve web info

local http = require "socket.http";

-- Get info from website

commandArray={}

domurl='http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue='

response, status = http.request(weburl)

-- Get level from info

response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')
print(response)

-- update text device IDX

-- Trial to insert a predefined value(500) :

response, status = http.request('http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue=500')

print(status,response)


return commandArray

from the log, I have this comment :

LUA: <html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html

but, when I' typing directly into my web browser "http://192.168.1.102:8080/json.htm?type ... svalue=500", the right value (500) is displayed on the device !

The device to be updated is a custom sensor.

Many thanks for your help

Seb

jvdz · Post by **jvdz** » Sunday 28 October 2018 19:51

Are you using Userid/Password security for Domoticz?
When the script is running on the same server you could make the url:

Code: Select all

response, status = http.request('http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue=500')

And make sure the 127.0.0.1 is in the Setup/Settings page defined in
Local Networks (no username/password): Networks:

Jos

cycloneseb · Post by **cycloneseb** » Tuesday 30 October 2018 22:53

Hi Jos;

I just come back to you to thank you for your precious help, now the lua script is working fine .. and I can manage the level of the lake every 15' (still increasing those days).

Many many thanks for your time and for sharing your knowledge !

for interested people, I join the code :

Code: Select all

-- Website
weburl='http://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'

-- domurl='http://192.168.1.102:8080/json.htm?type=command&param=udevice&idx=' .. DEVIDX .. '&nvalue=0&svalue='

-- LUA socket http
local http = require "socket.http"

-- Current time
t1=os.time()

-- Last upadte of the domoticz display
s = otherdevices_lastupdate['Niveau Lac Majeur']
year = string.sub(s, 1, 4)
month = string.sub(s, 6, 7)
day = string.sub(s, 9, 10)
hour = string.sub(s, 12, 13)
minutes = string.sub(s, 15, 16)
seconds = string.sub(s, 18, 19)


commandArray={}

print('Script LUA - Lake level')

-- 1° time difference

t2 = os.time{year=year, month=month, day=day, hour=hour, min=minutes, sec=seconds}
difference = (os.difftime (t1, t2))

if (difference > 900 and difference < 3600) then

        -- 2° website information
        response, status = http.request(weburl)

        response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')

        -- send info to the virtual device
        commandArray['UpdateDevice']='7180|0|'..response
end

return commandArray

lewy0111 · Post by **lewy0111** » Wednesday 11 September 2019 14:06

Hi,
there is any similar method for https pages? I tried to use local https = require "ssl.https", but it ends with error due to tlsv1 is not supported by this webpage. Thank you all for answer
regards
Pawel

elmortero · Post by **elmortero** » Wednesday 11 September 2019 16:12

lewy0111 wrote: ↑Wednesday 11 September 2019 14:06 Hi,
there is any similar method for https pages? I tried to use local https = require "ssl.https", but it ends with error due to tlsv1 is not supported by this webpage. Thank you all for answer
regards
Pawel

What page are you trying to get data from? Maybe they have an API available, which would make it a lot easier

lewy0111 · Post by **lewy0111** » Thursday 12 September 2019 7:42

There was API, but was restricted due to goverment train company in Poland has very strange politic... btw i would to get push notification and notification on my domoticz display about delay of my morning train. All of trains have gps and this data is avaiable and fresh, but it's reachable only by phone app or website, on website it's so hard to parse, but possible. If it's to hard to do using lua scripts I'll do it in python (which will update my custom 'train delay' sensor) and it will be runned by crontab, but embadding it in domoticz seems be more elegant

cycloneseb · Post by **cycloneseb** » Friday 01 November 2019 23:57

Hello all;

I'm back with my request to parse a value from a webpage, and more precisely the level of the lake near my house.

The LUA script developed by Jos was working perfectly since a few months. Now, the script cannot recover the data, and in the domoticz log I've this error : "attempt to concatenate global 'response' (a nil value)".

I'm wondering if this problem can come from the fact that the webpage is now https ????

If someone from the domoticz's expert team can give me a few advise to solve this problem, and recover this value, it'll be very helpful !

many thanks

Cycloneseb

jvdz · Post by **jvdz** » Saturday 02 November 2019 1:05

Have you seen my post with a specific question about your reported issue?

Jos

cycloneseb · Post by **cycloneseb** » Saturday 02 November 2019 12:32

Ciao Jos;

nice to read again !

No,sorry, I haven't seen it. When it was published ?

Seb.

jvdz · Post by **jvdz** » Saturday 02 November 2019 13:54

This one as you mention you changed to https.

Else I would need more info which line causes the concatenate error and you exact script that is used, as something is returning a nil value.

Jos

jvdz wrote: ↑Sunday 28 October 2018 19:51 Are you using Userid/Password security for Domoticz?
When the script is running on the same server you could make the url:
Code: Select all
response, status = http.request('http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=7180&nvalue=0&svalue=500')
And make sure the 127.0.0.1 is in the Setup/Settings page defined in
Local Networks (no username/password): Networks:

Jos

cycloneseb · Post by **cycloneseb** » Sunday 03 November 2019 21:20

Hi;

the code I used to perform test is the following ;

Code: Select all

-- Website
weburl='https://www.laghi.net/homepage.aspx?tab=3&subtab=2&idlago=3'

-- retrieve web info
local http = require ("socket.http")

-- FONCTION

commandArray={}

-- 2° website information
        response, status = http.request(weburl)

print ("page web :"..status)

response=response:match('Quota invaso.-<td class="col_right">(.-)</td>')
print("LAKE LEVEL : "..response)

return commandArray

According to the log of Domoticz :" Error: EventSystem: in /home/pi/domoticz/scripts/lua/script_time_lac2.lua: /home/pi/domoticz/scripts/lua/script_time_lac2.lua:17: attempt to concatenate global 'response' (a nil value)", so on this line "print("LAKE LEVEL : "..response)", this is strange for me...

tinus · Post by **tinus** » Saturday 10 October 2020 16:58

Hello i'am new to lua scripting and try to get a value from a website as well.
I found the value i'am loking for but can't get it as a usable value.

the value i'am trying to get is the "36.8" at the end of the following code:

<div id="ui-id-3" data-filteredid="ui-id-3" data-filtervalueid="907442" data-group="00_0010_2251_0100_006_2_0-ui-id-3-value" data-toggle="context" data-target="none" data-angle="0" data-type="value" data-fieldid="00_0010_2251_0100_006_2_0" data-symbol="false" data-src="false" class="font" style="position: absolute; top: 385.736px; left: 475.137px; z-index: 1; transform: rotate(0deg); font-size: 10.0191px; font-weight: bold; font-style: normal; font-family: sans-serif; color: rgb(51, 51, 51); text-decoration: none;" data-newfilter="yes">36.8 °C</div>

Thank for your help.

jvdz · Post by **jvdz** » Saturday 10 October 2020 17:15

One way it could be done:

Code: Select all

Web_Data='<div id="ui-id-3" data-filteredid="ui-id-3" data-filtervalueid="907442" data-group="00_0010_2251_0100_006_2_0-ui-id-3-value" data-toggle="context" data-target="none" data-angle="0" data-type="value" data-fieldid="00_0010_2251_0100_006_2_0" data-symbol="false" data-src="false" class="font" style="position: absolute; top: 385.736px; left: 475.137px; z-index: 1; transform: rotate(0deg); font-size: 10.0191px; font-weight: bold; font-style: normal; font-family: sans-serif; color: rgb(51, 51, 51); text-decoration: none;" data-newfilter="yes">36.8&nbsp;°C</div>'
tmp = Web_Data:match('data.newfilter="yes">(.-)°C</div>')
tmp = tmp:gsub("&nbsp;","")
print("tmp:"..tmp)

Jos

waaren · Post by **waaren** » Saturday 10 October 2020 17:29

tinus wrote: ↑Saturday 10 October 2020 16:58 the value i'am trying to get is the "36.8" at the end of the following code:

[Edit] Sorry, did not see the solution from Jos.. But what about when newfilter="No" ?

This should do it.

Code: Select all

local a = '<div id="ui-id-3" data-filteredid="ui-id-3" data-filtervalueid="907442" data-group="00_0010_2251_0100_006_2_0-ui-id-3-value" data-toggle="context" data-target="none" data-angle="0" data-type="value" data-fieldid="00_0010_2251_0100_006_2_0" data-symbol="false" data-src="false" class="font" style="position: absolute; top: 385.736px; left: 475.137px; z-index: 1; transform: rotate(0deg); font-size: 10.0191px; font-weight: bold; font-style: normal; font-family: sans-serif; color: rgb(51, 51, 51); text-decoration: none;" data-newfilter="yes">36.8&nbsp;°C</div>'

local b =  a:match("%b><") -- get everything between > and <
print('Chopped a to: ' .. b)
local c = b:match("%d+%.*%d*") -- get the first float (or integer) of the remaining string 
print('Found ' .. c .. ' as first float or integer in the chopped string')

Script LUA to read value on a webpage ?

Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Re: Script LUA to read value on a webpage ?

Who is online