Domoticz

Open source Home Automation System
https://forum.domoticz.com/

CURL : get rain data, save to file and read/process line by line

https://forum.domoticz.com/viewtopic.php?t=13136

Page 1 of 1

CURL : get rain data, save to file and read/process line by line

Posted: Monday 29 August 2016 23:59
by marmachine
Hi Folks,

I'm kinda stuck with the curl-part in the "IsItGonnaRain" function. (see this topic)
This scripts gets data from "Buienradar" and should save this into a tmp file
Next it should read this file line by line to process the data.

I am under the impression that the data/file is not saved actually.
Being no-expert in LUA, some help will be very welcome

here's part of my code

Code: Select all

function IsItGonnaRain(latitude,longitude,tempfilename,minutesinfuture,debug)

    local averagerain = 0
    local totalrain = 0
    local rainlines = 0
    
    -- construct the request URL
    local url = 'http://gps.buienradar.nl/getrr.php?lat='..latitude..'&lon='..longitude
    if (debug == 1) then print("| Request URL .............. : " .. url) end

    -- now get the data
    local trigger_action = 'curl -o '..tempfilename..' "'..url..'"' -- -s for Silent (no error messages) and -o for Write output to file instead of stdout.
    handle = os.execute(trigger_action)

    if (debug == 1) then
        print("| Trigger action ........... : " .. trigger_action)
        if handle then
            print("| CURL Data request ........ : Success")
        else
            print("! CURL Data request ........ : Failed")
        end
    end

    file = io.open(tempfilename, "r")

        -- now analyse the received lines, format is like 000|15:30 per line.
        while true do
            line = file:read("*line")
            if not line then break end
        
            if (debug == 1) then print("| Rain data (line) ......... : " .. line) end
        end
The debug result actually shows the content from the file, the content however isn't updated as you can see by the time.

Here's the debug result

Code: Select all

2016-08-29 23:56:00.509 LUA: +--------------------------------------------------+
2016-08-29 23:56:00.509 LUA: | =============== IS IT GONNA RAIN =============== |
2016-08-29 23:56:00.509 LUA: +--------------------------------------------------+
2016-08-29 23:56:00.509 LUA: | Request URL .............. : http://gps.buienradar.nl/getrr.php?lat=52.326080&lon=5.600445
2016-08-29 23:56:00.595 LUA: | Trigger action ........... : curl -o /home/pi/domoticz/scripts/tmp/rain.tmp "http://gps.buienradar.nl/getrr.php?lat=52.326080&lon=5.600445"
2016-08-29 23:56:00.596 LUA: | CURL Data request ........ : Success
2016-08-29 [color=#FF0000]23:56[/color]:00.596 LUA: | Rain data (line) ......... : 000|[color=#FF0000]19:45[/color]
2016-08-29 23:56:00.596 LUA: | Rain data # lines ........ : 0
2016-08-29 23:56:00.596 LUA: | Total rain ............... : 0

Re: CURL : get rain data, save to file and read/process line by line

Posted: Tuesday 30 August 2016 9:12
by Egregius
The url isn't changed, my curl in php is still up and running.

But, when you paste the gps.buienradar url in a browser you're redirected to http://gadgets.buienradar.nl/data/raintext?lat=... so maybe you should use that as url.

Re: CURL : get rain data, save to file and read/process line by line

Posted: Tuesday 30 August 2016 13:08
by marmachine
gordonb3 wrote:Ah!. My bad... :oops:
Just noticed that the uri in the wget output is incomplete, which happened to be due to the ampersand ('&') not being escaped. This is why wget did not return any output even though it followed the redirect. As it happens, curl does not do that unless you also specify '-L' as parameter. Without that it simply returns the output from the original uri, which is nothing.

But: this should also be what is in your output file and that is not the case because you have old content in it. Possibly your process does not have write access to that file?
Ohh the old content, actually the file entirely, is something i've put there to see if it finds and reads the file.
Also i tried to changes the file rights because i do expect the same issue;

Code: Select all

sudo chmod 777 /home/pi/domoticz/scripts/tmp/rain.tmp
However i am not sure if this is actually the case.
The CURL command in Putty also doesn't seem to work... (keep in mind i am unfamiliar with Linux and LUA)

Code: Select all

pi@raspberrypi:~$ curl -o /home/pi/domoticz/scripts/tmp/rain.tmp "http://gps.buienradar.nl/getrr.php?lat=52.326080&lon=5.600445"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
pi@raspberrypi:~$
Anyway, you're suggesting to try the -L parameter to the CURL and see what happens?

Re: CURL : get rain data, save to file and read/process line by line

Posted: Tuesday 30 August 2016 13:15
by marmachine
Egregius wrote:The url isn't changed, my curl in php is still up and running.

But, when you paste the gps.buienradar url in a browser you're redirected to http://gadgets.buienradar.nl/data/raintext?lat=... so maybe you should use that as url.
Thanks Egregius! Changing the URL actually is the solution to this! I have my file content updated now! Woohoo :D me happy!

Edit: i've shared my updated version of the full script here

Re: CURL : get rain data, save to file and read/process line by line

Posted: Tuesday 30 August 2016 17:48
by rolandbreedveld
Script isnt working since 29-aug-2016
after changing the url, it's working again:
url='http://gadgets.buienradar.nl/data/raint ... lon='..lon

Re: CURL : get rain data, save to file and read/process line by line

Posted: Tuesday 30 August 2016 18:12
by marmachine
gordonb3 wrote:
marmachine wrote:Anyway, you're suggesting to try the -L parameter to the CURL and see what happens?
Either that or change the uri to the secondary one. If the primary uri - the one you are using - is the documented one I would stick to that

Anyway, this is what you'll see with the -L parameter added:

Code: Select all

curl -Lo /home/pi/domoticz/scripts/tmp/rain.tmp "http://gps.buienradar.nl/getrr.php?lat=52.326080&lon=5.600445"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   264  100   264    0     0   3307      0 --:--:-- --:--:-- --:--:--  3307
The first line with all the zero's is from the original uri, the second one is from the redirect.
Thanks Gordonb3, that makes sence!
Below is my Domoticz log:

Code: Select all

2016-08-30 18:10:00.200 LUA: +--------------------------------------------------+
2016-08-30 18:10:00.200 LUA: | =============== IS IT GONNA RAIN =============== |
2016-08-30 18:10:00.200 LUA: +--------------------------------------------------+
2016-08-30 18:10:00.200 LUA: | Request URL .............. : http://gps.buienradar.nl/getrr.php?lat=52.326080&lon=5.600445
2016-08-30 18:10:00.347 LUA: | CURL Data request ........ : Success
2016-08-30 18:10:00.348 LUA: | Rain data (line) ......... : 000|18:10
2016-08-30 18:10:00.348 LUA: | Rain data (line) ......... : 000|18:15
2016-08-30 18:10:00.348 LUA: | Rain data (line) ......... : 000|18:20
2016-08-30 18:10:00.348 LUA: | Rain data (line) ......... : 000|18:25
2016-08-30 18:10:00.351 LUA: | Rain data received # lines : 24
2016-08-30 18:10:00.352 LUA: | In timerange # lines ..... : 4
2016-08-30 18:10:00.352 LUA: | Total rain ............... : 0
2016-08-30 18:10:00.352 LUA: | Average rain calculated .. : 0
2016-08-30 18:10:00.352 LUA: | Rain text ................ : Geen regen verwacht binnen komende 15 minuten
2016-08-30 18:10:00.352 LUA: +--------------------------------------------------+
2016-08-30 18:10:00.352 LUA: | --------------------- END ---------------------- |
2016-08-30 18:10:00.352 LUA: +--------------------------------------------------+
All times are UTC+01:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited