DzVents OpenURL response error

[ZN414]

Hi
I am running the Domoticz stable docker image on Raspberry Pi4.
The Pi 4 is running the current Pi OS based on Debian bookworm.

For a few years I have been using Domoticz to monitor the local river level using a DzVents script.
The script accesses the UK Environment Agency (EA) API.
An example script:

Jubilee river level.txt

(1.31 KiB) Downloaded 27 times

All was OK when the EA allowed access via http: but it changed to only supporting https: access.

I changed the script to use https: but then domoticz reported an error:

2025-04-17 19:00:04.432 Error: Error opening url: https://environment.data.gov.uk/flood-m ... h/measures
2025-04-17 19:00:04.465 Error: dzVents: HTTP/1.1 response: 403 ==>> Forbidden

I found I could stop this error by re-starting the docker instance.
Sometimes it took a number of restarts but then the script would run without error.

Now after a recent Pi kernel update, the error remains all the time.
Restarting the docker instance no longer fixes the problem.

If I SSH into the Pi and run:
curl -I https://environment.data.gov.uk/flood-m ... h/measures
It responds with:
HTTP/1.1 200 OK
Date: Thu, 17 Apr 2025 17:27:04 GMT
Content-Type: application/json
Content-Length: 775
Connection: keep-alive
Server: openresty/1.21.4.4
Vary: Accept
Cache-Control: no-transform, max-age=300
Last-Modified: Thu, 17 Apr 2025 17:18:23 GMT
Age: 57
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=15724800; includeSubDomains

No 403 forbidden error.....

Can anyone advise how I can troubleshoot this error?

Regards
Richard

[akamming]

Look like you really get a 403 error back from the server. don't know how to fix that...

But if the curl command works, you could work around it by giving that curl statement as an asynchronous shell command, see documentation:

https://wiki.domoticz.com/DzVents:_next ... tion_3.1.0

[waltervl]

What if you go into the docker container of Domoticz and do the curl command? Do you get 200 or 403 status?

[ZN414]

Hi

Thank you for the responses.

@akamming
So far I have not been able to work out how to use the asynchronous shell command.
I am not sure if I have to write a shell script or if I can directly access curl?
Also not sure what will be returned to the DzVents script.

@waltervl
I accessed the domoticz container and curl -I appears to work OK.

richard@neptune:/opt/domoticz $ docker exec -it domoticz bash
root@48d76438548a:/opt/domoticz# curl -I https://environment.data.gov.uk/flood-m ... h/measures
HTTP/1.1 200 OK
Date: Thu, 17 Apr 2025 20:13:29 GMT
Content-Type: application/json
Content-Length: 1817
Connection: keep-alive
Server: openresty/1.21.4.4
Vary: Accept
Cache-Control: no-transform, max-age=300
Last-Modified: Thu, 17 Apr 2025 20:03:21 GMT
Age: 0
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=15724800; includeSubDomains

[waltervl]

What is the -l curl option?
You use in your script:

Code: Select all

domoticz.openURL({
                url = baseurl .. stationid,
                method = 'GET',
                callback = 'JubileeLevel',
             })

Perhaps you need different OpenUrl options similar as the -l?
What if you remove the GET method?

[habahabahaba]

What if to use curl instead of domoticz.openurl ?

[ZN414]

Thank you for the ideas.

@ waltervl
curl -I https:/.......... only downloads the headers, I used it to show the status >> HTTP/1.1 200 OK
curl https:/.......... (with no switch) downlands the data (that in this case includes the river level information).
both curl commands appear to work OK.

Re the OpenUrl type, I removed the GET method and the 403 error is still reported.

The DzVents documentation states that GET is the default method.
https://wiki.domoticz.com/DzVents:_next ... P_requests

The other methods of POST PUT and DELETE do not seem appropriate for down loading data from a site.
The DzVents documentation does not explain the POST ,PUT & DELETE methods but I assume they are they same methods as when used with curl.
Using curl, GET is also the default and POST, PUT and DELETE appear to either transfer data out (up load) to a server or delete a record at the server.

@habahabahaba
I think using curl is probably the solution, I am working out how to implement using asynchronous shell command (as suggested by @akamming).

[habahabahaba]

I've just tested it from the Dzvents script by using curl and got 403 too Ж/

But from browser the link is working...

If it help you? my code:

Code: Select all

return {
    on = {
        timer = {
            'every minute', -- 

        }
    },

	logging = {
		level = domoticz.LOG_INFO,
		marker = 'Getting river level',
	},
	
    execute = function(domoticz)
        local url = 'https://environment.data.gov.uk/flood-monitoring/id/stations/2759th/measures'
        local command = 'curl -s ' .. url
        
        -- executing curl
        local handle = io.popen(command)
        local data = handle:read("*a")
        handle:close()
        
        if data and data ~= '' then
            domoticz.log('Data: ' .. data, domoticz.LOG_INFO)
            
        else
            domoticz.log('NO DATA ' , domoticz.LOG_ERROR)
        end
    end
}

The log:

2025-04-18 14:01:00.157 Status: dzVents: Info: Getting river level: ------ Start internal script: Script #1:, trigger: "every minute"
2025-04-18 14:01:01.024 Status: dzVents: Info: Getting river level: Data: <html>
2025-04-18 14:01:01.024 <head><title>403 Forbidden</title></head>
2025-04-18 14:01:01.024 <body>
2025-04-18 14:01:01.024 <center><h1>403 Forbidden</h1></center>
2025-04-18 14:01:01.024 <hr><center>Microsoft-Azure-Application-Gateway/v2</center>
2025-04-18 14:01:01.024 </body>
2025-04-18 14:01:01.024 </html>
2025-04-18 14:01:01.024
2025-04-18 14:01:01.024 Status: dzVents: Info: Getting river level: ------ Finished Script #1

[habahabahaba]

The same request from command line gives result:

{
"@context" : "http://environment.data.gov.uk/flood-mo ... ext.jsonld" ,
"meta" : {
"publisher" : "Environment Agency" ,
"licence" : "http://www.nationalarchives.gov.uk/doc/ ... version/3/" ,
"documentation" : "http://environment.data.gov.uk/flood-mo ... /reference" ,
"version" : "0.9" ,
"comment" : "Status: Beta service" ,
"hasFormat" : [ different URLs]
}
,
"items" : [ ]
}

[ZN414]

Hi habahabahaba

That is my fault. I used the wrong station id in my original post.
2759th appears to be offline.
I was changing the EA station id's to see if that was the problem.

If I change the id to 2029 curl works OK : 'https://environment.data.gov.uk/flood-m ... 9/measures'

I used the DzVents method:

local baseurl = 'https://environment.data.gov.uk/flood-m ... /stations/'
local stationid = '2029/measures' -- id & type

return {
on = {
timer = {'every 5 minutes'},
shellCommandResponses = {'RiverLevel'}
},

execute = function(domoticz, response)

if (response.isTimer or response.isDevice) then
domoticz.executeShellCommand({
command = 'curl ' .. baseurl .. stationid,
callback = 'RiverLevel', -- see shellCommandResponses above.
timeout = 20, -- max execution time in seconds
})
end


if (response.isShellCommandResponse) then -- triggered by the HTTP response from EA

if (response.statusCode == 0) then
local WaterLevel = domoticz.devices('River Level')
local index = string.find(response.data,"value" )
local Astring = string.sub(response.data,(index+8),(index+15))
WaterLevel.updateCustomSensor(tonumber(Astring))
else
domoticz.log('There was an error with accessing EA', domoticz.LOG_ERROR)
err = tostring(response.statusCode)
domoticz.log('Status Code: '..err, domoticz.LOG_ERROR)
err = tostring(response.statusText)
domoticz.log('Status text: '..err, domoticz.LOG_ERROR)
end
end

end
}

[habahabahaba]

Never mind

Both requests from Dz script - .openUrl, CURL - give an error 403