native buienradar Error getting http data

Moderator: leecollings

Gravityz
Posts: 587
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

native buienradar Error getting http data

Post by Gravityz »

lately i have been getting http data errrors with buienradar.

in the past i might have had 1 a week, not it is every hour

i think it has something to do with recent updates( i think it started after the upgrade to 4.11413)
anybody else havre these? if not please specify your version so we can pinpoint when things went wrong
2019-10-27 11:17:44.700 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 12:07:46.788 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 12:42:48.440 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 14:57:54.877 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 15:42:56.651 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 16:27:58.644 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 17:43:01.767 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 18:33:04.444 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 18:38:04.727 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 19:48:07.738 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 19:58:08.217 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 20:13:10.462 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 21:58:15.262 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 22:03:15.453 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 01:18:24.569 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 03:33:31.002 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 04:53:35.033 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 05:03:35.432 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 05:28:36.896 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 05:38:37.309 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 06:23:39.122 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 06:33:39.596 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 06:48:40.512 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 06:53:40.717 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 07:13:41.714 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 07:53:44.518 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 11:03:53.140 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 11:08:53.425 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 12:03:55.710 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 15:14:05.815 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 16:24:08.730 Error: BuienRadar: Error getting http data! (Check your internet connection!)
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by waaren »

Gravityz wrote: Monday 28 October 2019 16:42 Lately i have been getting http data errrors with buienradar. in the past i might have had 1 a week, now it is every hour
i think it has something to do with recent updates( i think it started after the upgrade to 4.11413)
anybody else have these?
I see more or less the same behavior on my systems. Raspberries 3 and 4 with Debian Buster and domoticz V4.11425
I believe It's caused by busy BuienRadar servers. (maybe combined with increased load from enthousiast domoticz fans)
These errors in the log can come from three different HTTP calls; all in use by BuienRadar hardware. I changed the error messages in BuienRadar.cpp for my test system to include the specific url. Hopefully that can help identifying if the message come from all three url's or only from one.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Gravityz
Posts: 587
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by Gravityz »

ok, if it is buienradar then there is nothing we can do.

however i remember that there were some changes regarding async tcp and firefox
if these changed timing or timeouts of some kind this might cause the http errors.

maybe we need to take another look at the buienradar code and change the timeout period or the number of retries before generating this error

i know we could do this with the script as well

Code: Select all

 --  get data from buienradar. curl 3 for 3 seconds. os.capture 3 for retry 3 times. adjust if getting errors in log
    local command = "curl --max-time 3 -L -s 'https://gpsgadget.buienradar.nl/data/raintext?lat=" .. lat .. "&lon=" .. lon .. "'"
    -- print("command: " .. command)
    local tmp = os.capture(command, 3)
User avatar
FireWizard
Posts: 1870
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: native buienradar Error getting http data

Post by FireWizard »

Hi all,

I noticed similar behaviour.
See:

Code: Select all

2019-10-27 19:58:42.683 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 19:58:42.683 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 19:58:42.683 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 20:14:05.879 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 20:17:33.013 Error: APC-UPS hardware (12) thread seems to have ended unexpectedly
2019-10-27 20:17:33.013 Error: Air Quality hardware (25) thread seems to have ended unexpectedly
2019-10-27 20:17:33.013 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 20:17:33.013 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 20:17:33.014 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 20:17:47.018 Error: APC-UPS hardware (12) thread seems to have ended unexpectedly
2019-10-27 20:17:47.018 Error: Air Quality hardware (25) thread seems to have ended unexpectedly
2019-10-27 20:17:47.018 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 20:17:47.018 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 20:17:47.019 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 20:18:01.024 Error: APC-UPS hardware (12) thread seems to have ended unexpectedly
2019-10-27 20:18:01.024 Error: Air Quality hardware (25) thread seems to have ended unexpectedly
2019-10-27 20:18:01.025 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 20:18:01.025 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 20:18:01.025 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 20:18:15.029 Error: APC-UPS hardware (12) thread seems to have ended unexpectedly
2019-10-27 20:18:15.030 Error: Air Quality hardware (25) thread seems to have ended unexpectedly
2019-10-27 20:18:15.030 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 20:18:15.030 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 20:18:15.030 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 20:18:29.042 Error: APC-UPS hardware (12) thread seems to have ended unexpectedly
2019-10-27 20:18:29.042 Error: Air Quality hardware (25) thread seems to have ended unexpectedly
2019-10-27 20:18:29.042 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 20:18:29.043 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 20:18:29.043 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 20:18:43.057 Error: Air Quality hardware (25) thread seems to have ended unexpectedly
2019-10-27 20:18:43.057 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 20:18:43.058 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 20:18:43.058 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 20:18:57.063 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 20:29:10.747 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 20:34:11.742 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 20:54:17.844 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 21:19:25.691 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 21:27:37.385 Error: Air Quality hardware (25) thread seems to have ended unexpectedly
2019-10-27 21:27:37.385 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 21:27:51.399 Error: APC-UPS hardware (12) thread seems to have ended unexpectedly
2019-10-27 21:27:51.399 Error: MQTT Discovery hardware (36) thread seems to have ended unexpectedly
2019-10-27 21:27:51.400 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 21:27:51.400 Error: Weeralarm hardware (45) thread seems to have ended unexpectedly
2019-10-27 21:28:05.404 Error: Buienradar hardware (39) thread seems to have ended unexpectedly
2019-10-27 21:54:37.477 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 22:49:53.749 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 22:54:54.764 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 23:09:59.868 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 23:20:02.926 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 23:25:03.853 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-27 23:30:05.850 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 00:10:19.277 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 00:30:25.078 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 00:55:32.707 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 01:55:50.004 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 02:00:51.830 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 02:10:54.680 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 02:36:01.940 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 02:56:10.574 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 03:11:15.175 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 03:31:20.864 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 03:36:21.864 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 03:46:24.831 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 04:31:39.094 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 05:31:56.860 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 06:07:12.938 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 07:02:29.733 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 07:07:30.694 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 07:47:42.751 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 08:37:57.971 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 08:48:01.129 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 09:18:15.011 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 09:38:21.245 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 09:58:27.012 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 10:28:36.135 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 10:38:39.668 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 10:48:43.389 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 11:08:49.594 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 11:18:52.928 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 11:28:55.806 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 11:32:07.705 Error: WebsocketHandler::Handle Exception: bad_weak_ptr
2019-10-28 12:01:18.736 Error: Error opening url: https://api.forecast.solar/estimate/52.179281/5.606528/10/-14/2.4
2019-10-28 12:01:18.937 Error: dzVents: Error: (2.4.29) solarData: response: 0 ==>>
2019-10-28 12:01:18.985 Error: dzVents: Error: (2.4.29) solarData: Problem with the JSON response
2019-10-28 13:01:08.541 Error: Error opening url: https://api.forecast.solar/estimate/52.179281/5.606528/10/-14/2.4
2019-10-28 13:01:08.783 Error: dzVents: Error: (2.4.29) solarData: response: 0 ==>>
2019-10-28 13:01:08.791 Error: dzVents: Error: (2.4.29) solarData: Problem with the JSON response
2019-10-28 14:55:00.921 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 15:30:16.934 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 15:45:22.362 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 16:10:29.460 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 16:20:32.189 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 16:45:40.014 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 16:55:42.820 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 17:05:47.287 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 17:20:51.322 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 17:25:53.579 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 17:35:56.323 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-10-28 17:40:57.406 Error: BuienRadar: Error getting http data! (Check your internet connection!) 
Beside that I have similar problems to get http data from the Buienradar site I noticed also a problem with all my plugins.
You can see that I still have the old, previous, Buienradar hardware (39) running. If it would be a "Buienradar" problem, you might expect that also the old hardware plugin would suffer from the same problem. This is not the case, but probably the old plugin polls the Buienradar service at another time.

Running Domoticz V4.11415 on Raspberry Pi 2

Regards
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by waaren »

Initial findings are that the HTPP errors only come from the call to https://gadgets.buienradar.nl/data/rain ... on=y.yyyyy and not from the other URL's.
All HTTP calls in the BuienRadar.cpp use the same technique ( HTTPClient::GET ) and the same default timeout.
When I see messages like

Code: Select all

The hardware (xx) thread seems to have ended unexpectedly  
on my systems, it is always related to a resource shortage or a lock. Most of the times the resource is IO but in some cases it is caused by a database lock triggered by a domoticz automatic backup.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
FireWizard
Posts: 1870
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: native buienradar Error getting http data

Post by FireWizard »

Hi

@waaren.

For your information:
Buienradar hardware (39) thread seems to have ended unexpectedly
This is related to the old "Buienradar" plugin and not to the new (native) "Buienradar" hardware.
Most of the times the resource is IO
What do you mean with "resource is IO"?
But don't obfuscate this thread with a different discussion about resources

Best regards
Gravityz
Posts: 587
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by Gravityz »

ok,
is it possible we try https://gpsgadget.buienradar.nl/data/ra ... t=51&lon=3
this is the adres which is currently advised on the buienradar site
it points to a different ip adres
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by waaren »

Gravityz wrote: Monday 28 October 2019 21:45 is it possible we try https://gpsgadget.buienradar.nl/data/ra ... t=51&lon=3
this is the adres which is currently advised on the buienradar site it points to a different ip adres
Sure. Let met start testing with this url. It should be clear quickly enough to see if it minimize the number of errors.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by waaren »

Gravityz wrote: Monday 28 October 2019 21:45 is it possible we try https://gpsgadget.buienradar.nl/data/ra ... t=51&lon=3
Sorry but it did not make a big diffence.

Buienradar test with modified url
2019-10-29 00:26:55.243 Status: BuienRadar: Calling URL ==>> https://gpsgadget.buienradar.nl/data/ra ... lon=Y.yyyy
and same call every 5 minutes

2019-10-29 00:26:55.008 Status: BuienRadar: Calling URL ==>> https://observations.buienradar.nl/1.0/ ... ation/6344
and same call every 10 minutes

2019-10-29 00:41:56.570 Error: BuienRadar: Error getting http data! (https://gpsgadget.buienradar.nl/data/ra ... lon=Y.yyyy
and 19 more same Errors at:
Spoiler: show

Code: Select all

2019-10-29 00:56:58
2019-10-29 01:11:59
2019-10-29 01:32:00
2019-10-29 01:47:02
2019-10-29 01:57:03
2019-10-29 02:02:15
2019-10-29 02:12:16
2019-10-29 02:57:20
2019-10-29 03:52:32
2019-10-29 04:17:35
2019-10-29 05:33:06
2019-10-29 05:53:17
2019-10-29 06:08:19
2019-10-29 07:08:24
2019-10-29 07:23:25
2019-10-29 09:04:00
2019-10-29 10:09:18
2019-10-29 10:14:18 (after this domoticz was down for some hours due to other activities. Unrelated to test)
2019-10-29 14:44:46
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
Gravityz
Posts: 587
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by Gravityz »

ok.

noticed Gizmocuz implemented the new url and also implemented some retries.

sofar looking good and no timeouts. i must say today before the update i only had 4 http errors so things on the buienradar end also improved
User avatar
FireWizard
Posts: 1870
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: native buienradar Error getting http data

Post by FireWizard »

Hi,

Tuesday waaren did some testing with both URL's. I decided to do some testing as well.
I created a timer that polled every 5 minutes both https://gadgets.buienradar.nl/data/rain ... n=y.yyyyyy and https://gpsgadget.buienradar.nl/data/ra ... n=y.yyyyyy at exactly the same moment in order to see, if there is a relationship between the 2 URL's, if data has not been received and also, if there is a relation with the Domoticz log file.

The test ran from Tuesday November 29, 22:50h until Thursday November 31, 16:25h. So total duration 41:35h and 500 polls.
My Domoticz logfile indicated in that period 6 errors, from which 5 were reported yesterday and only 1 today.

This confirms Gravityz observation that Buienradar has improved as well.

The results are, that https://gadgets.buienradar.nl/data/rain ... n=y.yyyyyy produced 6 missing http data and that https://gpsgadget.buienradar.nl/data/ra ... n=y.yyyyyy produced 47 missing http data. A huge difference!
Only today the measurement was 1 for the "gadgets" against 34 for "gpsgadget". An explanation might be that "gpsgadget" is more used then "gadgets", as the first one is mentioned on the Buienradar website. That Gravityz observes an improvement, I do understand, as the changed Buienradar file is not available, yet. What is even worse with the missing data from https://gpsgadget.buienradar.nl/data/ra ... n=y.yyyyyy is that almost all "misses" are 2 or more consecutive failures. So the change, that you miss data with "gpsgadget" will significantly increase.
If more retries will help, depends on the number and the time interval.

Tonight at 3:50h both URL's had a failure and that was exactly the moment that also my Domoticz log indicated missing http data.

I would recommend not to change anything regarding the URL's, unless Buienradar takes down the "gadgets" URL.
In the meantime I have seen that gizmocuz has changed the URL, but that will, in my opinion, not solve the problem, on the contrary, unless that the new URL improves significantly.

I wanted to share this information with you.
Let's see when the update reaches the users, but I'm not hopeful, as we move to an URL, that misses about 10% of the requests.

Regards
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by waaren »

FireWizard wrote: Thursday 31 October 2019 18:48 I wanted to share this information with you.
Let's see when the update reaches the users, but I'm not hopeful, as we move to an URL, that misses about 10% of the requests.
Thx for sharing your observations.
Besides changing the url. @Gizmocuz also added a retry functionality so if the url does not return valid data it will try again (once) after 2 seconds.
Let's see what the combination of these changes will do when used in a live environment.
If it gets worse, changing it back is a simple change in the source code.

Gving it an extra tought .. Maybe even better to make it a configurable item in the hardware setup page.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
FireWizard
Posts: 1870
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: native buienradar Error getting http data

Post by FireWizard »

Thanks waaren for your reaction,

A retry might help, but if I see even 3 consecutive failures, I doubt, if it wIll solve the problem, if the time between 2 requests is only 2 seconds. I noticed only one event that both URL' s failed in providing valid data.
Don' t make it too complex by adding another config option.

I suggest, that you leave the URL as it was before and, if it fails to provide valid data, do a retry after 1 or 2 seconds, but not the same URL, but take the other URL. I think, that increases the change you get valid data significantly.

Regards
Gravityz
Posts: 587
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by Gravityz »

thanks @Firewizard.
i think indeed that because gpsgadget is advertised on the site it might get more hits(and therefor more misses) than gadgets
if this is really the case then swithing back to the old url might be better.

however at the moment i see a lot of improvement and i can not believe it is only because the retries were added.
There might be indeed a relation between gadgets and gpsgadgets and maybee they should be used as eachothers backup including the retry functionality like you suggested.
For now i agree with @Waaren to wait and see who things will turn out the next days.
User avatar
FireWizard
Posts: 1870
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: native buienradar Error getting http data

Post by FireWizard »

Hi,

I'm running version 4.11415 and a newer version does not exist. The implementation of the "gpsgadget" URL and the retry functionality is very recent, as far as I know. We have to wait for the next beta, before we can make a conclusion. My test showed that also "gadgets" is behaving better.
Today 1 failure, yesterday 5 failures ("gpsgadget" is just the other way around; today 34 and yesterday 13 failures)

Let's wait,

Regards
Gravityz
Posts: 587
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by Gravityz »

i am using a synology build and those are build very fast after a new version is released (thanks to Jumbotroll)

the new url was implemented in build 433, i am currently at build 436

i only had 4 errors today so good url or not, the retries are really helping
installed build 436 at 18:00 and now 4.5 hours later still no errors
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by waaren »

FireWizard wrote: Thursday 31 October 2019 21:58 I'm running version 4.11415 and a newer version does not exist. The implementation of the "gpsgadget" URL and the retry functionality is very recent, as far as I know.
If you compile locally and only one aspect has changed. a new version can be created within minutes. It can be quite cumbersome at first but once you understand the pitfalls it's really helpful to quickly test implemented changes or locally changed settings.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
FireWizard
Posts: 1870
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: native buienradar Error getting http data

Post by FireWizard »

Ok Gravityz,

You area ahead. I'm running the latest beta on Raspberry Pi
Today I had one error at 3:50 a.m. with the "old" URL and no retries.
Keep posting the results.

@waaren. I never compiled myself so far, but as long as the old URL is error free, I will not see any improvement.

Regards
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: native buienradar Error getting http data

Post by waaren »

FireWizard wrote: Thursday 31 October 2019 22:51 @waaren. I never compiled myself so far, but as long as the old URL is error free, I will not see any improvement.
Can you please continue your tests without the parallel polling as it will influence your findings. The same cache like behavior is exactly the reason why a retry after 2 seconds significantly improve the error rate. I do not understand the theory behind it yet but did see the same behavior when working on an air quality script.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
FireWizard
Posts: 1870
Joined: Tuesday 25 December 2018 12:11
Target OS: Raspberry Pi / ODroid
Domoticz version: Beta
Location: Voorthuizen (NL)
Contact:

Re: native buienradar Error getting http data

Post by FireWizard »

Hi waaren,

I can continue ( I have stopped it for now).
Which URL? Buienradar and the test program run on different RPi.
So I suggest "gpsgadget"
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest