native buienradar Error getting http data

[waaren]

I can continue ( I have stopped it for now).
Which URL? Buienradar and the test program run on different RPi.
So I suggest "gpsgadget"

Your version still use the 'old' url so if you want to compare the results without influence, I suggest to stop that one now and once you upgraded to the version using the gspgadget url stop the parallel calls to that one.

[Gravityz]

since yesterday only one timeout
maybe with the old url it would have been 0

2019-10-31 22:34:01.541 Error: BuienRadar: Error getting http data! (Check your internet connection!)

[FireWizard]

Hi,

Just checked my logs and found that Domoticz, running the "old" URL did not report any Buienradar errors. The test RPi, polling the "new" URL, reported 5 missed data. Since 00:00h.
Times: 00:25 (2 consecutive misses), 08:35 (2 consecutive misses) and 10:00.

Testing continues.

Regards

[Gravityz]

ok
@waaren. Can we draw the conclusion that the old address is better?

should we do old url + retries and after that perhaps new url +retries

i think if we mis one once in a while it is not a big deal but then again we do not want errors in the log

[Gravityz]

hmmm. errors are getting worse.
maybe time to switch back to the old url gadgets

2019-10-31 22:34:01.541 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 12:50:43.133 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 13:20:55.572 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 13:25:55.859 Error: BuienRadar: RainPrediction has old data
2019-11-01 13:30:56.069 Error: BuienRadar: RainPrediction has old data
2019-11-01 13:55:57.229 Error: BuienRadar: RainPrediction has old data
2019-11-01 16:16:22.312 Error: BuienRadar: Error getting http data! (Check your internet connection!)

[FireWizard]

Hi.

This afternoon I saw indeed on my test Rpi more misses than before.
Since 10:00h, this morning I saw 7 missed data,
At 12:55 4 consecutive misses and further misses at 12:10 (2x),14:30,16:00, 16:15, 16:25 and 17:25(2x)
However I had also missed data on the "old" URL.

2019-11-01 12:44:08.026 Error: Darksky: Error getting http data!.
2019-11-01 13:06:12.337 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 13:31:18.978 Error: BuienRadar: RainPrediction has old data
2019-11-01 13:46:23.800 Error: BuienRadar: RainPrediction has old data
2019-11-01 14:01:28.018 Error: BuienRadar: RainPrediction has old data
2019-11-01 14:56:45.040 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 15:26:54.040 Error: BuienRadar: Error getting http data! (Check your internet connection!)

See also that I missed data from Darksky.

And your RainPrediction has old data matches with mine, about the same time.

[waaren]

Since 10:00h, this morning I saw 7 missed data,

On my test system I use a home made BuienRadar.cpp with new URL and max 4 retries with an increasing delay (after 1,2,3,4 seconds) .
Started at 07:00 this morning and no problems until 14:21
14:21 -- 4 retries ==> error
14:26 -- 4 retries ==> error
15:16 -- 4 retries ==> error
16:06 -- 4 retries ==> error
16:12 -- 2 retries ==> OK
16:27 -- 4 retries ==> error
17:48 -- 4 retries ==> error
18:43 -- 22:30 10 times -- 4 retries ==> error

On my 'production' system with 'old url' and without retries:
2019-11-01 11:35:52.387 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 13:31:26.406 Error: BuienRadar: RainPrediction has old data
2019-11-01 13:56:31.217 Error: BuienRadar: RainPrediction has old data
2019-11-01 15:36:53.333 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 16:27:03.801 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 17:02:11.450 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 17:22:15.521 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 19:42:45.155 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 20:37:56.282 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 21:33:08.666 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 21:38:09.351 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 22:03:17.420 Error: BuienRadar: Error getting http data! (Check your internet connection!)

I don't see a big difference in reliability between the two approaches. Next thing I will test is to use new url first and if it fails retry with the old one or vice-versa.
I don't expect a huge improvement but want to be sure.

[akamming]

hmmm… just noticed the same errors in my logging. Has anyone already tested if this was the rainprediction url or the actual conditions url (they both cause this error if there is no data...) ?

[waaren]

hmmm… just noticed the same errors in my logging. Has anyone already tested if this was the rainprediction url or the actual conditions url (they both cause this error if there is no data...) ?

Yes. allmost allways rainprediction.

[Gravityz]

i must say that things seem to improve
this is still my log and no new entries for almost 2 days.
this is with the gpsgadgets url

2019-10-31 22:34:01.541 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 12:50:43.133 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 13:20:55.572 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 13:25:55.859 Error: BuienRadar: RainPrediction has old data
2019-11-01 13:30:56.069 Error: BuienRadar: RainPrediction has old data
2019-11-01 13:55:57.229 Error: BuienRadar: RainPrediction has old data
2019-11-01 16:16:22.312 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-01 18:01:37.889 Error: BuienRadar: Error getting http data! (Check your internet connection!)

[FireWizard]

Hi,

As you know I still run Buienradar, polling the "old URL" (gadgets) and since last report (November 01, 17:58) I had no errors (Error: BuienRadar: Error getting http data! (Check your internet connection!)) in my logs. So the old URL is fine and also error free for almost 2 days.

My Rpi, that polled the "newURL" (gpsgadget) every 5 minutes, reported 29 misses on November 1, 19 misses on November 2 źand 15 misses November 3 until 14:00h. On November 1, I had 2 events, that reported 4 consecutive misses.

I think, it is time to take a decision, how to continue.
On the "newURL", I measured 63 misses, since November 1, 00:00h, while Gravityz reported "no errors" since November 1, 18:01h.
I think we may conclude that the mechanism of retries improves the reliability a lot.
However the "old URL" did not produce any errors as well, during the same time period. So I conclude that both the "old" URL and the "new"URL perform equally in this case.

In my opinion I think, that we should implement the retry mechanism and use both URL, as each other back-up.
If one fails, retry once. If it still fails, try the other, with eventually a retry.

I have stopped my test for now.

Regards

[Gravityz]

got the first message since 2 days, no timeout

2019-11-03 14:42:59.668 Error: BuienRadar: RainPrediction has old data

[KireStreve]

Hi,

I have the same error what was posted

2019-11-07 21:49:09.536 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-07 21:49:33.561 Error: BuienRadar: Error getting http data! (Check your internet connection!)

I run version 4.11468 on a Raspberry Pi 3B
The installation is two days old.

[FireWizard]

Hi,

The same here.
This afternoon 6 errors between 13:52h and 18:29h
Also running 4.11468.
Also yesterday I had errors.

Regards

[waaren]

This afternoon 6 errors between 13:52h and 18:29h

Today's results on my systems

Live system
using url https://gpsgadget.buienradar.nl

Code: Select all

2019-11-07 11:20:35.164  Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-07 11:40:45.668  Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-07 12:56:08.299  Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-07 16:02:18.390  Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-07 16:17:34.666  Error: BuienRadar: Error getting http data! (Check your internet connection!)

Test system (modified BuienRadar.cpp: after first failure it does max 2 retries with retry url
First url https://gadgets.buienradar.nl
Retry url https://gpsgadget.buienradar.nl

Code: Select all

2019-11-07 12:19:45.345  Status: BuienRadar Rainprediction: retry with different url
2019-11-07 12:19:47.531  Status: BuienRadar Rainprediction: retry with different url
2019-11-07 12:19:47.532  Error: BuienRadar Rainprediction: Error getting http data! (Check your internet connection!)
2019-11-07 12:39:55.449  Status: BuienRadar Rainprediction: retry with different url
2019-11-07 12:39:57.692  Status: BuienRadar Rainprediction: retry with different url
2019-11-07 12:39:57.693  Error: BuienRadar Rainprediction: Error getting http data! (Check your internet connection!)
2019-11-07 19:35:08.850  Status: BuienRadar Rainprediction: retry with different url
2019-11-07 19:35:11.043  Status: BuienRadar Rainprediction: retry with different url
2019-11-07 19:35:13.281  Error: BuienRadar Rainprediction: Error getting http data! (Check your internet connection!)
2019-11-07 19:35:13.281  Status: BuienRadar Rainprediction: retry with different url

My preliminary conclusion: Sometimes BuienRadar rainprediction system is too busy. I don't see these errors for the other BuienRadar and DarkSky url's and the used mechanism is exactly the same.

[Gravityz]

@waaren.
Maybe a final test where we swap primary and secondary link to make sure

[waaren]

Maybe a final test where we swap primary and secondary link to make sure

Will do

[Gravityz]

errors seem to increase again

cureently on build 484

Code: Select all

2019-11-14 19:09:49.242 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-14 23:41:19.162 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 04:52:18.292 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 06:47:36.944 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 07:58:02.107 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 08:13:03.839 Error: BuienRadar: RainPrediction has old data
2019-11-15 08:18:04.385 Error: BuienRadar: RainPrediction has old data
2019-11-15 08:23:04.661 Error: BuienRadar: RainPrediction has old data
2019-11-15 17:55:21.481 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 18:05:26.195 Error: BuienRadar: Error getting http data! (Check your internet connection!)

[Gravityz]

and even more strange errors

Code: Select all

2019-11-14 19:09:49.242 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-14 23:41:19.162 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 04:52:18.292 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 06:47:36.944 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 07:58:02.107 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 08:13:03.839 Error: BuienRadar: RainPrediction has old data
2019-11-15 08:18:04.385 Error: BuienRadar: RainPrediction has old data
2019-11-15 08:23:04.661 Error: BuienRadar: RainPrediction has old data
2019-11-15 17:55:21.481 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-15 18:05:26.195 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-16 07:33:46.679 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-16 07:59:06.643 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-16 14:46:54.092 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-16 14:51:58.380 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-16 16:38:06.379 Error: Buienradar hardware (9) thread seems to have ended unexpectedly
2019-11-16 16:38:20.382 Error: Buienradar hardware (9) thread seems to have ended unexpectedly
2019-11-16 16:38:34.384 Error: Buienradar hardware (9) thread seems to have ended unexpectedly
2019-11-16 16:38:34.646 Error: BuienRadar: Error getting http data! (Check your internet connection!)
2019-11-16 16:49:46.952 Error: Buienradar hardware (9) thread seems to have ended unexpectedly
2019-11-16 16:50:01.183 Error: Buienradar hardware (9) thread seems to have ended unexpectedly
2019-11-16 16:50:05.161 Error: BuienRadar: Error getting http data! (Check your internet connection!)

[FireWizard]

Hi,

You are not the only one with that error.
The other " Buienradar hardware thread seems to have ended unexpectedly" is related to the old "Buienradar" plugin. The (9) indicates the hardware idx number.

I see also this type of errors.
Is this your only plugin or do you have more plugins installed?

Regards