Import RIVM sensor data

[Derik]

kick....

Please

[ErikAtSpijk]

It might be a bit pure amateurism but the small piece of code works for me to get luchtmeetnet.nl data. Before using it you need to install lxml. (sudo apt-get install python3-lxml ) Save the pyhon script under /home/pi/domoticz/scripts and create a crontab job.

The scipt works by looking for all links (a herf) in the tabel from luchtmeetnet. It then loops though the data until it finds a location containing "Hoogvliet". You can change this for your own location. The value of the polution is then contained in the next value, just in the order as they are displayed on the website. If there is no data on the website for a component than it is just skipped. For every component following the script then uploads the data into the domoticz api, you need to fill in the domoticz username, password and ip adress plus the idx value of the virtual device that need to contain the data in the string "command ". Depeding on how many components are measured for your location you might need to remove or add a few uploads to the api.

It's pretty basic but does works for me.

Code: Select all

#! /usr/bin/python

from lxml import html
import requests
import subprocess

page = requests.get('https://www.luchtmeetnet.nl/tabel')
tree = html.fromstring(page.content)
value = tree.xpath('//a/text()')


tel =1
while tel < 500:
	if "Hoogvliet" in value[tel]:
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=1&nvalue=0&svalue='+str(float(value[tel+1]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=2&nvalue=0&svalue='+str(float(value[tel+2]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=3&nvalue=0&svalue='+str(float(value[tel+3]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=4&nvalue=0&svalue='+str(float(value[tel+4]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=5&nvalue=0&svalue='+str(float(value[tel+5]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=6&nvalue=0&svalue='+str(float(value[tel+6]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=7&nvalue=0&svalue='+str(float(value[tel+7]))+'"')
		output = subprocess.check_output(['bash','-c', command])
	tel = tel +1

[Derik]

It might be a bit pure amateurism but the small piece of code works for me to get luchtmeetnet.nl data. Before using it you need to install lxml. (sudo apt-get install python3-lxml ) Save the pyhon script under /home/pi/domoticz/scripts and create a crontab job.

The scipt works by looking for all links (a herf) in the tabel from luchtmeetnet. It then loops though the data until it finds a location containing "Hoogvliet". You can change this for your own location. The value of the polution is then contained in the next value, just in the order as they are displayed on the website. If there is no data on the website for a component than it is just skipped. For every component following the script then uploads the data into the domoticz api, you need to fill in the domoticz username, password and ip adress plus the idx value of the virtual device that need to contain the data in the string "command ". Depeding on how many components are measured for your location you might need to remove or add a few uploads to the api.

It's pretty basic but does works for me.

Code: Select all

#! /usr/bin/python

from lxml import html
import requests
import subprocess

page = requests.get('https://www.luchtmeetnet.nl/tabel')
tree = html.fromstring(page.content)
value = tree.xpath('//a/text()')


tel =1
while tel < 500:
	if "Hoogvliet" in value[tel]:
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=1&nvalue=0&svalue='+str(float(value[tel+1]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=2&nvalue=0&svalue='+str(float(value[tel+2]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=3&nvalue=0&svalue='+str(float(value[tel+3]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=4&nvalue=0&svalue='+str(float(value[tel+4]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=5&nvalue=0&svalue='+str(float(value[tel+5]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=6&nvalue=0&svalue='+str(float(value[tel+6]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=7&nvalue=0&svalue='+str(float(value[tel+7]))+'"')
		output = subprocess.check_output(['bash','-c', command])
	tel = tel +1

MMM
My system is not stable yet grrr.
When it is i give it a try..

Only is it not possible to use the intern python option in blockley?

Thanks for the great work

[Derik]

Dear....
I did try into the basis Domoticz Python option.
Only is this perhaps possible?
This is much better for the DB backup?

ScreenShot021.png (112.61 KiB) Viewed 2090 times

Could this work?
Or can you make this work?



Other question..
What is the range of strings:
What idx is the No2?
What idx is co etc etc

[ErikAtSpijk]

Hello, I am not familier with the Domiticz pyhton option as I struggled with urlib2 in the past. Pherhaps you can make it work do.

The string value[tel] will contain all links (<a herf>) on the website in the order in which they are shown on the website. In your case the loop will look trough the links until it finds "Nijmegen-Graafseweg". The next string in the array value[tel] are the componenst show. In your case they are NO2 (value(tel+1)), NO (value(tel+2)), PM10 (value(tel+3)), PM2.5 (value(tel+4)), roet (value(tel+5))and CO (value(tel+6)). The component not shown on the website are simply not in the array.

The idx of the vitrual divices are from Domitiocz (add hardware -> add device.) In your case divice 9277 seems to be NO2 (first component on the website) CO will be the 6th value in the array (value(tel+6))

There is no 7th components for you station, you can simply delete the last two lines

Hope this helps...

[Derik]

Dear ... connot get it to work...
Did intstall;

Code: Select all

sudo apt-get install python3-lxml

Did the cron:

Code: Select all

*/2 * * * * python /home/odroid/domoticz/scripts/python/rivm.py

Did the script:

ScreenShot046.png (8.59 KiB) Viewed 1950 times

Did set the login and ww to my versions..
Did change the dvice idx for my devices:

Code: Select all

@192.168.5.70:8080/json.htm?type=command&param=udevice&9277=1&nvalue=0&svalue='+str(float(value[tel+1]))+'"')

Only log data and the device is not changing
What do i mis?

[ErikAtSpijk]

Hello Derik,

Not sure I see the issue but a couple of thoughts:

• did you try running it from the command line? (./rivm.py?)

• Did you made an executable? (sudo chmod -x rivm.py)

• In the crontab I am not sure you need to include the word " python", I don't use it, I use "*/15 * * * * /home/pi/domoticz/scripts/luchtmeetnet.py"

Hope if helps.

Erik

[Derik]

Code: Select all

Traceback (most recent call last):
  File "./rivm.py", line 3, in <module>
    from lxml import html
ImportError: No module named lxml
odroid@odroid:~/domoticz/scripts/python$

When i try the command line ...

Only i did install:

Code: Select all

odroid@odroid:~$ sudo apt-get install python3-lxml
[sudo] password for odroid:
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-lxml is already the newest version (3.5.0-1build1).
The following packages were automatically installed and are no longer required:
  libllvm3.8 libllvm4.0 libmircommon5 libqmi-glib1
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.
odroid@odroid:~$

hope..

[EdwinK]

It looks to me that there isn't any data gathered from this resource for some time now.

At least since march 21 I only have n/a in the logs.

[ErikAtSpijk]

Edwin, the RIVM website seems to stop, the website luchtmeetnet.nl seems to be the new site.

Derik, it looks like lxml can't be found, does the link below helps?

http://lxml.de/3.0/installation.html

[Derik]

dear....
This is not the solution..Still get some errors... when installing this package.
Need to install other package:

Code: Select all

** make sure the development packages of libxml2 and libxslt are installed **

Code: Select all

    Could not find function xmlCheckVersion in library libxml2. Is libxml2 insta                                        lled?

Did reinstall and a new install, no luck

Code: Select all

odroid@odroid:~$ sudo apt-get install libxslt
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libxslt
odroid@odroid:~$

so hope there is help..

[Derik]

dear all..
I will get this option work

So i still hope someone will help me..

Code: Select all

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   8000      0 --:--:-- --:--:-- --:--:--  8000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   8000      0 --:--:-- --:--:-- --:--:-- 12000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0  12000      0 --:--:-- --:--:-- --:--:-- 12000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   6000      0 --:--:-- --:--:-- --:--:--  6000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   4000      0 --:--:-- --:--:-- --:--:--  4000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   8000      0 --:--:-- --:--:-- --:--:-- 12000
domoticz@odroid:~/domoticz/scripts/python$

When i do the script from putty.
I get this data....
Almost there???
Only when i try a other station my data is still the same.. grrr..

[ErikAtSpijk]

Hello Derik, not sure where is goes wrong, but what you see is the output of the lines:

command = ('curl -i "http://user:password@ip:8080/json.htm?t ... alue='+str(float(value[tel+1]))+'"')
output = subprocess.check_output(['bash','-c', command])

Because these lines are winthin the "while loop" it might suggest that some data is downloaded from luchtmeetnet and that the Nijmegen station is found. What you don't see in this output is the value it uploads. Suggest to add debug information like:

print tree at line 10
and
print value(tel) at line 15 and value(tel+1) at just underneath it.

[Derik]

Dear Erik...
Thanks

Only i do not understand what you mean..

Is the script ok?
Is there no data?
My dummys's are not working...
Is there perhaps someone that have this working?

[EdwinK]

Edwin, the RIVM website seems to stop, the website luchtmeetnet.nl seems to be the new site.

Too bad, I did like the older data, but guess need to live without it then

[ErikAtSpijk]

Derik, I guess you need to add debug information to the script in order to understand where it ges wrong.I can post a proposal later this week. The script works for me. (Fijnstof at Hoogvliet)