Domoticz

Open source Home Automation System
https://forum.domoticz.com/

Import RIVM sensor data

https://forum.domoticz.com/viewtopic.php?t=17334

Page 2 of 2

Re: Import RIVM sensor data

Posted: Tuesday 27 February 2018 17:24
by Derik
kick....

Please

Re: Import RIVM sensor data

Posted: Monday 12 March 2018 21:29
by ErikAtSpijk
It might be a bit pure amateurism but the small piece of code works for me to get luchtmeetnet.nl data. Before using it you need to install lxml. (sudo apt-get install python3-lxml ) Save the pyhon script under /home/pi/domoticz/scripts and create a crontab job.

The scipt works by looking for all links (a herf) in the tabel from luchtmeetnet. It then loops though the data until it finds a location containing "Hoogvliet". You can change this for your own location. The value of the polution is then contained in the next value, just in the order as they are displayed on the website. If there is no data on the website for a component than it is just skipped. For every component following the script then uploads the data into the domoticz api, you need to fill in the domoticz username, password and ip adress plus the idx value of the virtual device that need to contain the data in the string "command ". Depeding on how many components are measured for your location you might need to remove or add a few uploads to the api.

It's pretty basic but does works for me.

Code: Select all

#! /usr/bin/python

from lxml import html
import requests
import subprocess

page = requests.get('https://www.luchtmeetnet.nl/tabel')
tree = html.fromstring(page.content)
value = tree.xpath('//a/text()')


tel =1
while tel < 500:
	if "Hoogvliet" in value[tel]:
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=1&nvalue=0&svalue='+str(float(value[tel+1]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=2&nvalue=0&svalue='+str(float(value[tel+2]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=3&nvalue=0&svalue='+str(float(value[tel+3]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=4&nvalue=0&svalue='+str(float(value[tel+4]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=5&nvalue=0&svalue='+str(float(value[tel+5]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=6&nvalue=0&svalue='+str(float(value[tel+6]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=7&nvalue=0&svalue='+str(float(value[tel+7]))+'"')
		output = subprocess.check_output(['bash','-c', command])
	tel = tel +1

Re: Import RIVM sensor data

Posted: Monday 12 March 2018 21:34
by Derik
ErikAtSpijk wrote: ↑Monday 12 March 2018 21:29 It might be a bit pure amateurism but the small piece of code works for me to get luchtmeetnet.nl data. Before using it you need to install lxml. (sudo apt-get install python3-lxml ) Save the pyhon script under /home/pi/domoticz/scripts and create a crontab job.

The scipt works by looking for all links (a herf) in the tabel from luchtmeetnet. It then loops though the data until it finds a location containing "Hoogvliet". You can change this for your own location. The value of the polution is then contained in the next value, just in the order as they are displayed on the website. If there is no data on the website for a component than it is just skipped. For every component following the script then uploads the data into the domoticz api, you need to fill in the domoticz username, password and ip adress plus the idx value of the virtual device that need to contain the data in the string "command ". Depeding on how many components are measured for your location you might need to remove or add a few uploads to the api.

It's pretty basic but does works for me.

Code: Select all

#! /usr/bin/python

from lxml import html
import requests
import subprocess

page = requests.get('https://www.luchtmeetnet.nl/tabel')
tree = html.fromstring(page.content)
value = tree.xpath('//a/text()')


tel =1
while tel < 500:
	if "Hoogvliet" in value[tel]:
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=1&nvalue=0&svalue='+str(float(value[tel+1]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=2&nvalue=0&svalue='+str(float(value[tel+2]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=3&nvalue=0&svalue='+str(float(value[tel+3]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=4&nvalue=0&svalue='+str(float(value[tel+4]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=5&nvalue=0&svalue='+str(float(value[tel+5]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=6&nvalue=0&svalue='+str(float(value[tel+6]))+'"')
		output = subprocess.check_output(['bash','-c', command])
		command = ('curl -i  "http://user:password@ip:8080/json.htm?type=command&param=udevice&idx=7&nvalue=0&svalue='+str(float(value[tel+7]))+'"')
		output = subprocess.check_output(['bash','-c', command])
	tel = tel +1

MMM
My system is not stable yet grrr.
When it is i give it a try..

Only is it not possible to use the intern python option in blockley?

Thanks for the great work

Re: Import RIVM sensor data

Posted: Sunday 18 March 2018 12:55
by Derik
Dear....
I did try into the basis Domoticz Python option.
Only is this perhaps possible?
This is much better for the DB backup?
ScreenShot021.png
ScreenShot021.png (112.61 KiB) Viewed 2098 times
Could this work?
Or can you make this work?



Other question..
What is the range of strings:
What idx is the No2?
What idx is co etc etc

Re: Import RIVM sensor data

Posted: Sunday 18 March 2018 19:54
by ErikAtSpijk
Hello, I am not familier with the Domiticz pyhton option as I struggled with urlib2 in the past. Pherhaps you can make it work do.

The string value[tel] will contain all links (<a herf>) on the website in the order in which they are shown on the website. In your case the loop will look trough the links until it finds "Nijmegen-Graafseweg". The next string in the array value[tel] are the componenst show. In your case they are NO2 (value(tel+1)), NO (value(tel+2)), PM10 (value(tel+3)), PM2.5 (value(tel+4)), roet (value(tel+5))and CO (value(tel+6)). The component not shown on the website are simply not in the array.

The idx of the vitrual divices are from Domitiocz (add hardware -> add device.) In your case divice 9277 seems to be NO2 (first component on the website) CO will be the 6th value in the array (value(tel+6))

There is no 7th components for you station, you can simply delete the last two lines

Hope this helps...

Re: Import RIVM sensor data

Posted: Saturday 31 March 2018 22:09
by Derik
Dear ... connot get it to work...
Did intstall;

Code: Select all

sudo apt-get install python3-lxml
Did the cron:

Code: Select all

*/2 * * * * python /home/odroid/domoticz/scripts/python/rivm.py
Did the script:
ScreenShot046.png
ScreenShot046.png (8.59 KiB) Viewed 1958 times
Did set the login and ww to my versions..
Did change the dvice idx for my devices:

Code: Select all

@192.168.5.70:8080/json.htm?type=command&param=udevice&9277=1&nvalue=0&svalue='+str(float(value[tel+1]))+'"')
Only log data and the device is not changing
What do i mis?

Re: Import RIVM sensor data

Posted: Sunday 01 April 2018 22:25
by ErikAtSpijk
Hello Derik,

Not sure I see the issue but a couple of thoughts:
  • did you try running it from the command line? (./rivm.py?)
  • Did you made an executable? (sudo chmod -x rivm.py)
  • In the crontab I am not sure you need to include the word " python", I don't use it, I use "*/15 * * * * /home/pi/domoticz/scripts/luchtmeetnet.py"
Hope if helps.

Erik

Re: Import RIVM sensor data

Posted: Sunday 01 April 2018 22:44
by Derik

Code: Select all

Traceback (most recent call last):
  File "./rivm.py", line 3, in <module>
    from lxml import html
ImportError: No module named lxml
odroid@odroid:~/domoticz/scripts/python$
When i try the command line ...

Only i did install:

Code: Select all

odroid@odroid:~$ sudo apt-get install python3-lxml
[sudo] password for odroid:
Reading package lists... Done
Building dependency tree
Reading state information... Done
python3-lxml is already the newest version (3.5.0-1build1).
The following packages were automatically installed and are no longer required:
  libllvm3.8 libllvm4.0 libmircommon5 libqmi-glib1
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 15 not upgraded.
odroid@odroid:~$
hope..:-)

Re: Import RIVM sensor data

Posted: Monday 02 April 2018 11:12
by EdwinK
It looks to me that there isn't any data gathered from this resource for some time now.

At least since march 21 I only have n/a in the logs.

Re: Import RIVM sensor data

Posted: Monday 02 April 2018 21:30
by ErikAtSpijk
Edwin, the RIVM website seems to stop, the website luchtmeetnet.nl seems to be the new site.

Derik, it looks like lxml can't be found, does the link below helps?

http://lxml.de/3.0/installation.html

Re: Import RIVM sensor data

Posted: Friday 04 May 2018 10:28
by Derik
dear....
This is not the solution..Still get some errors... when installing this package.
Need to install other package:

Code: Select all

** make sure the development packages of libxml2 and libxslt are installed **

Code: Select all

    Could not find function xmlCheckVersion in library libxml2. Is libxml2 insta                                        lled?
Did reinstall and a new install, no luck

Code: Select all

odroid@odroid:~$ sudo apt-get install libxslt
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libxslt
odroid@odroid:~$
so hope there is help..

Re: Import RIVM sensor data

Posted: Tuesday 22 May 2018 21:38
by Derik
dear all..
I will get this option work

So i still hope someone will help me..

Code: Select all

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   8000      0 --:--:-- --:--:-- --:--:--  8000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   8000      0 --:--:-- --:--:-- --:--:-- 12000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0  12000      0 --:--:-- --:--:-- --:--:-- 12000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   6000      0 --:--:-- --:--:-- --:--:--  6000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   4000      0 --:--:-- --:--:-- --:--:--  4000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    24  100    24    0     0   8000      0 --:--:-- --:--:-- --:--:-- 12000
domoticz@odroid:~/domoticz/scripts/python$
When i do the script from putty.
I get this data....
Almost there???
Only when i try a other station my data is still the same.. grrr..

Re: Import RIVM sensor data

Posted: Tuesday 22 May 2018 22:14
by ErikAtSpijk
Hello Derik, not sure where is goes wrong, but what you see is the output of the lines:

command = ('curl -i "http://user:password@ip:8080/json.htm?t ... alue='+str(float(value[tel+1]))+'"')
output = subprocess.check_output(['bash','-c', command])

Because these lines are winthin the "while loop" it might suggest that some data is downloaded from luchtmeetnet and that the Nijmegen station is found. What you don't see in this output is the value it uploads. Suggest to add debug information like:

print tree at line 10
and
print value(tel) at line 15 and value(tel+1) at just underneath it.

Re: Import RIVM sensor data

Posted: Tuesday 22 May 2018 23:58
by Derik
Dear Erik...
Thanks

Only i do not understand what you mean..

Is the script ok?
Is there no data?
My dummys's are not working...
Is there perhaps someone that have this working?

Re: Import RIVM sensor data

Posted: Saturday 02 June 2018 21:29
by EdwinK
ErikAtSpijk wrote: ↑Monday 02 April 2018 21:30 Edwin, the RIVM website seems to stop, the website luchtmeetnet.nl seems to be the new site.

Too bad, I did like the older data, but guess need to live without it then :(

Re: Import RIVM sensor data

Posted: Wednesday 06 June 2018 23:30
by ErikAtSpijk
Derik, I guess you need to add debug information to the script in order to understand where it ges wrong.I can post a proposal later this week. The script works for me. (Fijnstof at Hoogvliet)
All times are UTC+01:00
Page 2 of 2

Powered by phpBB® Forum Software © phpBB Limited