capture and import smartmeter data from Web Solar Log

[pgit]

I use Web Solar Log (http://websolarlog.com) on a Raspberry Pi to monitor, gather and display data from my PV inverter (Omnik 3K-TL) using the Omnik-Data-Logger script https://github.com/Woutrrr/Omnik-Data-Logger and from my Dutch Smart Meter (ISKRA ME382) using an USB/serial cable attached to the P1 port.
This in itself works great and gives great insight into my solar energy production data and my household consumption of electricity and gas.

WSL also uploads the data to PVoutput, but in it's interpretation of what consumption is differs from mine. It graphs and uploads consumption as negative when production is higher than consumption. I want to see my actual consumption as how much energy my household is consuming, wether it be produced, imported or a combination of both, (hope this makes sense to you). Since negative figures are ignored by PVoutput this results in inacurate energy usage figures there.

Perhaps I should add that the smart meter records totals of imported energy and exported energy and power levels at the grid, but is not aware of my inverter's production.

I also run Domoticz on the raspi and of course I want to display my PV production and smart meter readings, mainly energy usage (and perhaps use that to trigger some automagic later on).I tried monitoring the smart meter from Domoticz simultaneously, but this disturbs WSL completely, making it sending out alarms every few minutes about not recieving data from the smart meter. I experimented a little with the polling rate, but I found out that it's just a bad idea to poll the meter from both WSL and from Domotics.

Since WSL does not have an option to query Domoticz, but does have a webpage that shows smart meter data (among others) that can be queried through JSON, I now want to use that to feed "virtual meters" in Domoticz with the elctricity and gas consumption data.
I (think I) managed to create a meters with virtual sensors for energy and gas, and I gues I will have to create a lua script (or two) now to query WSL and feed them. My problem is, I don't know how. I found some posts like http://www.domoticz.com/wiki/Capturing_ ... ua_Scripts that treat part of what I want, but I get confused and can't put two and two together.
I'm (still) a noob on lua and JSON, but know a (tiny) little bit about php and bash scripting.
Can I put everything in one lua script or do I need one for both meters? Can it only be used for updating Domoticz or also for querying WSL? Should it be a device- or a time script? Will I need a temporary file to store the values? etc.
What should be my next steps?

I also started a discussion athttps://groups.google.com/forum/#!topic ... Z9xS8rqw5E about where to retrieve what (smart meter) data to export to Domoticz.
It seems the WSL summary page provides some of the needed daily (actually current) values but not the "totals" and the power levels of the smart meter which I'd also like to export to the virtual meters in Domoticz:
- Total (lifetime) amount of imported electricity at low rate in kWh (1.8.1),
- Total (lifetime) amount of imported electricity at high rate in kWh (1.8.2)
- Total (lifetime) amount of exported electricity at low rate, in KWh (2.8.1)
- Total (lifetime) amount of exported electricity at high rate in KWh (2.8.2)
- Total (lifetime) amount of gas used in m3 (beneath 24.3.0)
- Current power usage in kW (1.7.0)
- Currently power returns in kW

My WSL page http://ip.adress.of.wsl/websolarlog/api.php/Summary/ will display:

{"data":{"metering":[{"lowReturn":0,"lowReturnCO2":0,"lowReturnTrees":0,"lowUsage":2.647,"lowUsageCO2":1164.68,"lowUsageTrees":43.14,"highReturn":2.33,"highReturnCO2":1025.2,"highReturnTrees":37.97,"highUsage":0.153,"highUsageCO2":67.32,"highUsageTrees":2.49,"gasUsage":0.439,"gasUsageCO2":0.97,"gasUsageTrees":0.04,"gasUsageCosts":" u20ac 0,29","returnKWH":2.33,"returnCO2":1.03,"returnCosts":" \u20ac 0,47","usageKWH":2.8,"usageCosts":" \u20ac 0,14","usageCO2":1232,"effUsageKWH":0.47,"effUsageCosts":" \u20ac 0,00"}],"weather":
[{"time":"1407495000","weatherSamples":70,"beaufort":3,"avgBeaufort":2,"avgTemp":18,"avgWindSpeed":3.2,"avgWindChill":21.5,"currentTemp":
21.5,"currentWindChill":0,"countTemp":70,"sunInfo":{"sunrise":1407471164,"sunset":1407525599,"transit":1407498382,"civil_twilight_begin":1407468766,"civil_twilight_end":1407527997,"nautica
l_twilight_begin":1407465592,"nautical_twilight_end":1407531172,"astronomical_twilight_begin":1407461402,"astronomical_twilight_end":1407535361},"minTemp":20.9,"maxTemp":22.4,"degreeDays":0.030000000000001,"windDirection":100,"humidity":"77","pressure":"1012","conditionId":"800","wind_speed":"4.6","rain1h":0,"rain3h":0,"clouds":"0"}],"production":[{"KWH":3.6,"costs":" \u20ac 0,72","CO2avoid":1.58,"trees":97.2}]},"total":{"metering":{"lowReturn":0,"lowReturnCO2":0,"lowReturnTrees":0,"lowUsage":2.647,"lowUsageCO2":1164.68,"lowUsageTrees":43.14,"highReturn":2.33,"highReturnCO2":1025.2,"highReturnTrees":37.97,"highUsage":0.153,"highUsageCO2":67.32,"highUsageTrees":2.49,"gasUsage":0.439,"gasUsageCO2":0.97,"gasUsageTrees":0.04,"gasUsageCosts":" \u20ac 0,29","returnKWH":2.33,"returnCO2":1.03,"returnCosts":" \u20ac 0,47","usageKWH":2.8,"usageCosts":" \u20ac 0,14","usageCO2":1232,"effUsageKWH":0.47,"effUsageCosts":" \u20ac 0,00"},"weather":{"time":1407495000,"weatherSamples":70,"beaufort":3,"avgBeaufort":2,"avgTemp":18,"avgWindSpeed":3.2,"avgWindChill":21.5,"currentTemp":21.5,"currentWindChill":0,"countTemp":70,"sunrise":1407471164,"sunset":1407525599,"transit":1407498382,"civil_twilight_begin":1407468766,"civil_twilight_end":1407527997,"nautical_twilight_begin":1407465592,"nautical_twilight_end":1407531172,"astronomical_twilight_begin":1407461402,"astronomical_twilight_end":1407535361,"minTemp":20.9,"maxTemp":22.4,"degreeDays":0.030000000000001,"windDirection":100,"humidity":77,"pressure":1012,"conditionId":800,"wind_speed":4.6,"rain1h":0,"rain3h":0,"clouds":0},"production":{"KWH":3.6,"costs":" \u20ac 0,72","CO2avoid":1.58,"trees":97.2},"totalUsagekWh":4.07,"totalUsageKWHCosts":" \u20ac 0,81","totalUsageKWHCO2":1.79,"totalUsageKWHTrees":66.3,"usedBeforeMeterKWH":1.27,"usedBeforeMeterCosts":" \u20ac 0,01","usedBeforeMeterCO2":0.56,"usedBeforeMeterTrees":20.7,"householdCO2":2.76,"householdUsage":2.8,"householdCosts":" \u20ac 0,56","householdTrees":102,"trees":102,"co2CompensationTree":27,"degreeDays":0.03,"m3PerdegreeDays":14.633,"costkwh":0.2,"costGas":0.65,"sunDown":false},"lang":
{"subscriptTrees":"nodig","usage":"verbruik","used":"gebruikt","gas":"gas","harvested":"Opgewekt","generated":"gegenereerd","trees":"bomen","power":"Vermogen","weather":"weer"},"timezoneOffset":2}


Below is a php script I made to retrieve and output the array and some of it's values:

#!/usr/bin/php
<?php
//$date=date +"%d-%m-%y";
$today = date ('mm-dd-Y');
$time = time ('H-i-s');
echo "date: " . $today . PHP_EOL;
echo "time: " . $time . PHP_EOL;
$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Summary/'. urlencode($today) . '");
//$json_string = file_get_contents("http://127.0.0.1/websolarlog/api.php/Summary/");
$parsed_json = json_decode($json_string, true);


$STDOUT = fopen("php://stdout", "w");
fwrite($STDOUT, print_r($parsed_json, TRUE));
fclose($STDOUT);

$parsed_json = $parsed_json['data']['weather'][0];
$time = $parsed_json['time'];
echo "Time ".$time."\n";

$parsed_json = json_decode($json_string, true);
$parsed_json = $parsed_json['data']['metering'][0];
$lowUsage = $parsed_json['lowUsage'];
echo "Low Import Today ".$lowUsage."kWh"."\n";

echo "Lifetime Low Import ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);
$parsed_json = $parsed_json['data']['metering'][0];
$highUsage = $parsed_json['highUsage'];
echo "High Import Today ".$highUsage."kWh"."\n";

echo "Lifetime High Import ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);
$parsed_json = $parsed_json['data']['metering'][0];
$usageKWH = $parsed_json['usageKWH'];
echo "Total Import Today ".$usageKWH."kWh"."\n";

echo "Lifetime Total Import ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);
$parsed_json = $parsed_json['data']['metering'][0];
$lowReturn = $parsed_json['lowReturn'];
echo "Low Return Today ".$lowReturn."kWh"."\n";

echo "Lifetime Low Export ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);
$parsed_json = $parsed_json['data']['metering'][0];
$highReturn = $parsed_json['highReturn'];
echo "High Return Today ".$highReturn."kWh"."\n";

echo "Lifetime High Export ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);
$parsed_json = $parsed_json['data']['metering'][0];
$returnKWH = $parsed_json['returnKWH'];
echo "Total Return Today ".$returnKWH."kWh"."\n";

echo "Lifetime Export Total ??? kWh"."\n";

$parsed_json = json_decode($json_string, true);
$parsed_json = $parsed_json['data']['metering'][0];
$gasUsage = $parsed_json['gasUsage'];
echo "Gas Usage Today ".$gasUsage."m3"."\n";

echo "Gas Lifetime Usage ??? m3"."\n";


?>

[gizmocuz]

i assume you know how to create a dummy hardware device, and dummy sensors, and post data from your php script to domoticz via the json command (udevice) as is explained in the wiki ?
So your problem is you want the right values from your p1 smart meter also in domoticz?

can't you use another p1 smart meter cable connected directly to domoticz, and put that cable in parallel with the p1 cable you already use ?

[pgit]

Yes, want the right values from my smart meter in domoticz.
I know how to create a dummy hardware device, dummy sensors, but thought I needed a lua script to post data to domoticz via the json command. I see now that the posting can be done from the (same) php script also, as you are sugesting,
Using another p1 smart meter cable won't be necesary, since both Domonticz and Web Solar Log are running on the same machine. The problem is however, that (I understood that) the smart meter will output all zero's for the data, if it is polled more then once within 10 seconds. I can adjust the polling rate in WSL, but don't know how to do that in Domoticz.
Either way, one will be "overtaking" the other at some point, so it's best to only have one system poll the meter physically and let the other system query for the gathered data.
It would be great if WSL had a way to import the smart meter data through json, then I would have Domotics poll the smart meter through a "real" device, but it doesn't.
Now I'm kinda stuck, because allthough WSL reads all of the smart meter data, I don't see where to "get" the "totals".
I was hoping somebody else allready managed to do this, or has some other way of getting the smart meter data in WSL and in Domotics "directly" (not through PVoutput) without messing up WSL...

[gizmocuz]

The smart meter is not 'polled', it is just 'sending' the status every 10 seconds, so a parallel cable would work.
from your current logger, if you get the 'counter' values, you can use these to send the data to a virtual energy/gas meter in domoticz

[pgit]

The guys from WSL told me the meter will not "respond" if it is"polled" more then once within a 60 second interval. Please see https://groups.google.com/forum/embed/? ... Z9xS8rqw5E ("First: it's not(!) recommended to let two devices poll the smartmeter. An smartmeter telegram could only be read once so, if the second device poll the smartmeter within 60 seconds after the last poll the smartmeter will return null.") Maybe that's inccorrect and the meter just only sends a telegram every x number of seconds. Then I don't understand why WSL loses communication with the meter, if I configure a "real" P1 meter in Domoticz also. Could it be that the OS won't let me use use /dev/USB0 from WSL and Domoticz simultaneously?
I don't have another serial/usb cable and I want to save the other USB port on my raspi. Wouldn't it be possible to create a "paralel" Serial/USB connection by creating some kind of "virtual" or "pseudo" port by redirecting (actually splitting) /dev/USB0 within the OS (instead of physically)?
The current logger in WSL can get the counter values. It has is a test option that shows the whole telegram. But I don't know if all counter values are saved in it's database and I don't know how to query the (now) missing ones.

[pepijn]

According to the P1 Protocol specifications you can use a 2 P1 cables in parallel. Just go to your local hardware store and buy a telephone splitter, order an P1 SmartMeter cable and small USB hub.

As Gizmocuz suggested, the best option without adding additional hardware is to send data from WSL to a Domoticz Virtual Sensor using JSON (or the other way around of course)

[pgit]

Thanks for the scheme.
I just got a reaction from the WSL developers that they are going to look into extending their API to allow for JSON queries on ALL of the smart meter data (and possibly other data also).
So i'll wait a bit before ordering extra hardware...

[sincze]

I tried to start with WSL, but as I have my Domoticz already installed I created some virtual devices and query the Omnik directly using CRON.

The basics are described in https://github.com/Woutrrr/Omnik-Data-Logger.
I modified the script to fill the appropriate devices.

Would be nice if this feature would implemented by default in Domoticz.

[mschut]

Ow wow, this would be great, to monitor my Omnik datalogger in Domoticz, I agree with sincze

Would be nice if this feature would implemented by default in Domoticz.

.

I notice a wiki page, but since I am a bit of a noob (compared to tweaking in code like people), and I have Domoticz running on my NAS, I am not sure if it's a good idea to try this on my installation.

Does anyone think this would also be possible? Or is this very very maybe coming as a "standard hardware/monitor" option soon?

Regards

[sincze]

Ow wow, this would be great, to monitor my Omnik datalogger in Domoticz, I agree with sincze

Would be nice if this feature would implemented by default in Domoticz.

.

I notice a wiki page, but since I am a bit of a noob (compared to tweaking in code like people), and I have Domoticz running on my NAS, I am not sure if it's a good idea to try this on my installation.

Does anyone think this would also be possible? Or is this very very maybe coming as a "standard hardware/monitor" option soon?

Regards

Your NAS should be able to deal with cronjobs I would think. However a default HW device always very nice.

[natnek]

Hi to all

I am trying also to get the Omnik data to domoticz and followed the adjustment instuctions but the script that i use gives me an error can someone help me with this one ?
python OmnikExport.py
Gives me

Traceback (most recent call last):
File "OmnikExport.py", line 121, in <module>
response = urllib2.urlopen(request_object)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 399, in open
req = meth(req)
File "/usr/lib/python2.7/urllib2.py", line 1110, in do_request_
raise URLError('no host given')
urllib2.URLError: <urlopen error no host given>

[ThinkPad]

It expects an address/URL, but you didn't provide the script that address, so it complains it can't find the host.

[natnek]

Well

What will be the url of the domoticz host then?

I provide a Ip adress of my local host (domoticz) in the config file

################
### Settings ###
################

[domoticz]
domoticz_enabled = true
domoticz_host = 192.168.1.13
domoticz_port = 8080
domoticz_url = json.htm

# Provide IDX here of the specific devices
domoticz_temp = 53
domoticz_Input_PV1 = idx
domoticz_Input_PV2 = idx
domoticz_AC_Output = idx
domoticz_Cur_Total = idx
domoticz_AMP_1_2_3 = idx

[kaatje]

Did you get it 2 work?, Im getting the same error..meaning put the host in the config file but the script returns an error


Ive got similar settings as above here's the part of the script that gives no host given error

# --------------------------------------------------

if domoticz_enabled:
url = ("http://" + domoticz_host + ":" + domoticz_port + "/" + domoticz_url)

if use_temp:
get_data = {
'svalue': msg.getVPV(1),
'type': 'command',
'param': 'udevice',
'idx' : domoticz_Input_PV1,
'nvalue': '0'
}

else:
get_data = {
'svalue': msg.getVPV(1),
'type': 'command',
'param': 'udevice',
'idx' : domoticz_Input_PV1,
'nvalue': '0'
}

get_data_encoded = urllib.urlencode(get_data)
request_object = urllib2.Request(url + '?' + get_data_encoded)
response = urllib2.urlopen(request_object)

# --------------------------------------------------


raceback (most recent call last):
File "/home/pi/domoticz/scripts/Omnik-Data-Logger/OmnikExport.py", line 148, in <module>
response = urllib2.urlopen(request_object)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 399, in open
req = meth(req)
File "/usr/lib/python2.7/urllib2.py", line 1110, in do_request_
raise URLError('no host given')
urllib2.URLError: <urlopen error no host given>

[natnek]

Nope

still looking for a solution

[kaatje]

my settings
[domoticz]
domoticz_enabled = true
domoticz_host = 192.168.1.100
domoticz_port = 8080
domoticz_url = json.htm

# Provide IDX here of the specific devices
domoticz_temp = idx
domoticz_Input_PV1 = idx
domoticz_Input_PV2 = idx
domoticz_AC_Output = idx
domoticz_Cur_Total = idx
domoticz_AMP_1_2_3 = idx

(my settings are all idx your temp = 53? )

[kaatje]

I discovered the error is gone if you put the values directly in the script (omnikexport.py), however i still dont see any data in domoticz

like this
# Receive data with a socket
ip = config.get('inverter','ip')
port = config.get('inverter','port')
use_temp = config.getboolean('inverter','use_temperature')
wifi_serial = config.getint('inverter', 'wifi_sn')

mysql_enabled = config.getboolean('mysql', 'mysql_enabled')
mysql_host = config.get('mysql','mysql_host')
mysql_user = config.get('mysql','mysql_user')
mysql_pass = config.get('mysql','mysql_pass')
mysql_db = config.get('mysql','mysql_db')

domoticz_enabled = 'true'
domoticz_host = '192.168.1.10'
domoticz_port = '8080'
domoticz_url = 'json.htm'

[natnek]

idx 53 was the idx number i had made in domoticz for temp

I replaced idx for the id number.

[kaatje]

I got it to work, did you? idx values in config. and host port url directly in script, only not sure what to put in virtual sensor omnik ampere is it a voltage and do i put 3 idx values combined?

[natnek]

Good job

Can you place the code of the config.cfg and Omnikexport.py ? so i can see the right format