Updating dummy electric sensor over JSON shows incorrect data

[Roeldebest]

Hi all,

My first post on this forum after happily using Domoticz for quite some time already.
A couple of months ago we got solar panels installed and I have connected a Raspberry Pi to the inverter to read out the statistics of the current and daily production. So far, so good.

From this Raspberry Pi I'm calling the JSON API to set the value for this dummy sensor using the following URL:
http://192.168.178.23:8080/json.htm?typ ... e=22;15600

Where 22 is the current power of the solar panels and 15600 the total amount of energy produced today. When this API endpoint is called, I can see under the Devices tab that the dummy sensor indeed shows 15.600 kWh as the amount of energy being produced today. However, when I go to the Utilities tab, the Today section of the widget shows only 15.500 kWh. The current power is correct. Why does Domoticz seem to substract 100 wH here?

Device.PNG (6.37 KiB) Viewed 3535 times

Widget.PNG (9.8 KiB) Viewed 3535 times

Also, when I go to the graphs, in the Generated Last 24 hours graph, the amount of energy stays at 0 kWh during the day, while the power is again correct. However, the energy generated over the week is correct (even though there is 100 wH missing).

Graphs.PNG (74.49 KiB) Viewed 3536 times

The sensor is set to Energy read: From device.

Does anyone know a solution to both of these minor issues?

Thanks in advance.

[waaren]

From this Raspberry Pi I'm calling the JSON API to set the value for this dummy sensor using the following URL:
http://192.168.178.23:8080/json.htm?typ ... e=22;15600

Where 22 is the current power of the solar panels and 15600 the total amount of energy produced today.

The today's kwH is calculated by domoticz from the total KwH so you should send the 22;<whatever overall total you have>.
The first day will be wrong but after that it will be fine. Please note that it can take a couple of minutes before the daily total will be shown on the Utility tab.

[Roeldebest]

Thanks for your quick reply. Quite annoying that the inverter rounds floors it to kW so then I can't get really accurate data, but I will find a way to get around that.

[FireWizard]

Hello waaren,

What do you mean with:
"The today's kwH is calculated by domoticz from the total KwH"

I also noticed strange things in the log regarding the graphs.

I have the following sensor:

Screenshot_solar production sensor.png (50.33 KiB) Viewed 3512 times

It is configured as follows:

Screenshot_solar production sensor-config.png (27.61 KiB) Viewed 3512 times

This sensor is calculated by Node-Red and updated by MQTT with the following command:
msg.payload = {"command":"udevice","idx":279,"nvalue":0,"svalue":(msg.payload.wattsNow.toString()) + ";" + (msg.payload.wattHoursToday.toString())};

So the data is sent correctly and it updates the sensor.

However if you look into the log you will see that the Energy Production (wattHoursToday) is not logged.
See:

Screenshot_solar production sensor-log.png (232.2 KiB) Viewed 3512 times

You see that the energy production is not updated in the log. You will see it tomorrow, like you see the yesterdays Energy production at the left.
Another strange thing is that you see in the weekly overview that today's energy production is equal to yesterdays production, while the day has not finished yet. It is corrected tomorrow.

Perhaps there is an explanation.

Regards

[waaren]

What do you mean with:
"The today's kwH is calculated by domoticz from the total KwH"

Let me try to illustrate what I mean with an example
If I look in the database (copy from 2019-07-27 12:06) at the records for a device of this type, I see:

in table DeviceStatus, field sValue: 17608759;1090 -- Total wH ;Current Watt

table Meter (this table is updated every 5 minutes by a background process from values in DeviceStatus)
DeviceRowID|Value|Usage|Date
....
133|17608495|896|2019-07-27 11:50:01 -- ID | Total wH |Current Watt | Timestamp
133|17608553|757|2019-07-27 11:55:00
133|17608638|1056|2019-07-27 12:00:00
133|17608728|1145|2019-07-27 12:05:00

table Meter_Calendar (this table is updated daily by a background process from values in Meter)
DeviceRowID|Value|Counter|Date
...
133|27468|17479405|2019-07-21 -- ID | Daily Total |Overall total| DateStamp
133|29848|17509253|2019-07-22
133|26686|17535939|2019-07-23
133|5461|17561101|2019-07-24
133|24086|17585187|2019-07-25
133|21226|17606413|2019-07-26 -- 17606413 - 17585187 = 21226 ==>> Calculated Daily total

[FireWizard]

Thank you waaren for your reply and explanation.

I think that I understand your explanation. However I don't want to hijack Roeldebest's topic, but I believe that his issue is the same as I want to address. I understand how the tables are updated (5 minutes and daily), but I do not understand why in the "last 24h" view the bars are not updated during the day, but only the day after. Also I do not understand why in the "last week" overview the bar for the current day (which is not calculated yet) shows the same value as the previous day.

As you can see in the next screenshot I have 2 sensors of the same type, which gives more or less the same result:

Screenshot_solar production double-sensor.png (97.29 KiB) Viewed 3498 times

The first one (Solar Panel Production) gets its data from Node Red through MQTT.
Node queries an URL and receives:
{
"wattHoursToday": 11510,
"wattHoursSevenDays": 84054,
"wattHoursLifetime": 888225,
"wattsNow": 99
}
Node Red parses the data wattHoursToday and wattsNow and send these to Domoticz.

The other one receives the data from the Domoticz hardware device Enphase Envoy with LAN (HTTP) interface.
This polls another URL and receives the following data:
{
"production"
"0"
"type" "inverters"
"activeCount" 8
"readingTime" 1564244588
"wNow" 99
"whLifetime" 888225
}

The first one gives the following log:

Screenshot_solar log 1.png (241.21 KiB) Viewed 3498 times

while the second sensor (Enphase hardware) gives the next log:

Screenshot_solar log 2.png (243.34 KiB) Viewed 3498 times

The only difference is that the hardware Enphase is calculated by Domoticz as the URL gives only wNow.

I think that the log is always calculated the way you explained regardless of the JSOn input.
The JSON input is only shown in the sensors "Today" field.
Or is this sensor not the correct type for this function?

Regards and your contributions are appreciated very much.

[waaren]

The only difference is that the hardware Enphase is calculated by Domoticz as the URL gives only wNow.

I don't know in detail how the GUI deals with the information that has not yet made it to the database but if the database entries are similar I expect also similar graphs.

Can you please share the database records for these devices ?

from the CLI

Code: Select all

cp <domoticz dir>/domoticz.db workcopy.db
sudo sqlite3 workcopy.db
sqlite> .header on
sqlite> .tables
sqlite> select * from DeviceStatus where id in (<id1>, <id2>); -- replace <id#> by your idx
sqlite> select * from Meter where DeviceRowID in (<id1>, <id2>) order by date desc limit 20;
sqlite> select * from Meter_Calendar where DeviceRowID in (<id1>, <id2>) order by date desc limit 20;

[FireWizard]

Hello waaren,

I will do that and will post it on Monday.
Have a nice weekend.

Regards

[FireWizard]

Hello waaren,

As said earlier see below the results:

Code: Select all

pi@RPi2:~/domoticz $ sudo sqlite3 workcopy.db
SQLite version 3.16.2 2017-01-06 16:32:41
Enter ".help" for usage hints.
sqlite> .header on
   
sqlite> select * from DeviceStatus where id in (279);         
ID|HardwareID|DeviceID|Unit|Name|Used|Type|SubType|SwitchType|Favorite|SignalLevel|BatteryLevel|nValue|sValue|LastUpdate|Order|AddjValue|AddjMulti|AddjValue2|AddjMulti2|StrParam1|StrParam2|LastLevel|Protected|CustomImage|Description|Options|Color
279|41|00082279|1|Solar Panel Production|1|243|29|4|0|12|255|0|1667;8930|2019-07-29 15:57:11|254|0.0|1.0|0.0|1.0|||0|0|0||EnergyMeterMode:MA==|
sqlite> select * from Meter where DeviceRowID in (279) order by date desc limit 20;
DeviceRowID|Value|Usage|Date
279|8791|18080|2019-07-29 15:55:00
279|8640|18760|2019-07-29 15:50:01
279|8483|12040|2019-07-29 15:45:00
279|8382|8470|2019-07-29 15:40:00
279|8372|6170|2019-07-29 15:35:00
279|8326|4950|2019-07-29 15:30:00
279|8285|3670|2019-07-29 15:25:00
279|8255|1300|2019-07-29 15:20:01
279|8226|4110|2019-07-29 15:15:00
279|8226|4110|2019-07-29 15:10:00
279|8226|4110|2019-07-29 15:05:00
279|8193|320|2019-07-29 15:00:01
279|8187|13250|2019-07-29 14:55:00
279|8187|13250|2019-07-29 14:50:00
279|8076|18430|2019-07-29 14:45:00
279|7922|19850|2019-07-29 14:40:01
279|7756|19830|2019-07-29 14:35:00
279|7590|20610|2019-07-29 14:30:00
279|7418|12260|2019-07-29 14:25:00
279|7316|5490|2019-07-29 14:20:00
sqlite> select * from Meter_Calendar where DeviceRowID in (279) order by date desc limit 20;
DeviceRowID|Value|Counter|Date
279|11869|11869|2019-07-28
279|12936|12936|2019-07-27
279|12936|12936|2019-07-26
279|12925|12925|2019-07-25
279|13594|13594|2019-07-24
279|13594|13594|2019-07-23
279|10538|10538|2019-07-22
279|10004|10004|2019-07-21
279|10884|10884|2019-07-20
279|10884|10884|2019-07-19
279|12935|12935|2019-07-18
279|12935|12935|2019-07-17
279|5310|5310|2019-07-16
279|5417|5417|2019-07-15
279|6335|6335|2019-07-14
279|6335|6335|2019-07-13
279|6522|6522|2019-07-12
279|6522|6522|2019-07-11
279|11448|11448|2019-07-10
279|11448|11448|2019-07-09
sqlite>

I hope it is useful.

Regards

[waaren]

As said earlier see below the results:

The difference between your values and mine:
Yours: the sValue field of the DeviceStatus table are Daily total wH ;current W
Mine: the sValue field of the DeviceStatus table are Lifetime total wH; current W

That is why you see in your Meter_Calender table that Value and Counter fields are equal.
To correct this your nodered should not parse wattHoursToday and wattsNow and send these to Domoticz,
but parse wattHoursLifetime and wattsNow and send these to Domoticz.

The first day will not show you the right values but subsequent days should be OK.

Please let me know if this works for you.

[FireWizard]

Hi waaren,

I will change that and keep you updated.

Thanks for the support,

Regards

[FireWizard]

Hello waaren,

I think that your suggestion has worked for me.

In the next screenshots you will see some results:

Screenshot_Solar Production and Enphase.png (91.2 KiB) Viewed 3450 times

Just to remember, the right sensor is the Enphase "hardware" sensor.
The left sensor comes from Node-Red and MQTT. Node-Red polls a different URL then the Domoticz "hardware".
That explains the different production.

If we look to the logs.

Screenshot_Enphase-Log.png (251.71 KiB) Viewed 3450 times

Above you see the log of the Domoticz "hardware"sensor, which is correct and what is to expected.
Below you will see the log of the sensor that comes from Node_Red with MQTT.

Screenshot_Solar-Log.png (254.96 KiB) Viewed 3450 times

At first sight it looks that it doesn't work.
As you already mentioned the first day (in my case on Monday) the value in the log would not be correct.
That is true.
I has been able to delete the spike of the wrong value in the "Last Week" view with <Shift-Ctrl> + Left Mouse Button.
So therefore it is empty on Monday.
The <Shift-Ctrl> + Left Mouse Button did not work for the "Generated Last 24 Hours" view.
Because of the spike at 20:00 h the automatic scaling at the left Y-axis went sky-high and that means that the bars at the right do not show a significant change. However if you hoover over it with your mouse, you can see that there is data.

Any suggestion how to get the bar at 20:00 h removed? How to get the left Y-axis back to normal.
Or is this automatically achieved, when that bar is out of view tomorrow?

Once more, thank for you support and suggestions. It is appreciated.

Regards

[FireWizard]

Hi

This question has answered itself.
As soon as the bar disappears at the left side, the scale of the left y-axis returns to "normal".

Screenshot_Solar-Log2.png (249.28 KiB) Viewed 3437 times

Perhaps it has been useful for Roeldebest as well.
@waaren. Thank your very much for your suggestions and support.

Regards

[Wiley]

Wow, I've had this same problem forever and I just couldn't figure out what happened, because the device itself shows the right values. It's just the graphs that are wrong.
The documentation is so unclear about how to update the custom counter devices. One can notice that the project has been growing evolutionary instead of by design

So in summary, when using Electric (actual and counter) you should update with lifetime energy and not with daily (reset to 0) energy?
There's no device that will just display (in the graphs) current W and Wh that you update them with?

[SunHopper]

I have the same problem. I read here it has to do with sending lifetime KWh.
I'm using the same virual sensor device for loggin my Solar inverters.
The inverters sending the data current Watt and Day KWh with a JSON string over WiFi to Domoticz (server on Windows).
This works fine. The inverters start every day with 0KWh! The Weekly Graphs log's show the daily delivered power as expected.
I have made an Event Script which adding all for a total generated day KWh.
This value and the total live Watts are send to a new virtual electric device.
The KWh value's are wrong!!!
Why does it work Ok if I send data external and doesn't work if I send the value from an internal script?

[FireWizard]

Hi, @SunHopper,

Can you publish the script?

Regards

[SunHopper]

@FireWizard
I think I solved it.
Because the Inverters Switch of after Sundown the last KWh values are not set to 0 after midnight.
The last values are stil added to a total and send to the virtual device until the Inverters Switch on.
Because the inveters do not switch on on the same time the total KWh value isn't starting at 0.
I made a rule which sets the values of my 3 inverters to zero at midnight.
Tomorrow I know more.

if hour == 0 then
SetMeterkast.updateElectricity(0,0)
SetXS1.updateElectricity(0,0)
SetXS2.updateElectricity(0,0)
end

[SunHopper]

Working Ok for me now