dzvents: Gas meter - deviation between 'counter' and 'gasUpdate'

[jake]

Status:
The (virtual) gasmeter displays the value of 28.482 in the Utility tab. This is also the value that is visible in the 'devices' page.

When I run a dzvents script and print the gasmeter.counter value, it also outputs 28.482

However, when I run the command gasmeter.updateGas(gasmeter.counter + 1) to simulate gas usage of 1dm3, the whole meter is gone haywire, displaying the impossible high value. I was able to delete the record with sqlbrowser via this topic. Only then I realized that the updateGas command requires the 'raw' value in dm3, meaning that I had to change the command to gasmeter.update(gasmeter.counter * 1000 + 1) to reach an appropriate value of 28.483

So, please either update the wiki page, explaining that the .counter displays in m3 and the .updateGas in dm3, or change dzvents to make .counter and .updateGas the same units (in that case, please inform upfront, so I can remove the factor 1000 from my script )

[dannybloe]

Interesting indeed. Never noticed this. I will adapt the documentation.

[jake]

After logging for weeks to find the root cause of constant resetting of the gasmeters, I think I found a possible error:

The .counter value for a gasmeter jumps from total value to daily consumption (0), after the first update of that day.

I have to say that my 2 virtual gasmeters, where I split the P1 gasmeter in a meter for heating (CH) and hot water (DHW), are defined as gasmeter as well. Recently I saw that domoticz calls them P1 gas meter, while I only update the new total value to them. So, I don't update usage. That might be part of the issue/bug.

The attached Excel file with the data:
I log every minute the:
standard lua svalues value
dzVents.counter value

I did that for both virtual meters, plus the P1 gasmeter.counter as well to check the actual gas usage.

It can be seen in the file that the svalues value displays correct and updates correct 1 time per day, since just before the first update, both svalues and .counter were equal. However, my script uses the .counter value (new value = xxx.counter + usage), so at the next update it will be (0 + usage), usually resulting in a new LOWER total value. This makes the counter function in Domoticz go crazy with crazy high counter numbers.

Please check the values of 29-12-2017, since I repaired the database last nigh with sqlitebrowser to get rid of the low - high - low values in the gasmeter table.

[dannybloe]

I must say that I have a hard time following exactly what is going on here (still early I guess). But, can you give me some steps on how to reproduce or simulate this (and also with a script perhaps)? I have a P1 meter myself (but I don't know about hot water usage on my system (unfortunately)).

[jake]

I must say that I have a hard time following exactly what is going on here (still early I guess). But, can you give me some steps on how to reproduce or simulate this (and also with a script perhaps)? I have a P1 meter myself (but I don't know about hot water usage on my system (unfortunately)).

I can imagine that it is not easy to understand. I have the script to split the P1 meter reading into 2, but I now made a 2nd script (for easier understanding) where I simplify it to 1 meter, because the essence is the same. This script updates a new virtual P1 gas meter every time the original P1 gasmeter updates. And it updates it with the gas usage between actual and previous reading of the P1 gas meter.

I will run this script for a day or what to see if the behaviour with this simplied script is the same.

BTW: I see in my script (that I translated from a std LUA script) that I use the funcion 'round'

Code: Select all

function round(num, numDecimalPlaces) 
  return tonumber(string.format("%." .. (numDecimalPlaces or 0) .. "f", num)) 
end

In the dzVents Wiki I see that dzVents also has a function 'round'. Can those functions interfere?

[dannybloe]

Well, you should never define global functions in lua like this. Always prefix with local:

Code: Select all

local function round()
    --
end

The same applies for variables. Otherwise you infect the global scope and you surely will have undesired and unexpected side-effects some day.

But in this case it won't interfere.

And the round function in dzVents is much better than this one

[willemd]

The .counter value for a gasmeter jumps from total value to daily consumption (0), after the first update of that day.

I think this bug still exists.

1. I have the P1 gas meter. An increase of 0.021 is detected on that meter counter at 00:02 hours on 2022-09-03.
2. This triggers my script, which determines the increase is as result of hot water usage and it therefore tries to update the HW-gas meter with an increase of 0.021. For this it takes the original value * 1000 plus the increase * 1000 and calls the updateGas function of that meter. The original value was 0.832 so the new total should be 0.853.
3. A few minutes later the script is triggered again and the counter of the HW-gas meter shows 0.
4. The Meter table in the database shows 853 starting at 00:05 and then every 5 minutes until 03:00 (no new values after 03:00 until the seconds gas increase of 11:00 below). The Meter_Calendar table shows 0.832 for the 2022-09-02 (which is correct)
6. The Meter table shows a new value later in the day (at 11:00) equal to a second gas increase of 0.112, as-if it starts to count from 0 again.
7. The graph for the HW-gas meter shows a negative value for the current day of 0.741 (=0.112-0.853)

So, the meter table is correct between 00:05 and 03:00 but when querying the counter it returns a 0 value. This is the bug. After 03:00 no new records are inserted into the Meter table until a second increase later in the day, at which point the value of the increase is the new total (since it is added to the counter which is reported as 0).

Gas meters seem to be very buggy, see also my other bug report about gas meter graphs that I posted yesterday. I think I will switch to dummy counters instead and see whether that works better.

[willemd]

OK. I did some further analysis of this bug. It is definitely related to gas devices (point 2 below)

1) I have a real Gas P1 meter which shows in increase of 0.06 at 00:02 hrs.
This is shown as the daily total in the top right corner of the icon.
This is shown in the graph and related cvs at 00:00 hrs.
This is shown in the Meter table as an increase at 00:05:00 hrs and remains there with every further update.

2) This value is copied to a second GAS device.
The value is not shown in the top right corner (since not part of today's total)
This is shown in the graph and cvs at 23:00 the previous day.
This is shown in the Meter tables as an increase at 00:05:00 hrs and remains there until 03:00:00.
After 03:00:00 no new records are seen until a new value comes in later in the day, which is then shown as the new total. The 0.06 total disappears.

3) The value of point 1) is copied to a CUSTOM device.
The value is shown as part of the total in the top right corner of the icon.
It is shown in the graph and cvs as an increase at 00:05:00 hrs and remains there until 01:00:00
It is shown in the Percentage table at 00:05:00 hrs and remains there until 01:00:00
The new value that comes in later in the day is added to the total.

4) The value of point 1) is copied to a COUNTER device.
The value is part of the daily total shown in the top right corner of the icon.
It is shown in the graph and csv at 00:05:00 and remains there until 01:00:00
The new value later in the day is added to the total.

The code that does the insertions for point 2,3 and 4 is:

Code: Select all

    		            
    		          currentValue=domoticz.devices(idxHotWaterGas).counter
                        domoticz.devices(idxHotWaterGas).updateGas(increasegas*1000+currentValue*1000)  -- update value must be in dm3
                        currentValue=domoticz.devices(idxHotWaterGas2).sensorValue
                        domoticz.devices(idxHotWaterGas2).updateCustomSensor(increasegas+currentValue)  
                        currentValue=domoticz.devices(idxHotWaterGastest).counter
                        domoticz.devices(idxHotWaterGastest).updateCounter(increasegas*1000+currentValue*1000)  -- update value must be in dm3

The problem is that for point 2, i.e. the GAS device, when querying the currentValue in the script later in the day (after 03:00:00) it is reported as 0 and then as result the new value replaces the total instead of being added to the total.

For the update at 00:02:00 and the update at 10:01:00 later in the day the logfile shows the following device values before the update:
2) for the second GAS device at 00:02 a counter of 0.616 and countertoday of 0.367
and at 10:01 counter 0 and countertoday 0, so the 0.06 is no longer there.
3) for the CUSTOM device at 00:02 a value of 0.505 and at 10:01 a value of 0.564, so the 0.06 was added correctly
4) for the COUNTER device at 00:02 a counter of 0.424 and countertoday of 0.367 (which sould be 0 since the new day just started)
and at 10:01 a counter of 0.484 and countertoday of 0.06, so the 006 was added correctly (if countertoday was 0 to start with after all)

Also the icon of the GAS meter shows in the top right corner the value of the last update and shows the total day value in small digits in the icon, as a total.

So definitely some real problems with GAS device of point 2. (icon shows p1 smart meter,gas) whereas a standard counter device works correctly and also the custom device works correctly.

I will now remove and re-create the 3 gas devices so they all start at the same value.

[willemd]

The problem has not occurred again after I created 3 new test devices, although I still see the inconsistent timing issues as reported in this post. Sometimes the copied value onto test devices is shown in the graph one hour earlier than on the original GAS P1 device.

[willemd]

I might have found the issue in the code.

I saw in the log file a message "EventSystem: reset all device statuses...", after which suddely the counter of a manually created dummy GAS sensor was 0, while counterToday was still correct and the graphs still showed values as well.

The log file section is as follows (I left out non relevant log messages):
2022-09-07 11:53:16.685 Status: dzVents: Info: gas usage classification: Cooking Gas value 0.15
...
2022-09-07 11:53:16.690 Status: dzVents: Info: gas usage classification: ------ Finished gas classification
..
2022-09-07 11:54:00.376 Status: EventSystem: reset all device statuses...
2022-09-07 11:54:16.786 Status: dzVents: Info: Handling events for: "Gas P1 meter", value: "13894000.0"
2022-09-07 11:54:16.786 Status: dzVents: Info: gas usage classification: ------ Start internal script: gas classification: Device: "Gas P1 meter (Homewizard P1 meter)", Index: 2
2022-09-07 11:54:16.787 Status: dzVents: Info: gas usage classification: Device Gas P1 meter was updated
..
2022-09-07 11:54:16.790 Status: dzVents: Info: gas usage classification: Cooking Gas value 0.0
Note that the script did not do an update of cooking gas device.

I searched for this message in the source code and found it in EventSystem.cpp on line 422.

Looking further on line 451 I see an exception being made for incremental counters, BUT NOT FOR GAS COUNTERS (which is just a special type of incremental counter?)

I don't have sufficient knowledge to fully analyse the C-code but my impression is that some kind of reset is taking place (don't know what is triggering the reset) and that as result all devices that have not been updated for some (?) time are reset to zero. An exception is and should be made for incremental counters but this exception should also be made for dummy GAS counters. Instead these GAS counters are reset to zero.

Can anyone confirm this? Should I raise it as an issue on github?

Also one issue is that I don't always see this reset in the log file while the problem still occurs, maybe a similar action is done elsewhere in the code?

[willemd]

I will now create an issue report on github. The problem occurred again, without the "reset all device statuses". I collected all the evidence.

The problem actually is that the counter value of the virtual gas device is incorrectly set to 0, whereas this does not happen for a counter device of subtype gas. The icon display, the graphs, the database values are all still correct, only the computed counter value is wrong, so when this value is used for further updates in the day those updates will also be wrong.

Only a full domoticz restart brings the counter value back to the correct level.

[willemd]

Issue created on github

https://github.com/domoticz/domoticz/issues/5334