counterToday and counterDeliveredToday [SOLVED]

[JanvdW]

I made a dxvents script which uses counterToday and counterDeliveredToday to calculate the energy costs. This script is executed every 15 minutes. It appeared that counterToday and counterDeliveredToday get the values of the previous day till 00:15, which messes up my graphs.

Investigating this issue I saw another strange thing: despite the dzvents script that calculates the costs is scheduled every 15 minutes, the Domoticz database keeps values for every 5 minutes. I don't understand why?

[waaren]

I made a dxvents script which uses counterToday and counterDeliveredToday to calculate the energy costs. This script is executed every 15 minutes. It appeared that counterToday and counterDeliveredToday get the values of the previous day till 00:15, which messes up my graphs.

Investigating this issue I saw another strange thing: despite the dzvents script that calculates the costs is scheduled every 15 minutes, the Domoticz database keeps values for every 5 minutes. I don't understand why?

If you report a bug please state your version and include your full script and relevant loglines.

Domoticz way of storing history data is to take the current sensor data and store this in the short term history tables every 5 minutes. Once a day the long term history tables are updated using the data from the short term history tables.

[JanvdW]

Right, that explains the update every 5 minutes. I am sorry, hereby the additional info. I am running version 2020.2 (build 12067). It's about the following script:

Code: Select all

return { on = { timer = { "every 15 minutes", }, }, 
    execute = function(domoticz, device, timer)
        -- Parameters huidige energiecontract
        local Normaaltarief = <normal tariff>
        local Daltarief  = <low tariff>
        local kwhPrijsVast = <fixed costs in Euro's per day>
        local gasM3Prijs = <gas price in Euro's / M3)
        local gasM3PrijsVast = <fixed costs in Euro's per day>
        local Korting = <discount per year>

        --Select tariff, calculate and store costs
        local kwhPrijs = Normaaltarief
        if (domoticz.time == 'Between 23:00 and 07:00') or (domoticz.day == 'Saturday') or (domoticz.day == 'Sunday') then
            kwhPrijs = Daltarief
        end

        local kwhKosten             = tonumber(domoticz.utils.round((kwhPrijs * (domoticz.devices(120).counterToday - domoticz.devices(120).counterDeliveredToday)) + kwhPrijsVast,2))   
        local GasM3Kosten           = tonumber(domoticz.utils.round((gasM3Prijs * domoticz.devices(127).counterToday) + gasM3PrijsVast,2))
        local OpbrengstSolar        = tonumber(domoticz.utils.round((kwhPrijs * domoticz.devices(23).counterToday),2))
        local Kostentotaal          = tonumber(domoticz.utils.round((kwhKosten + GasM3Kosten - Korting/365),2))

        domoticz.devices('Stroomkosten').updateCustomSensor(kwhKosten)      
        domoticz.devices('Gaskosten').updateCustomSensor(GasM3Kosten)
        domoticz.devices('Solaropbrengst').updateCustomSensor(OpbrengstSolar)
        domoticz.devices('Energiekosten').updateCustomSensor(Kostentotaal)
        domoticz.devices('Virtuele kosten energiegebruik').updateCustomSensor(Kostentotaal + OpbrengstSolar)
    end
}

Around midnight this results in the following entries in the domoticz database (Percentage table) for device 166 (similar entries for the other custom sensors that store the energy costs):

Code: Select all

166	3.27	2020-05-22 23:50:00
166	3.27	2020-05-22 23:55:00
166	3.27	2020-05-23 00:00:00
166	3.3	2020-05-23 00:05:00
166	3.3	2020-05-23 00:10:00
166	3.3	2020-05-23 00:15:00
166	-0.08	2020-05-23 00:20:00
166	-0.08	2020-05-23 00:25:00

Note: the negative costs on 00:20 and 00:25 are caused by a negative fixed price, so this is expected.

[jvdz]

I made a dxvents script which uses counterToday and counterDeliveredToday to calculate the energy costs. This script is executed every 15 minutes. It appeared that counterToday and counterDeliveredToday get the values of the previous day till 00:15, which messes up my graphs.

Doesn't this make sense when you run the script every 15 minutes.
The execution at midnight (00:00:00) will likely still contain yesterday's data and the run at 00:15:00 will contain today's data, but is finished after Domoticz took the 00:15:00 value for that sensor.
you just need to make sure the sensor is updated before the 5 minute mark.

[JanvdW]

Right, probably that causes the issue. I'll try if updating at 00:03 solves the issue. But even in that case I suppose that the value on 00:00 will cause that the dashticz graphs will show the totals of yesterday as the today value (as long as they are higher), right? Do you have a suggestion how to fix that?

[waaren]

Right, probably that causes the issue. I'll try if updating at 00:03 solves the issue. But even in that case I suppose that the value on 00:00 will cause that the dashticz graphs will show the totals of yesterday as the today value (as long as they are higher), right? Do you have a suggestion how to fix that?

counterToday is not a stored value but calculated with formula

currentTotal - ( Last stored total ( from 23:55:00 previous day ) ) -> at 00:00 if the script started sooner then domoticz history aggregate function you will see yesterdays..

So you could either do the math in the script using persistent data or (probably easier ) skip the 00:00:00 script using

if domoticz.time.matchesRule('at 00:00-00:01') then
return
end

[jvdz]

Let's think about this a little: Don't you want the last value, taken for any sensor on any day, be the correct total value for that day? In this case the total usage of Energy.
The last snapshot for a day is at 24:00/00:00 so that needs to be the total for that day and then the value in the 00:05 slot should contain the value for the usage between 00:00-00:05. Agree?
I am not using dashticz graphs so can't help with that.

Jos

[JanvdW]

You're right I suppose, Jos, but missing some minutes of a day is negligible. The problem is that the value at 00:00 is seen as the first value for the next day. So, I think that the best thing I can do is to reset the values to '0' at 23:59. I'll try if that works.

[jvdz]

Normally for Energy counter, no data is "lost" as you feed Domoticz every time with the latest total counter value of the P1 and Domoticz calculates the difference with the previous value to come up with the consumption.
Is that not what you see?

Jos

[JanvdW]

Yes, that's right. The data for the energy meter is fine; it's reflecting the valid data on the right day.

Based on these values my script calculates the corresponding costs in Euros. My problem is that the costs of 24:00/00:00 is so to say the value of 24:00 that is stored as if it is the data of 00:00. In other words, the last value of yesterday, is stored as the first value of today. Which ruin my Dashticz graphs of course. So I prefer to store '0' values at 00:00, which (for the costs involved) causes data loss of several minutes.

[jvdz]

Just add a test for 00:00 and set that value simply to 0?

Jos

[JanvdW]

Yes, indeed; that's what I did. Waiting now

[waaren]

Yes, indeed; that's what I did. Waiting now

Any news on this ?

[JanvdW]

Yes, I tested different scenarios and optimized the script. I am content with my current solution. The script runs every quarter and around midnight at 23:54, 23:59 and 00:04. The run at 23:54 updates the sensors with last today values for storage in the Domoticz short term history table at 23:55. Runs at 23:59 and 00:04 stores fixed costs, without values for counterToday and counterDeliveredToday. This will be stored in Domoticz short term history table at 00:00 and 00:05. So, my problem is solved.

[Martini77]

I still find this to be a bug. I’m using the Enphase solar panel plugin and like to use the counterToday value to calculate the actual consumption of energy for today (P1 usage + solar production - P1 delivery).
From midnight till the first moment any energy like 0.001 kWh is being produced i get the counterToday for production being yesterday’s value..

It doesn’t seem right to manipulate the Solar production values.

Also have this issue for Water measurements (device type = RFXMeter counter). When on holiday and no water is being used/reported, I get counterToday being the last value the day I consumed some water. These daily usages I report to myself via a script via Telegram message.