Ignoring bad value in kwh meter

[goldorak]

Hi Domoticz team,

I'm monitoring electric consumption of some of my equipment with an OWL CM device.

Sometimes, it gave totally erratic value, like that :
C2-Pompe-a-chaleur Usage is 1054889 [> 6000.0 ]

So I can have notifications of these wrong values (like above (6000 is for 6kwh)), and that's great.
But I have to manually modify the sql database to remove these wrong values, in order to keep my consumption above months and years real.

A great feature could be to be able to ignore a value if it's above a max defined by user ?

Anymway, I'm using Domoticz for around 5 years, and it's still great to use !
Thanks for all the good work !
Kind Regards
Benoit.

[kimhav]

Agree, would be nice to be able to edit specific values and since we can view the table it would be nice to have a edit button for each registered value so it would be a simple click to edit of a specific value. So similar to this previous suggestion.

[erem]

Being a (very) old software developer, i remember a concept called "exit points".
basically, this is an initially empty routine called just before the save operation.
the user can then populate this routine with logic to create the desired result.
this can be implemented in various ways. up to the developer.

just my €.02

[lost]

Agree, would be nice to be able to edit specific values and since we can view the table it would be nice to have a edit button for each registered value so it would be a simple click to edit of a specific value.

Hello,

You can already remove manually a bad point in the DB from the graphs in the UI: That's a shift+click the point job.
Anyway, would be nice to be able to discard values even before they reach the DB by setting a max value for instance. This may already be done, just setting the hardware reponsible for the measurement to unused & having a device script that feeds a virtual sensor of the right type and can filter out impossible values if needed.

I wrote something similar for my home electrical power measurements done by a qubino smart-meter: Did this to log the max power reached for 10mn time-slices and remove some bad values (about 1 per week) that made me believe I should setup a nuclear plant in my garden ;o). Can send the code when back home if this may help.

Regards

EDIT: Code sample dump:

$ cat domoticz/scripts/lua/script_device_SmartMeterPeak.lua

Code: Select all

-- Smart-Meter (per 10mn/600s default time slice max) peak power consumption
-- Only use slices > 5mn Domoticz log/graph record time in DB or some max will
-- be overwritten!
--
-- Changelog:
--   11/01/2019, YL, 1st version.

-- User editabe settings:
devSmartMeterPwr='Smart-Meter W' -- Real PWR consumtion measurement device name
devPeakPwrVrtSen='Puissance MAX' -- Virtual sensor type "electricity" name
maxLastUpdateVrt=600 -- Store current power value if last max record is too old...
maxPeakPwr=13000     -- Device limit is 63A, measures exceeding are filtered.

commandArray = {}

--
-- Internal fct :
-- Get current script exec time & compute last 'device' update time.
--
local function timeLastUpdate(device)
    t1  = os.time()
    sT0 = otherdevices_lastupdate[device]

    -- Returns a date time like 2016-12-02 15:30:10
    -- => Format as os.time & compute diff :

    year    = string.sub(sT0, 1, 4)
    month   = string.sub(sT0, 6, 7)
    day     = string.sub(sT0, 9, 10)
    hour    = string.sub(sT0, 12, 13)
    minutes = string.sub(sT0, 15, 16)
    seconds = string.sub(sT0, 18, 19)
    t0      = os.time{year=year, month=month, day=day, hour=hour, min=minutes, sec=seconds}
    tDiff   = os.difftime(t1, t0)

    return(tDiff)
end

--
-- Update virtual sensor
--
local function updateNum(dev, value1)
    local cmd = string.format("%d|0|%d", otherdevices_idx[dev], value1)
    table.insert (commandArray, { ['UpdateDevice'] = cmd } )
end

--
-- MAIN
--
if (devicechanged[devSmartMeterPwr]) then
    curPwr = tonumber(devicechanged[devSmartMeterPwr])
    
    -- Immediately log & return on bad pwr figures...
    if (curPwr > maxPeakPwr) then
        print('WARNING : '..devSmartMeterPwr..' reports '..curPwr..'W !!!')
        return commandArray
    end
    
    pkPwr  = tonumber(otherdevices[devPeakPwrVrtSen])
    tLastPk= timeLastUpdate(devPeakPwrVrtSen)
        
    -- Debug :
    --print(devSmartMeterPwr..': P='..curPwr..'W ('..type(curPwr)..')')
    --print(devPeakPwrVrtSen..': P='..pkPwr..'W ('..type(pkPwr)..') ; Last : '..tLastPk..'s')
    
    -- Peak Pwr is stored rounded...
    if (curPwr > (pkPwr + 1)) or (tLastPk > maxLastUpdateVrt) then
       --Debug :
       --print('Update '..devPeakPwrVrtSen..' to '..curPwr..'W')
       updateNum(devPeakPwrVrtSen, curPwr)
    end
end

return commandArray

As you can see, on bad pwr figures (exceeding device limit, 63A *220V ~ 13kW) script returns commandArray unchanged while if current PWR is over current time slice last max, virtual device (to create separately as a virtual sensor) is updated as this was my need to log true max pwr reached (to compare vs my electricity provider contract).

Domoticz log powers every 5mn, so this gives an idea of power usage vs time but most of the times this misses true max that may not last for long. Max are often underestimated by 3KW in default domoticz logs. For a 9KW contract, that's huge.

You may be able to modify this script for your own needs (a few debug prints can be commented out for debug purpose using domoticz logs).

[goldorak]

Whaow, I discover this feature of shift+click to remove a value...
That's great, and respond to my first need !

Thanks a lot !

Hello,

You can already remove manually a bad point in the DB from the graphs in the UI: That's a shift+click the point job.

[kimhav]

You can already remove manually a bad point in the DB from the graphs in the UI: That's a shift+click the point job.
Anyway, would be nice to be able to discard values even before they reach the DB by setting a max value for instance. This may already be done, just setting the hardware reponsible for the measurement to unused & having a device script that feeds a virtual sensor of the right type and can filter out impossible values if needed.

Yes, aware one can remove data entries in the graph problem is that one then removes the entire value for the day and it would be absolutely so much better if it would be possible to add a logical rule as suggested. Also would be quite nice if there also was a setting to either use an average value instead for a certain time period or just that the system should use the latest incoming value.

So pushing again for this one since I once a while get erroneous values on the 3 meters that I have.

[ld57]

Hi!
Any news on this? For example with cacti, you can configure min-max valid data values.
Thanks

[kimhav]

Badly needs this one as as you mention just being able to define a certain delta which a value should be within or just a percentage would be of help to avoid having crazy value to be populated. If the input value, let's say differs more than +/- 200% then it would either use the 1) previous recent value or an 2) average value.

[waltervl]

For Zwave devices, some are known to send illegal values so a filter has been setup to ignore those. This max Power Value can be set in Menu Setup Settings, tab Meters/Counters. See also the Wiki https://www.domoticz.com/wiki/Applicati ... ters_Setup

[kimhav]

In my case it's 433MHz related devices and not ZWave.

[waltervl]

Perhaps you should look at your device because these 433 devices are not known to give strange values.
It also would not harm to extend the Max Power limit to other hardware type devices but then the source code has to be modified.
I do not know the reason why it is only limited to Zwave.

[kimhav]

With regards to the 433MHz devices that I'm using as pulse meters for electricity meters aren't the issue. The issue that the 433Mhz is sensitive for interference which makes the data transmitted go of the charts. Among other previously I noticed that the sparks plugs used in the garage tend to mess with the 433MHz sensors.