Median filter for temperatures

[mpx2]

Is there a simple way to add median filter for temperature sensors? My 433 MHz sensors someties report erroneous values, very high or very low. I would like all my sensors to use the median value of the last three measurements, like median(t0,t1,t2).

[waltervl]

No, the values are directly uploaded in the device. You can make a dummy temperature device for every real temperature sensor and with a dzvents script that can do the filtering or calculations and fill the dummy sensor with the filtered data from your real sensor.

[gizmocuz]

@mpx2, throw away that temperature sensor? If it reports false values it is of no use.
If your device is to far away from the 433Mhz transceiver, try a bigger antenna

[mpx2]

@mpx2, throw away that temperature sensor? If it reports false values it is of no use.
If your device is to far away from the 433Mhz transceiver, try a bigger antenna

I have Telldus 313160 (FT007TH) temperature/humidity sensors viewtopic.php?p=212188#p212188 .

Erroneous temperatures are quite rare but those are bad for plots and statistics. This was one of the coldest days in last year but the highest received temperature is +186.5 C.

wrong temperature.png (25.19 KiB) Viewed 1471 times

I had an older version of RTL_433 (https://github.com/merbanan/rtl_433). I updated to the lastest version. Let's see if this helps.


There's a CRC error detection used in temp/humidity reports but I don't know how these are handled. I don't have any logs when erroneous values are sent.

rtl_433.png (13.52 KiB) Viewed 1471 times

I tested this DzEvents script. A new value is received and updated to the sensor but then this scripts will calculate the median value and updates it to the sensors.

Code: Select all

return {
	on = {
		devices = {	465 } -- idx of the sensors
	},
	data = {
            temperatures = { history = true, maxItems = 3 },
            humidities = { history = true, maxItems = 3 },
            medianT = { initial = 0 },
            medianH = { initial = 0 },
    },
	logging = {
		level = domoticz.LOG_DEBUG,
		marker = 'template',
	},
	execute = function(domoticz, sensor)
	    -- add new data
        domoticz.data.temperatures.add(sensor.temperature)
        domoticz.data.humidities.add(sensor.humidity)
        -- calculate median
        domoticz.data.medianT = domoticz.data.temperatures.med()
        domoticz.data.medianH = domoticz.data.humidities.med()
        -- update median values to sensor. silent() does not trigger the script again causing a loop.
	    domoticz.devices(sensor.id).updateTempHum(domoticz.data.medianT, domoticz.data.medianH).silent()
	    -- domoticz.devices(sensor.id).updateTempHum(-10, domoticz.data.medianH).silent()
	    -- debug
		domoticz.log('New temperature value is ' .. sensor.temperature .. ' . Median value is ' .. domoticz.data.medianT .. ' .' , domoticz.LOG_INFO)
		domoticz.log('New humidity value is ' .. sensor.humidity .. ' . Median value is ' .. domoticz.data.medianH .. ' .' , domoticz.LOG_INFO)
		if (domoticz.data.median ~= sensor.temperature) then
                domoticz.log('New temperature value is not Median value.', domoticz.LOG_INFO)
        end

	end
}

For testing I used the line "domoticz.devices(sensor.id).updateTempHum(-10, domoticz.data.medianH).silent()" . Every minute temperature 21 C was received and then replaced with a temperature -10 C.

Short spikes of 21 C are not shown in plots.

Filtered-test.png (39.13 KiB) Viewed 1471 times

[lost]

There's a CRC error detection used in temp/humidity reports but I don't know how these are handled.

If CRC is reported, IMO it's not ignored & such invalid data is already discarded. I also have some zwave sensors that are secure-included (so I'm pretty sure any altered message will be discarded) that sometimes send 0°C values, usually for a few 10's of minutes so several values to manually discard.

So this one may not be handled by some easy range filtering (0°C is not uncommon temp in winter!) as this was done for power-meters that allows you to setup a max consumption & easily discard high erroneous values (sometimes up to need a nuclear power plant in my garden, visually flattening whole curves with a single value).

The issue I sometimes observe with my device may also fool some statistics based filtering as several erroneous values may influence a mean for instance.

=> Not easy to have a setup that may apply to any buggy hardware + such radio devices may sometimes not be able to reach the controller due to range/perturbations issues: If that's a few hours between night and day, you may then filter values that finally make their way on statistics that do not make sense anymore & get trapped there until temp reports match the old stats, so last valid data should also be a tunable to reset your stat based filter. And any such tunable should be device specific, of course! Aspirin needed!

[mpx2]

I compiled the latest code for rtl_433 and added the same sensor by RFLink. rtl_433 shows an erroneous temperature value but RFLink does not show it.

rtl_433-vs-RFLink.png (21.64 KiB) Viewed 902 times

[gizmocuz]

It might be an issue with rtl_433 then...

But you cannot compare a RFXCom transceiver (or RFLink) with a USB stick that was not designed for supporting weather sensors.

I do think it's a great project! But I know that RFXCom does some sensible checks inside

This is also not a Domoticz issue, it just logs what its reported.

[lost]

No, the values are directly uploaded in the device. You can make a dummy temperature device for every real temperature sensor and with a dzvents script that can do the filtering or calculations and fill the dummy sensor with the filtered data from your real sensor.

For data like temp/hum that does not change quickly vs 5mn Domoticz sampling period, filtering is only needed to handle sometimes misbehaving sensors. But, as I already said in the past, some calculate options (like a discrete integration based average) would be helpful for sensors getting data that can vary a lot shortly (as a power-meter): Currently, Domoticz graphs for such sensors does not show true min/max or 5mn average.

[waltervl]

That can be done by dzvents script with persistent data analysis that is triggered on every device update it receives. So not limited to the 5 min delta in the database.

[lost]

@waltervi:
I know & this is what I did (in pure Lua as I my brain never managed to fit with dzVent syntax), but that's not so straightforward just to double store values that makes sense (true min/max/time slice average) compared to the basic sampling done. Not useful for each sensor, but storing current values is almost useless as they do not reflect anything for sensors with quick variations.
Another issue is the update & event handling: Lua table update or http API behave differently, especially if you want to have a device script attached to the virtual meter getting overpower instantly to manage SW offloading for instance.
Believe me, handling this from the outside of domoticz core needs some tricks (+pure Lua does not allow persistent data)! I already published this in the past.
+ you need a device script, so not precise slice possible: On my side I use a 10mn one (any over 5mn would do) but that's only a target that'll always be exceeded, as only 1st measure received after target can stop current slice: So we are sit between the 2 chairs of device and time!

[waltervl]

Get over the syntax and use dzvents..... Every device update is triggered so for example a sudden overpower will be noticed as soon as device data has been sent to Domoticz.

But it is still domotica and not real time processing. Then you need other hardware than for example an RPi..

[lost]

dzVent being based on Lua, I think we have the same issue there: Update using table will not trigger event system for another device script to be able to notice. Only using HTTP API will, and also not using domoticz OpenURL that'll have side effect.

Here is my update function that permits both depending on use case (peak power needs immediate update as I have SW offloading and typically 30s to 1 mn, depending on power load excess vs provider contract, to go back in the limits or whole home is shutdown!):

Code: Select all

--
-- Update virtual sensor (using normal 'UpdateDevice' or http/JSON if useHttpApi set to true),
-- take care only useHttpApi will trigger other device script(s) on a virtual sensor/meter update.
-- Uses urlDz as global base URL variable for Domoticz (f.i: urlDz='http://127.0.1.1:8080').
--
local function updateNum(dev, value1, useHttpApi)
    if (useHttpApi == true) then
        local cmd = string.format(urlDz..'/json.htm?type=command&param=udevice&idx=%d&svalue=%d', otherdevices_idx[dev], math.floor(value1))
        --print(cmd) -- DEBUG

        -- OpenURL (being async) may trigger multiple events if base HW retriggers before API call is processed => use curl.
        -- table.insert (commandArray, { ['OpenURL'] = cmd } )

        local handle = assert(io.popen('curl --retry 1 --connect-timeout 1 -m 3 \"'..cmd..'\"'))
        result = handle:read('*all')
        -- Returned JSON format : {"status" : "OK","title" : "Update Device"}
        handle:close()

        if not string.find(tostring(result), "OK") then
            print('ERROR : '..cmd)
            print('ERROR : updateNum/useHttpApi returned '..tostring(result))
        end
    else
        local cmd = string.format('%d|0|%d', otherdevices_idx[dev], math.floor(value1))
        --print(cmd) -- DEBUG
        table.insert (commandArray, { ['UpdateDevice'] = cmd } )
    end
end

And both cases use case in my real peak and average power virtual sensors feeding logic:

Code: Select all

-- Smart-Meter Mean/Peak :
if (devicechanged[devSmartMeterPwr]) then
    curPwr = tonumber(devicechanged[devSmartMeterPwr])

    -- Immediately log & return on bad pwr figures...
    if (curPwr > maxPeakPwr) then
        print('WARNING : '..devSmartMeterPwr..' reports '..curPwr..'W !!!')
        return commandArray
    end

    -----------------------
    -- Update Peak power --
    -----------------------

    pkPwr  = tonumber(otherdevices[devPeakPwrVrtSen])
    tLastPk= timeLastUpdate(devPeakPwrVrtSen, false)

    -- DEBUG :
    --print(devSmartMeterPwr..': P='..curPwr..'W ('..type(curPwr)..')')
    --print(devPeakPwrVrtSen..': P='..pkPwr..'W ('..type(pkPwr)..') ; Last : '..tLastPk..'s')

    -- Peak Pwr is stored rounded...
    if (curPwr > (pkPwr + 1)) then
       -- Log all intermediate peaks (with events trigger to allow overload management)
       updateNum(devPeakPwrVrtSen, curPwr, true)
    else
       -- Log last slice peak + "reset" new intermediate peak
       -- to current pwr (without events trigger) if slice time exceeded.
       if (tLastPk > maxLastUpdPeak) then
           print(string.format('%s : New time slice (since %ds) => Got Pmax=%.2fW', devPeakPwrVrtSen,  tLastPk, math.max(pkPwr, curPwr)))
           updateNum(devPeakPwrVrtSen, curPwr, false)
       end
    end

    -----------------------
    -- Update Mean power --
    -----------------------

    tLastMean = timeLastUpdate(devMeanPwrVrtSen, false)

    -- DEBUG :
    --print(devSmartMeterPwr..': P='..curPwr..'W ; Last : '..tLastMean..'s')

    pwrLast   = uservariables[varPwrLast]
    pwrSum    = uservariables[varPwrSum]
    sumDeltaT = timeLastUpdate(varPwrSum, true)

    -- Calc pwr sum + store. Set current mean pwr if time slice exceeded & reset user vars.
    if (tLastMean > maxLastUpdMean) then
       meanPwr = (pwrSum + pwrLast*sumDeltaT) / tLastMean

       --Log update :
       print(string.format('%s : New time slice (after %ds) => Record Pmean=%.2fW', devMeanPwrVrtSen,  tLastMean, meanPwr))

       updateNum(devMeanPwrVrtSen, meanPwr, true)
       commandArray['Variable:'..varPwrSum]  = '0'
       commandArray['Variable:'..varPwrLast] = tostring(curPwr)
    else
       pwrSum = pwrSum + pwrLast*sumDeltaT

       -- DEBUG :
       --print('Update current Psum='..pwrSum..'Ws ; deltaT='..sumDeltaT..'s')

       commandArray['Variable:'..varPwrLast] = tostring(curPwr)
       commandArray['Variable:'..varPwrSum]  = tostring(pwrSum)
    end
end

Using last updates & on the fly (value*time) discrete integration, with a bit of brain use, also allows to avoid using some local storage available in dzVent (a hack, IMO, as this use a file) only.

Just to get valid figures for data that change often in a 5mn time frame, that's really hacky IMO. And going into this also shows some event system characteristics that one may not foresee, so that's hacks over hacks + needs 2 added virtual sensors on top of HW driven one in my use case.

And can tell you charts from HW sensors are mostly really far from reality & currently store garbage.

[waltervl]

What is SW loading?
What do you mean with update using table?

Not sure what your problem exactly is but I experience when you update a device from dzvents it can trigger another script that looks at that specific device.

And yes, the charts show avarages of 5 min. For normal smart home operation no problem......

[lost]

That's power offloading, as every high consumer in my home is SW controlled it's done by SW & need instant notice when over contract limits.