Median filter for temperatures

[mpx2]

Is there a simple way to add median filter for temperature sensors? My 433 MHz sensors someties report erroneous values, very high or very low. I would like all my sensors to use the median value of the last three measurements, like median(t0,t1,t2).

[waltervl]

No, the values are directly uploaded in the device. You can make a dummy temperature device for every real temperature sensor and with a dzvents script that can do the filtering or calculations and fill the dummy sensor with the filtered data from your real sensor.

[gizmocuz]

@mpx2, throw away that temperature sensor? If it reports false values it is of no use.
If your device is to far away from the 433Mhz transceiver, try a bigger antenna

[mpx2]

@mpx2, throw away that temperature sensor? If it reports false values it is of no use.
If your device is to far away from the 433Mhz transceiver, try a bigger antenna

I have Telldus 313160 (FT007TH) temperature/humidity sensors viewtopic.php?p=212188#p212188 .

Erroneous temperatures are quite rare but those are bad for plots and statistics. This was one of the coldest days in last year but the highest received temperature is +186.5 C.

wrong temperature.png (25.19 KiB) Viewed 1202 times

I had an older version of RTL_433 (https://github.com/merbanan/rtl_433). I updated to the lastest version. Let's see if this helps.


There's a CRC error detection used in temp/humidity reports but I don't know how these are handled. I don't have any logs when erroneous values are sent.

rtl_433.png (13.52 KiB) Viewed 1202 times

I tested this DzEvents script. A new value is received and updated to the sensor but then this scripts will calculate the median value and updates it to the sensors.

Code: Select all

return {
	on = {
		devices = {	465 } -- idx of the sensors
	},
	data = {
            temperatures = { history = true, maxItems = 3 },
            humidities = { history = true, maxItems = 3 },
            medianT = { initial = 0 },
            medianH = { initial = 0 },
    },
	logging = {
		level = domoticz.LOG_DEBUG,
		marker = 'template',
	},
	execute = function(domoticz, sensor)
	    -- add new data
        domoticz.data.temperatures.add(sensor.temperature)
        domoticz.data.humidities.add(sensor.humidity)
        -- calculate median
        domoticz.data.medianT = domoticz.data.temperatures.med()
        domoticz.data.medianH = domoticz.data.humidities.med()
        -- update median values to sensor. silent() does not trigger the script again causing a loop.
	    domoticz.devices(sensor.id).updateTempHum(domoticz.data.medianT, domoticz.data.medianH).silent()
	    -- domoticz.devices(sensor.id).updateTempHum(-10, domoticz.data.medianH).silent()
	    -- debug
		domoticz.log('New temperature value is ' .. sensor.temperature .. ' . Median value is ' .. domoticz.data.medianT .. ' .' , domoticz.LOG_INFO)
		domoticz.log('New humidity value is ' .. sensor.humidity .. ' . Median value is ' .. domoticz.data.medianH .. ' .' , domoticz.LOG_INFO)
		if (domoticz.data.median ~= sensor.temperature) then
                domoticz.log('New temperature value is not Median value.', domoticz.LOG_INFO)
        end

	end
}

For testing I used the line "domoticz.devices(sensor.id).updateTempHum(-10, domoticz.data.medianH).silent()" . Every minute temperature 21 C was received and then replaced with a temperature -10 C.

Short spikes of 21 C are not shown in plots.

Filtered-test.png (39.13 KiB) Viewed 1202 times

[lost]

There's a CRC error detection used in temp/humidity reports but I don't know how these are handled.

If CRC is reported, IMO it's not ignored & such invalid data is already discarded. I also have some zwave sensors that are secure-included (so I'm pretty sure any altered message will be discarded) that sometimes send 0°C values, usually for a few 10's of minutes so several values to manually discard.

So this one may not be handled by some easy range filtering (0°C is not uncommon temp in winter!) as this was done for power-meters that allows you to setup a max consumption & easily discard high erroneous values (sometimes up to need a nuclear power plant in my garden, visually flattening whole curves with a single value).

The issue I sometimes observe with my device may also fool some statistics based filtering as several erroneous values may influence a mean for instance.

=> Not easy to have a setup that may apply to any buggy hardware + such radio devices may sometimes not be able to reach the controller due to range/perturbations issues: If that's a few hours between night and day, you may then filter values that finally make their way on statistics that do not make sense anymore & get trapped there until temp reports match the old stats, so last valid data should also be a tunable to reset your stat based filter. And any such tunable should be device specific, of course! Aspirin needed!

[mpx2]

I compiled the latest code for rtl_433 and added the same sensor by RFLink. rtl_433 shows an erroneous temperature value but RFLink does not show it.

rtl_433-vs-RFLink.png (21.64 KiB) Viewed 633 times

[gizmocuz]

It might be an issue with rtl_433 then...

But you cannot compare a RFXCom transceiver (or RFLink) with a USB stick that was not designed for supporting weather sensors.

I do think it's a great project! But I know that RFXCom does some sensible checks inside

This is also not a Domoticz issue, it just logs what its reported.

[lost]

No, the values are directly uploaded in the device. You can make a dummy temperature device for every real temperature sensor and with a dzvents script that can do the filtering or calculations and fill the dummy sensor with the filtered data from your real sensor.

For data like temp/hum that does not change quickly vs 5mn Domoticz sampling period, filtering is only needed to handle sometimes misbehaving sensors. But, as I already said in the past, some calculate options (like a discrete integration based average) would be helpful for sensors getting data that can vary a lot shortly (as a power-meter): Currently, Domoticz graphs for such sensors does not show true min/max or 5mn average.

[waltervl]

That can be done by dzvents script with persistent data analysis that is triggered on every device update it receives. So not limited to the 5 min delta in the database.

[lost]

@waltervi:
I know & this is what I did (in pure Lua as I my brain never managed to fit with dzVent syntax), but that's not so straightforward just to double store values that makes sense (true min/max/time slice average) compared to the basic sampling done. Not useful for each sensor, but storing current values is almost useless as they do not reflect anything for sensors with quick variations.
Another issue is the update & event handling: Lua table update or http API behave differently, especially if you want to have a device script attached to the virtual meter getting overpower instantly to manage SW offloading for instance.
Believe me, handling this from the outside of domoticz core needs some tricks (+pure Lua does not allow persistent data)! I already published this in the past.
+ you need a device script, so not precise slice possible: On my side I use a 10mn one (any over 5mn would do) but that's only a target that'll always be exceeded, as only 1st measure received after target can stop current slice: So we are sit between the 2 chairs of device and time!

[waltervl]

Get over the syntax and use dzvents..... Every device update is triggered so for example a sudden overpower will be noticed as soon as device data has been sent to Domoticz.

But it is still domotica and not real time processing. Then you need other hardware than for example an RPi..