Median filter for temperatures

[mpx2]

Is there a simple way to add median filter for temperature sensors? My 433 MHz sensors someties report erroneous values, very high or very low. I would like all my sensors to use the median value of the last three measurements, like median(t0,t1,t2).

[waltervl]

No, the values are directly uploaded in the device. You can make a dummy temperature device for every real temperature sensor and with a dzvents script that can do the filtering or calculations and fill the dummy sensor with the filtered data from your real sensor.

[gizmocuz]

@mpx2, throw away that temperature sensor? If it reports false values it is of no use.
If your device is to far away from the 433Mhz transceiver, try a bigger antenna

[mpx2]

@mpx2, throw away that temperature sensor? If it reports false values it is of no use.
If your device is to far away from the 433Mhz transceiver, try a bigger antenna

I have Telldus 313160 (FT007TH) temperature/humidity sensors viewtopic.php?p=212188#p212188 .

Erroneous temperatures are quite rare but those are bad for plots and statistics. This was one of the coldest days in last year but the highest received temperature is +186.5 C.

wrong temperature.png (25.19 KiB) Viewed 301 times

I had an older version of RTL_433 (https://github.com/merbanan/rtl_433). I updated to the lastest version. Let's see if this helps.


There's a CRC error detection used in temp/humidity reports but I don't know how these are handled. I don't have any logs when erroneous values are sent.

rtl_433.png (13.52 KiB) Viewed 301 times

I tested this DzEvents script. A new value is received and updated to the sensor but then this scripts will calculate the median value and updates it to the sensors.

Code: Select all

return {
	on = {
		devices = {	465 } -- idx of the sensors
	},
	data = {
            temperatures = { history = true, maxItems = 3 },
            humidities = { history = true, maxItems = 3 },
            medianT = { initial = 0 },
            medianH = { initial = 0 },
    },
	logging = {
		level = domoticz.LOG_DEBUG,
		marker = 'template',
	},
	execute = function(domoticz, sensor)
	    -- add new data
        domoticz.data.temperatures.add(sensor.temperature)
        domoticz.data.humidities.add(sensor.humidity)
        -- calculate median
        domoticz.data.medianT = domoticz.data.temperatures.med()
        domoticz.data.medianH = domoticz.data.humidities.med()
        -- update median values to sensor. silent() does not trigger the script again causing a loop.
	    domoticz.devices(sensor.id).updateTempHum(domoticz.data.medianT, domoticz.data.medianH).silent()
	    -- domoticz.devices(sensor.id).updateTempHum(-10, domoticz.data.medianH).silent()
	    -- debug
		domoticz.log('New temperature value is ' .. sensor.temperature .. ' . Median value is ' .. domoticz.data.medianT .. ' .' , domoticz.LOG_INFO)
		domoticz.log('New humidity value is ' .. sensor.humidity .. ' . Median value is ' .. domoticz.data.medianH .. ' .' , domoticz.LOG_INFO)
		if (domoticz.data.median ~= sensor.temperature) then
                domoticz.log('New temperature value is not Median value.', domoticz.LOG_INFO)
        end

	end
}

For testing I used the line "domoticz.devices(sensor.id).updateTempHum(-10, domoticz.data.medianH).silent()" . Every minute temperature 21 C was received and then replaced with a temperature -10 C.

Short spikes of 21 C are not shown in plots.

Filtered-test.png (39.13 KiB) Viewed 301 times

[lost]

There's a CRC error detection used in temp/humidity reports but I don't know how these are handled.

If CRC is reported, IMO it's not ignored & such invalid data is already discarded. I also have some zwave sensors that are secure-included (so I'm pretty sure any altered message will be discarded) that sometimes send 0°C values, usually for a few 10's of minutes so several values to manually discard.

So this one may not be handled by some easy range filtering (0°C is not uncommon temp in winter!) as this was done for power-meters that allows you to setup a max consumption & easily discard high erroneous values (sometimes up to need a nuclear power plant in my garden, visually flattening whole curves with a single value).

The issue I sometimes observe with my device may also fool some statistics based filtering as several erroneous values may influence a mean for instance.

=> Not easy to have a setup that may apply to any buggy hardware + such radio devices may sometimes not be able to reach the controller due to range/perturbations issues: If that's a few hours between night and day, you may then filter values that finally make their way on statistics that do not make sense anymore & get trapped there until temp reports match the old stats, so last valid data should also be a tunable to reset your stat based filter. And any such tunable should be device specific, of course! Aspirin needed!