Prevent script from overloading external server

[riko]

For my Evohome system I have a script that switches the thermostat (through web API) based on presence, fireplace, door open etc. This works well, but after implementing the fireplace I became blocked twice by the Evohome server. After contacting the customer service they renewed my account, but I'm affraid they are not so helpful a third time.

I found the root cause: my gas fire place is sending an update 4 times per second with the status (for instance 'On'). The Evohome thermostat is programmed to switch off when the fireplace switches on. Because I previously didn't validate the current status of the heating system as a requirement before sending the switch off signal, the script was firing thousands of messages to the Evohome server.

I hoped to prevent this by configuring the 'Poll Interval' in the Evohome hardware tab with a higher number. But this concerns only the reading from the evohome server. Sending to the Evohome server is unlimited.

I solved it now (by checking the actual status of the thermostat before sending the off signal), but a little mistake in the script would bring the same error back. Normally I would use the checkFirst() command for each of the commands to the thermostat. But this one is unfortunately not available for .setMode() nor for .updateSetPoint()

Does anyone have a solution to limit the number of updates to a certain device on the script level? I was thinking about a lastUpdate object, but cannot wrap my mind around that one at the moment.

Code: Select all

package.path = package.path .. ';' .. '/home/pi/domoticz/scripts/lua/Telegram.lua'
local telegram = require('Telegram')


local home_status 			= 197					-- 221 is test, 197 echt
local pir_wk_delay			= 333
local pir_tk				= 224
local pir_tk_delay			= 334 					-- 334 echt, 230 is test
local evohome_controller 	= 331
local evohome_tuinkamer		= 332
local evohome_woonkamer		= 330
local haard_status			= 252					-- Haard status vanuit Node-RED
local calendar_atwork		= 268
local tuindeur				= 73					-- Deur


---local variable = 'PIR count tuinkamer' -- define as type integer; initial value 0 <- VARIABELE MOET NOG GEWIST WORDEN IN DOMOTICZ


local test_button			= 164					-- 164

local trigger1 =	'every 15 minutes'

return 
{
	active = true,
    on = 
	{   
        timer = { trigger1,  },    
        devices = { home_status, calendar_atwork, pir_tk_delay, pir_tk, test_button, haard_status, tuindeur }
	},      
	data = 
		{
        pir_counter = { initial = 0 } -- Counts mogen niet meer dan 5 minuten oud zijn
        },
        
    logging = { level = domoticz.LOG_DEBUG, marker = 'Evohome' },   
    
    execute = function(dz, item)
		
        local Time = require('Time')
 
		-- uitschakelen van script
		local manualOverrideSwitch = 336
			if (manualOverrideSwitch and dz.devices(manualOverrideSwitch).state == 'On') then
				--telegram.sendText(telegram.getId('rik'), 'Evohome control is handmatig uitgeschakeld (disable script switches)')
				--dz.log('Evohome control is handmatig uitgeschakeld (disable script switches)', dz.LOG_DEBUG)
				return
			end	

		-- Start script
        local home_status = dz.devices(home_status) -- Home status device
		local pir_wk_delay = dz.devices(pir_wk_delay)
		local pir_tk = dz.devices(pir_tk)
		local pir_tk_delay = dz.devices(pir_tk_delay)
		local evohome_controller = dz.devices(evohome_controller)
		local evohome_tuinkamer = dz.devices(evohome_tuinkamer)
		local evohome_woonkamer = dz.devices(evohome_woonkamer)
 		local haard_status = dz.devices(haard_status)
 		local tuindeur = dz.devices(tuindeur)


 		local calendar_atwork = dz.devices(calendar_atwork)
 		local test_button = dz.devices(test_button)

        --local variable = dz.variables(variable)
		

		-- Counter obv PIR tuinkamer
		if item == pir_tk and pir_tk.state == 'Open' then
            dz.data.pir_counter = dz.data.pir_counter + 1
			--telegram.sendText(telegram.getId('rik'), 'Evohome: Beweging gedetecteerd, ' .. dz.data.pir_counter .. ' keer')	
			if (dz.data.pir_counter == 5) then
				telegram.sendText(telegram.getId('rik'), 'Tuinkamer waarschijnlijk in gebruik')	
				--variable.set(9999)
				dz.data.pir_counter = 0 -- reset the counter
				if calendar_atwork.state == 'Off' and evohome_tuinkamer.mode == 'TemporaryOverride' then
					evohome_tuinkamer.updateSetPoint(dz.EVOHOME_MODE_AUTO)
					telegram.sendText(telegram.getId('rik'), 'Evohome: Beweging gesignaleerd in tuinkamer en Rik niet naar kantoor, verwarming weer aan')
				elseif calendar_atwork.state == 'On' then
					--evohome_tuinkamer.updateSetPoint(dz.EVOHOME_MODE_AUTO)
					telegram.sendText(telegram.getId('rik'), 'Evohome: Beweging gesignaleerd in tuinkamer en Rik wel aan het werk, verwarming weer aan (SCHAKELT NIET)')
				end				
			end		
		end
		
		if item.isTimer then
			dz.data.pir_counter = 0 -- reset the counter
			--telegram.sendText(telegram.getId('rik'), 'PIR counter elke 3 minuten gereset')	
			return
		end

		-- Home status verandering
		if item == home_status and (home_status.state == 'Away' or home_status.state == 'Vacation') then	
			evohome_controller.setMode(dz.EVOHOME_MODE_AWAY)
			telegram.sendText(telegram.getId('rik'), 'Evohome: Huis verlaten - thermostaat op afwezig')
		elseif item == home_status and (home_status.state == 'Home' or home_status.state == 'Kids') then	
			evohome_controller.setMode(dz.EVOHOME_MODE_AUTO)
			telegram.sendText(telegram.getId('rik'), 'Evohome: Thuis - thermostaat op auto programma')	
			if calendar_atwork.state == 'On' then -- om te zorgen dat bij thuiskomst de tuinkamer nog op werkmodus blijft
				evohome_tuinkamer.updateSetPoint(10, dz.EVOHOME_MODE_TEMPORARY_OVERRIDE, dz.time.rawDate .. 'T23:59:00Z')
				telegram.sendText(telegram.getId('rik'), 'Evohome: Rik vandaag nog op kantoor - tuinkamer blijft 10 graden')	
			end
		elseif item == home_status and home_status.state == 'Sleep' then	
			evohome_controller.setMode(dz.EVOHOME_MODE_AWAY, dz.time.addMinutes(120).getISO() )
			telegram.sendText(telegram.getId('rik'), 'Evohome: Sleep - programma voor 2u onderbroken')	
		end

		-- Work modus aan voor tuinkamer
		if item == calendar_atwork and calendar_atwork.state == 'On' then
			evohome_tuinkamer.updateSetPoint(10, dz.EVOHOME_MODE_TEMPORARY_OVERRIDE, dz.time.addMinutes(1440).getISO())
			telegram.sendText(telegram.getId('rik'), 'Evohome: Vandaag naar kantoor - tuinkamer 10 graden')
		elseif item == calendar_atwork and calendar_atwork.state == 'Off' then
			evohome_tuinkamer.updateSetPoint(dz.EVOHOME_MODE_AUTO)
			telegram.sendText(telegram.getId('rik'), 'Evohome: Terug van kantoor - tuinkamer auto')	
		end
 
 		-- Haard - UITKIJKEN MET DEZE OMDAT DE OPENHAARD EEN PAAR KEER PER SECONDE DE STATUS DOORSTUURT!
		if item == haard_status then
			if haard_status.state == 'On' and evohome_woonkamer.mode ~= 'PermamentOverride' then
				--evohome_woonkamer.updateSetPoint(14, dz.EVOHOME_MODE_PERMANENT_OVERRIDE)
				telegram.sendText(telegram.getId('rik'), 'Evohome: Haard gaat aan, verwarming woonkamer op 14 graden (SCHAKELT NIET)')	
			elseif haard_status.state == 'Off' and evohome_woonkamer.mode ~= 'FollowSchedule' then
				--evohome_woonkamer.updateSetPoint(dz.EVOHOME_MODE_FOLLOW_SCHEDULE)
				telegram.sendText(telegram.getId('rik'), 'Evohome: Haard gaat uit, verwarming woonkamer op 14 graden (SCHAKELT NIET)')	
			end
		end
 
		-- Tuindeur, verarming woonkamer uit
		if item == tuindeur then
			if tuindeur.state == 'Open' then
				evohome_woonkamer.updateSetPoint(19, dz.EVOHOME_MODE_PERMANENT_OVERRIDE)
				telegram.sendText(telegram.getId('rik'), 'Evohome: Tuindeur gaat open, verwarming woonkamer op 10 graden')	
			elseif tuindeur.state == 'Closed' then
				evohome_woonkamer.updateSetPoint(dz.EVOHOME_MODE_FOLLOW_SCHEDULE)
			end
		end 
		
		-- Bewegingssensor voor tuinkamer
		if item == pir_tk_delay and pir_tk_delay.state == 'Off' and (evohome_tuinkamer.mode == 'FollowSchedule' or evohome_tuinkamer.mode == 'Auto') then
			evohome_tuinkamer.updateSetPoint(10, dz.EVOHOME_MODE_TEMPORARY_OVERRIDE, dz.time.rawDate .. 'T23:59:00Z')
			telegram.sendText(telegram.getId('rik'), 'Evohome: Geen beweging meer in tuinkamer, verwarming weer 10 graden')	
		end

     end
}
	

[rrozema]

Just put an if around it, like this?

Code: Select all

if evohome_controller.mode ~= dz.EVOHOME_MODE_AUTO then
    evohome_controller.setMode(dz.EVOHOME_MODE_AUTO)
end

[rrozema]

The real problem is however not the lack of checkFirst() or an equivalent, it's in your coding.

First of all, why do you re-use the same variable names for both a device's idx value and the device itself?
For example:

Code: Select all

local home_status 			= 197

and later:

Code: Select all

local home_status = dz.devices(home_status) -- Home status device

You're making it very hard on yourself this way to understand which is which. It's not a prescribed way, but I like to write my 'globals' in all capitals, and local variables i put in small letters. So in the above example this would be:

Code: Select all

local HOME_STATUS 			= 197

and

Code: Select all

local home_status = dz.devices(HOME_STATUS) -- Home status device

[rrozema]

One other point to note in your script: most motion sensors send an On message when motion is detected, then stay on for a predefined period (for example 30 seconds, often this is configurable in the sensors settings). After the predefined period is over and no new motion was detected, the sensor will send an Off message. If however new motion was detected, it will wait longer before it sends the Off signal. In other words, if someone keeps moving around in the room, the sensor may give one On message and stay on for a long time -even minutes- before sending an Off message.

Though I like your way of thinking with counting the number of times motion is detected, I don't think this will work very well given how the motion sensors behave. I think a better way is to look at the state and the lastUpdate value for the motion sensor: if the sensor is On or the sensor is Off and lastUpdate for the sensor is less than X amount of seconds ago, the room must be occupied, otherwise the room is empty. X is for you to choose: the time before the room will be considered empty after someone left the room is the sensor's period + X seconds.

[rrozema]

I put some time in it and rewrote your script as much as I could understand the intentions of your script. I think this does what you wanted to achieve without risking to send to many switch commands. I obviously haven't been able to test it, so some errors could still be in here.

Code: Select all

package.path = package.path .. ';' .. '/home/pi/domoticz/scripts/lua/Telegram.lua'
local telegram = require('Telegram')


local HOME_STATUS 			= 197					-- 221 is test, 197 echt
--local PIR_WK_DELAY			= 333
local PIR_TK				= 224
--local PIR_TK_DELAY			= 334 					-- 334 echt, 230 is test
local EVOHOME_CONTROLLER 	= 331
local EVOHOME_TUINKAMER		= 332
local EVOHOME_WOONKAMER		= 330
local HAARD_STATUS			= 252					-- Haard status vanuit Node-RED
local CALENDAR_ATWORK		= 268
local TUINDEUR				= 73					-- Deur
local MANUALOVERRIDESWITCH	= 336


---local variable = 'PIR count tuinkamer' -- define as type integer; initial value 0 <- VARIABELE MOET NOG GEWIST WORDEN IN DOMOTICZ


local TEST_BUTTON			= 164					-- 164

return 
{
	active = true,
    on = 
	{   
        devices = { 
			HOME_STATUS, 
			CALENDAR_ATWORK, 
--			PIR_WK_DELAY,
			PIR_TK, 
			TEST_BUTTON, 
			HAARD_STATUS, 
			TUINDEUR,
			MANUALOVERRIDESWITCH
		}
	},      
        
    logging = { level = domoticz.LOG_DEBUG, marker = 'Evohome' },   
    
    execute = function(dz, item)
        
		-- Start script
                local home_status = dz.devices(HOME_STATUS) -- Home status device
--		local pir_wk_delay = dz.devices(PIR_WK_DELAY)
		local pir_tk = dz.devices(PIR_TK)
--		local pir_tk_delay = dz.devices(PIR_TK_DELAY)
		local evohome_controller = dz.devices(EVOHOME_CONTROLLER)
		local evohome_tuinkamer = dz.devices(EVOHOME_TUINKAMER)
		local evohome_woonkamer = dz.devices(EVOHOME_WOONKAMER)
 		local haard_status = dz.devices(HAARD_STATUS)
 		local tuindeur = dz.devices(TUINDEUR)
		local manualOverrideSwitch = dz.devices(MANUALOVERRIDESWITCH)

 		local calendar_atwork = dz.devices(CALENDAR_ATWORK)
-- 		local test_button = dz.devices(TEST_BUTTON)
		
		if nil == home_status then
			dz.log("Device " .. tostring(HOME_STATUS) .. " not found", dz.LOG_ERROR)
--		elseif nil == pir_wk_delay then
--			dz.log("Device " .. tostring(PIR_WK_DELAY) .. " not found", dz.LOG_ERROR)
		elseif nil == pir_tk then
			dz.log("Device " .. tostring(PIR_TK) .. " not found", dz.LOG_ERROR)
--		elseif nil == pir_tk_delay then
--			dz.log("Device " .. tostring(PIR_TK_DELAY) .. " not found", dz.LOG_ERROR)
		elseif nil == evohome_controller then
			dz.log("Device " .. tostring(EVOHOME_CONTROLLER) .. " not found", dz.LOG_ERROR)
		elseif nil == evohome_tuinkamer then
			dz.log("Device " .. tostring(EVOHOME_TUINKAMER) .. " not found", dz.LOG_ERROR)
		elseif nil == evohome_woonkamer then
			dz.log("Device " .. tostring(EVOHOME_WOONKAMER) .. " not found", dz.LOG_ERROR)
		elseif nil == haard_status then
			dz.log("Device " .. tostring(HAARD_STATUS) .. " not found", dz.LOG_ERROR)
		elseif nil == tuindeur then
			dz.log("Device " .. tostring(TUINDEUR) .. " not found", dz.LOG_ERROR)
		elseif nil == manualOverrideSwitch then
			dz.log("Device " .. tostring(MANUALOVERRIDESWITCH) .. " not found", dz.LOG_ERROR)

		else

			-- This is just to send out a message if the manual override changes state.
			if item.idx == MANUALOVERRIDESWITCH then
				if item.active then
					--telegram.sendText(telegram.getId('rik'), 'Evohome control handmatig uitgeschakeld (disable script switches).')
					dz.log('Evohome control is handmatig uitgeschakeld (disable script switches).', dz.LOG_DEBUG)
				else
					--telegram.sendText(telegram.getId('rik'), 'Evohome control ingeschakeld (enable script switches).')
					dz.log('Evohome control is ingeschakeld (enable script switches).', dz.LOG_DEBUG)
				end
			end
		
			-- If the manual override switch is On, all ends here: we will not execute the remainder of the script.
			if manualOverrideSwitch.active then

				-- exit

			else
				-- We only get to the rest of the script, below here, if manual override switch is Off.
				
				-- Conditions for tuinkamer
				if calendar_atwork.active then
					-- Work modus aan voor tuinkamer
					if evohome_tuinkamer.setPoint ~= 10 or evohome_tuinkamer.mode ~= dz.EVOHOME_MODE_TEMPORARY_OVERRIDE then
						evohome_tuinkamer.updateSetPoint(10, dz.EVOHOME_MODE_PERMANENT_OVERRIDE)
					end
				elseif not pir_tk.timedOut and (pir_tk.active or pir_tk.lastUpdate.secondsAgo < 300) then
					if evohome_tuinkamer.mode ~= dz.EVOHOME_MODE_AUTO then
						evohome_tuinkamer.setMode(dz.EVOHOME_MODE_AUTO)
					end
				else
					if evohome.tuinkamer.setPoint ~= 10 or evohome_tuinkamer.mode ~= dz.EVOHOME_MODE_TEMPORARY_OVERRIDE then
						evohome_tuinkamer.updateSetPoint(10, dz.EVOHOME_MODE_PERMANENT_OVERRIDE)
						telegram.sendText(telegram.getId('rik'), 'Evohome: Geen beweging meer in tuinkamer, verwarming weer 10 graden')
					end
				end

				-- Conditions for woonkamer
				if tuindeur.active and not tuindeur.timedOut then
					if evohome_woonkamer.setPoint ~= 10 or evohome_woonkamer.mode ~= dz.EVOHOME_MODE_PERMANENT_OVERRIDE then
						evohome_woonkamer.updateSetPoint(10, dz.EVOHOME_MODE_PERMANENT_OVERRIDE)
						telegram.sendText(telegram.getId('rik'), 'Evohome: Tuindeur gaat open, verwarming woonkamer op 10 graden')	
					end
				elseif haard_status.active and not haard.timedOut then
					if evohome_woonkamer.setPoint ~= 14 or evohome_woonkamer.mode ~= dz.EVOHOME_MODE_PERMANENT_OVERRIDE then
						evohome_woonkamer.updateSetPoint(14, dz.EVOHOME_MODE_PERMANENT_OVERRIDE)
						telegram.sendText(telegram.getId('rik'), 'Evohome: Haard gaat aan, verwarming woonkamer op 14 graden (SCHAKELT NIET)')
					end
				else
					if evohome_woonkamer.mode ~= dz.EVOHOME_MODE_FOLLOW_SCHEDULE then
						evohome_woonkamer.updateSetPoint(19, dz.EVOHOME_MODE_FOLLOW_SCHEDULE)
					end
				end
				
				-- Conditions for controller
				if home_status.idx == HOME_STATUS then
					if home_status.state == 'Home' or home_status.state == 'Kids' then
						if evohome_controller.mode ~= dz.EVOHOME_MODE_AUTO then
							evohome_controller.setMode(dz.EVOHOME_MODE_AUTO)
							telegram.sendText(telegram.getId('rik'), 'Evohome: Thuis - thermostaat op auto programma')
						end
					elseif home_status.state == 'Away' or home_status.state == 'Vacation' then
						if evohome_controller.mode ~= dz.EVOHOME_MODE_AWAY then
							evohome_controller.setMode(dz.EVOHOME_MODE_AWAY)
							telegram.sendText(telegram.getId('rik'), 'Evohome: Huis verlaten - thermostaat op afwezig')
						end
					elseif home_status.state == 'Sleep' then
						if evohome_controller.mode ~= dz.EVOHOME_MODE_AWAY then
							evohome_controller.setMode(dz.EVOHOME_MODE_AWAY)
							telegram.sendText(telegram.getId('rik'), 'Evohome: Sleep')
						end
					end
				end
			end
		end
    end
}

[riko]

Thanks for cleaning up my script and the suggestions, rrozema! As you see I'm a self-learner with some strange twists in my mind.

• I now understand that I was wrongly using the 'renaming' of the global variables in my scripts. A lot of work to clean up all my other scripts

• I didn't know the .active function. Looked this up in the DzVents guide but is not described, only used in some examples. Does it really mean it points at the device that is triggering the script?

• The PIR solution including the count funtion is intended (and seems to work) to measure the 'intensity' of use in the room. As my children regularly just walk in and out to grab some stuff it should not turn on the thermostat all the time. By having the count function I only turn on the thermostat when the room is really in use. This only works if the PIR switches on/off within a few seconds. As you can see I used the PIR delay device, which has a delay of 30 minutes to turn off the thermostat as soon there was no movement for 30 minutes. To turn it off according to your methodology

About my initial question: I now use the

Code: Select all

if evohome_controller.mode ~= dz.EVOHOME_MODE_AUTO then
    evohome_controller.setMode(dz.EVOHOME_MODE_AUTO)
end

trick to validate and prevent the heating switching on for multiple subsequent times. But it still leaves the challenge: what if one of my other devices are flipping on/off too often (for instance if the presence detection is working errornously). I would like to have a fail-safe to prevent the controller / setpoints to be updated 'back and forth'. Still something with lastUpdate()?

Any ideas left on that?

[rrozema]

device.active is an alternative to device.bState, which is true when the device is "On", "Open" or whatever is the active state for this type of device. It is documented here: https://www.domoticz.com/wiki/DzVents:_ ... ll_devices.

The device that caused the execute() function to get called is in the 2nd parameter to the execute() function: https://www.domoticz.com/wiki/DzVents:_ ... gerInfo.29

And yes, I do have a suggestion if you want to make sure you don't send to much commands in a specific time frame: historical data.
You can do all sorts of tricks using these "history" persistent data variables. For example in your case you could create a data item:

Code: Select all

	data  = {
		evohomecalls = { history = true, maxMinutes = 5}
	},

And every time you make a call to evohome, you add an item with value 1 to this data evohomecalls. Now you can check the number of calls made in the last 5 minutes by taking the sum of all items in the evohomecalls data and if it exceeds your limit, skip the command or send a notification, or whatever else you want to do. In below example I skip sending the command if 10 or more commands were sent during the last 5 minutes.

Code: Select all

if evohome_controller.mode ~= dz.EVOHOME_MODE_AUTO then
	dz.data.evohomecalls.add(1)
	if dz.data.evohomecalls.sum() >= 10 then
		dz.log('Not sending this command because to many calls were made in the last 5 minutes.', dz.LOG_ERROR)
	else
		evohome_controller.setMode(dz.EVOHOME_MODE_AUTO)
	end
end

This is of course just an example, you can implement similar logic for whatever purpose you want to count something (I think a count attribute exists too, but it's not in the documentation, so I just add 1's in the set and sum them to get a count). And the time can be specified easily in the data declaration.

Using the same trick you can for example also implement your motion counter for the tuinkamer.

[riko]

Thanks for your clear reply. I'll experiment with your suggestion. I suppose I make a function for all the different evohome calls I make in the script (setting different type of setpoints etc).

Last question, whether you have ideas how instead of skipping sending the command the command can be send with a delay based on the outcome of the count and the maximum number of commands. So postponing instead of canceling.

[rrozema]

You could postpone the command using customEvents (more information is in the dzVents documentation). But I would advice against queueing the commands: if to many calls are made, you don't want to delay the commands to have them executed at some later time. Just think what can happen if you delay the commands: for example, your heating could turn off a hour after the tuindeur was opened: you wouldn't even know anymore why it turns off... And lots of other strange things can happen.

It's an event driven sytem: you want the action to be performed when the event is signaled, any delays are going to annoy you at some point or make no sense anymore to execute at a later time. If you decide to not execute some command because you think the system may get overloaded, you could decide to have your system go into some 'error state' for some defined period, where you choose some "safe settings" and don't respond to any more new triggers for a specific time.

I personally wouldn't bother blocking any commands. If I were afraid sending to many commands would be harmful, I would only put in a notification if to many commands are sent out. The system would still work as intended, and if I receive the notification I can decide to enable manual override and investigate why so many commands were sent.

[rrozema]

So I would suggest doing something like this. At every point where a command is executed, add a line to add 1 to the set of evohomecalls. So for example:

Code: Select all

if evohome_controller.mode ~= dz.EVOHOME_MODE_AUTO then
	dz.data.evohomecalls.add(1)
	evohome_controller.setMode(dz.EVOHOME_MODE_AUTO)
end

Then at the end of the script add one more if to send a notification if the number of items in the evohomecalls is over my set limit:

Code: Select all

local calls = dz.data.evohomecalls.sum()
if calls >= 10 then
	dz.log( tostring(calls) .. ' calls were made in the last 5 minutes.', dz.LOG_ERROR)
	telegram.sendText(telegram.getId('rik'), 'Evohome: ' .. tostring(calls) .. ' calls were made in the last 5 minutes!')
end

and of course add the data element in the top of the script:

Code: Select all

	data  = {
		evohomecalls = { history = true, maxMinutes = 5}
	},

[riko]

Thanks for thinking along. I agree on your view on the risks of sending later. I'll use the notification instead!