Some LUA magic: one script to rule them all..

[dannybloe]

Update: Moved the code to github https://github.com/dannybloe/ezVents and called it ezVents. Kid gotta have a name
Update: Version 0.9.6. Fixed a bug.
Update: Version 0.9.5 makes it almost trivial to make script schedules from code. Not only can you easily trigger time scripts per minute but now also all kinds of intervals, specific times, days of the week, sunset/rise etc. Check it out! http://www.domoticz.com/forum/viewtopic ... =20#p76871.

Update: The whole thing is simplified in v0.9.2, see post below. The idea is still the same but I made it much simpler. No need for special bindings file. Simply put your scripts in a subfolder and make it a Lua module (very easy)...

Old post:
======================================
Well, two actually.... bare with me

A lot of people complained a bit in the past about how long it takes for Domoticz to run all the event scripts each update cycle. I think this is mainly because it runs every script out there even when it is not needed (hence the notorious devicechanged checks everywhere) and that it fetches all device states in between over and over again. Some people suggested to create just one script and put all you logic in there.

So, I decided to make something just like and move it a step further and make it simple and generic so everybody can use it. I must say it works like a charm and fast (for me course). So here it goes:

There is only one script for timer events (runs every minute) and one script for device change events. These two scripts both read a file called event_bindings.lua. This file defines which scripts should be run every minute and which scripts should be run on specific device changes. The main scripts read the bindings and dispatch script execution to these event scripts but only if there is a need to. So, if there is binding for mySwitch, it only runs the script when mySwitch has actually changed! So no longer you need to check in every script if you device actually did change. Yay!

There are a couple of 'niceties':

• Each script that you bind to an event (device or timer) is a Lua module with a main function. See the example script. The main function receives the value of the changed device (just a convenience). Every script has access to all global tables created by Domoticz like devicechanged etc.
• Each script returns his part of the commandArray and an optional boolean value that can stop any further processing of scripts when set to false.
• Event bindings can use device ids instead of the device name since the name may change, the id stays the same. You need one of the newer versions of Domoticz for this because it uses the otherdevices_idx table. See the explanation in event_bindings.lua.
• In the logs you can see exactly what script runs and what the final list of commands is that is sent back to Domoticz
• Each cycle can define one or more preprocess and postprocess scripts. So, at the beginning of each cycle (e.g. device change) you can run a script that may do some checking. Perhaps make a switch 'Run scripts' that is checked and when active, the script returns false and execution of all scripts stop. Same for postprocess. It's optional but might be handy.
• You can bind multiple scripts to one event and use the same script for different events.

So, as I said before: I am running this for a little while now and so far I haven't seen any problems. But before I move this to the wiki it would be great to have other people test and use this as well.

So, how does it work or better how can you use it?

• The zip contains 5 files: copy them over to the Lua script folder inside your Domoticz folder: e.g. /home/pi/domoticz/scripts/lua.
• The idea is that the two main scripts script_device_main.lua and script_time_main.lua, are the only scripts that are started by Domoticz event system. So you should rename all your other scripts to something with _demo in the name so that Domoticz ignores them. Or move them out of the way. Of course you can do this at a later stage. This system doesn't conflict with what you already have in place!
• Edit event_bindings.lua. This file is the glue between the event system and your event scripts. You have a section for timer events and for device events. The file contains lots of information and instructions but basically it is a list of timer scripts and device+script combinations. It looks like this:

  Code: Select all

  return {
  	device = {
  		preprocess = {'myPreprocessScript1', 'myPrepocessScript2'},
  		postprocess = {'doSomeStuffAtTheEnd'},
  		devices = {
  			['Kitchen light'] = {'handle_kitchen_lights', 'notify_me'},
  			['125'] = {'calculate_energy_usage', 'notify_me'}
  		}
  	},
  	timer = {
  		preprocess = {},
  		postprocess = {},
  		scripts = {
  			'control_bathroom_humidity',
  			'check_windows',
  			'another script'
  		}
  	}
  }
  

  Note that you do not have to add .lua to the script names.
  I have considered using XML or something like that but this is pure Lua and is always faster. All sections in this bindings file are optional.
• Change your existing scripts into 'requireable' lua modules. This is very simple and you can use example_module.lua as the base. Basically you rename your script into something like 'handle_button_blabla.lua' and put some code around your existing code:

  Code: Select all

  return {
  	main = function(value)
  		local commandArray = {}
  		
  		-- your code here
  		
  		return commandArray, true -- return false if you want to stop further script processing  
  	end
  }
  

  
  Maybe you can remove your devicechanged checks that you may have in your existing scripts because they are likely not needed anymore since script_device_main.lua does that for you.

Well, that is all there is to it. Let me know what you think, if it helps you or even if this is a wacky/insane/stupid idea. I know that people suggeted to change the underlying event system in Domoticz completely but until then this might be a good alternative.

Danny

Small update: I didn't include a script for variable changes and security changes. script_variable_xxx doesn't actually work on all user variable updates (only GUI changes) so I assume people don't use this feature anyway (but I could be wrong). And I think that people don't have many security scripts that have to be processed in sequence. But if that is required then I can add that quite easily.

[simonrg]

So, I decided to make something just like and move it a step further and make it simple and generic so everybody can use it. I must say it works like a charm and fast (for me course).

Interesting comprehensive approach and great documentation -

I too would like to make Domoticz response to be just a little bit faster.

I have thought about trying your approach, but I am uncomfortable with modifying my working scripts and I can't see a way with your approach of implmenting a number of my general scripts. A general point is that the sensible implementation plan would be to do 1 at a time, as I could still keep old scripts running directly while adding 1 at a time to a new framework.

I have scripts which react to multiple devices based on the structure of the device name or a property of the device and so pick up new devices automatically without knowing their names - your structure seems to require explicit device names which need to be individually entered in the binding file.

I have taken a simpler approach to reduce to 1 the number of scripts Domoticz has to call, while not needing me to alter my working scripts at all, just move them.

I am assuminig that there is no problem with multiple array access etc., only with script calls as all the arrays have to be reloaded etc..

So I have taken all my original scripts and just moved them into a resources sub-directory, so that Domoticz does not see them anymore.

I have then created a single script with elseifs in order of time criticality, which calls the original script from the resources sub-directory.

The elseif statement is just the if statement from the script to be called.

I leave the setting up of the commandArray in each sub-script and it is returned at the end of the sub-script, so any one device change will only activate 1 sub-script.

Code: Select all

-- /home/pi/domoticz/scripts/lua/script_device_framework.lua

function doit(scriptname)
    print('Executing '..scriptname)
    dofile('/home/pi/domoticz/scripts/lua/resources/'..scriptname)
end

commandArray = {}

if devicechanged['Owl'] then
    doit('script_device_owl_log.lua')
elseif (tostring(next(devicechanged)):sub(1,3) == 'PIR') then
    doit('script_device_aapirs.lua')
elseif (tostring(devicechanged[next(devicechanged)]):sub(1,5) == 'Group') then
    doit('script_device_moodscene.lua')
elseif devicechanged['Alarm Set'] then
    doit('script_device_alarm.lua')
elseif (tostring(next(devicechanged)):sub(1,3) == 'Hue') then
    doit('script_device_hue.lua')
elseif (devicechanged[uservariables['PhilipsHueOnCaptureSwitch']]) then
    doit('script_device_hue_scene_capture.lua')
elseif (devicechanged['Hue Effect']) then
    doit('script_device_hue_scene_random.lua')
elseif (devicechanged[uservariables['PhilipsHueIncrementSwitch']]) then
    doit('script_device_hue_scene_increment.lua')
end

return commandArray

It would useful to understand more where you are using pre / post-processes and where multiple script calls are used, currently I can't see where I would need them.

Thanks.

[dannybloe]

I understand your approach and if you have something in place I'd leave it as it is. I don't think your approach is much different other than that my approach is more declarative, less of a pile of if-statements glueing things together. That makes it more manageable to me. And with my approach you can still have scripts that can respond to multiple device changes. I can easily adapt the event_bindings rules a bit to support regular expression so you can have a script kick in when the expression is true. Or, since I hate regexps, change the binding structure a bit so you can create a group of device-events and link them to one script. Again more descriptive and manageable.

Owh and you don't have to change your scripts to make it work with my approach, just put a main function around them and all stays the same. With a slight adaptation you can put your scripts in subfolders as well, just add the path to the package.path statement and you'r done.

[nayr]

this looks pretty nice, subscribed to check out at a later time..

can yeh make it so it just loads up everything out of a 'event' and 'time' folder so dont have to edit multiple files when creating a script?

[dannybloe]

this looks pretty nice, subscribed to check out at a later time..

can yeh make it so it just loads up everything out of a 'event' and 'time' folder so dont have to edit multiple files when creating a script?

Mmm.. interesting... I am not sure if I understand you correctly. Are you suggesting that you don't need the event_bindings.lua file and that you just place your scripts in the appropriate folder and they will be executed?

I have been playing with the idea that an event script not only exposes a main function but also a getTriggers function. The getTriggers function returns a list of device events it wants to listen to. At startup, all the scripts are loaded and for each script (in a folder as you suggest) the triggers are collected so the main script knows when to fire which script when a certain event occurs.

Downside for this is that you cannot easily disable a script or combine scripts. Right now all you have to do is edit the bindings file, put a comment mark before a line and the script is ignored. It also gives you a nice overview of all the events. It is more declarative.
On the other hand now that I think of it... you could have both ways. You could choose to put stuff in the bindings file but also choose to define the triggers in the script file themselves. That shouldn't be too hard.

Of course, for timer scripts you don't have a rule. Unless..... hehe... ... I could define some specific time triggers there too like:

Code: Select all

return {
	main = function (
		local commandArray = {}
		-- do your stuff		
		return commandArray
	end,
	getTriggers = function()
		return { 'hourly', 'every-other-minute', '12:00', '10:00', 'odd minutes' }
	end
}

That would save you a lot of nasty horrible Lua time manipulation/computation code (I hate Lua's way of dealing with date/time methods). The main script could do that for you and only trigger your time script when it matches the rules it returns. We could define a nice list of predefined time triggers here.

[dannybloe]

We could define a nice list of predefined time triggers here.

Heck yeah, you could even return a function that is evaluated each cycle to see if your time-script has to be called.

Of course you could define these time-triggers also in the event-bindings file. That way you don't have to change your script code if want to change the call schedule a bit.

[dannybloe]

Of course, defining time triggers like this is a risky way of scheduling because there is no guarantee that the main time-script is indeed called every minute so you could miss a heartbeat here. But you could extend the time triggers to a time period: '12:00-13:00' which will call your script only when the current time is within that interval.

[dannybloe]

... and I'm thinking of a persistent storage mechanism per script where each script receives his own storage and writes back to that storage so you don't need those nasty uservariable storage anymore to keep state between script runs. Allows you to create arrays, dictionaries, temporary values, history data etc. Shouldn't be too hard to make.

Mmm... so many opportunities to make scripting so much more flexible and powerful.

[simonrg]

... and I'm thinking of a persistent storage mechanism per script where each script receives his own storage and writes back to that storage so you don't need those nasty uservariable storage anymore to keep state between script runs. Allows you to create arrays, dictionaries, temporary values, history data etc. Shouldn't be too hard to make.

I quite like uservariables, basically by converting arrays, dictionaires to strings then up to 255 characters you can store quite a lot of stuff and Domoticz takes care of everything so you don't have a load of extra files to worry about.

In http://domoticz.com/wiki/Philips_Hue_Li ... Hue_Scenes the script captures a status of my Hue lights, so I can call them back later and if I create a nice effect on the Hue bridge then I can just capture it with the script creating a new variable to store it in - crude but effective.

Equally with your binding mechanism you could send data to SQL or create specific files.

[dannybloe]

There may be a need for uservariables but for me it is a wrong solution for keeping state between script runs. It's like creating global variables in your code. It's just a bad way with all kinds of complications. Also, having to go to Domoticz, create variables and then back to my code.. .to me that is nasty, it creates a dependency that it is not needed and it is hard to maintain because you easily end up with a huge pile of variables. Globals are a big red flag in software coding. I want proper scoping in my scripts and keep stuff where it belongs.

I don't know yet if and how I am going to do it but I'm thinking of some kind of json-based sidecar file per script that is maintained automatically and can be removed when necessary. After all, there is one thread/process running the script so there's no concurrency issue I believe.

My script could look like this:

Code: Select all

return {
   main = function (value, persistence)
      local commandArray = {}
      -- do your stuff      
      ...
      persistence.previousTemp = currentTemp
      persistence.myHistory.push(currentTemp)
      persistence.onTime(now)
      ...
      return commandArray, persistence
   end
}

[dannybloe]

Ok, I created a new version. Much simpler. There is no need for a bindings file. All you have to do is create a subfolder called scripts and put your scripts in there. Each script is a module that returns a lua object with two parts:

1. A on section listing all the triggers
2. An execute method which is called when on of the tirggers occurs.

Code: Select all

return {
	on = {
		'<device name>', -- name of the device
		258, -- index of the device
		'timer'  -- causes this script to be called every minute
	},
	execute = function(value)
		local commandArray = {}

		-- do your stuff

		return commandArray, true
	end
}

So usually you will place one trigger in the on-section but there could me more than one. I have several virtual switches that are triggered when people in my family come home. So for this script it looks like this:

Code: Select all

return {
	on = {
		'Danny@home', 'Thyrsa@home', 'Heleen@home'
	},
	execute = function(value)
		--
	end

The two main scripts make sure that all your event scripts are called when needed. Simply put 'timer' as a trigger and it is handled by the timer scrip.
I also removed the preprocess and postprocess handlers. Guess they are not very useful after all. Keep it simple

[dannybloe]

Ok, made it even simpeler (yep, still possible) and even a bit more versatile.

I removed the option to return a boolean from your script. All you have to return now is a commandArray. Each script will return its own portion of this commandArray and the code will nicely merge them together without them interfering with each other.

I added an active key to the event script module so you can easily disable a script. This can even be a function that can do some checking (perhaps check for a switch state so you can switch scripts on or off easily, or groups of scripts).

I added a '*' trigger. When added to the 'on' section it will make the script run in every cycle. No matter what.

This is the example:

Code: Select all

return {
	active = false,    -- set to true to activate this script, can also be a function returning either true or false
	on = {
		'My switch',            -- name of the device
		'My sensor_Temperature',
		'My sensor',
		258,                    -- index of the device
		'timer',                -- causes this script to be called every minute
		'*'                     -- script is always executed
	},

	execute = function(value)
		local commandArray = {}

		-- example
		if (value == 'On') then
			commandArray['SendNotification'] = 'I am on!'
		end

		return commandArray
	end
}

And finally created a readme.md with some instructions.

I must say that it runs quite nicely over here. And fast.

[dannybloe]

this looks pretty nice, subscribed to check out at a later time..

can yeh make it so it just loads up everything out of a 'event' and 'time' folder so dont have to edit multiple files when creating a script?

I think that I did just that. Except that I didn't create different folders for timed scripts and trigger scripts. Just put your scripts in one folder and set the triggers and you're done.

[NietGiftig]

Nice, going to try this

For my old and slowly decaying grey mass inside my head:

In the "on" part of my own script I only have to put the index of the device?
or only the name?
or both?
So it's not good to have a device named "timer"

[dannybloe]

Nice, going to try this

For my old and slowly decaying grey mass inside my head:

In the "on" part of my own script I only have to put the index of the device?
or only the name?
or both?

If you put them both in the on part then the script will be triggered twice. Just put the index there (without quotes, a number).

So it's not good to have a device named "timer"

That's correct. I could change that if that's a problem but I didn't think it would be

[dannybloe]

Of course you can put multiple devices in the on-part. Like if you have a couple of window sensors that should all trigger one script:

Code: Select all

return {
	on = { 12,13,14,15,16},
	active=true,
	execute=function(value) 
		-- help... burglars in da house!!	
	end
}

[NietGiftig]

I'm programming in LUA now for 2 weeks, so i'm rather blanco on this subject

Would you be so kind to put this small (working for me in Domoticz) script in your example so that i can see if I understand the format.
Assume that the idx from the lux sensor is 999

Code: Select all

local lux_sensor = 'LUX Achter' -- name of the lux sensor
-- Get values from the Lux sensor
local V = otherdevices_svalues[lux_sensor]
local lux_base = 'LuxNow'

-- Function to strip charachters
function
 stripchars(str, chrs)
 local s = str:gsub("["..chrs.."]", '')
 return s
end

commandArray = {}

-- Strip " Lux" from V
w = stripchars( V, " Lux" )
-- print('Lux reading ' .. ' = ' .. w)

commandArray['Variable:' .. lux_base] = w

return commandArray

[dannybloe]

I'm programming in LUA now for 2 weeks, so i'm rather blanco on this subject

Would you be so kind to put this small (working for me in Domoticz) script in your example so that i can see if I understand the format.
Assume that the idx from the lux sensor is 999

Code: Select all

-- Function to strip charachters
function stripchars(str, chrs)
	local s = str:gsub("["..chrs.."]", '')
	return s
end

return {
	active = true,
	on = {
		999   --'LUX Achter'
	},

	execute = function(value)
		
		-- you may check what value is here.. could be that 
		-- you don't have to fetch it from otherdevices_svalues at all

		local V = otherdevices_svalues[lux_sensor]
		
		print('value: ' .. value)
		print('svalue: ' .. V)

		local commandArray = {}
		
		commandArray['Variable:LuxNow'] = stripchars( V, " Lux" )

		return commandArray
	end
}

Something like that

[NietGiftig]

Ah, I see, other function(s) in script are outside the returning function, of course, silly me

Thanks, learning all the way

[dannybloe]

Indeed, the scripchars function is in the local scope of the script module. Nice encapsulation, not accessible from the outside. Love that.

However you could easily create the function inside the execute function as well. It's better to keep it apart.