dzVents errors

Easy to use, 100% Lua-based event scripting framework.

Moderator: leecollings

roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

dzVents errors

Post by roblom »

Yesterday I was experimenting with mqtt and node-r ed. But it seems I have broken something in Domoticz. Domoticz is quite unstable and crashes often. When I turn on DzEvents I get the following errors.

Code: Select all

2020-11-23 22:11:37.745 Error: dzVents: Error: (3.0.16) Consumption: Problem with module: __data_consumption

2020-11-23 22:11:37.745 Error: EventSystem: in /home/pi/domoticz/dzVents/runtime/dzVents.lua: /home/pi/domoticz/dzVents/runtime/EventHelpers.lua:101: attempt to index a boolean value (local 'fileStorage')

2020-11-23 22:11:45.213 Status: dzVents: Info: Handling events for: "Energie (P1)", value: "8071351;9648730;2817226;6193098;590;0"
It seems it's related to my energy consumption script because when I remove that script from the DzEvents script folder the errors are gone.
The problem is that I have no clue how to solve this so hope someone can help.

My Domoticz
Version: 2020.2 (build 12697)
Build Hash: 34c800ea6-modified
Compile Date: 2020-11-22 13:14:05
dzVents Version: 3.0.16
Python Version: 3.7.3 (default, Jul 25 2020, 13:03:44) [GCC 8.3.0]

Last edited by roblom on Monday 23 November 2020 22:40, edited 2 times in total.
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

When I add my script that uploads data to pvoutput I get different errors.

Code: Select all

Error: dzVents: Error: (3.0.16) PVOutput: Error: the return from PVOutput was :

Error: Error opening url: HTTPS://pvoutput.org/service/r2/addstatus.jsp
Error: dzVents: Error: (3.0.16) PVOutput: HTTP/1.1 response: 28 ==>> Timeout was reached
To be clear, I did not change anything to these scripts and they where working without any errors.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Monday 23 November 2020 22:40

Code: Select all

2020-11-23 22:11:37.745 Error: dzVents: Error: (3.0.16) Consumption: Problem with module: __data_consumption
The problem is that I have no clue how to solve this so hope someone can help.
The datafile of consumption.lua seems to be corrupt. You can delete it and it will be rebuild automatic at the next execution of consumption.lua

Code: Select all

sudo rm <domoticz dir>/scripts/dzVents/data/__data_consumption.lua
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Monday 23 November 2020 22:57 When I add my script that uploads data to pvoutput I get different errors.
To be clear, I did not change anything to these scripts and they where working without any errors.
Seems not directly related to the other error. Maybe more corruption happened on your system?
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

waaren wrote:
roblom wrote: Monday 23 November 2020 22:40

Code: Select all

2020-11-23 22:11:37.745 Error: dzVents: Error: (3.0.16) Consumption: Problem with module: __data_consumption
The problem is that I have no clue how to solve this so hope someone can help.
The datafile of consumption.lua seems to be corrupt. You can delete it and it will be rebuild automatic at the next execution of consumption.lua

Code: Select all

sudo rm <domoticz dir>/scripts/dzVents/data/__data_consumption.lua
This seems indeed the cause, after removing the file the error didn't came back.
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

waaren wrote:
roblom wrote: Monday 23 November 2020 22:57 When I add my script that uploads data to pvoutput I get different errors.
To be clear, I did not change anything to these scripts and they where working without any errors.
Seems not directly related to the other error. Maybe more corruption happened on your system?
Thank you for the tip, I'm going to check for a corrupted card.
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

Checking the SD card for bad sectors is harder than I thought. It's there something else I could try to solve the error because it seems something hangs in DzEvents.

Code: Select all

Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
And it seems that not all the DzEvents scripts does cause the error.
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

Code: Select all

sudo badblocks -vsn /dev/sdq
Checking for bad blocks in non-destructive read-write mode
From block 0 to 15558143
Checking for bad blocks (non-destructive read-write test)
Testing with random pattern: done
Pass completed, 0 bad blocks found. (0/0/0 errors)
So no bad blocks but still it seems DzEvents gives errors. Is it possible to reinstall Domoticz? Our is it better to start over from scratch? This is for about Domoticz not a big issue but I also had for example node-red and Mosquito installed so this is less easy.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Tuesday 24 November 2020 18:34

Code: Select all

Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
Can you activate dzVents debug logging? [setup][settings][other] (close to the bottom) It might reveal some useful information and / or what is causing this.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

I don't see anything useful only this below.

Code: Select all

2020-11-27 00:54:40.700 Error: dzVents: Error: (3.0.17) PVOutput: HTTP/1.1 response: 28 ==>> Timeout was reached

2020-11-27 00:54:40.701 Error: dzVents: Error: (3.0.17) PVOutput: Error: the return from PVOutput was :

2020-11-27 00:55:12.886 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-27 00:55:24.372 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
I also removed the Domoticz folder, reinstalled Domoticz and recovered the database and put back the lua and DzEvents scripts. But the problem stays the same.
It looks like there is no response from the internet possible causing DzEvents keeps waiting for response or something.
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Friday 27 November 2020 0:58 It looks like there is no response from the internet possible causing dzVents keeps waiting for response for something.
Could be if you use os.execute(curl ...) or io.popen(curl ... ) in a script but not when using openURL. The openURL passes control to domoticz via the commandURL and does not wait for a response. The response can be catched by another instance (could be the same- but also zero or more other scripts) based on the set callback string but this is an optional- and none blocking mechanism.

It looks like one of the domoticz threads "hangs" on a HTTP call causing the EventSystem thread not getting enough time to process its tasks.
Maybe you can catch relevant information when you activate domoticz debug logging. (warning: It will produce an awful lot of log lines)

The domoticz logfile location and other settings are defined in /etc/init.d/domoticz.sh
the relevant settings are in set to the DAEMON_ARGS var.

with an editor of choice

Code: Select all

DAEMON_ARGS="$DAEMON_ARGS -log /var/log/domoticz.log" # or any other OS file
DAEMON_ARGS="$DAEMON_ARGS -loglevel  normal, status, error, debug" #enable debug 
#DAEMON_ARGS="$DAEMON_ARGS -loglevel  normal,status,error" # default loglevels (# commented now)
DAEMON_ARGS="$DAEMON_ARGS -debuglevel normal,hardware,received,webserver,eventsystem,python,thread_id"

Code: Select all

sudo systemctl daemon-reload
sudo service domoticz stop
sudo service domoticz start

sudo tail -f /var/log/domoticz.log
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Friday 27 November 2020 0:58 I don't see anything useful only this below.

Code: Select all

2020-11-27 00:54:40.700 Error: dzVents: Error: (3.0.17) PVOutput: HTTP/1.1 response: 28 ==>> Timeout was reached
I added an extra debug file (module.log) in build 12707 to investigate the issue a bit more and I when I saw the message
Error: EventSystem: Warning!, lua script domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds
once, I could now determine that it was caused by a combination of many time triggered scripts activated in one dzVents.lua pass, one of them in which I have to use wget to get information from my Enphase solar panels. The wget took much longer then normal and this caused the 10 seconds Error.
So maybe if you update to beta 12707 or later and activate dzVents debug mode you can also identify what is causing the message.

btw. I now made the wget script kind of async by separating the wget part from the data processing part using customEvents.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

I tried to update but it didn't go higher than my current 12705 version. Also not with a manual update using ./updatebeta.
I also saw the error below.

Code: Select all

2020-11-28 08:22:34.044 Error: FCM: Could not send message, HTTP Error
2020-11-28 08:22:34.173 Error: Pushover: Could not send message!
Could that be related?
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Saturday 28 November 2020 9:15 I tried to update but it didn't go higher than my current 12705 version. Also not with a manual update using ./updatebeta.
Seems that the Linux tar for donload got corrupted somehow. Until someone fix this:

if You replace <domoticz dir>/dzVents/runtime/EventHelpers.lua with below file, you will effectively have the same as in build 12707.
I also saw the error below.

Code: Select all

2020-11-28 08:22:34.044 Error: FCM: Could not send message, HTTP Error
2020-11-28 08:22:34.173 Error: Pushover: Could not send message!
Could that be related?
Related in a sense that the issue is triggered by a network hickup of some kind.

EventHelpers.lua (from build 12707)

Code: Select all

local _ = require('lodash')
local GLOBAL_DATA_MODULE = 'global_data'
local utils = require('Utils')
local persistence = require('persistence')
local HTTPResponse = require('HTTPResponse')
local Timer = require('Timer')
local Security = require('Security')
local SystemEvent = require('SystemEvent')
local CustomEvent = require('CustomEvent')
local HistoricalStorage = require('HistoricalStorage')

local function EventHelpers(domoticz, mainMethod)

	local globalsDefinition

	local currentPath = globalvariables['script_path']

	if (_G.TESTMODE) then
		-- make sure you run the tests from the tests folder !!!!
		_G.scriptsFolderPath = currentPath .. 'scripts'

		package.path =
			currentPath .. 'scripts/?.lua;' ..
			currentPath .. 'data/?.lua;' ..
			currentPath .. '/../?.lua;' ..
			package.path
	end

	local validEventTypes = 'devices,timer,security,customEvents,system,httpResponses,scenes,groups,variables'
	local inValidEventTypes = 'on,logging,active,data,execute'

	local webRoot = globalvariables['domoticz_webroot']
	local _url = 'http://127.0.0.1:' .. (tostring(globalvariables['domoticz_listening_port']) or "8080")

	local settings = {
		['Log level'] = tonumber(globalvariables['dzVents_log_level']) or 1,
		['Domoticz url'] = _url,
		url = _url,
		webRoot = tostring(webRoot),
		serverPort = globalvariables['domoticz_listening_port'] or '8080',
		dzVentsVersion = globalvariables.dzVents_version,
		domoticzVersion = globalvariables.domoticz_version,
		location = {
			name = utils.urlDecode(globalvariables['domoticz_title'] or "Domoticz"),
			latitude = globalvariables.latitude or 0,
			longitude = globalvariables.longitude or 0,
		}
	}

	if (webRoot ~= '' and webRoot ~= nil) then
		settings['Domoticz url'] = settings['Domoticz url'] .. '/' .. tostring(webRoot)
	end

	_G.logLevel = settings['Log level']

	if (domoticz == nil) then
		local Domoticz = require('Domoticz')
		domoticz = Domoticz(settings)
	end

	local self = {
		['utils'] = utils, -- convenient for testing and stubbing
		['domoticz'] = domoticz,
		['settings'] = settings,
	}

	if (_G.TESTMODE) then
		self.scriptsFolderPath = scriptsFolderPath
		self.dataFolderPath = _G.dataFolderPath
		function self._getUtilsInstance()
			return utils
		end
	end

	function self.getStorageContext(storageDef, module)

		local storageContext = {}
		local fileStorage, value, ok
		--require('lodash').print(1, storageDef)
		if (storageDef ~= nil) then
			-- load the datafile for this module
			ok, fileStorage = pcall(require, module)
			if type(fileStorage) == 'boolean' then
				utils.log('Problem with module: ' .. module, utils.LOG_ERROR)
			end
			package.loaded[module] = nil -- no caching
			if (ok) then
				-- only transfer data as defined in storageDef
				for _var, _def in pairs(storageDef) do
					local var, def

					if (type(_var) == 'number' and type(_def) == 'string') then
						var = _def
						def = {}
					else
						var = _var
						def = _def
					end

					if (def.history ~= nil and def.history == true) then
						storageContext[var] = HistoricalStorage(fileStorage[var], def.maxItems, def.maxHours, def.maxMinutes, def.getValue)
					else
						if (fileStorage[var] == nil) then
							-- obviously this is a var that was added later
							-- initialize it
							storageContext[var] = def.initial
						else
							storageContext[var] = fileStorage[var]
						end
					end
				end
			else
				for _var, _def in pairs(storageDef) do

					local var, def

					if (type(_var) == 'number' and type(_def) == 'string') then
						var = _def
						def = {}
					else
						var = _var
						def = _def
					end

					--require('lodash').print('>', var, def, '<')
					if (def.history ~= nil and def.history == true) then
						-- no initial value, just an empty history
						storageContext[var] = HistoricalStorage(fileStorage[var], def.maxItems, def.maxHours, def.maxMinutes, def.getValue)
					else
						if (def.initial ~= nil) then
							storageContext[var] = def.initial
						else
							storageContext[var] = nil
						end
					end
				end
			end

		end
		storageContext.initialize = function(varName)
			if (storageContext[varName] ~= nil and storageDef[varName].history == nil ) then
				storageContext[varName] = storageDef[varName].initial
			end
		end
		fileStorage = nil
		return storageContext
	end

	function self.writeStorageContext(storageDef, dataFilePath, dataFileModuleName, storageContext)

		local data = {}

		if (storageDef ~= nil) then
			-- transfer only stuf as described in storageDef
			for var, def in pairs(storageDef) do

				if (type(var) == 'number' and type(def) == 'string') then
					data[def] = storageContext[def]
				else
					if (def.history ~= nil and def.history == true) then
						data[var] = storageContext[var]._getForStorage()
					else
						data[var] = storageContext[var]
					end
				end
			end
			local ok, err = pcall(persistence.store, dataFilePath, data)

			-- make sure there is no cache for this 'data' module
			package.loaded[dataFileModuleName] = nil
			if (not ok) then
				utils.log('There was a problem writing the storage values', utils.LOG_ERROR)
				utils.log(err, utils.LOG_ERROR)
			end
		end
	end

	local function getEventInfo(eventHandler, mode)
		local res = {}
		res.type = mode
		res.scriptName = eventHandler.name
		if (eventHandler.trigger ~= nil) then
			res.trigger = eventHandler.trigger
		end
		return res
	end

	local function deprecationWarning(moduleName, key, value, quoted)
		local msg

		if quoted then
			msg = 'dzVents deprecation warning: please use "on = { [\'' .. key .. '\'] = { \'' .. tostring(value) .. '\' } }". Module: ' .. moduleName
		else
			msg = 'dzVents deprecation warning: please use "on = { [\'' .. key .. '\'] = { ' .. tostring(value) .. ' } }". Module: ' .. moduleName
		end

		utils.log(msg, utils.LOG_ERROR)
	end

	function self.callEventHandler(eventHandler, subject)

		local useStorage = false

		if (eventHandler['execute'] ~= nil) then

			-- ==================
			-- Prepare storage
			-- ==================
			if (eventHandler.data ~= nil) then
				useStorage = true
				local localStorageContext = self.getStorageContext(eventHandler.data, eventHandler.dataFileName)

				if (localStorageContext) then
					self.domoticz.data = localStorageContext
				else
					self.domoticz.data = {}
				end
			end

			if (globalsDefinition) then
				local globalStorageContext = self.getStorageContext(globalsDefinition, '__data_global_data')
				self.domoticz.globalData = globalStorageContext
			else
				self.domoticz.globalData = {}
			end

			-- ==================
			-- Run script
			-- ==================
			local ok, res, info

			local baseType = subject and subject.baseType or ''

			if (baseType == domoticz.BASETYPE_DEVICE) then
				info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_DEVICE)
				ok, res = pcall(eventHandler['execute'], self.domoticz, subject, info)

			elseif (baseType == domoticz.BASETYPE_VARIABLE) then
				info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_VARIABLE)
				ok, res = pcall(eventHandler['execute'], self.domoticz, subject, info)

			elseif (baseType == domoticz.BASETYPE_SECURITY) then
				info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_SECURITY)
				local security = Security(self.domoticz, info.trigger)
				ok, res = pcall(eventHandler['execute'], self.domoticz, security, info)

			elseif (baseType == domoticz.BASETYPE_SCENE or baseType == domoticz.BASETYPE_GROUP) then
				if (baseType == domoticz.BASETYPE_SCENE) then
					info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_SCENE)
				else
					info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_GROUP)
				end
				ok, res = pcall(eventHandler['execute'], self.domoticz, subject, info)

			elseif (baseType == domoticz.BASETYPE_HTTP_RESPONSE) then
				info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_HTTPRESPONSE)
				info.trigger = subject.callback
				local response = HTTPResponse(self.domoticz, subject)
				ok, res = pcall(eventHandler['execute'], self.domoticz, response, info)

			elseif (baseType == domoticz.BASETYPE_SYSTEM_EVENT) then
				info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_DOMOTICZ)
				info.trigger = subject.type
				local dze = SystemEvent(self.domoticz, subject)
				ok, res = pcall(eventHandler['execute'], self.domoticz, dze, info)
			elseif (baseType == domoticz.BASETYPE_CUSTOM_EVENT) then
				info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_CUSTOM)
				info.trigger = subject.type
				local custom = CustomEvent(self.domoticz, subject)
				ok, res = pcall(eventHandler['execute'], self.domoticz, custom, info)
			else
				-- timer
				info = getEventInfo(eventHandler, self.domoticz.EVENT_TYPE_TIMER)
				local timer = Timer(self.domoticz, info.trigger)
				ok, res = pcall(eventHandler['execute'], self.domoticz, timer, info)
			end

			if (ok) then

				-- ==================
				-- Persist storage
				-- ==================

				if (useStorage) then
					self.writeStorageContext(eventHandler.data,
						eventHandler.dataFilePath,
						eventHandler.dataFileName,
						self.domoticz.data)
				end

				if (globalsDefinition) then
					self.writeStorageContext(globalsDefinition,
						_G.dataFolderPath .. '/__data_global_data.lua',
						_G.dataFolderPath .. '/__data_global_data',
						self.domoticz.globalData)
				end

				self.domoticz.data = nil
				self.domoticz.globalData = nil

				return res
			else
				utils.log('An error occurred when calling event handler ' .. eventHandler.name, utils.LOG_ERROR)
				utils.log(res, utils.LOG_ERROR) -- error info
			end
		else
			utils.log('No "execute" function found in event handler ' .. eventHandler, utils.LOG_ERROR)
		end

		self.domoticz.data = nil
		self.domoticz.globalData = nil
	end

	function self.scandir(directory, type)
		local pos, len
		local i, t, popen = 0, {}, io.popen
		local sep = string.sub(package.config, 1, 1)
		local cmd
		local namesLookup = {}

		if (directory == nil) then
			return {}
		end

		if (sep == '/') then
			cmd = 'ls -a "' .. directory .. '"'
		else
			-- assume windows for now
			cmd = 'dir "' .. directory .. '" /B'
		end

		t = {}
		local pfile = popen(cmd)
		for filename in pfile:lines() do
			pos, len = string.find(filename, '.lua', 1, true)
			if (pos and pos > 0 and filename:sub(1, 1) ~= '.' and len == string.len(filename)) then

				local name = string.sub(filename, 1, pos - 1)
				table.insert(t, {
					['type'] = type,
					['name'] = name
				})

				namesLookup[name] = true -- for quick lookup for filename

				--utils.log('Found module in ' .. directory .. ' folder: ' .. t[#t].name, utils.LOG_DEBUG)
			end
		end
		pfile:close()
		return t, namesLookup
	end

	function self.processTimeRuleFunction(fn)

		local ok, res = pcall(fn, self.domoticz)

		if (not ok) then
			utils.log('Error executing custom timer function.', utils.LOG_ERROR)
			utils.log(res, utils.LOG_ERROR)
			if (_G.TESTMODE) then
				print(res)
			end
			return false
		end
		return res
	end

	function self.handleEvents(events, subject)

		local originalLogLevel = _G.logLevel -- a script can override the level

		local function restoreLogging()
			_G.logLevel = originalLogLevel
			_G.logMarker = nil
		end

		if (type(events) ~= 'table') then
			restoreLogging()
			return
		end

		for eventIdx, eventHandler in pairs(events) do

			if (eventHandler.logging) then
				if type(eventHandler.logging) ~= 'table' then
					if type(eventHandler.logging) == 'number' then
						local level = eventHandler.logging
						eventHandler.logging = {}
						eventHandler.logging.level = level
					end
					utils.log(eventHandler.type .. ' script ' .. eventHandler.name .. '.lua has a malformed logging section. Check the documentation.', utils.LOG_FORCE)
				end

				if (eventHandler.logging.level ~= nil) then
					_G.logLevel = eventHandler.logging.level
				end
				if (eventHandler.logging.marker ~= nil) then
					_G.logMarker = eventHandler.logging.marker
				end
			end

			local moduleLabel
			local moduleLabelInfo = ''
			local triggerInfo
			local scriptType = eventHandler.type == 'external' and 'external script: ' or 'internal script: '

			if (eventHandler.type == 'external') then
				moduleLabel = eventHandler.name .. '.lua'
			else
				moduleLabel = eventHandler.name .. ''
			end

			domoticz.moduleLabel = eventHandler.name:gsub('%.lua','') -- used in dynamic logmarker

			local baseType = subject and subject.baseType or ''

			if (baseType == domoticz.BASETYPE_DEVICE) then
				moduleLabelInfo = ' Device: "' .. subject.name .. ' (' .. subject.hardwareName .. ')", Index: ' .. tostring(subject.id)
			elseif (baseType == domoticz.BASETYPE_VARIABLE) then
				moduleLabelInfo = ' Variable: "' .. subject.name .. '" Index: ' .. tostring(subject.id)
			elseif (baseType == domoticz.BASETYPE_SECURITY) then
				moduleLabelInfo = ' Security: "' .. subject.name .. '"'
			elseif (baseType == domoticz.BASETYPE_SCENE or baseType == domoticz.BASETYPE_GROUP) then
				moduleLabelInfo = (subject.baseType == 'scene' and ' Scene' or ' Group') .. ': "' .. subject.name .. '", Index: ' .. tostring(subject.id)
			elseif (baseType == domoticz.BASETYPE_HTTP_RESPONSE) then
				moduleLabelInfo = ' HTTPResponse: "' .. subject.callback .. '"'
			elseif (baseType == domoticz.BASETYPE_SYSTEM_EVENT) then
				moduleLabelInfo = ' Domoticz event: "' .. subject.name .. '"'
			elseif (baseType == domoticz.BASETYPE_CUSTOM_EVENT) then
				moduleLabelInfo = ' Custom event: "' .. subject.name .. '"'
			end

			triggerInfo = eventHandler.trigger and ( ', trigger: "' .. eventHandler.trigger .. '"' ) or ''

			local clockTimeStampAtStart = os.clock()
			local timeStampAtStart = os.time()

			utils.log('------ Start ' .. scriptType .. moduleLabel ..':' .. moduleLabelInfo .. triggerInfo, utils.LOG_MODULE_EXEC_INFO)
			self.callEventHandler(eventHandler, subject)

			local clockTimeSpend = os.clock() - clockTimeStampAtStart
			local realTimeSpend = os.time() - timeStampAtStart

			if realTimeSpend > 9 or clockTimeSpend > 7 then
				utils.log('------ Finished ' .. moduleLabel .. ' after >' .. realTimeSpend .. ' seconds. (using '.. tostring(clockTimeSpend):sub(1,5) .. ' seconds CPU time !)' , utils.LOG_ERROR)
			elseif realTimeSpend > 6 or clockTimeSpend > 5 then
				utils.log('------ Finished ' .. moduleLabel .. ' after >' .. realTimeSpend .. ' seconds. (using '.. tostring(clockTimeSpend):sub(1,5) .. ' seconds CPU time !)' , utils.LOG_FORCE)
			else
				utils.log('------ Finished ' .. moduleLabel , utils.LOG_MODULE_EXEC_INFO)
			end

			if (tonumber(globalvariables['dzVents_log_level']) == utils.LOG_DEBUG or TESTMODE ) then
				local moduleSummary = globalvariables.script_path  .. 'module.log'
				utils.log('Debug: Writing module summary to ' .. moduleSummary ,utils.LOG_FORCE)

				local f = io.open(moduleSummary, 'a' )
				f:write(
					os.date('%x %X - ',timeStampAtStart) .. os.date('%x %X ') .. '(' ..
					string.format('%02d', realTimeSpend) .. ' - ' .. string.format('%.4f',clockTimeSpend) ..
					') ' .. string.format('%35s',moduleLabel) .. ' <<' .. moduleLabelInfo ..
					( eventHandler.trigger and ( ' timer: "' .. eventHandler.trigger .. '"' ) or '') ,'\n')
				f:close()
			end

			restoreLogging()
		end
	end

	function self.processTimeRules(timeRules, testTime)
		-- accepts a table of timeDefs, if one of them matches with the
		-- current time, then it returns true
		-- otherwise it returns false

		local now
		if (testTime == nil) then
			now = self.domoticz.time
		else
			now = testTime
		end

		for i, _rule in pairs(timeRules) do

			if (type(_rule) == 'function') then
				return self.processTimeRuleFunction(_rule), 'function'
			end

			local rule = string.lower(_rule)
			if (now.matchesRule(rule)) then
				return true, _rule
			end
		end

		return false
	end

	function self.checkSecurity(securityDefs, security)

		for i, def in pairs(securityDefs) do
			if (def == security) then
				return true, def
			end
		end

		return false
	end

	local function addBindingEvent(bindings, event, module)
		if (bindings[event] == nil) then
			bindings[event] = {}
		end
		table.insert(bindings[event], module)
	end

	local function loadEventScripts()
		local scripts = {}
		local errModules = {}
		local internalScripts
		local ok, diskScripts, externalNames, moduleName, i, event, j, err
		local modules = {}
		local scripts = {}

		ok, diskScripts, externalNames = pcall(self.scandir, _G.scriptsFolderPath, 'external')

		if (not ok) then
			utils.log(diskScripts, utils.LOG_ERROR)
		else
			modules = diskScripts
		end

		ok = true

		ok, internalScripts = pcall(self.scandir, _G.generatedScriptsFolderPath, 'internal')

		if (not ok) then
			utils.log(internalScripts, utils.LOG_ERROR)
		else
			for i, internal in pairs(internalScripts) do
				if (externalNames[internal.name]) then
					-- oops already there, skipping
					utils.log('There is already an external script with the name "' .. internal.name .. '.lua". Please rename in the internal script.', utils.LOG_ERROR)
				else
					table.insert(modules, internal)
				end
			end
		end

		-- extract global_data module and make sure it is the first in the list
		-- because then peeps can create some globals available everywhere before the other modules load
		local globalIx
		for i, moduleInfo in pairs(modules) do
			if (moduleInfo.name == GLOBAL_DATA_MODULE) then
				globalIx = i
				break
			end
		end

		if (globalIx ~= nil and globalIx > 1) then
			local globalModule = modules[globalIx]
			table.remove(modules, globalIx)
			table.insert(modules, 1, globalModule)
		end

		for i, moduleInfo in ipairs(modules) do

			local module, skip

			local moduleName = moduleInfo.name
			local logScript = (moduleInfo.type == 'external' and 'Script ' or 'Internal script ')

			_G.domoticz = {
				['LOG_INFO'] = utils.LOG_INFO,
				['LOG_MODULE_EXEC_INFO'] = utils.LOG_MODULE_EXEC_INFO,
				['LOG_DEBUG'] = utils.LOG_DEBUG,
				['LOG_ERROR'] = utils.LOG_ERROR,
				['SECURITY_DISARMED'] = self.domoticz.SECURITY_DISARMED,
				['SECURITY_ARMEDAWAY'] = self.domoticz.SECURITY_ARMEDAWAY,
				['SECURITY_ARMEDHOME'] = self.domoticz.SECURITY_ARMEDHOME,
			}

			ok = true

			ok, module = pcall(require, moduleName)

			_G.domoticz = nil

			if (ok) then

				if (moduleName == GLOBAL_DATA_MODULE) then
					if (module.data ~= nil) then
						globalsDefinition = module.data
						if (_G.TESTMODE) then
							self.globalsDefinition = globalsDefinition
						end
					end

					if (module.helpers ~= nil) then
						self.domoticz.helpers = module.helpers
					end

				else
					if (type(module) == 'table') then
						skip = false

						if (module.active ~= nil) then
							local active = false
							if (type(module.active) == 'function') then
								active = module.active(self.domoticz)
							else
								active = module.active
							end

							if (not active) then
								skip = true
							end
						end

						if (not skip) then
							if (module.on ~= nil and module['execute'] ~= nil) then
								module.name = moduleName
								module.type = moduleInfo.type
								module.dataFileName = '__data_' .. moduleName
								module.dataFilePath = _G.dataFolderPath .. '/__data_' .. moduleName .. '.lua'

								table.insert(scripts, module)
							else
								utils.log(logScript .. moduleName .. '.lua has no "on" and/or "execute" section, not a dzVents module. Skipping', utils.LOG_DEBUG)
								--table.insert(errModules, moduleName)
							end
						end
					else
						utils.log(logScript .. moduleName .. '.lua is not a dzVents module. Skipping', utils.LOG_DEBUG)
						--table.insert(errModules, moduleName)
					end
				end
			else
				table.insert(errModules, moduleName)
				utils.log(module, utils.LOG_ERROR)
			end
		end

		return scripts, errModules
	end

	function self.getEventBindings(mode, testTime)
		local bindings = {}
		local ok, i, event, j, device, err
		local modules = {}

		if not self.scripts then
			self.scripts, self.errModules = loadEventScripts()
		end

		if (mode == nil) then mode = 'device' end

		for i, module in pairs(self.scripts) do

			local logScript = (module.type == 'external' and 'Script ' or 'Internal script ')

			for j, event in pairs(module.on) do
				if type(j) ~= 'string' or type(event) ~= 'table' or validEventTypes:find(j) == nil then
					if not self.scripts[i].invalidOnSection then
						self.scripts[i].invalidOnSection = true
						if type(j) == "string" and validEventTypes:find(j) == nil then
							if inValidEventTypes:find(j) then
								utils.log('You entered "' .. tostring(j) .. '" in the on = section. Probably you misplaced your curly brackets', utils.LOG_FORCE)
							else
								utils.log('Valid eventTypes are: ' .. validEventTypes, utils.LOG_DEBUG )
								utils.log('You entered "' .. tostring(j) .. '" in the on = section. Maybe you meant "' .. utils.fuzzyLookup(j, utils.stringSplit(validEventTypes,',')) ..'"?', utils.LOG_FORCE)
							end

						else
							utils.log('You entered "' .. utils.toStr(event) .. '" as trigger but the eventType is not (properly) set')
						end
						utils.log(logScript .. module.name .. '.lua has an invalid on = section;  please check the wiki. Skipping it until fixed.', utils.LOG_ERROR)
					end

				else
					if (mode == 'timer' and j == 'timer') then
						-- { ['timer'] = { 'every minute ', 'every hour' } }

						if type(event) ~= 'table' then
							utils.log(logScript .. module.name .. '.lua has a malformed timer = section. Check the documentation.', utils.LOG_FORCE)
							event = { event }
						end

						local triggered, def = self.processTimeRules(event)
						if (triggered) then
							-- this one can be executed
							module.trigger = def
							event.type = 'timer'
							table.insert(bindings, module)
						end
					elseif (mode == 'device' and j == 'devices') then

						-- { ['devices'] = { 'devA', ['devB'] = { ..timedefs }, .. }

						for devIdx, devName in pairs(event) do

							-- detect if devName is of the form ['devB'] = { 'every hour' }
							if (type(devName) == 'table') then
								local triggered, def = self.processTimeRules(devName, testTime)
								if (triggered) then
									addBindingEvent(bindings, devIdx, module)
								end
							else
								-- a single device name (or id)
								addBindingEvent(bindings, devName, module)
							end
						end
					elseif (mode == 'scenegroups' and (j == 'scenes' or j == 'groups')) then

						-- { ['scenes'] = { 'scA', ['scB'] = { ..timedefs }, .. }

						for scgrpIdx, scgrpName in pairs(event) do

							-- detect if scgrpName is of the form ['devB'] = { 'every hour' }
							if (type(scgrpName) == 'table') then
								local triggered, def = self.processTimeRules(scgrpName, testTime)
								if (triggered) then
									addBindingEvent(bindings, scgrpIdx, module)
								end
							else
								-- a single scene or group name (or id)
								addBindingEvent(bindings, scgrpName, module)
							end
						end
					elseif (mode == 'variable' and j == 'variables') then
						-- { ['variables'] = { 'varA', 'varB' }
						for varIdx, varName in pairs(event) do
							addBindingEvent(bindings, varName, module)
						end
					elseif (mode == 'security' and j == 'security') then
						local triggered, def = self.checkSecurity(event, self.domoticz.security)
						if (triggered) then
							table.insert(bindings, module)
							module.trigger = def
						end
					elseif (mode == 'httpResponse' and j == 'httpResponses') then
						-- { ['httpResponses'] = { 'callbackA', 'callbackB' }
						for i, callbackName in pairs(event) do
							addBindingEvent(bindings, callbackName, module)
						end
					elseif (mode == 'system' and j == 'system') then
						-- { ['system'] = { 'start', 'end' }
						for evtIdx, systemEvent in pairs(event) do
							-- detect if system is of the form ['start'] = { 'every hour' }
							if (type(systemEvent) == 'table') then
								local triggered, def = self.processTimeRules(systemEvent, testTime)
								if (triggered) then
									addBindingEvent(bindings, evtIdx, module)
								end
							else
								-- a single system event
								addBindingEvent(bindings, systemEvent, module)
							end
						end
					elseif (mode == 'customEvents' and j == 'customEvents') then
						-- { customEvents = { 'myEvent', ['anotherEvent'] = { ... timeDefs ... } }

						for evtIdx, customEventName in pairs(event) do
							-- detect if customEvent is of the form ['myEvent'] = { 'every hour' }
							if (type(customEventName) == 'table') then
								local triggered, def = self.processTimeRules(customEventName, testTime)
								if (triggered) then
									addBindingEvent(bindings, evtIdx, module)
								end
							else
								-- a single custom event
								addBindingEvent(bindings, customEventName, module)
							end
						end
					end
				end
			end
		end
		return bindings, self.errModules
	end

	function self.dumpCommandArray(commandArray, fromIndex, force)
		local printed = false
		local level = utils.LOG_DEBUG

		if (fromIndex == nil) then
			fromIndex = 1
		end

		if (force == true and force ~= nil or globalvariables['testmode'] == true) then
			level = utils.LOG_INFO
		end

		for k, v in pairs(commandArray) do
			if ((fromIndex ~= nil and k >= fromIndex) or fromIndex == nil) then
				if (not printed) then
					utils.log('Commands sent to Domoticz: ', level)
				end
				if (type(v) == 'table') then
					for kk, vv in pairs(v) do
						utils.log('- ' .. kk .. ' = ' .. _.str(vv), level)
					end
				else
					utils.log(k .. ' = ' .. v, level)
				end
				printed = true
			end
		end
		if (printed) then utils.log('=====================================================', level) end
	end

	function self.findScriptForTarget(target, allEventScripts)
		-- event could be like: myPIRLivingRoom
		-- or myPir(.*)

		--[[

			allEventScripts is a dictionary where
			each key is the name or id of a device and the value
			is a table with all the modules for this device

			{
				['myDevice'] = {
					modA, modB, modC
				},
				['anotherDevice'] = {
					modD
				},
				12 = {
					modE, modF
				},
				['myDev*'] = {
					modG, modH
				}
			}

		]]--

		local modules

		-- only search for named and wildcard triggers,
		-- id is done later

		for scriptTrigger, scripts in pairs(allEventScripts) do

			local function strFind(str, key)
				return string.find(str, key)
			end

			local ok, res = pcall(strFind, scriptTrigger, '*');
			if not ok then
				utils.log('Script name: ' .. scripts[1].name  .. '.lua, has a malformed on = section. The trigger = ' .. _.str(scriptTrigger) , utils.LOG_ERROR)
				allEventScripts[scriptTrigger] = ''
			elseif res then -- a wild-card was used
				scriptTrigger = ('^' .. scriptTrigger:gsub("[%^$]","."):gsub("*", ".*") .. '$')

				local function sMatch(text, match) -- specialized sanitized match function to allow combination of Lua magic chars in wildcards
					local sanitizedMatch = match:gsub("([%%%(%)%[%]%+%-%?])", "%%%1") -- escaping all 'magic' chars except *, ., ^ and $
					return text:match(sanitizedMatch)
				end

				if sMatch(target, scriptTrigger) then
					if modules == nil then modules = {} end

					for i, mod in pairs(scripts) do
						table.insert(modules, mod)
					end

				end

			else
				if (scriptTrigger == target) then
					-- there is trigger for this target

					if modules == nil then modules = {} end

					for i, mod in pairs(scripts) do
						table.insert(modules, mod)
					end

				end
			end
		end

		return modules
	end

	function self.dispatchDeviceEventsToScripts(domoticz)

		if (domoticz == nil) then -- you can pass a domoticz object for testing purposes
			domoticz = self.domoticz
		end

		local allEventScripts = self.getEventBindings()

		domoticz.changedDevices().forEach( function(device)

			local scriptsToExecute = self.findScriptForTarget(device.name, allEventScripts)
			local idScripts = allEventScripts[device.id]

			local caSize = _.size(self.domoticz.commandArray)

			if (idScripts ~= nil) then
				-- merge id scripts with name scripts
				if (scriptsToExecute == nil) then
					scriptsToExecute = {}
				end
				for i, mod in pairs(idScripts) do
					table.insert(scriptsToExecute, mod)
				end
			end

			if (scriptsToExecute ~= nil) then
				utils.log('Handling events for: "' .. device.name .. '", value: "' .. tostring(device.state) .. '"', utils.LOG_MODULE_EXEC_INFO)
				self.handleEvents(scriptsToExecute, device)
				self.dumpCommandArray(self.domoticz.commandArray, caSize + 1)
			end

		end)

		return self.domoticz.commandArray
	end

	function self.dispatchSceneGroupEventsToScripts(domoticz)

		if (domoticz == nil) then -- you can pass a domoticz object for testing purposes
			domoticz = self.domoticz
		end

		local allEventScripts = self.getEventBindings('scenegroups')

		local processItem = function(item, loglabel)
			utils.log(loglabel .. '-event for: ' .. item.name .. ' value: ' .. tostring(item.state), utils.LOG_DEBUG)

			local scriptsToExecute = self.findScriptForTarget(item.name, allEventScripts)
			local idScripts = allEventScripts[item.id]
			local caSize = _.size(self.domoticz.commandArray)

			if (idScripts ~= nil) then
				-- merge id scripts with name scripts
				if (scriptsToExecute == nil) then
					scriptsToExecute = {}
				end
				for i, mod in pairs(idScripts) do
					table.insert(scriptsToExecute, mod)
				end
			end

			if (scriptsToExecute ~= nil) then
				utils.log('Handling events for: "' .. item.name .. '", value: "' .. tostring(item.state) .. '"', utils.LOG_MODULE_EXEC_INFO)
				self.handleEvents(scriptsToExecute, item)
				self.dumpCommandArray(self.domoticz.commandArray, caSize + 1)
			end

		end

		domoticz.changedScenes().forEach(function(scene)
			processItem(scene, 'Scene')
		end)

		domoticz.changedGroups().forEach(function(group)
			processItem(group, 'Group')
		end)

		return self.domoticz.commandArray
	end

	function self.dispatchTimerEventsToScripts()
		local scriptsToExecute = self.getEventBindings('timer')

		self.handleEvents(scriptsToExecute)
		self.dumpCommandArray(self.domoticz.commandArray)

		return self.domoticz.commandArray
	end

	function self.dispatchSecurityEventsToScripts()

		local updates =_G.securityupdates

		if (updates ~= nil) then

			for i, securityState in pairs(updates) do

				local security = {
					baseType = domoticz.BASETYPE_SECURITY,
					name = securityState
				}

				local caSize = _.size(self.domoticz.commandArray)

				self.domoticz.security = securityState

				local scriptsToExecute = self.getEventBindings('security', nil)

				self.handleEvents(scriptsToExecute, security)

				self.dumpCommandArray(self.domoticz.commandArray, caSize + 1)
			end
		end

		return self.domoticz.commandArray
	end

	function self.dispatchVariableEventsToScripts(domoticz)
		if (domoticz == nil) then -- you can pass a domoticz object for testing purposes
			domoticz = self.domoticz
		end

		local allEventScripts = self.getEventBindings('variable')

		domoticz.changedVariables().forEach(function(variable)

			local caSize = _.size(self.domoticz.commandArray)

			-- first search by name

			local scriptsToExecute = self.findScriptForTarget(variable.name, allEventScripts)
			if (scriptsToExecute == nil) then
				-- search by id
				scriptsToExecute = allEventScripts[variable.id]
			end

			if (scriptsToExecute ~= nil) then
				utils.log('Handling variable-events for: "' .. variable.name .. '", value: "' .. tostring(variable.value) .. '"', utils.LOG_MODULE_EXEC_INFO)
				self.handleEvents(scriptsToExecute, variable)
				self.dumpCommandArray(self.domoticz.commandArray, caSize + 1)
			end
		end)

		return self.domoticz.commandArray
	end

	function self.dispatchHTTPResponseEventsToScripts(domoticz)
		if (domoticz == nil) then -- you can pass a domoticz object for testing purposes
			domoticz = self.domoticz
		end

		local httpResponseScripts = self.getEventBindings('httpResponse')

		local responses =_G.httpresponse

		if (responses ~= nil) then
			for i, response in pairs(responses) do

				response.baseType = domoticz.BASETYPE_HTTP_RESPONSE
				local callback = response.callback
				local caSize = _.size(self.domoticz.commandArray)

				local scriptsToExecute = self.findScriptForTarget(callback, httpResponseScripts)

				if (scriptsToExecute ~= nil) then
					utils.log('Handling httpResponse-events for: "' .. callback .. '"', utils.LOG_MODULE_EXEC_INFO)
					self.handleEvents(scriptsToExecute, response)
					self.dumpCommandArray(self.domoticz.commandArray, caSize + 1)
				end

			end

		end

		return self.domoticz.commandArray

	end

	function self.dispatchSystemEventsToScripts(domoticz)
		if (domoticz == nil) then
			-- you can pass a domoticz object for testing purposes
			domoticz = self.domoticz
		end

		if (_G.notification == nil) then
			return
		end

		local systemEventScripts = self.getEventBindings('system', nil)
		local systemEvents =_G.notification.domoticz

		if (systemEvents ~= nil) then
			for i, event in pairs(systemEvents) do

				event.name = SystemEvent(self.domoticz, event)["type"]
				if event.name == nil then
					utils.log('System event "' .. event.type .. '" is not supported yet.', utils.LOG_ERROR)
				else
					event.baseType = domoticz.BASETYPE_SYSTEM_EVENT

					local caSize = _.size(self.domoticz.commandArray)
					local scriptsToExecute = self.findScriptForTarget(event.name, systemEventScripts)

					if (scriptsToExecute ~= nil) then
						utils.log('Handling system event for: "' .. event.name .. '"', utils.LOG_MODULE_EXEC_INFO)
						self.handleEvents(scriptsToExecute, event)
						self.dumpCommandArray(self.domoticz.commandArray, caSize + 1)
					end
				end
			end
		end

		return self.domoticz.commandArray
	end

	function self.dispatchCustomEventsToScripts(domoticz)
		if (domoticz == nil) then
			-- you can pass a domoticz object for testing purposes
			domoticz = self.domoticz
		end

		if (_G.notification == nil or _G.notification.customevent == nil ) then
			return
		end

		for _, row in ipairs(_G.notification.customevent) do
			 if row.data.name:match('^___%a*__$') then
				table.insert(self.domoticz.commandArray, { [ row.data.name:sub(4,-3) ] = row.data.data })
			 end
		end

		local customEventScripts = self.getEventBindings('customEvents', nil)
		local customEvents = _G.notification.customevent

		if (customEvents ~= nil) then
			for i, customEvent in pairs(customEvents) do

				customEvent.name = customEvent.data.name
				customEvent.baseType = domoticz.BASETYPE_CUSTOM_EVENT

				local caSize = _.size(self.domoticz.commandArray)

				local scriptsToExecute = self.findScriptForTarget(customEvent.name, customEventScripts)

				if (scriptsToExecute ~= nil) then
					utils.log('Handling Domoticz custom event for: "' .. customEvent.name .. '"', utils.LOG_MODULE_EXEC_INFO)
					self.handleEvents(scriptsToExecute, customEvent)
					self.dumpCommandArray(self.domoticz.commandArray, caSize + 1)
				end
			end
		end

		return self.domoticz.commandArray
	end

	function self.getEventSummary(domoticz)
		if (domoticz == nil) then -- you can pass a domoticz object for testing purposes
			domoticz = self.domoticz
		end

		local items = {}
		local length = 0

		domoticz.changedDevices().forEach( function(device)
			table.insert(items, '- Device: ' .. device.name)
			length = length + 1
		end)

		domoticz.changedVariables().forEach(function(variable)
			table.insert(items, '- Variable: ' .. variable.name)
			length = length + 1
		end)

		local securityUpdates =_G.securityupdates
		if (securityUpdates ~= nil) then
			for i, securityState in pairs(securityUpdates) do
				table.insert(items, '- Security: ' .. securityState)
				length = length + 1
			end
		end

		domoticz.changedScenes().forEach(function(scene)
			table.insert(items, '- Scene: ' .. scene.name)
			length = length + 1
		end)

		domoticz.changedGroups().forEach(function(group)
			table.insert(items, '- Group: ' .. group.name)
			length = length + 1
		end)

		local responses =_G.httpresponse

		if (responses ~= nil) then
			for i, response in pairs(responses) do
				table.insert(items, '- HTTPResponse: ' .. response.callback)
				length = length + 1
			end
		end

		local domoticzEvents = _G.notification and _G.notification.domoticz or nil

		if (domoticzEvents ~= nil) then
			for i, domoticzEvent in pairs(domoticzEvents) do
				table.insert(items, '- Domoticz: ' .. domoticzEvent.type) -- .. ' - ' .. domoticzEvent.status)
				length = length + 1
			end
		end

		local customEvents = _G.notification and _G.notification.customEvents or nil

		if (customEvents ~= nil) then
			for i, customEvent in pairs(customEvents) do
				table.insert(items, '- Custom: ' .. customEvent.type) -- .. ' - ' .. customEvent.status)
				length = length + 1
			end
		end

		return items, length;

	end

	return self
end

return EventHelpers
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

The update runs normal but it installs version 12705. Just checked and it still says that there is no update available.

So I changed the EventHelpers.lua file. But now what? I still see the same errors in the log, do I need to look somewhere else or do I have to look for something specific?
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Sunday 29 November 2020 11:28 So I changed the EventHelpers.lua file. But now what? I still see the same errors in the log, do I need to look somewhere else or do I have to look for something specific?
If you have Debug on for dzVents you now should see a file <domoticz dir>/dzVents/scripts/module.log

sudo tail -f <domoticz dir>/dzVents/scripts/module.log should show you lines like

Code: Select all

11/29/20 11:54:02 - 11/29/20 11:54:02 (00 - 0.0129)                        PVOutput.lua << timer: "every 6 minutes"
11/29/20 11:54:02 - 11/29/20 11:54:02 (00 - 0.0237)                         splitP1.lua << timer: "every 1 minutes"
11/29/20 11:54:03 - 11/29/20 11:54:03 (00 - 0.0505)                  getEnergyCosts.lua << HTTPResponse: "dailyEnergyCost_energy"
11/29/20 11:54:04 - 11/29/20 11:54:04 (00 - 0.0038)                        PVOutput.lua << HTTPResponse: "PVOutput"
11/29/20 11:55:00 - 11/29/20 11:55:00 (00 - 0.0671)               Bathroom humidity.lua << timer: "every minute between 05:15 and 23:30"
11/29/20 11:55:00 - 11/29/20 11:55:01 (01 - 0.0131)                      splitDaily.lua << timer: "every 5 minutes"
between () you see clocktime in seconds - CPU time in seconds)
Maybe this will give you more information on your issue.
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

Don't see anything strange

Code: Select all

11/29/20 12:32:10 - 11/29/20 12:32:11 (01 - 0.3582)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:32:16 - 11/29/20 12:32:18 (02 - 0.4877)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:32:23 - 11/29/20 12:32:25 (02 - 0.3597)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:32:35 - 11/29/20 12:32:36 (01 - 0.3578)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:32:43 - 11/29/20 12:32:45 (02 - 0.3946)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:32:57 - 11/29/20 12:32:58 (01 - 0.3883)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:33:03 - 11/29/20 12:33:04 (01 - 0.3678)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:33:16 - 11/29/20 12:33:17 (01 - 0.4505)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:33:23 - 11/29/20 12:33:24 (01 - 0.3717)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:33:35 - 11/29/20 12:33:37 (02 - 0.4186)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:33:42 - 11/29/20 12:33:43 (01 - 0.3773)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:33:54 - 11/29/20 12:33:56 (02 - 0.3425)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:34:02 - 11/29/20 12:34:03 (01 - 0.3821)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:34:14 - 11/29/20 12:34:16 (02 - 0.3911)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:34:28 - 11/29/20 12:34:29 (01 - 0.4024)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:34:40 - 11/29/20 12:34:41 (01 - 0.3648)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:34:46 - 11/29/20 12:34:47 (01 - 0.4038)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:34:54 - 11/29/20 12:34:55 (01 - 0.3683)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:35:12 - 11/29/20 12:35:12 (00 - 0.0868)                          blinds.lua << timer: "every 5 minutes"
11/29/20 12:35:12 - 11/29/20 12:35:15 (03 - 0.8917)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:35:20 - 11/29/20 12:35:22 (02 - 0.4792)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:35:28 - 11/29/20 12:35:30 (02 - 0.4508)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
11/29/20 12:35:30 - 11/29/20 12:35:30 (00 - 0.0726)                          blinds.lua << HTTPResponse: "AM43-Blinds_Response"
11/29/20 12:35:36 - 11/29/20 12:35:37 (01 - 0.4677)                     consumption.lua << Device: "Energie (P1) (P1-poort)", Index: 175
While the log of Domoticz shows

Code: Select all

2020-11-29 12:29:53.382 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:30:05.560 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:30:23.294 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:30:36.331 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:30:48.827 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:31:00.565 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:35:12.368 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:38:27.314 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:38:41.116 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:38:55.260 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:38:57.131 Error: SMTP Mailer: Error sending Email to: <[email protected]> !

2020-11-29 12:38:57.131 Error: libcurl: (28)

2020-11-29 12:38:57.131 Error: SSL connection timeout

2020-11-29 12:38:57.131

2020-11-29 12:38:57.132 Error: Failed to send Email notification!

2020-11-29 12:38:57.379 Error: FCM: Could not send message, HTTP Error

2020-11-29 12:38:58.808 Error: Pushover: Could not send message!

2020-11-29 12:39:07.607 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:39:23.316 Error: dzVents: Error: (3.0.17) Consumption: ------ Finished consumption.lua after >10 seconds. (using 1.967 seconds CPU time !)

2020-11-29 12:39:23.605 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:39:36.298 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:39:56.567 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

2020-11-29 12:40:13.964 Error: EventSystem: Warning!, lua script /home/pi/domoticz/dzVents/runtime/dzVents.lua has been running for more than 10 seconds

I think at this moment it's easier to do a complete reinstall, or do you want to learn from this and need some more info?
User avatar
waaren
Posts: 6028
Joined: Tuesday 03 January 2017 14:18
Target OS: Linux
Domoticz version: Beta
Location: Netherlands
Contact:

Re: dzVents errors

Post by waaren »

roblom wrote: Sunday 29 November 2020 12:50 Don't see anything strange
I think at this moment it's easier to do a complete reinstall, or do you want to learn from this and need some more info?
What I consider strange is that your consumption script takes a relatively long time and that your system seems to be very slow. Cannot tell if this is because of CPU / Memory or IO but the values you show here are not what I see on any of my systems.

Can you share the consumption.lua script and your hardware details? (CPU, memory, storage, LAN)
Debian buster, bullseye on RPI-4, Intel NUC.
dz Beta, Z-Wave, RFLink, RFXtrx433e, P1, Youless, Hue, Yeelight, Xiaomi, MQTT
==>> dzVents wiki
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

Since 21 November the CPU is really high, average is about 92%.
When I use "top" it shows Domoticz is taking up 90%.
I use a Raspberry 3B, storage is 16Gb SanDisk sd-card which is 17% in use. Maximum memory usage is also risen since the 21th.

Consumption script

Code: Select all

--[[
	This script collects the values below from Domoticz
	  * The Power and energy values (import and export) from a smartmeter 
	  * The Power and energy values from a Solar power inverter
	
	It then calculates the (average) consumed power and energy from the values above with the formula's
	  * EnergyConsumption = EnergyGeneration + EnergyImport - EnergyExport
	  * PowerConsumption = PowerGeneration + PowerImport - PowerExport
	The average power is calculated because it fluctuates much and can cause negative consumption due to timing issues.
	
	It then updates a virtual device which displays the consumed power and energy in Domoticz
	
	The base of this scrip comes from
	https://www.domoticz.com/forum/viewtopic.php?f=61&t=4714&p=251778#p251778
]]--

local smartMeterIDX = 175	-- IDX of Smart meter device
local solarPanelsIDX = 96	-- IDX of Solar Panels device
local consumptionIDX = 93	-- IDX of Own Usage / calculated value / dummy Elekta kWh + counter

return {

	active = true,
	
	on =
	{
		devices = 	
		{
			-- Devices on which this calculation script should trigger.
			smartMeterIDX,
			solarPanelsIDX,
		}
	},

	logging =
	{
		--level = domoticz.LOG_DEBUG,
		level = domoticz.LOG_ERROR,
		marker = 'Consumption',
	},
		
	data =
	{
		PowerImportHistory = { history = true, maxMinutes = 4 },
		PowerExportHistory = { history = true, maxMinutes = 4 },
		PowerGenerationHistory = { history = true, maxItems = 24 },
	},

		execute = function(dz, item)
		----------------------------------------------------------------------------------------------------------
		-- Domoticz IDX of the needed devices
		----------------------------------------------------------------------------------------------------------
		local Smartmeter = dz.devices(smartMeterIDX)
		local Generation = dz.devices(solarPanelsIDX)
		local Consumption = dz.devices(consumptionIDX)

		----------------------------------------------------------------------------------------------------------
		-- Smartmeter
		----------------------------------------------------------------------------------------------------------
		-- Energy import and export
		local EnergyImportLow = Smartmeter.usage1
		dz.log('EnergyImportLow	 = ' .. EnergyImportLow/1000 .. ' kWh ', dz.LOG_DEBUG)
		local EnergyImportHigh = Smartmeter.usage2
		dz.log('EnergyImportHigh = ' .. EnergyImportHigh/1000 .. ' kWh ', dz.LOG_DEBUG)
		local EnergyExportLow = Smartmeter.return1
		dz.log('EnergyExportLow	 = ' .. EnergyExportLow/1000 .. ' kWh ', dz.LOG_DEBUG)
		local EnergyExportHigh = Smartmeter.return2
		dz.log('EnergyExportHigh = ' .. EnergyExportHigh/1000 .. ' kWh ', dz.LOG_DEBUG)

		local EnergyImport = EnergyImportLow + EnergyImportHigh
		local EnergyExport = EnergyExportLow + EnergyExportHigh
		
		-- Power import and export
		-- Add new data for calculating average
		if item == Smartmeter then
			dz.data.PowerImportHistory.add(Smartmeter.usage)
			dz.data.PowerExportHistory.add(Smartmeter.usageDelivered)
		end


		----------------------------------------------------------------------------------------------------------
		-- Generation
		----------------------------------------------------------------------------------------------------------
		-- Energy generated
		local EnergyGeneration = Generation.WhTotal
		dz.log('EnergyGeneration = ' .. EnergyGeneration .. ' Wh ', dz.LOG_DEBUG)
		
		-- Power generated
		-- Add new data for calculating average
		if item == Generation then
			dz.data.PowerGenerationHistory.add(Generation.actualWatt)
		end


		----------------------------------------------------------------------------------------------------------
		-- Check historical storage entries
		----------------------------------------------------------------------------------------------------------
		if dz.data.PowerImportHistory.size < 3 or dz.data.PowerGenerationHistory.size < 3 then
			dz.log('Not enough data points yet to produce something useful', dz.LOG_DEBUG)
			dz.log('PowerImportHistory entries = ' .. dz.data.PowerImportHistory.size, dz.LOG_DEBUG)
			dz.log('PowerGeneration entries = ' .. dz.data.PowerGenerationHistory.size, dz.LOG_DEBUG)
			return
		end


		----------------------------------------------------------------------------------------------------------
		-- Calculate
		----------------------------------------------------------------------------------------------------------
		-- Average power generated
		local PowerGeneration = dz.utils.round(dz.data.PowerGenerationHistory.avgSince('00:02:00'), 2)
		dz.log('PowerGeneration = ' .. PowerGeneration .. ' W ', dz.LOG_DEBUG)

		-- Average power imported and exported
		local PowerImport = dz.utils.round(dz.data.PowerImportHistory.avgSince('00:02:00'), 2)
		dz.log('PowerImport = ' .. PowerImport .. ' W ', dz.LOG_DEBUG)
		local PowerExport = dz.utils.round(dz.data.PowerExportHistory.avgSince('00:02:00'), 2)
		dz.log('PowerExport = ' .. PowerExport .. ' W ', dz.LOG_DEBUG)

		-- Power consumption
		local PowerConsumption = dz.utils.round(PowerGeneration + PowerImport - PowerExport, 0)
		dz.log('PowerConsumption = ' .. PowerConsumption .. ' W ', dz.LOG_DEBUG)

		-- Energy consumption
		local EnergyConsumption = dz.utils.round(EnergyGeneration + EnergyImport - EnergyExport, 0)
		dz.log('EnergyConsumption = ' .. EnergyConsumption/1000 .. ' kWh ', dz.LOG_DEBUG)
		
		----------------------------------------------------------------------------------------------------------
		-- Debug for negative power consumption
		----------------------------------------------------------------------------------------------------------
		if PowerConsumption < 0 then
		dz.log('(NEGATIVE) PowerConsumption = PowerGeneration + PowerImport - PowerExport', dz.LOG_ERROR)
			dz.log('' .. PowerConsumption .. ' W = ' .. PowerGeneration .. ' W + ' .. PowerImport .. ' W - ' .. PowerExport, dz.LOG_ERROR)
		else
			dz.log('PowerConsumption = PowerGeneration + PowerImport - PowerExport', dz.LOG_INFO)
			dz.log('' .. PowerConsumption .. ' W = ' .. PowerGeneration .. ' W + ' .. PowerImport .. ' W - ' .. PowerExport, dz.LOG_INFO)
		end
		
		----------------------------------------------------------------------------------------------------------
		-- Update device
		----------------------------------------------------------------------------------------------------------
		Consumption.updateElectricity(PowerConsumption, EnergyConsumption)

	end

}
roblom
Posts: 408
Joined: Wednesday 26 February 2014 15:28
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: the Netherlands
Contact:

Re: dzVents errors

Post by roblom »

When DzEvents is on and no scripts in the script folder, CPU is about 10-15%. As soon as I put one script in the DzEvents script folder the CPU usage raises to a 50-60%. Which script doesn't seems to make any difference.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 1 guest