Use an error in log as a trigger

[bldewit]

Hello,

It would be great if we could use an error message in the log as a trigger for a script. This way one could send a message after not being able to connect to a URL when switching network connected devices or for instance send the same command again after X seconds.

Is there a way to read the log and use the text in (e.g.) a LUA script?

Thx!

Bart

[emme]

it would be better to increase the error handling in the script rather than use an error as a trigger


you'd better have a look to LUA instructions like assert() and pcall()

http://luatut.com/assert.html
https://www.lua.org/pil/8.4.html

[waaren]

It would be great if we could use an error message in the log as a trigger for a script. This way one could send a message after not being able to connect to a URL when switching network connected devices or for instance send the same command again after X seconds.
Is there a way to read the log and use the text in (e.g.) a LUA script?

Maybe this can get you started.

When not yet familiar with dzVents please start with reading Get started Before implementing. Special attention please for
"In Domoticz go to Setup > Settings > Other and in the section EventSystem make sure the checkbox 'dzVents disabled' is not checked. Also make sure that in the Security section in the settings you allow 127.0.0.1 to not need a password. dzVents uses that port to send certain commands to Domoticz. Finally make sure you have set your current location in Setup > Settings > System > Location, otherwise there is no way to determine nighttime/daytime state."

[bldewit]

Thx @waaren and @emme, that seems to do the check i am looking for. Never worked with dzVents yet, so i will have to get into that. I will check it out!

Check the log for the error, check the state of the corresponding switch and set that switch to the same state again, that should do the trick then...
Any coding suggestions? All help is welcome since I am a newbee to dzVents...

[waaren]

Any coding suggestions? All help is welcome since I am a newbee to dzVents...

The post that I pointed to, contains already an example. If you start with that you can already extract the log message you want to work with, Take it from there. As dzVents is 100% Lua it should not give you any problems after you read the introduction. Don't get overwhelmed by the unknown names and structures you see. These are all just table- and functionNames.
Any dzVents script written by users are in fact just Lua functions.

[bldewit]

I have copied the script and adapted some commands. When testing I get the following error

2019-06-13 16:50:00.953 Status: dzVents: Info: getFailedLoging: ------ Start internal script: getFailedVent:, trigger: every 5 minutes
2019-06-13 16:50:00.960 Status: dzVents: Info: getFailedLoging: ------ Finished getFailedVent
2019-06-13 16:50:01.130 Error: Error opening url: http://127.0.0.1:8079/json.htm?type=com ... loglevel=4
2019-06-13 16:50:02.101 Status: dzVents: Info: Handling httpResponse-events for: "getDomoticzErrorLog
2019-06-13 16:50:02.110 Status: dzVents: Info: getFailedLoging: ------ Start internal script: getFailedVent: HTTPResponse: "getDomoticzErrorLog"
2019-06-13 16:50:02.114 Status: dzVents: Error (2.4.19): getFailedLoging: Access problem to log
2019-06-13 16:50:02.120 Status: dzVents: Info: getFailedLoging: ------ Finished getFailedVent

In passwordfree local networks I have

127.0.0.1;192.168.1.*;192.168.2.*

Any suggestions as to why this message is occurring? (@waaren maybe?)

[waaren]

I have copied the script and adapted some commands.
Any suggestions as to why this message is occurring?

In the original script this message will only be triggered if the response from the API call is not OK.
I have no idea what you changed so let start with you sharing your script in it's current state. I seem to have some problems receiving PM so you will have to share it here.

[bldewit]

Basically, in the first part I changed very little. Just the scriptname, marker and the errormessage to check for. Then in the business-end I check for a switch-state, resend the switchcommand that went wrong and notify the action. Obviously it doesn't get there right now:

Code: Select all

-- getFailedVent.lua

myHttpResponse = "getDomoticzErrorLog"

return {
    on      =   {   httpResponses   =   { myHttpResponse },
                    timer           =   { "every 5 minutes"},
                --    devices         =   { "getErrorTrigger"}   -- Only used for development and testing
                },

    logging =   {   level           =   domoticz.LOG_INFO,
                    marker          =   "getFailedLoging"    },

    data    =   {   lastlogtime     =   { initial = 0 }}, 


    execute = function(dz, trigger)
        
        local violationMessage = "Error opening url: http://192.168.1.123"          -- Change to reflect the messages you want to catch
        
        local function askDomoticzForLogLines()
            local lastLogTime = dz.data.lastlogtime            -- get time last logrequest
            local logLevel    = 4                              -- loglevel (1=normal,2=Status,4=error,268435455=all
            local jsonString  = "/json.htm?type=command" .. 
                                "&param=getlog" ..
                                "&lastlogtime=" .. tostring(lastLogTime) ..
                                "&loglevel=" .. tostring(logLevel)
            
            local logURL      =  dz.settings["Domoticz url"] .. jsonString
            
            dz.openURL  ({ url      = logURL,
                           method   = "GET", 
                           callback = myHttpResponse 
                        })

            dz.data.lastlogtime = os.time(os.date('*t'))        -- store current Time as seconds from epoch  
            dz.log(logURL,dz.LOG_DEBUG)
        end

        local function findViolation()
           local violation = ""
           if trigger.json.result ~= nil then               -- Only when there are errormessages in the log
               local resultTable = trigger.json.result
               for i = #resultTable,1,-1 do                 -- traverse backwards to get latest violation first
                    if      string.find(resultTable[i].message,violationMessage) and
                        not(string.find(resultTable[i].message,"dzVents")) then     -- We don't want debug messages to interfere
                            violation = violation .. "\n".. resultTable[i].message
                    end
                end
            end
            return violation
        end

        if trigger.isDevice or trigger.isTimer then
            askDomoticzForLogLines()                     -- get the relevant loglines
        elseif trigger.ok then
            local violation = findViolation()
            if violation == "" then
                dz.log("No (new) violations found",dz.LOG_DEBUG)
            else    
                dz.log("violation(s) found: " .. violation,dz.LOG_INFO)
                if domoticz.devices('Ventiltie').state == On then
                    domoticz.devices('Another switch').switchOn()
                end
                 if domoticz.devices('Ventiltie').state == Off then
                    domoticz.devices('Another switch').switchOff()
                end
                domoticz.notify('Ventilatieschakeling herhaald n.a.v. foutmelding',
                         domoticz.PRIORITY_LOW)
--                dz.notify("Access violation",violation, dz.PRIORITY_NORMAL,nil,nil,dz.NSS_PUSHOVER)
            end    
        else
            dz.log("Access problem to log",dz.LOG_ERROR)
       end
    end
}

[bldewit]

Hardcoding my Domoticz-server's IP-address seems to solve the problem. OK for a workaroud, but is should do it's job properly....

[waaren]

Hardcoding my Domoticz-server's IP-address seems to solve the problem. OK for a workaroud, but is should do it's job properly....

Added some logging. Also: you used domoticz.devices but you should use dz.devices as that is the name of the table passed to the main function. I also noticed that the Domoticz url used port 8079. If you see the message that the openURL cannot access this port can you please also check if you can connect to that port and get to the domoticz GUI ? or from the CLI give the command

Code: Select all

sudo curl 127.0.0.1:8079

One other place where your setup might differ from mine is the authentication of the web-site Protection (in settings) It should be set to Basic Auth

Code: Select all

myHttpResponse = "getDomoticzErrorLog"

return {
    on      =   {   httpResponses   =   { myHttpResponse },
                    timer           =   { "every 5 minutes"},
                },

    logging =   {   level           =   domoticz.LOG_DEBUG,
                    marker          =   "getStr from log"    },

    data    =   {   lastlogtime     =   { initial = 0 }}, 

    execute = function(dz, trigger)
        
        local searchStr = "Error opening url: http://192.168.1.123"          -- Change to reflect the messages you want to catch

        local function askDomoticzForLogLines()
            local lastLogTime = dz.data.lastlogtime            -- get time last logrequest
            local logLevel    = 4                              -- loglevel (1=normal,2=Status,4=error,268435455=all
            local jsonString  = "/json.htm?type=command" .. 
                                "&param=getlog" ..
                                "&lastlogtime=" .. tostring(lastLogTime) ..
                                "&loglevel=" .. tostring(logLevel)
            
            local logURL      =  dz.settings["Domoticz url"] .. jsonString
            
            dz.openURL  ({ url      = logURL,
                           method   = "GET", 
                           callback = myHttpResponse 
                        })

            dz.data.lastlogtime = dz.time.dDate        -- store current Time as seconds from epoch  
        end

        local function findString()
           if trigger.json.result ~= nil then               -- Only when there are errormessages in the log
               local resultTable = trigger.json.result
               for i = #resultTable,1,-1 do                 -- traverse backwards to get latest violation first
                    local s = resultTable[i].message
                    local found = s:find(searchStr,1,true) -- searcg plain string starting at pos 1 (no pattern) 
                    local dzVents = s:find("dzVents")
                    if found and not(dzVents) then
                       return true
                    end
                end
            end
        end

        if trigger.isDevice or trigger.isTimer then
            askDomoticzForLogLines()                     -- get the relevant loglines
        elseif trigger.ok then
            dz.log("Show ALL: \n" .. trigger.data,dz.LOG_DEBUG)
            if findString() then
                dz.log("string " .. searchStr .. " found! ",dz.LOG_DEBUG)
                if dz.devices('Ventiltie').state == On then -- Should it not be Ventilatie ?
                    dz.devices('Another switch').switchOn()
                else
                    dz.devices('Another switch').switchOff()
                end
                dz.notify('Ventilatieschakeling herhaald n.a.v. foutmelding',dz.PRIORITY_LOW)
                -- dz.notify("searchStr", searchStr, dz.PRIORITY_NORMAL, nil, nil, dz.NSS_PUSHOVER)
            else    
                dz.log("No (new) occurrences of string " .. searchStr .. " in the domoticz log ",dz.LOG_DEBUG)
            end    
        else
            dz.log("Access problem to log. Data found: " .. trigger.data,dz.LOG_ERROR)
       end
    end
}

[bldewit]

First of all, thanks for your support!

My authentication is also set to basic-auth. Using the CURL-command from the commandline on my rPi does spit out a ton of result which looks like an HTML-page to me (text-version of course), so the port does seem to be open.

Using your adapted script (after correcting the error in switch-name you so sharply found, thx) generates the following log:

2019-06-19 20:20:01.380 Status: dzVents: Info: getStr from log: ------ Start internal script: getFailedVent2:, trigger: every 5 minutes

2019-06-19 20:20:01.385 Status: dzVents: Debug: getStr from log: OpenURL: url = http://127.0.0.1:8079/json.htm?type=com ... loglevel=4

2019-06-19 20:20:01.385 Status: dzVents: Debug: getStr from log: OpenURL: method = GET

2019-06-19 20:20:01.385 Status: dzVents: Debug: getStr from log: OpenURL: post data = nil

2019-06-19 20:20:01.386 Status: dzVents: Debug: getStr from log: OpenURL: headers = nil

2019-06-19 20:20:01.386 Status: dzVents: Debug: getStr from log: OpenURL: callback = getDomoticzErrorLog

2019-06-19 20:20:01.389 Status: dzVents: Info: getStr from log: ------ Finished getFailedVent2

2019-06-19 20:20:01.952 Status: dzVents: Info: Handling httpResponse-events for: "getDomoticzErrorLog

2019-06-19 20:20:01.953 Status: dzVents: Info: getStr from log: ------ Start internal script: getFailedVent2: HTTPResponse: "getDomoticzErrorLog"

2019-06-19 20:20:01.956 Status: dzVents: Error (2.4.19): getStr from log: Access problem to log. Data found:

2019-06-19 20:20:01.959 Status: dzVents: Info: getStr from log: ------ Finished getFailedVent2

2019-06-19 20:20:01.417 Error: Error opening url: http://127.0.0.1:8079/json.htm?type=com ... loglevel=4

So,, basically still the same problem, the script does not even get executed fully because it never receives log info....

[waaren]

Using the CURL-command from the commandline on my rPi does spit out a ton of result which looks like an HTML-page to me (text-version of course), so the port does seem to be open.

Kind of puzzled by this. Seems to me there must be a difference in setup on your system compared to mine and other users of this script.
Can you enter the curl commands in the code window from the CLI and post the results (via PM) and remove any sensitive stuff.
Both commands should return a (potential big) JSON string.

Code: Select all

curl 'http://127.0.0.1:8079/json.htm?type=command&param=getlog&lastlogtime=1560968100&loglevel=4'
curl 'http://127.0.0.1:8079/json.htm?type=settings'

[bldewit]

Actually, there must be somethng more goïng on. Looking at my log again (when running with the hard-coded ip address) i do not get the logline "no (new) error messages found"

When I am back on my laptop again I will do your checks and add some logging on reading the logfile...

[bldewit]

Well, the answer seems to be clear: Unauthorized!

<html><head><title>Unauthorized</title></head><body><h1>401 Unauthorized</h1></body></html>

So, no logliines to process, so no output....

But next: What is the solution....

Knipsel2.png (25.18 KiB) Viewed 6403 times

[waaren]

Well, the answer seems to be clear: Unauthorized!

Based on the config you showed it's still unclear to me.
To make sure I completely understand what you did. You logged on to the system where domoticz is running and gave the curl command ?

What do you see when you entere these commands from the CLI on the domoticz system ?

ps -ef | grep domoticz
netstat -nao | grep 127.0
ping -c5 -4 localhost
nmap 127.0.0.1
ifconfig | grep 127

[bldewit]

pi@UniPi:~ $ ps -ef | grep domoticz
root 373 1 8 jun10 ? 1-15:18:33 /home/pi/domoticz/domoticz -daemon -www 8079 -sslwww 442
pi 12416 12388 0 09:38 pts/0 00:00:00 grep --color=auto domoticz

pi@UniPi:~ $ netstat -nao | grep 127.0
tcp 0 0 127.0.0.1:49928 127.0.0.1:42477 ESTABLISHED off (0.00/0/0)
tcp 0 0 127.0.0.1:42477 127.0.0.1:49928 ESTABLISHED off (0.00/0/0)

pi@UniPi:~ $ ping -c5 -4 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.191 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.299 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.270 ms
64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.263 ms
64 bytes from localhost (127.0.0.1): icmp_seq=5 ttl=64 time=0.214 ms

--- localhost ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4103ms
rtt min/avg/max/mdev = 0.191/0.247/0.299/0.041 ms

pi@UniPi:~ $ nmap 127.0.0.1
-bash: nmap: opdracht niet gevonden

pi@UniPi:~ $ ifconfig | grep 127
inet 127.0.0.1 netmask 255.0.0.0

[waaren]

pi@UniPi:~ $ nmap 127.0.0.1
-bash: nmap: opdracht niet gevonden

Use this command to install nmap on your pi

Code: Select all

sudo apt install nmap

the nmap wil show which ports can be accessed so it will be useful to see the result of the nmap 127.0.0.1 command

Can you also have a look in your /etc/hosts file to check if it contains

Code: Select all

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback

[bldewit]

OK, the loopbacks are in the HOSTS-file, but I'm hitting a big stop-sign on nmap. When installing nmap my Pi also wants to uninstall Unifi, a software package from Ubiquiti. From past experience on this particular configuration I know that uninstalling Unifi will brick my config....

My only alternative would be to do a clean install for this rPi of install everything on a nanopi neo and see where that takes me....

It will take some time to get there though...

Any suggestions without using nmap?

[waaren]

Any suggestions without using nmap?

What I try to find is the reason why a command that works for me and more users of dzVents, does not work on your system. the nmap command is needed to check if your domoticz port is accessible from the system itself. From your earlier reply I get the impression that the response you get from the curl command is not from the domoticz process.
So if you have an other way of doing that ?

[bldewit]

Because of the Unifi software on this rPi i did tell Domoticz to use a different port than the default one. This was done during the installation procedure, though....

Might this be the cause?

Otherwise I will really have to try a complete reinstall. That will take some time though, so this thread might go silent for a while in that case...