Maximum of lua scripts?

[JohnnySK]

still same error

2021-03-31 23:08:04.971 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3650.err
2021-03-31 23:08:28.844 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3659.out
2021-03-31 23:08:28.844 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3659.err
2021-03-31 23:10:43.734 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3714.out
2021-03-31 23:10:43.734 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3714.err

[manjh]

when using os.execute you should use the background operator

Code: Select all

os.execute('/home/pi/domoticz/scripts/speedtest2.sh &')

Excellent. That does it! Thank you!
It comes to mind that I need to educate on dzvents. I'm quite happy using LUA, dzVents looks different in approach.
What is a good place to start? Step by step documentation?

[waaren]

What is a good place to start? Step by step documentation?

os.execute is 100% Lua and the background (&) operator is Linux.

The wiki and the video @dannyBloe posted here should get you started

[waaren]

still same error

2021-03-31 23:08:04.971 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3650.err
2021-03-31 23:08:28.844 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3659.out
2021-03-31 23:08:28.844 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3659.err
2021-03-31 23:10:43.734 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3714.out
2021-03-31 23:10:43.734 Error: unable to remove file /home/pi/domoticz/scripts/dzVents/data/domscript3714.err

can you share the os script and the content of these files in home/pi/domoticz/scripts/dzVents/data/ ?
Also please the output of

Code: Select all

sudo ps -ef | grep  '[d]omoticz' | cut -d' ' -f1
sudo find /home/pi/domoticz/scripts/dzVents -type d -exec stat -c '%A - %U:%G  %n' {} +
sudo find /home/pi/domoticz/domoticz -type f -exec stat -c '%A - %U:%G  %n' {} +
sudo find /home/pi/domoticz/scripts/dzVents/data/*err -type f -exec stat -c '%A - %U:%G  %n' {} +

[JohnnySK]

Code: Select all

pi@raspberrypi:~/domoticz/scripts $ sudo ps -ef | grep  '[d]omoticz' | cut -d' ' -f1
root
root
root
pi@raspberrypi:~/domoticz/scripts $ sudo find /home/pi/domoticz/scripts/dzVents -type d -exec stat -c '%A - %U:%G  %n' {} +
drwxr-xr-x - pi:pi  /home/pi/domoticz/scripts/dzVents
drwxr-xr-x - pi:pi  /home/pi/domoticz/scripts/dzVents/scripts
drwxr-xr-x - pi:pi  /home/pi/domoticz/scripts/dzVents/examples
drwxr-xr-x - pi:pi  /home/pi/domoticz/scripts/dzVents/examples/templates
drwxr-xr-x - pi:pi  /home/pi/domoticz/scripts/dzVents/generated_scripts
drwxr-xr-x - pi:pi  /home/pi/domoticz/scripts/dzVents/dumps
drwxrwxrwx - pi:pi  /home/pi/domoticz/scripts/dzVents/data
pi@raspberrypi:~/domoticz/scripts $ sudo find /home/pi/domoticz/domoticz -type f -exec stat -c '%A - %U:%G  %n' {} +
-rwxr-xr-x - pi:pi  /home/pi/domoticz/domoticz
pi@raspberrypi:~/domoticz/scripts $ sudo find /home/pi/domoticz/scripts/dzVents/data/*err -type f -exec stat -c '%A - %U:%G  %n' {} +
-rw------- - root:root  /home/pi/domoticz/scripts/dzVents/data/domscript4307.err
-rw------- - root:root  /home/pi/domoticz/scripts/dzVents/data/domscript4308.err

content of file in home/pi/domoticz/scripts/dzVents/data/
empty, 0 bytes, just filenames

script, connect to webserver, get values, calculate and push it to domoticz

Code: Select all

#!/bin/bash

content=`curl -s http://admin:[email protected]/rect.cgi?A=4&B=Select`
current=`html2text <<< "$content"  | grep "Current_(A)_____" | column -ts'|' | column -ts'Current_(A)_____  '`
voltage=`html2text <<< "$content"  | grep "Voltage_(V)_____" | column -ts'|' | column -ts'Voltage_(V)_____  '`
power=`awk '{print $1*$2}' <<<"$current $voltage"`
url="http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=240&nvalue=0&svalue=$power"
report=`curl -s $url`

[waaren]

content of file in home/pi/domoticz/scripts/dzVents/data/
empty, 0 bytes, just filenames

Can you try again after changing the dzVents and bash to the below ones. It might not directly solve the issue but will show more information on what happens.
The power value will be returned to dzVents for further processing.

btw. It should also be possible to directly call the http://admin:[email protected]/rect.cgi?A=4&B=Select with a openURL call and bypass the OS script completely.

Your domoticz user is root but the directories are owned by pi:pi
that might be the reason the temp files (*err / *out) cannot be deleted

Code: Select all

local scriptVar = 'getPower'

return
{
    on =
    {
        devices =
        {
            'Batt Napatie',
        },
        shellCommandResponses =
        {
            scriptVar,
        },
    },

    logging =
    {
        level = domoticz.LOG_DEBUG,
        marker = scriptVar,
    },

    execute = function(dz, item)

        local function osScript(script, delay)
            dz.executeShellCommand(
            {
                command = '/home/pi/domoticz/scripts/' .. script,
                callback = scriptVar,
                timeout = 30,                          -- max execution time in seconds
            }).afterSec(delay or 0)
        end

        if item.isDevice or item.isTimer then
           osScript('pv_fpv1.sh')
           osScript('pv_fpv2.sh', 10)
        elseif item.ok then
            dz.log('data from script: ' .. item.data, dz.LOG_DEBUG) -- need more processing
        else
            dz.log('There was an error', dz.LOG_ERROR)
            dz.log(item, dz.LOG_DEBUG)
        end
    end
}

bash script

Code: Select all

#!/bin/bash

content=`curl -s http://admin:[email protected]/rect.cgi?A=4&B=Select`
current=`html2text <<< "$content"  | grep "Current_(A)_____" | column -ts'|' | column -ts'Current_(A)_____  '`
voltage=`html2text <<< "$content"  | grep "Voltage_(V)_____" | column -ts'|' | column -ts'Voltage_(V)_____  '`
power=`awk '{print $1*$2}' <<<"$current $voltage"`
echo $power

[azonneveld]

Yes,
maybe this is problem.. I have os.execute in one script.. I will move this one to the dzVents.

You could try, in lua:

Code: Select all

os.execute ('(/home/.../script.sh > /dev/null)&')

This makes the script continue without waiting for the externally script to finish.

[waaren]

This makes the script continue without waiting for the externally script to finish.

True but in dzVents > 3.1 the executeShellCommand() is available and that command will not only execute any shell command truly async and in a separate thread but does also offers the option to capture the result and returncode by means of the callback.
So if you are on a recent domoticz build the preferred way of executing any os command from within a dzVents script is to use the executeShellCommand()

[JohnnySK]

I have some work now but I will try it. And it's possible get from bash more than one value?
Because I already started work on get voltage from BASH too.. see this

Code: Select all

#!/bin/bash

content=`curl -s http://admin:[email protected]/rect.cgi?A=4&B=Select`
current=`html2text <<< "$content"  | grep "Current_(A)_____" | column -ts'|' | column -ts'Current_(A)_____  '`
voltage=`html2text <<< "$content"  | grep "Voltage_(V)_____" | column -ts'|' | column -ts'Voltage_(V)_____  '`
power=`awk '{print $1*$2}' <<<"$current $voltage"`
url="http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=240&nvalue=0&svalue=$power"
report=`curl -s $url`
#echo "$current"A
#echo "$voltage"V
#echo "$power"W
#echo "$content"



replace=""
firstStep=`echo "${content//<\/TR>/$replace}"`
secondStep=`echo "${firstStep//<TR>/$replace}"`
lastStep=`echo "${firstStep//BORDER CELLPADDING="3"/$replace}"`
#whole=`html2text <<< "$lastStep"`
#echo "$whole"

values=`html2text <<< "$lastStep"  | grep "Input_Voltage_(V)" | column -ts'| °()_-' | column -ts'abcdefghijklmnoprstuvqxyzTIVSNC'`
#echo $values

IFS=' ' read -ra params <<< "$values"

#Print the split string
#for i in "${params[@]}"
#do
#    echo $i
#done

sekundy=`date +%S`
#echo $sekundy
remainder=$(( sekundy % 15 ))
if [ "$remainder" -eq 0 ]; then
        #echo $sekundy
        pvVoltage=${params[1]}
        #echo "$pvVoltage"V
        url="http://127.0.0.1:8080/json.htm?type=command&param=udevice&idx=331&nvalue=0&svalue=$pvVoltage"
        report=`curl -s $url`
        #echo $url
fi

[waaren]

I have some work now but I will try it. And it's possible get from bash more than one value?

Yes. The amount of data to return to dzvents is not unlimited but I tested up to 64K without issues.

can you share the result of the first curl command

Code: Select all

curl -s http://admin:[email protected]/rect.cgi?A=4&B=Select

It might help in creating a dzVents performing the same task but without the need of curl and/or OS script.

[JohnnySK]

Code: Select all

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<HTML>
<HEAD>
<META http-equiv="refresh" content="30;URL=rect.cgi">
</HEAD>
<BODY>
<H2>Rectifier</H2>
<FORM METHOD=GET action=rect.cgi>
<P>Rectifier Address
<SELECT NAME=A><OPTION >1<OPTION SELECTED>4<OPTION >5</SELECT>
<INPUT TYPE=SUBMIT NAME=B VALUE="Select">
</P>
</FORM>
<!--
<FORM METHOD=GET action=rect.cgi>
Enabled <INPUT TYPE=CHECKBOX NAME=C CHECKED>
<INPUT TYPE=SUBMIT NAME=D VALUE="Submit"  >
</FORM>
-->
<H2>Rectifier Status</H2>
<TABLE BORDER CELLPADDING="3">
<TR>
<TH>Voltage (V)</TH>
<TD>52.1</TD>
</TR>
<TR>
<TH>Current (A)</TH>
<TD>14.9</TD>
</TR>
<TR>
<TH>Mean Current (A)</TH>
<TD>98.5</TD>
</TR>
<TR>
<TH>Temperature (&deg;C)</TH>
<TD>20</TD>
</TR>
<TH>Input Voltage (V)</TH>
<TD>201</TD>
</TR>
<TH>Version</TH>
<TD>50.56</TD>
</TR>
<TH>Serial Number</TH>
<TD>541</TD>
</TR>
</TABLE>
<H2>Alarm Status</H2>
<P><IMG SRC=ok.gif ALT="O"> Input Voltage OK<BR><IMG SRC=ok.gif ALT="O"> Low Voltage<BR><IMG SRC=ok.gif ALT="O"> Module Alarm<BR><IMG SRC=ok.gif ALT="O"> OV Shutdown<BR><IMG SRC=ok.gif ALT="O"> Current Limiting<BR><IMG SRC=ok.gif ALT="O"> Power On<BR><IMG SRC=ok.gif ALT="O"> OVP Enabled<BR><IMG SRC=ok.gif ALT="O"> Current Limit Reduced<BR><IMG SRC=ok.gif ALT="O"> Fan Fail<BR><IMG SRC=ok.gif ALT="O"> Current Share<BR><IMG SRC=ok.gif ALT="O"> Temp. Alarm<BR><IMG SRC=ok.gif ALT="O"> Phase Failure<BR><IMG SRC=ok.gif ALT="O"> Temp. SD<BR><IMG SRC=ok.gif ALT="O"> Internal Failure<BR></P>
</BODY>
</HTML>

[waaren]

Code: Select all

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
<HTML>

It's worth a try to see the result of below dzVents script that would get the values without using curl or OS scripts.
Please feel free to ask for clarification if and when needed.

Code: Select all

local scriptVar = 'getRectData'

return
{
    on =
    {
        timer =
        {
            'every 5 minutes',
        },

        httpResponses =
        {
            scriptVar,
        }
    },
    logging =
    {
        level = domoticz.LOG_INFO,
        marker = scriptVar,
    },

    execute = function(dz, item)

        local function extractData(myData)

            -- get Values
            values = {}
            for value in string.gmatch(myData, "<TD>([%d%.]*)</TD>") do
                table.insert(values, value)
            end

            -- get Labels and make table
            local counter = 1
            local data = {}
            for label in string.gmatch(myData, "<TH>([%s%w%(%)%&%;]*)</TH>") do
                data[label] = values[counter]
                counter = counter + 1
            end

            return data
        end

        if item.isTimer then
            dz.openURL(
            {
                url = 'http://admin:[email protected]/rect.cgi?A=4&B=Select',
                callback = scriptVar,
            })
        elseif item.isHTTPResponse and item.ok then
            local data = extractData(item.data)
            dz.utils.dumpTable (data) -- This will show the content of the datatable. From here it will be relatively easy to populate the various virtual sensors (volt, ampere, etc)
        else
            dz.log('There was a problem handling the request', dz.LOG_ERROR)
            dz.log(item, dz.LOG_ERROR)
        end

    end
}

[JohnnySK]

but timer offer minutes timer only.. and my script must run maximal each 10seconds..

aaand, I must to turn off dzVents.. I have few devices updated by web api (json svalue update by web get) but each 6-8seconds... and devices get higher and higher delay. After 12hours values in domoticz and reality was so far ... only restart of domoticz service helps. It looks like dzVents create some big load on web server.

[waaren]

but timer offer minutes timer only.. and my script must run maximal each 10seconds..

For that restriction an easy workaround exists that was used before for similar cases.
See below

Code: Select all

local scriptVar = 'getRectData'

return
{
    on =
    {
        timer =
        {
            'every 5 minutes',
        },

        httpResponses =
        {
            scriptVar,
        }
    },
    logging =
    {
        level = domoticz.LOG_INFO,
        marker = scriptVar,
    },

    execute = function(dz, item)
        local frequency = 6 -- every 60 / frequency seconds (frequency 6 ==>> every 10 seconds)

        local function extractData(myData)

            -- get Values
            values = {}
            for value in string.gmatch(myData, "<TD>([%d%.]*)</TD>") do
                table.insert(values, value)
            end

            -- get Labels and make table
            local counter = 1
            local data = {}
            for label in string.gmatch(myData, "<TH>([%s%w%(%)%&%;]*)</TH>") do
                data[label] = values[counter]
                counter = counter + 1
            end

            return data
        end

        local function sendURL(delay)
            dz.openURL(
            {
                url = 'http://admin:[email protected]/rect.cgi?A=4&B=Select',
                callback = scriptVar,
            }).afterSec(delay or 0)
        end

        if item.isTimer then
            local interval = math.floor(60/frequency)
            for delay = interval, (60 - interval), interval do
                sendURL(delay)
           end
        elseif item.isHTTPResponse and item.ok then
            local data = extractData(item.data)
            dz.utils.dumpTable (data) -- This will show the content of the datatable. From here it will be relatively easy to populate the various virtual sensors (volt, ampere, etc)
        else
            dz.log('There was a problem handling the request', dz.LOG_ERROR)
            dz.log(item, dz.LOG_ERROR)
        end

    end
}

I have few devices updated by web api (json svalue update by web get) but each 6-8seconds... and devices get higher and higher delay. After 12hours values in domoticz and reality was so far ... only restart of domoticz service helps.

So to summarize you have similar response problems as you reported before in January 2019 when you use a couple of classic Lua scripts and also after converting scripts to dzVents when updating a number of devices every 5 seconds.
Then the conclusion might well be that given your current hardware, scripting is not the solution for you if the current update frequency and number of devices are a requirement.

[JohnnySK]

Maybe..
But, on latest release was this delay with same scripts under minute. (not realtime, but not so bad)

Today, I have turned off dzVents, I migrated two big measurement scripts to MQTT and LUA but on beta, it looks like web interface gets big delay after few minutes of run. (delay more than 5minutes, really bad)

Yes I had simmilar problem before. I migrated lot off devices to MQTT and upgraded to latest release with new RPi and data was almost in realtime..

[JohnnySK]

How LUA scripts update values.. Maybe will be better use red-node for prepare data and push ti domoticz only final values. V 2020.2 support consuption calculation by itself.. (send Watt only) I have this scripts because in earlier version was it on device update.