Pass2PHP

[sincze]

Modified the code to 100% Egregius: Reading the forum requires patience and a big screen`

Code: Select all

2016-12-21 20:25:39.798 User: Admin initiated a switch command (226/Motion Keuken/On)
2016-12-21 20:25:39.799 (Virtual Device) Lighting 2 (Motion Keuken)
2016-12-21 20:25:39.962 => Pass2PHP de functie reversed: Motion_Keuken
2016-12-21 20:25:39.966 => SETLEVEL 40
2016-12-21 20:25:39.969 User: Admin initiated a switch command (193/Lamp (Keukentafel)/Set Level)
2016-12-21 20:25:39.970 (RFXCom) Lighting 2 (Lamp (Keukentafel)))

From PIR ON: 2016-12-21 20:25:39.798 User: Admin initiated a switch command (226/Motion Keuken/On)
To LIGHT ON: 2016-12-21 20:25:39.969 User: Admin initiated a switch command (193/Lamp (Keukentafel)/Set Level)

Total: 171ms

script_device_pass2php.lua

Code: Select all

c = ''
s = ''
i = ''
t = ''
for k,v in pairs(devicechanged) do c = c..k.."|"..v.."#" end
for k,v in pairs(otherdevices) do s = s..k.."|"..v.."#" end
for k,v in pairs(otherdevices_idx) do i = i..k.."|"..v.."#" end
for k,v in pairs(otherdevices_lastupdate) do t = t..k.."|"..v.."#" end
os.execute('curl -s --data "c='..c..'&s='..s..'&i='..i..'&t='..t..'" http://127.0.0.1/pass2php.php &')

start of pass2php.php

Code: Select all

<?php
error_reporting(E_ALL);
ini_set("display_errors","on");
date_default_timezone_set('Europe/Brussels');
define('time',$_SERVER['REQUEST_TIME']);
$c=ex($_REQUEST['c']);
$s=ex($_REQUEST['s']);
$i=ex($_REQUEST['i']);
$t=ex($_REQUEST['t']);
function ex($x){
    $return=array();
    $pieces=explode('#',$x);
    foreach($pieces as $piece){
        $keyval=explode('|',$piece);
        if(count($keyval)>1)$return[$keyval[0]]=$keyval[1];
        else $return[$keyval[0]]='';
    }
    return $return;
}

......
 

[sincze]

And 100% Sneezydevil.

Domoticz Log

Code: Select all

2016-12-21 20:45:55.824 User: Admin initiated a switch command (226/Motion Keuken/On)
2016-12-21 20:45:55.825 (Virtual Device) Lighting 2 (Motion Keuken)
2016-12-21 20:45:56.055 => Pass2PHP de functie reversed: Motion_Keuken
2016-12-21 20:45:56.061 => SETLEVEL 40
2016-12-21 20:45:56.064 User: Admin initiated a switch command (193/Lamp (Keukentafel)/Set Level)
2016-12-21 20:45:56.065 (RFXCom) Lighting 2 (Lamp (Keukentafel))
2016-12-21 20:45:56.759 RFLink: 2 level: 15
2016-12-21 20:45:56.759 (RFLink Gateway) Light/Switch (Lamp (Keukentafel) (RF))

So:
from: PIR ON: 2016-12-21 20:45:55.824 User: Admin initiated a switch command (226/Motion Keuken/On)
to: LIGHT ON: 2016-12-21 20:45:56.064 User: Admin initiated a switch command (193/Lamp (Keukentafel)/Set Level)

240ms including filling up the domoticz log.

script_device_pass2php.lua

Code: Select all

local cb64json = require "cb64json"
c=cb64json.b64encode(cb64json.encode(devicechanged))
s=cb64json.b64encode(cb64json.encode(otherdevices))
i=cb64json.b64encode(cb64json.encode(otherdevices_idx))
t=cb64json.b64encode(cb64json.encode(otherdevices_lastupdate))
os.execute('curl -s --data "c='..c..'&s='..s..'&i='..i..'&t='..t..'" http://127.0.0.1/pass2php.php &')
commandArray={}
return commandArray

start of pass2php.php

Code: Select all

<?php
error_reporting(E_ALL);
ini_set("display_errors","on");
date_default_timezone_set('Europe/Brussels');
define('time',$_SERVER['REQUEST_TIME']);
$c=json_decode(base64_decode($_REQUEST['c']),true);
$s=json_decode(base64_decode($_REQUEST['s']),true);
$i=json_decode(base64_decode($_REQUEST['i']),true);
$t=json_decode(base64_decode($_REQUEST['t']),true);
.....

[Egregius]

On what hardware and how many devices?
You could add print('encode start') and some more in the lua script to determine where the time goes.
Tought you had nicer numbers...

[sincze]

On what hardware and how many devices?
You could add print('encode start') and some more in the lua script to determine where the time goes.
Tought you had nicer numbers...

ok, I will try.

Devices well...
213 used devices
601 total devices in database. Maybe delete a few unused ones to speed up things??

All running on cubietruck a20 (ARM), should be sufficient ?

Code: Select all

2016-12-21 22:45:25.301 User: Admin initiated a switch command (226/Motion Keuken/On)
2016-12-21 22:45:25.356 LUA: encode start
2016-12-21 22:45:25.359 LUA: encode finished
2016-12-21 22:45:25.302 (Virtual Device) Lighting 2 (Motion Keuken)
2016-12-21 22:45:25.467 => Pass2PHP de functie reversed: Motion_Keuken
2016-12-21 22:45:25.471 => SETLEVEL 40
2016-12-21 22:45:25.475 User: Admin initiated a switch command (193/Lamp (Keukentafel)/Set Level)

[Egregius]

Well, that is a whole bunch of devices...
I have 106 of them. I don't use any power meting at all, don't care about that. Most of my devices are for heating. Dummy setpoint, thermometer and valve(s) for each room.
Or dou you use a lot of dummy devices to group or handle states?

Looks like mst of the time get lost in the curl. Encoding is done in 3 msec so that's not the issue.
Is that lg command the first in php, or after handling stuff?

[sincze]

Well, that is a whole bunch of devices...
I have 106 of them. I don't use any power meting at all, don't care about that. Most of my devices are for heating. Dummy setpoint, thermometer and valve(s) for each room.
Or dou you use a lot of dummy devices to group or handle states?

Looks like mst of the time get lost in the curl. Encoding is done in 3 msec so that's not the issue.
Is that lg command the first in php, or after handling stuff?

lg function is in pass2php.php pretty much at the location it wass in your 2.2 version before SMS and after Telegram.
Only thing I added is 'generate_snapshots'.... in case of mailbox & doorbell.. And some if statements for debugging that are now: "false" so should be skipped.
Thats it. The rest is in separate files. No not that many virtual devices 1-3.
A lot is picked up stuff from rflink that seems to generate devices without having 'enable new hardware for 5 minutes' ticket on.

[Egregius]

Think you can easily remove the unused rflink devices, no?

I mean where you put 'the functie reversed'...

Ow, you triggered the motion? The timestamps ar unordered, didn't see that. So,it takes 53msec between motion switch and start lua script. Are there any other scripts or blocklies?
Mine always starts after 3 msec, 2msec encoding, 10 msec curl and the rest in php.

[sincze]

Think you can easily remove the unused rflink devices, no?

I mean where you put 'the functie reversed'...

Ow, you triggered the motion? The timestamps ar unordered, didn't see that. So,it takes 53msec between motion switch and start lua script. Are there any other scripts or blocklies?
Mine always starts after 3 msec, 2msec encoding, 10 msec curl and the rest in php.

Sure RFLINK ... delete unused no problem.
A yes some blockly left and some other lua scripts in Database less time critical (update thermostat, house_mode, that kind of stuff).

The start of my pass2php looks like:

Code: Select all

 if(file_exists('pass2php/'.key($c).'.php'))                // Added this function to have individual files ;-) .php directory
     { include 'pass2php/'.key($c).'.php';
         if($debug3=="true")
        {
        lg('Step 4: Pass2PHP running de file bestaat: '.key($c));
         lg('Step 5: Pass2PHP running echo file functie is: '.$c2);
        }     
     }
 elseif(file_exists('pass2php/'.($r).'.php'))                // Added this function to have individual files ;-) .php directory
     { include 'pass2php/'.($r).'.php';
         if($debug3=="true")
        {
        lg('Step 6: Pass2PHP running de file bestaat REVERSED: '.key($r));
         lg('Step 7: Pass2PHP running echo file functie is REVERSED: '.$r);
        }     
     }

 if(function_exists(key($c)))
    {
     if($debug=="true")
        {
        lg('Pass2PHP de functie bestaat: '.key($c));
         lg('Pass2PHP echo functie is: '.$c2);
        } 
//     key($c)();
    $c2();
    } 
  elseif(function_exists($r))
    {
     if($debug=="true")
        {
        lg('Pass2PHP de functie reversed: '.$r);
        } 
        $r();
    }

The lg lines are for debugging ... maybe execution of $r and $c can already be done in the first 2 functions..

[sincze]

after moving the $c() and $r() into the first 2 functions.

Code: Select all

2016-12-21 23:20:24.182 User: Admin initiated a switch command (226/Motion Keuken/On)
2016-12-21 23:20:24.052 (OpenTherm) Temp (Max. Vloerverwarming)
2016-12-21 23:20:24.202 (OpenTherm) General/Percentage (Maximum Relative Modulation Level)
2016-12-21 23:20:24.234 LUA: Encode Start time: 120.43
2016-12-21 23:20:24.237 LUA: Encode End time: 120.43
2016-12-21 23:20:24.205 (OpenTherm) Thermostat (Room Setpoint)
2016-12-21 23:20:24.346 (OpenTherm) General/Percentage (Relative modulation level)
2016-12-21 23:20:24.379 LUA: Encode Start time: 120.56
2016-12-21 23:20:24.381 LUA: Encode End time: 120.57
2016-12-21 23:20:24.348 (OpenTherm) Temp (Room Temperature)
2016-12-21 23:20:24.524 LUA: Encode Start time: 120.70
2016-12-21 23:20:24.527 LUA: Encode End time: 120.70
2016-12-21 23:20:24.494 (OpenTherm) Temp (Boiler Water Temperature)
2016-12-21 23:20:24.636 (OpenTherm) Temp (DHW Temperature)
2016-12-21 23:20:24.669 LUA: Encode Start time: 120.83
2016-12-21 23:20:24.672 LUA: Encode End time: 120.84
2016-12-21 23:20:24.639 (OpenTherm) Temp (OTGW - Outside Temperature)
2016-12-21 23:20:24.816 LUA: Encode Start time: 120.97
2016-12-21 23:20:24.818 LUA: Encode End time: 120.97
2016-12-21 23:20:24.785 (OpenTherm) Temp (Return Water Temperature)
2016-12-21 23:20:24.958 LUA: Encode Start time: 121.10
2016-12-21 23:20:24.961 LUA: Encode End time: 121.10
2016-12-21 23:20:24.927 (OpenTherm) Thermostat (DHW Setpoint)
2016-12-21 23:20:25.108 LUA: Encode Start time: 121.24
2016-12-21 23:20:25.111 LUA: Encode End time: 121.24
2016-12-21 23:20:25.072 (OpenTherm) Thermostat (Max_CH Water Setpoint)
2016-12-21 23:20:25.254 LUA: Encode Start time: 121.38
2016-12-21 23:20:25.257 LUA: Encode End time: 121.38
2016-12-21 23:20:25.220 (Virtual Device) Lighting 2 (Motion Keuken)
2016-12-21 23:20:25.369 => SETLEVEL 40
2016-12-21 23:20:25.372 User: Admin initiated a switch command (193/Lamp (Keukentafel)/Set Level)
2016-12-21 23:20:25.409 LUA: Encode Start time: 121.52
2016-12-21 23:20:25.412 LUA: Encode End time: 121.52
2016-12-21 23:20:25.374 (RFXCom) Lighting 2 (Lamp (Keukentafel))
2016-12-21 23:20:25.775 RFLink: 2 level: 15
2016-12-21 23:20:25.816 LUA: Encode Start time: 121.66
2016-12-21 23:20:25.818 LUA: Encode End time: 121.66
2016-12-21 23:20:25.775 (RFLink Gateway) Light/Switch (Lamp (Keukentafel) (RF))
2016-12-21 23:20:26.130 => Motion Keuken, Lamp aangezet 40%

[Nautilus]

Ow, you triggered the motion? The timestamps ar unordered, didn't see that. So,it takes 53msec between motion switch and start lua script. Are there any other scripts or blocklies?
Mine always starts after 3 msec, 2msec encoding, 10 msec curl and the rest in php.

I had similar results with the delay to begin the lua script after motion device is triggered, 55ms when there's a script file in directory, over 250ms with a db script. I have several LUA scripts and around 350 devices (some z-wave ones are unused but don't know if they can be removed as they are "part" of the main device).

With Blockly the trigger time is around 15ms so considerably faster than LUA. Then with z-wave switch it takes about 20ms to trigger the command and most of the time is spent to relay the command to the node (usually 100-500ms).

I've been thinking should I venture down the road to "pass2php" but with my current understanding the most I'd be able to cut away would be few milliseconds from the 15+20=35ms and still spend the majority of time waiting for the actual switch command to go through - is this correct?

[G3rard]

pass2php 3.1 might come faster than I guessed...

2016-12-19 18:03:12.106 (ZWAVE) Light/Switch (pirkeuken)
2016-12-19 18:03:12.142 User: Admin initiated a switch command (61/wasbak/On)
2016-12-19 18:03:12.142 OpenZWave: Domoticz has send a Switch command! NodeID: 6 (0x06)

Found a possible way to avoid the json encode. Need some hours to see if everything still works...

Was just wondering if you have done some testing with MQTT as well in the need for speed
The eDomoticz plugin for Siri is using MQTT and that's really quick. When I press a switch in Domoticz, that switch is immediately updated in the app on my phone. And the MQTT out messages only contains the switch that is being updated and not all other statuses.
And googling for PHP and MQTT gives a lot of hits (eg https://www.cloudmqtt.com/docs-php.html).

[Egregius]

Is this the trigger and action?

2016-12-21 23:20:25.220 (Virtual Device) Lighting 2 (Motion Keuken)
2016-12-21 23:20:26.130 => Motion Keuken, Lamp aangezet 40%

Why is it called 'virtual device'?

And you have both ways together? Seperate files and functions?

Lot's of lg calls means also calls to domoticz to log it... You should only use them while debugging (at least if you go for the fastest possible)

And yes, all those other blocklies and script causes for delays.

btw, found a better way today for file_exists... Updated github.
These are timings of a single file check, OK is for an existing file, NOK for a not existing file:
OK file_exists 0.21600723266602
NOK file_exists 0.2291202545166
OK stream_resolve_include_path 0.032901763916016
NOK stream_resolve_include_path 0.020027160644531
So, stream_resolve_include_path is about 10 times faster

[Egregius]

I've been thinking should I venture down the road to "pass2php" but with my current understanding the most I'd be able to cut away would be few milliseconds from the 15+20=35ms and still spend the majority of time waiting for the actual switch command to go through - is this correct?

Feel free, it's fun

But yes, I don't get mine any faster than 20msec. And then I need to put everything in cache and only sent the devicechanged table. Restored my previous setup because I didn't really liked the idea to sync everything with possible errors and ... just returned back to sending everything from lua to php on each device update. It takes only 5 msec longer, so why risk a out of sync system for that?

And yes again, 'we' can only control how fast we react when domoticz notifies us of a change and send the response back. What domoticz does with that, or the underlying protocol (zwave, rfx, ...) is totally out of our control.

[Egregius]

Was just wondering if you have done some testing with MQTT as well in the need for speed

Nope, seen the term before, but that's it. I have no idea at all what MQTT is capable of.
Visually I don't think you can see the difference between 20 and 40msec in a user interface.
Once it reaches several hundreds yes...

[DarkAllMan]

Installed your script, but seeing an error in my log:
Error: EventSystem: in /home/user/domoticz/scripts/lua/script_device_pass2php.lua: .../user/domoticz/scripts/lua/script_device_pass2php.lua:1: attempt to call a nil value

What did I miss?

[Egregius]

Wich version did you use?
There's something wrong in the first line. a nil value means that there's a call to a variable that doesn't exist.
Latest version of my lua script is at https://github.com/Egregius/LUA-Pass2PH ... ss2php.lua

[sincze]

btw, found a better way today for file_exists... Updated github.
These are timings of a single file check, OK is for an existing file, NOK for a not existing file:
OK file_exists 0.21600723266602
NOK file_exists 0.2291202545166
OK stream_resolve_include_path 0.032901763916016
NOK stream_resolve_include_path 0.020027160644531
So, stream_resolve_include_path is about 10 times faster

Do you think we can make this work with my functions using "_" in the name or a "<space>"

Code: Select all

foreach($c as $device=>$status)
	if(!empty($status))
		if(false!==stream_resolve_include_path('pass2php/'.$device.'.php'))include 'pass2php/'.$device.'.php';
include 'pass2php/__CRON.php';

DId not test it yet myself to be honest.

[DarkAllMan]

ok, that seems to have fixed the errors...
I see that in this script, the base64.lua and JSON.lua are no longer called....
I can remove these scripts?

[sincze]

Yes, for Egregius script you don't need these files anymore.
He solved it indeed a different way.

[Egregius]

Correct, by just concatenating all statusses in to a csv string I avoided the need of json and base64 wich seems to save about 10msec of response time.
Currently I'm at 21msec by just sending devicechanged and otherdevices but that requires quite some recoding of scripts to store all idx's and lastupdate times in APCu cache.

For the switches with a space in them, probably this will do:

Code: Select all

foreach($c as $device=>$status)
   if(!empty($status)){
      $devicer=str_replace(" ","_",$device);
      if(false!==stream_resolve_include_path('pass2php/'.$device.'.php'))include 'pass2php/'.$device.'.php';
      elseif(false!==stream_resolve_include_path('pass2php/'.$devicer.'.php'))include 'pass2php/'.$devicer.'.php';
   }   
include 'pass2php/__CRON.php';