Pass2PHP

[sincze]

Tnx for the update.

Need to look at the github repository to see the changes and how to move to tmpfs.

[Egregius]

Keep in mind dat tmpfs can cause data loss! Don't use it if you find your data of the day important!

First try the other stuff to improve and do that only if you want it at the maximum.

In short:
the secure/atreboot.sh script copies the files of domoticz and mysql to /temp.
/temp is created in /etc/fstab as tmpfs
the secure/atshutdown.sh syncs the changes from /temp to disk. I run it by cron once a day.
In /etc/init.d/domoticz.sh I added the same sync in the do_stop() function to sync changes to disk when domoticz stops.
I made a symlink from /var/lib/mysql to /temp/mysql
And probably some other things I don't think of right now...

[sincze]

Ok so first start with :
wget instead of curl in the lua script (biggest change).

Got it.

[Egregius]

Tell me, tell me... How big is the change on your system? Is it also 5 msec? I went form 15 to 10 with that.

[sincze]

Current intermediate setup. (due to still in migration process)

Changed the LUA.

Code: Select all

os.execute('wget -O /dev/null -o /dev/null "http://192.168.1.200/secure/pass2php.php?d='..d..'&s='..s..'" &')

So.
- A Domoticz Pi3 B machine (192.168.1.100) executes the LUA as this machine is running Domoticz.

LUA script is calling pass2php.php that can be found on a different Raspberry Pi4 (192.168.1.200)
This PI4 has NGINX + PHP 7.4 and MariaDB running. That PI will talk to Domoticz (192.168.1.100) to actually switch on/off the light.

Code: Select all

2020-11-20 17:30:20.709 Status: --->> (Pass2PHP SWITCH) | Lamp (CV Ruimte) (RF) => Off
2020-11-20 17:30:20.722 Status: User: Admin initiated a switch command (531/Lamp (CV Ruimte) (RF)/Off)

Code: Select all

2020-11-20 17:31:56.039 Status: --->> (Pass2PHP SWITCH) | Function Licht_Badkamer On device: Deur Badkamer with status: Open
2020-11-20 17:31:56.054 Status: User: Admin initiated a switch command (1104/Lamp Badkamer/On)

[Egregius]

I don't see the triggering device line in your log. Here I see a delay of 15msec between pass2php and domoticz, that'll be caused by it installed on another Pi I guess. Those timings should be 0.

[Egregius]

Isn't this interesting?
Keep in mind that these aren't real life tests. In all tests other than pass2php there was only one small tiny script.

Code: Select all

domoticz_main
#!/bin/sh
wget -O /dev/null -o /dev/null "http://127.0.0.1/secure/pass2php.php?d=$6&s=$4" &

2020-11-21 07:12:17.379  (ZWAVE) Light/Switch (pirkeuken)
2020-11-21 07:12:18.387  Status: Executing script: /temp/domoticz/scripts/domoticz_main
2020-11-21 07:12:18.400  (STORE)                => pirkeuken    => On   (Pass2PHP)
2020-11-21 07:12:18.400  (SWITCH)               =>keuken=>On (functions.php:1027)
2020-11-21 07:12:18.400  Status: User: Admin initiated a switch command (11/keuken/On)
2020-11-21 07:12:18.400  OpenZWave: Domoticz has send a Switch command! NodeID: 4 (0x04)
1008 msec

Code: Select all

dzVents
return {
	on = {
		devices = {
			'pirkeuken'
		}
	},
	execute = function(domoticz, device)
		domoticz.devices('keuken').switchOn()
	end
}
2020-11-21 06:20:16.682  (ZWAVE) Light/Switch (pirkeuken)
2020-11-21 06:20:16.729  OpenZWave: Domoticz has send a Switch command! NodeID: 4 (0x04)
47 msec

Code: Select all

lua file
commandArray = {}
for deviceName,deviceValue in pairs(devicechanged) do
    if (deviceName=='pirkeuken') then
            commandArray['keuken'] = "On"
    end
end
return commandArray
2020-11-21 06:29:25.006  (ZWAVE) Light/Switch (pirkeuken)
2020-11-21 06:29:25.036  OpenZWave: Domoticz has send a Switch command! NodeID: 4 (0x04)
30 msec

Code: Select all

Blockly
2020-11-21 06:07:23.155  (ZWAVE) Light/Switch (pirkeuken)
2020-11-21 06:07:23.180  OpenZWave: Domoticz has send a Switch command! NodeID: 4 (0x04)
25 msec

Code: Select all

lua database
2020-11-21 06:27:17.113  (ZWAVE) Light/Switch (pirkeuken)
2020-11-21 06:27:17.132  OpenZWave: Domoticz has send a Switch command! NodeID: 4 (0x04)
19 msec

Code: Select all

pass2php from lua database
2020-11-21 06:35:05.848  (ZWAVE) Light/Switch (pirkeuken)
2020-11-21 06:35:05.864  (STORE)                => pirkeuken    => On   (Pass2PHP)
2020-11-21 06:35:05.864  (SWITCH)               =>keuken=>On (functions.php:1027)
2020-11-21 06:35:05.864  Status: User: Admin initiated a switch command (11/keuken/On)
2020-11-21 06:35:05.864  OpenZWave: Domoticz has send a Switch command! NodeID: 4 (0x04)
16 msec

Code: Select all

pass2php from lua file
2020-11-21 06:37:34.572  (ZWAVE) Light/Switch (pirkeuken)
2020-11-21 06:37:34.582  (STORE)                => pirkeuken    => On   (Pass2PHP)
2020-11-21 06:37:34.582  (SWITCH)               =>keuken=>On (functions.php:1027)
2020-11-21 06:37:34.582  Status: User: Admin initiated a switch command (11/keuken/On)
2020-11-21 06:37:34.582  OpenZWave: Domoticz has send a Switch command! NodeID: 4 (0x04)
10 msec

[sincze]

All running on the Pi4 now.

Xiaomi ZIgbee door sensor triggering a Tasmota (Sonoff Basic)

Code: Select all

2020-11-21 13:41:52.730 Status: {"cmd":"report","model":"magnet","sid":"158d000171ea5e","short_id":19206,"data":"{\"status\":\"open\"}"}
2020-11-21 13:41:52.745 Status: {"cmd":"report","model":"magnet","sid":"158d000171ea5e","short_id":19206,"data":"{\"status\":\"open\"}"}
2020-11-21 13:41:52.824 Status: --->> Deur Badkamer: Open
2020-11-21 13:41:52.825 Status: --->> (Pass2PHP SWITCH) | Function Licht_Badkamer On device: Deur Badkamer with status: Open
2020-11-21 13:41:52.826 Status: User: Admin initiated a switch command (1104/Lamp Badkamer/On)

Xiaomi ZIgbee door sensor / motion triggering a 433 MHZ Inwall kaku

Code: Select all

2020-11-21 14:30:40.544 (GW-Xiaomi-Boven) Light/Switch (Deur WC Boven)
2020-11-21 14:30:40.549 Status: {"cmd":"report","model":"magnet","sid":"158d0001e5be3a","short_id":11032,"data":"{\"status\":\"open\"}"}
2020-11-21 14:30:40.552 (GW-Xiaomi-Boven) Light/Switch (Deur WC Boven)
2020-11-21 14:30:40.555 Status: {"cmd":"report","model":"magnet","sid":"158d0001e5be3a","short_id":11032,"data":"{\"status\":\"open\"}"}
2020-11-21 14:30:40.575 Status: LUA: URL HAS BEEN CALLED for: Deur WC Boven
2020-11-21 14:30:40.596 Status: --->> (Pass2PHP SWITCH) | Function Licht_WC_Boven On device: Deur WC Boven with status: Open
2020-11-21 14:30:40.597 Status: User: Admin initiated a switch command (1134/Lamp WC Boven/On)
2020-11-21 14:30:40.770 (RFXCom) Lighting 2 (Unknown)
2020-11-21 14:30:41.517 (GW RFLink LAN) Light/Switch (Lamp WC Boven)

Xiaomi Motion Sensor trigger Tasmota lightswitch.

Code: Select all

2020-11-21 17:52:49.048 Status: {"cmd":"report","model":"motion","sid":"158d000222ef6e","short_id":2560,"data":"{\"status\":\"motion\"}"}
2020-11-21 17:52:49.085 Status: {"cmd":"report","model":"motion","sid":"158d000222ef6e","short_id":2560,"data":"{\"status\":\"motion\"}"}
2020-11-21 17:52:49.121 Status: --->> (Pass2PHP SWITCH) | Motion Logeerkamer
2020-11-21 17:52:49.125 Status: User: Admin initiated a switch command (1285/Lamp Logeerkamer/On)

[ropske]

My Ring doorbell is working with IFTTT but that sucks, sometimes the alerts arrive hours later, even with a premium IFTTT account. I've put a Python script to poll the Ring alerts every 3 seconds. That's running on a server with another ip and another ring account because I got locked for some reason. It's running good now, but still like to handle it in PHP just because I know that better.

The only problem I have at the moment with Zwave is for the Qubino ZMNHBA2 Flush 2 Relays that don't report their correct status. Everything else works as it should. I'm running Domoticz 2020.2 Build 12485 with openzwave 1.6-1392-gb0afd4c6. The Qubino's also have the right status in the OZW control panel, but not in Domoticz. I'm using a kind of refresh script to have the right status but it's not ideal.
Oh, wait. I disabled that again because it did to much on the zwave network and caused delays.
It was something like this: if ($d['pirkeuken']['s']=='On') RefreshZwave(4);

seems you are running a newer Domoticz with openzwave 1.6
mine is: Version: 1.6-1080-g5db357e3

is it possible to update the openzwave seperately?

[ropske]

btw, i'm running the stable version

[Egregius]

@Sincze: Do I see it correctly that there are great differences between calls? That's pretty strange. I always get the same respone time.
@Ropske: I'm running beta

[ropske]

ok, maybe i better try to update to beta version then and see how the openzwave 1.6 is reacting (hopefully better than now lol)

[sincze]

@Sincze: Do I see it correctly that there are great differences between calls? That's pretty strange. I always get the same respone time.

This is a Xiaomi ZIGBEE doorsensor triggering a Tasmota light.

Code: Select all

2020-11-22 19:50:54.309 Status: {"cmd":"report","model":"magnet","sid":"158d00029b0dfb","short_id":13505,"data":"{\"status\":\"open\"}"}
2020-11-22 19:50:54.318 Status: {"cmd":"report","model":"magnet","sid":"158d00029b0dfb","short_id":13505,"data":"{\"status\":\"open\"}"}
2020-11-22 19:50:54.372 Status: --->> (Pass2PHP SWITCH) | Deur Logeerkamer Open
2020-11-22 19:50:54.374 Status: User: Admin initiated a switch command (1285/Lamp Logeerkamer/On)

Have no Clue,

Domoticz log is pretty filling up with all the events that are happening

Code: Select all

lg('(STORE) Pass2php: Called '.$device.' = Received '.$status.' status');

Not enabling that function all the time

Also many interactions with the influxDB server on a remote machine to store all the updates from the temperature and energy sensors.
System load is 2.60 and memory in use only 1 GB of the 8 GB in total. PHP is consuming 365 MB.

monit.PNG (70.57 KiB) Viewed 2714 times

[Egregius]

I would start by saying dump monit...
Tried it this morning. When freshly started my system load was 0.6-0.7.
Configured monit to monitor domoticz, php-fpm, nginx, mysql, cron. Several hours later my system load was 1.5-1.6???

How much data is sent to influx db? Maybe on option to only have it by cron?
How much calls to pass2php do you have approx? If you have APCu enabled you could count this by adding apcu_inc('pass2php_raw'); to pass2php.php

Maybe you need to filter the number of calls. Either on the lua side, or die on the php side. Your system is a lot more stressed than mine.

[Egregius]

If you change pass2php to this you'll see what's doing most calls:

Code: Select all

apcu_inc('pass2php_raw');
require '/var/www/html/secure/functions.php';
$device=$_REQUEST['d'];
$status=$_REQUEST['s'];

if (endswith($device, '_Temperature')) die('Ignore these doubles');
elseif (endswith($device, '_Utility')) die('Ignore these doubles');

apcu_inc('pass2php_net');
apcu_inc($device);

$d=fetchdata();
if ($d[$device]['dt']=='dimmer'||$d[$device]['dt']=='rollers'||$d[$device]['dt']=='luifel') {
	if ($status=='Off'||$status=='Open') {
		$status=0;
	} elseif ($status=='On'||$status=='Closed') {
		$status=100;
	} else {
		$status=filter_var($status, FILTER_SANITIZE_NUMBER_INT);
	}
} elseif (in_array($device, array('badkamer_temp'))) {
	$status=explode(';', $status);
	$status=$status[0];
} elseif ($device=='achterdeur') {
	if ($status=='Open') {
		$status='Closed';
	} else {
		$status='Open';
	}
} elseif ($device=='sirene') {
	if ($status=='Group On') {
		$status='On';
	} else {
		$status='Off';
	}
}
store($device, $status, 'Pass2PHP');
if(@include '/var/www/html/secure/pass2php/'.$device.'.php')apcu_inc('pass2php_effective');

[sincze]

I would start by saying dump monit...
Tried it this morning. When freshly started my system load was 0.6-0.7.
Configured monit to monitor domoticz, php-fpm, nginx, mysql, cron. Several hours later my system load was 1.5-1.6???

How much data is sent to influx db? Maybe on option to only have it by cron?
How much calls to pass2php do you have approx? If you have APCu enabled you could count this by adding apcu_inc('pass2php_raw'); to pass2php.php

Maybe you need to filter the number of calls. Either on the lua side, or die on the php side. Your system is a lot more stressed than mine.

Forgot to install ACPU so will start with that first

Egregius-APCU.PNG (33.46 KiB) Viewed 2687 times

Let me start with that.

All device updates that contain TEMP / Watt are pushed to influxdb.

Let me try your other solutions as well. This is what my system is running next to Domoticz.

Dockers.PNG (51.32 KiB) Viewed 2687 times

[ropske]

gonna update tonight to beta, fingers crossed

[sincze]

If you change pass2php to this you'll see what's doing most calls:

Code: Select all

apcu_inc('pass2php_raw');

Modification Done (via Github) example.

Also downloaded acpu.php and acpu.css from your website.

Egregius-APCU2.PNG (54.44 KiB) Viewed 2683 times

Values time are updated , however hits stays 0.
Maybe I missed a command to activate it ?


for the sake of.. I added

Code: Select all

lg($device.' count '.apcu_fetch($device));
lg('pass2php_net '.apcu_fetch('pass2php_net'));

So at least we know something is working.

Code: Select all

2020-11-23 20:24:48.234 Status: --->> Airco_Temperature count 91
2020-11-23 20:24:48.235 Status: --->> pass2php_net 8124
2020-11-23 20:24:48.250 Status: --->> Airco count 91
2020-11-23 20:24:48.251 Status: --->> pass2php_net 8125
2020-11-23 20:24:48.331 Status: --->> Airco (kWh) count 91
2020-11-23 20:24:48.333 Status: --->> pass2php_net 8126
2020-11-23 20:24:48.347 Status: --->> Airco (kWh)_Utility count 91
2020-11-23 20:24:48.349 Status: --->> pass2php_net 8127
2020-11-23 20:24:48.441 Status: --->> Stroom Keuken TV (kWh) count 91
2020-11-23 20:24:48.442 Status: --->> pass2php_net 8128
2020-11-23 20:24:48.479 Status: --->> Airco (Watt) count 91
2020-11-23 20:24:48.481 Status: --->> pass2php_net 8129
2020-11-23 20:24:48.506 Status: --->> Airco (Watt)_Utility count 91
2020-11-23 20:24:48.514 Status: --->> pass2php_net 8130
2020-11-23 20:24:48.544 Status: --->> Stroom Keuken TV (Watt) count 91
2020-11-23 20:24:48.545 Status: --->> pass2php_net 8131
2020-11-23 20:24:48.589 Status: --->> Stroom Keuken TV (V)_Utility count 91
2020-11-23 20:24:48.591 Status: --->> pass2php_net 8132
2020-11-23 20:24:48.605 Status: --->> Stroom Keuken TV (V) count 91
2020-11-23 20:24:48.607 Status: --->> pass2php_net 8133
2020-11-23 20:24:48.656 Status: --->> Stroom Keuken TV (A) count 91
2020-11-23 20:24:48.657 Status: --->> pass2php_net 8134

OPcache.

Egregius-APCU2.PNG (54.44 KiB) Viewed 2683 times

[Egregius]

How long was it running to reach pass2php_net 8134?

Mine has now been running 18 hours:
pass2php_raw 6489 request
pass2php_net 4551 after filtering _Temperature and _Utility devices
pass2php_effective 2983 executed because file exists in pass2php folder.

[Egregius]

gonna update tonight to beta, fingers crossed

If you have a good backup there should be a risk. I'm running beta since the beginning 5 years ago.