Python plugin: Presence detection from wireless router

Python and python framework

Moderator: leecollings

EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

1. Did you manage to install the 64-bit version of raspbian on your rpi3b? I don't think there is a official release for that(?) 64-bit requires rpi4 and up.
2. Domoticz is not yet compatible with Debian bookworm, so it's better to install the 'legacy' raspbian (bullseye)
3. You don't need to install pssh from source when you already installed it from the repo by sudo pip install parallel-ssh (so skip your step 4). This might have broken pssh (not saying it has, but it might)
Fredom
Posts: 140
Joined: Saturday 19 September 2020 21:02
Target OS: Raspberry Pi / ODroid
Domoticz version: 2022.2
Location: Krimpen aan den IJssel
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Fredom »

EscApe wrote: Friday 10 November 2023 13:26 1. Did you manage to install the 64-bit version of raspbian on your rpi3b? I don't think there is a official release for that(?) 64-bit requires rpi4 and up.
2. Domoticz is not yet compatible with Debian bookworm, so it's better to install the 'legacy' raspbian (bullseye)
3. You don't need to install pssh from source when you already installed it from the repo by sudo pip install parallel-ssh (so skip your step 4). This might have broken pssh (not saying it has, but it might)
Hi EscApe,

Bookworm 64 bit runs well on the raspberry, chrome is a bit more unstable but it does work and is slow.
How can I undo step 4?
See photo 64 bit
Attachments
64 BIT-.jpg
64 BIT-.jpg (242.5 KiB) Viewed 10648 times
Yours sincerely,
Fred

Rasberry Pi 3B+ - Debian Buster - Domoticz 2022.2
RFLink - RFXCom - Zigbee (CC2531)
P1 Smart Meter - KaKu
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

I stand corrected.. Didn't know a 64-bit image was available now.
I don't know how you can undo a manual install from source or if rerunning "sudo pip install parallel-ssh" would simply overwrite it.
There are several mentions on the Domoticz Forum and tweakers.net of Domoticz not being compatible with bookworm, because of the ssl version in bookworm. That is probably unrelated to the pssh problem, but I would still stick with bullseye for now.

There is a difference between the python environment in Domoticz and the python environment when running python from the commandline as the same (root) user. I don't have the skills or knowledge to troubleshoot that, so besides of some experimenting I can't do much more to solve this issue.

To see the difference you can login to the same rpi, start python3 en run the following code (make sure to edit the password):

Code: Select all

from pssh.clients.ssh import SSHClient
client = SSHClient('127.0.0.1', port=22, user='pi', password='<yourpassword>', timeout=5)
output = client.run_command('ls -al')
ssh_output='\n'.join(map(str,output.stdout))
print(ssh_output)
The code ran fine on my rpi3 running 32-bit raspberry os, but the same code would fail inside a (otherwise empty) plugin on the same rpi.
Fredom
Posts: 140
Joined: Saturday 19 September 2020 21:02
Target OS: Raspberry Pi / ODroid
Domoticz version: 2022.2
Location: Krimpen aan den IJssel
Contact:

Re: Python plugin: Presence detection from wireless router

Post by Fredom »

EscApe wrote: Friday 10 November 2023 16:36 I stand corrected.. Didn't know a 64-bit image was available now.
I don't know how you can undo a manual install from source or if rerunning "sudo pip install parallel-ssh" would simply overwrite it.
There are several mentions on the Domoticz Forum and tweakers.net of Domoticz not being compatible with bookworm, because of the ssl version in bookworm. That is probably unrelated to the pssh problem, but I would still stick with bullseye for now.

There is a difference between the python environment in Domoticz and the python environment when running python from the commandline as the same (root) user. I don't have the skills or knowledge to troubleshoot that, so besides of some experimenting I can't do much more to solve this issue.

To see the difference you can login to the same rpi, start python3 en run the following code (make sure to edit the password):

Code: Select all

from pssh.clients.ssh import SSHClient
client = SSHClient('127.0.0.1', port=22, user='pi', password='<yourpassword>', timeout=5)
output = client.run_command('ls -al')
ssh_output='\n'.join(map(str,output.stdout))
print(ssh_output)
The code ran fine on my rpi3 running 32-bit raspberry os, but the same code would fail inside a (otherwise empty) plugin on the same rpi.
Hi,
I have tried your code in a number of different ways but I always get the same error, a syntax error. I think it is better to go back to Bulsseye then but with the IP instead of the MAC addresses in iDetect
Yours sincerely,
Fred

Rasberry Pi 3B+ - Debian Buster - Domoticz 2022.2
RFLink - RFXCom - Zigbee (CC2531)
P1 Smart Meter - KaKu
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

Yesterday I updated Domoticz to the newest version 2023.2 and not the plugin does not want to start, i'm getting the following erros:

I also have updated the plugin itself.

Code: Select all

2023-11-16 09:34:33.077 Error: Presence detection: 192.168.0.100 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:34:43.108 Error: Presence detection: 192.168.0.104 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:34:53.120 Error: Presence detection: 192.168.0.103 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:35:03.171 Error: Presence detection: 192.168.0.102 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:35:53.089 Error: Presence detection: 192.168.0.100 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:36:03.118 Error: Presence detection: 192.168.0.104 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:36:13.129 Error: Presence detection: 192.168.0.103 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:36:23.180 Error: Presence detection: 192.168.0.102 ====> SSH Could not connect (using password). Exception: Error reading SSH protocol banner
2023-11-16 09:37:38.110 Error: Presence detection: 192.168.0.100 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 09:37:48.136 Error: Presence detection: 192.168.0.104 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 09:37:58.147 Error: Presence detection: 192.168.0.103 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 09:38:08.207 Error: Presence detection: 192.168.0.102 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 09:37:53.135 Presence detection: 192.168.0.103 Not (yet) fully initialized for polling
2023-11-16 09:37:53.186 Presence detection: 192.168.0.102 Not (yet) fully initialized for polling
What do I do wrong here?

I'm running Domoticz inside Docker and installed 'paramiko' within the docker with the dockerfile:

Code: Select all

FROM domoticz/domoticz

EXPOSE 8080
EXPOSE 444

WORKDIR /opt/domoticz

RUN pip install --upgrade pip
RUN pip3 install --upgrade pip

RUN pip install pylgtv paramiko
RUN pip install paramiko

RUN pip3 install wheel
RUN pip3 install pyfujitsu

ENTRYPOINT ["docker-entrypoint.sh"]

CMD ["/opt/domoticz/domoticz"]
When I disable and enable the plugin, I'm getting the following errors:

Code: Select all

2023-11-16 09:42:02.678 Error: Presence detection: Call to function 'onStart' failed, exception details:
2023-11-16 09:42:02.683 Error: Presence detection: Traceback (most recent call last):
2023-11-16 09:42:02.684 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/plugin.py", line 431, in onStart
2023-11-16 09:42:02.684 Error: Presence detection: _plugin.onStart()
2023-11-16 09:42:02.684 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/plugin.py", line 212, in onStart
2023-11-16 09:42:02.684 Error: Presence detection: from trackers import poll_methods
2023-11-16 09:42:02.684 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/trackers/__init__.py", line 6, in <module>
2023-11-16 09:42:02.684 Error: Presence detection: from trackers.ssh_tracker import ssh_tracker
2023-11-16 09:42:02.684 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/trackers/ssh_tracker.py", line 8, in <module>
2023-11-16 09:42:02.684 Error: Presence detection: import paramiko
2023-11-16 09:42:02.684 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/paramiko/__init__.py", line 22, in <module>
2023-11-16 09:42:02.684 Error: Presence detection: from paramiko.transport import (
2023-11-16 09:42:02.684 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/paramiko/transport.py", line 33, in <module>
2023-11-16 09:42:02.684 Error: Presence detection: from cryptography.hazmat.primitives.ciphers import algorithms, Cipher, modes
2023-11-16 09:42:02.684 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/cryptography/hazmat/primitives/ciphers/__init__.py", line 11, in <module>
2023-11-16 09:42:02.684 Error: Presence detection: from cryptography.hazmat.primitives.ciphers.base import (
2023-11-16 09:42:02.684 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/cryptography/hazmat/primitives/ciphers/base.py", line 10, in <module>
2023-11-16 09:42:02.684 Error: Presence detection: from cryptography.exceptions import (
2023-11-16 09:42:02.684 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/cryptography/exceptions.py", line 9, in <module>
2023-11-16 09:42:02.684 Error: Presence detection: from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
2023-11-16 09:42:02.684 Error: Presence detection: ImportError: PyO3 modules may only be initialized once per interpreter process
What I have tried:
- Rebooting the whole system
- Installing pip install parallel-ssh inside the docker
- Run pip install cryptography==40.0.2 --upgrade inside the docker

Edit:

I have it working again. I have changed the dockerfile to this below and then rebooted the whole system

Code: Select all

FROM domoticz/domoticz

EXPOSE 8080
EXPOSE 444

WORKDIR /opt/domoticz

RUN pip install --upgrade pip
RUN pip3 install --upgrade pip

RUN pip install pylgtv paramiko
RUN pip install paramiko
RUN pip install parallel-ssh

RUN pip3 install wheel
RUN pip3 install pyfujitsu

ENTRYPOINT ["docker-entrypoint.sh"]

CMD ["/opt/domoticz/domoticz"]
Edit2:
I had another plugin which did not work correctly, that was the deconz plugin. But when I disable and enable this plugin, both plugins crash:

Code: Select all

2023-11-16 11:21:18.766 Presence detection: 192.168.0.100 Could not be polled
2023-11-16 11:21:18.766 Error: Presence detection: 192.168.0.100 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 11:21:19.831 Presence detection: 192.168.0.104 Could not be polled
2023-11-16 11:21:19.830 Error: Presence detection: 192.168.0.104 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 11:21:20.664 Presence detection: 192.168.0.103 Could not be polled
2023-11-16 11:21:20.664 Error: Presence detection: 192.168.0.103 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 11:21:21.619 Presence detection: 192.168.0.102 Could not be polled
2023-11-16 11:21:21.619 Error: Presence detection: 192.168.0.102 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-16 11:21:23.092 Conbee II: ### Request config
2023-11-16 11:21:23.093 Error: Conbee II: Your python version is missing the requests library
2023-11-16 11:21:23.093 Error: Conbee II: To install it, type : sudo -H pip3 install requests | sudo -H pip install requests
Edit2600:

I have It all working, apparently the request library was needed in this version, so i have added that to the dockerfile aswell:

Code: Select all

FROM domoticz/domoticz

EXPOSE 8080
EXPOSE 444

WORKDIR /opt/domoticz

RUN pip install --upgrade pip
RUN pip3 install --upgrade pip

#FOR LG plugin
RUN pip install pylgtv paramiko
#FOR Precense detection
RUN pip install paramiko
RUN pip install parallel-ssh
#FOR conbee II plugin
RUN pip install requests
RUN pip3 install requests

#FOR airco
RUN pip3 install wheel
RUN pip3 install pyfujitsu

ENTRYPOINT ["docker-entrypoint.sh"]

CMD ["/opt/domoticz/domoticz"]

Result:

Code: Select all

2023-11-16 11:32:21.039 Status: Presence detection: 192.168.0.103 Tracker initialized
2023-11-16 11:32:21.040 Status: Presence detection: Starting address:192.168.0.102, port:None, user:admin, keyfile:, class:ssh_autodetect and poll interval:10
2023-11-16 11:32:21.379 Status: Presence detection: 192.168.0.102 ====> SSH connection established
2023-11-16 11:32:21.533 Status: Presence detection: 192.168.0.102 Tracker initialized
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

Troubles again :(

I have updated my docker version as well to get some other things working, and not I'm getting the following errors:

Code: Select all

2023-11-23 08:07:08.631 Error: Presence detection: Call to function 'onStart' failed, exception details:
2023-11-23 08:07:08.636 Error: Presence detection: Traceback (most recent call last):
2023-11-23 08:07:08.636 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/plugin.py", line 431, in onStart
2023-11-23 08:07:08.636 Error: Presence detection: _plugin.onStart()
2023-11-23 08:07:08.636 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/plugin.py", line 212, in onStart
2023-11-23 08:07:08.636 Error: Presence detection: from trackers import poll_methods
2023-11-23 08:07:08.636 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/trackers/__init__.py", line 6, in <module>
2023-11-23 08:07:08.636 Error: Presence detection: from trackers.ssh_tracker import ssh_tracker
2023-11-23 08:07:08.636 Error: Presence detection: File "/opt/domoticz/userdata/plugins/iDetect2/trackers/ssh_tracker.py", line 8, in <module>
2023-11-23 08:07:08.636 Error: Presence detection: import paramiko
2023-11-23 08:07:08.636 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/paramiko/__init__.py", line 22, in <module>
2023-11-23 08:07:08.636 Error: Presence detection: from paramiko.transport import (
2023-11-23 08:07:08.636 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/paramiko/transport.py", line 33, in <module>
2023-11-23 08:07:08.637 Error: Presence detection: from cryptography.hazmat.primitives.ciphers import algorithms, Cipher, modes
2023-11-23 08:07:08.637 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/cryptography/hazmat/primitives/ciphers/__init__.py", line 11, in <module>
2023-11-23 08:07:08.637 Error: Presence detection: from cryptography.hazmat.primitives.ciphers.base import (
2023-11-23 08:07:08.637 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/cryptography/hazmat/primitives/ciphers/base.py", line 10, in <module>
2023-11-23 08:07:08.637 Error: Presence detection: from cryptography.exceptions import (
2023-11-23 08:07:08.637 Error: Presence detection: File "/usr/local/lib/python3.9/dist-packages/cryptography/exceptions.py", line 9, in <module>
2023-11-23 08:07:08.637 Error: Presence detection: from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
2023-11-23 08:07:08.637 Error: Presence detection: ImportError: PyO3 modules may only be initialized once per interpreter process
Any thoughts? I have rebuild the docker container, restarted the plugin, restarted Domoticz and restarted the entire system. All without success.
Last edited by rensbr on Thursday 23 November 2023 8:17, edited 1 time in total.
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

rensbr wrote: Thursday 23 November 2023 8:09 Troubles again :(

I have updated my docker version as well to get some other things working, and not I'm getting the following errors:

Code: Select all

2023-11-23 08:07:08.637 Error: Presence detection: ImportError: PyO3 modules may only be initialized once per interpreter process
This is a known problem in the Domoticz framework and also occurs with other plugins. Restarting a plugin seems to not really reset the python environment to the initial state and some modules choke on that. Restarting Domoticz is the only way to safely restart plugins at the moment. There some mentions about this in the forum and GitHub issue log.
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

EscApe wrote: Thursday 23 November 2023 8:16
rensbr wrote: Thursday 23 November 2023 8:09 Troubles again :(

I have updated my docker version as well to get some other things working, and not I'm getting the following errors:

Code: Select all

2023-11-23 08:07:08.637 Error: Presence detection: ImportError: PyO3 modules may only be initialized once per interpreter process
This is a known problem in the Domoticz framework and also occurs with other plugins. Restarting a plugin seems to not really reset the python environment to the initial state and some modules choke on that. Restarting Domoticz is the only way to safely restart plugins at the moment. There some mentions about this in the forum and GitHub issue log.
Thanks for the fast reply. I was busy with editing my post, but I have done the following already: Rebuild the docker container, restarted the plugin, restarted Domoticz and restarted the whole system. All ended up with the same error.

Edit, when I look in the logger from the container I also see the following error:

Code: Select all

Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/dist-packages/paramiko/transport.py", line 2219, in run
    emsg = "{} ({:d})".format(e.args[1], e.args[0])
IndexError: tuple index out of range
2023-11-23 08:22:08.453  Presence detection: 192.168.0.100 Not (yet) fully initialized for polling
2023-11-23 08:22:08.487  Presence detection: 192.168.0.104 Not (yet) fully initialized for polling
2023-11-23 08:22:08.506  Error: Presence detection: 192.168.0.103 ====> SSH Could not connect (using password). Exception: No existing session
2023-11-23 08:22:13.520  Presence detection: 192.168.0.102 Not (yet) fully initialized for polling
2023-11-23 08:22:13.520  Status: Presence detection: 192.168.0.102 tracker initialization (still) not completed after waiting at least 30 seconds. Retrying 5 more times
Exception in thread Thread-57:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/paramiko/transport.py", line 2129, in run
    ptype, m = self.packetizer.read_message()
  File "/usr/local/lib/python3.9/dist-packages/paramiko/packet.py", line 453, in read_message
    header = self.read_all(self.__block_size_in, check_rekey=True)
  File "/usr/local/lib/python3.9/dist-packages/paramiko/packet.py", line 301, in read_all
    x = self.__socket.recv(n)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/dist-packages/paramiko/transport.py", line 2219, in run
    emsg = "{} ({:d})".format(e.args[1], e.args[0])
IndexError: tuple index out of range
2023-11-23 08:22:28.455  Presence detection: 192.168.0.100 Not (yet) fully initialized for polling
2023-11-23 08:22:28.476  Domoticz Server: General/Percentage (CPU_Usage)
2023-11-23 08:22:28.478  Domoticz Server: General/Percentage (Memory Usage)
2023-11-23 08:22:28.480  Domoticz Server: General/Custom Sensor (Process Usage)
2023-11-23 08:22:28.488  Presence detection: 192.168.0.104 Not (yet) fully initialized for polling
2023-11-23 08:22:28.508  Presence detection: 192.168.0.103 Not (yet) fully initialized for polling
2023-11-23 08:22:28.531  Presence detection: 192.168.0.102 Not (yet) fully initialized for polling
Edit26:

I have changed the following and now it works again:
1. I have executed the following command in the docker container: 'pip install cryptography==40.0.2 --upgrade' as mentioned in viewtopic.php?p=305792#p305792
2. Changed the Tracker syntax from: 192.168.0.100#user=USER&password=PASSWORD&interval=10 to: 192.168.0.100.
3. Rebooted Domoticz container for the 1000 time.

No idea what from the above steps did the trick, but it works again. Thanks for the support!

Edit27:

That still did not quite do the trick. I have some conflicts with the Domoticz-Fujitsu-AC plugin. When that is enabled and I restart the container, I'm getting the same errors as above. When I disable the Presence detection plugin, restart the container. Let everything start-up and then enable the Presence detection plugin it works without any problems. Very strange, but for now this is a work arround.
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

I'm not sure what you meen by "the same error", because your logs show different errors, but I think you are on the right track. Maybe there is a conflicting (version) requirement between the imported modules. I have had lots of trouble with because of some underlying modules changed.

Looks like you are already Installed parallel-ssh. You could try the beta iDetect plugin that also uses parallel-ssh instead of paramiko.

Code: Select all

sudo git clone -b DomoticzEx-based-beta https://github.com/d-EScape/Domoticz_iDetect.git iDetect
The beta will require you te remove existing (tracked) devices from Domoticz and new devices will be created automatically.

You might also consider using the standard Domoticz docker image and installing the required modules using a "customstart.sh" file in userdata. Mine looks like this:

Code: Select all

#!/bin/bash

if [ -f /opt/domoticz/FIRSTRUN ]; then
echo 'Domoticz container already prepped'
else
echo 'Preparing Domoticz container for FIRSTRUN'
echo 'installing python modules'
pip install cryptography setuptools requests parallel-ssh paramiko motionblinds python_picnic_api paho-mqtt gTTS pychromecast hwmon fritzconnect>
echo 'updating linux packages'
apt-get -qq update
echo 'installing linux packages'
apt-get install -y \
    iputils-ping \
    nano
echo 'creating FIRSTRUN file so script does not run again on container start'
touch /opt/domoticz/FIRSTRUN
cd /opt/domoticz || return
fi
Works like a charm and is easier to maintain.

I just verified and I have no problem running the stable version of iDetect from a fresh docker container using the above approach. Have not yet tested any possibly conflicting plugins.
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

EscApe wrote: Friday 24 November 2023 8:06 I'm not sure what you meen by "the same error", because your logs show different errors, but I think you are on the right track. Maybe there is a conflicting (version) requirement between the imported modules. I have had lots of trouble with because of some underlying modules changed.
With the same error I mean this one:

Code: Select all

Exception in thread Thread-57:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/paramiko/transport.py", line 2129, in run
    ptype, m = self.packetizer.read_message()
  File "/usr/local/lib/python3.9/dist-packages/paramiko/packet.py", line 453, in read_message
    header = self.read_all(self.__block_size_in, check_rekey=True)
  File "/usr/local/lib/python3.9/dist-packages/paramiko/packet.py", line 301, in read_all
    x = self.__socket.recv(n)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/dist-packages/paramiko/transport.py", line 2219, in run
    emsg = "{} ({:d})".format(e.args[1], e.args[0])
IndexError: tuple index out of range
You might also consider using the standard Domoticz docker image and installing the required modules using a "customstart.sh" file in userdata. Mine looks like this:

Code: Select all

#!/bin/bash

if [ -f /opt/domoticz/FIRSTRUN ]; then
echo 'Domoticz container already prepped'
else
echo 'Preparing Domoticz container for FIRSTRUN'
echo 'installing python modules'
pip install cryptography setuptools requests parallel-ssh paramiko motionblinds python_picnic_api paho-mqtt gTTS pychromecast hwmon fritzconnect>
echo 'updating linux packages'
apt-get -qq update
echo 'installing linux packages'
apt-get install -y \
    iputils-ping \
    nano
echo 'creating FIRSTRUN file so script does not run again on container start'
touch /opt/domoticz/FIRSTRUN
cd /opt/domoticz || return
fi
I see that you are using pip, instead of pip3 to install modules, what is the reason you do this? Or what is the difference?
Looks like you are already Installed parallel-ssh. You could try the beta iDetect plugin that also uses parallel-ssh instead of paramiko.

Code: Select all

sudo git clone -b DomoticzEx-based-beta https://github.com/d-EScape/Domoticz_iDetect.git iDetect
The beta will require you te remove existing (tracked) devices from Domoticz and new devices will be created automatically.
I have installed this beta plugin and it looks like it works. When I restart Domoticz I just works without any issues or errors. So, great thanks! I'm going to use the beta plugin from now-on.
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

Glad that the new approach (beta version) is working for you.

There is no real reason for using pip instead of pip3. I noticed it does exactly the same on systems that only have python3, so I just stopped typing the 3. Call me lazy ;-)
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

EscApe wrote: Monday 27 November 2023 16:49 Glad that the new approach (beta version) is working for you.

There is no real reason for using pip instead of pip3. I noticed it does exactly the same on systems that only have python3, so I just stopped typing the 3. Call me lazy ;-)
Good to know. Thanks.

After the last installation of Tuesday I'm running into memory leak problems, is that something you recognize from the beta plugin?
Within out 6 to 8 hours the whole memory runs full, then Domoticz does 'something', I have no idea what that is, since everything is unresponsive when the memory is full. The memory is cleaned up then and it will run full again over time. Here you can see the log of the memory:
Screenshot 2023-11-30 094414.png
Screenshot 2023-11-30 094414.png (47.58 KiB) Viewed 10385 times
I'm not sure if this has anything to do with your plugin, but I have no idea how I can look into the docker container to look for the separate RAM usages.
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

Nope, no (noticeable) memory leaks on my system.
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

EscApe wrote: Thursday 30 November 2023 17:19 Nope, no (noticeable) memory leaks on my system.
Thanks for all the support and responses sofar.

I have looked into some stuff. I'm sure that the problem is caused by the plug-in. When I disable the plugin the problem is gone. When is change the interval from 10 to 60 seconds it takes more time to fill the memory.

Is there something I can do to look what happens or where the problem may be?
Attachments
Screenshot_20231203-101513~2.png
Screenshot_20231203-101513~2.png (131.77 KiB) Viewed 10349 times
Last edited by rensbr on Sunday 03 December 2023 15:57, edited 1 time in total.
User avatar
waltervl
Posts: 5715
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Python plugin: Presence detection from wireless router

Post by waltervl »

There is a memory leak reported on the python plugin framework when using the plugin device touch function: https://github.com/domoticz/domoticz/issues/5881

I did not check if this plugin uses this function.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

waltervl wrote: Sunday 03 December 2023 13:19 There is a memory leak reported on the python plugin framework when using the plugin device touch function: https://github.com/domoticz/domoticz/issues/5881

I did not check if this plugin uses this function.
Thanks for the tip! The plugin doesn't use the touch function but I appreciate your help!
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

rensbr wrote: Sunday 03 December 2023 10:19
EscApe wrote: Thursday 30 November 2023 17:19 Nope, no (noticeable) memory leaks on my system.
...
Is there something I can do to look what happens or where the problem may be?
I must revise my first answer. After setting up another test environment I DO see a memory leak. My other system wasn't even using ssh as a poll method, so that was ... uh ... well .... uh .... never mind :oops:

I will have to investigate, but I'm not sure when I will have the time. Maybe other forum members can help find the cause (probably in the use of pssh)?
Looks like the beta is not yet ready for daily use with a ssh based tracker.
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

EscApe wrote: Sunday 03 December 2023 14:49
rensbr wrote: Sunday 03 December 2023 10:19
EscApe wrote: Thursday 30 November 2023 17:19 Nope, no (noticeable) memory leaks on my system.
...
Is there something I can do to look what happens or where the problem may be?
I must revise my first answer. After setting up another test environment I DO see a memory leak. My other system wasn't even using ssh as a poll method, so that was ... uh ... well .... uh .... never mind :oops:
No worries. Thanks for the support so far. I think it is good that you now having the problem to. Makes reproducing easier.
I will have to investigate, but I'm not sure when I will have the time. Maybe other forum members can help find the cause (probably in the use of pssh)?
Looks like the beta is not yet ready for daily use with a ssh based tracker.
Is there something I can do? I'm a PLC programmer, so I know some code. But i have no knowledge of phyton or something.
rensbr
Posts: 62
Joined: Friday 27 July 2018 19:01
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Python plugin: Presence detection from wireless router

Post by rensbr »

Maybe I found a right direction to look. When I look in the log I see that the on/off status is triggered every minute (the check is set to 60s)

I can imagine this is not entirely how it should work and maybe cause the problem. See below:
Screenshot_20231209-001839.png
Screenshot_20231209-001839.png (256.15 KiB) Viewed 10251 times
EscApe
Posts: 535
Joined: Thursday 02 April 2015 8:46
Target OS: Linux
Domoticz version: 2020+
Location: The Netherlands
Contact:

Re: Python plugin: Presence detection from wireless router

Post by EscApe »

Thanks for your help in diagnosing this issue.
What grace period did you set? If it is also 60 seconds or less then this is normal behavior. The presence will be set to off after the grace period unless presence has been confirmed by a positive poll. So poll interval must be shorter than the grace period.
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest