Python plugin: Presence detection from wireless router

[rensbr]

Thanks for your help in diagnosing this issue.
What grace period did you set? If it is also 60 seconds or less then this is normal behavior. The presence will be set to off after the grace period unless presence has been confirmed by a positive poll. So poll interval must be shorter than the grace period.

Ah those where both set to 60 seconds. Changed it to 60 and 70 and now let see.

Probably that is not the issue then. Since I also got the problem when these values where different.

[EscApe]

I prefer a higher polling frequency, like 10 or 15 seconds and a grace period that includes several polls. That way it will detect new presence quicker and the occasional false negatives don’t matter as much.

The problem is probably in the base ssh tracker that now uses parallel-ssh. Maybe a buffer that fills up. The module is new to me, so it will take me some time to investigate.

[rensbr]

I prefer a higher polling frequency, like 10 or 15 seconds and a grace period that includes several polls. That way it will detect new presence quicker and the occasional false negatives don’t matter as much.

The problem is probably in the base ssh tracker that now uses parallel-ssh. Maybe a buffer that fills up. The module is new to me, so it will take me some time to investigate.

Yeah me too. But having the polling faster makes the memory runs full faster.

[EscApe]

<snip>

After the last installation of Tuesday I'm running into memory leak problems, is that something you recognize from the beta plugin?
Within out 6 to 8 hours the whole memory runs full, then Domoticz does 'something', I have no idea what that is, since everything is unresponsive when the memory is full. The memory is cleaned up then and it will run full again over time. Here you can see the log of the memory:

Screenshot 2023-11-30 094414.png

I'm not sure if this has anything to do with your plugin, but I have no idea how I can look into the docker container to look for the separate RAM usages.

I have done some more testing but the outcome did not really narrow it down

With or without the iDetect plugin enabled the docker container running Domoticz would show an increase in memory use of around 1MB per minute. iDetect was monitoring a FritzBox, so no SSH. When I also enabled (a second) iDectect that was using SSH the memory increased a bit quicker. Might be the ssh, the additional Domoticz 'hardware' of the addition python plugin. That still needs te be tested, but I don't have the time this week.

Also wrote a little testprogram using the exact same SSH approach as the plugin and left it running for a few hours and a poll interval of 5 seconds. It showed no noticeable memory increase.

[rensbr]

<snip>

After the last installation of Tuesday I'm running into memory leak problems, is that something you recognize from the beta plugin?
Within out 6 to 8 hours the whole memory runs full, then Domoticz does 'something', I have no idea what that is, since everything is unresponsive when the memory is full. The memory is cleaned up then and it will run full again over time. Here you can see the log of the memory:

Screenshot 2023-11-30 094414.png

I'm not sure if this has anything to do with your plugin, but I have no idea how I can look into the docker container to look for the separate RAM usages.

I have done some more testing but the outcome did not really narrow it down

With or without the iDetect plugin enabled the docker container running Domoticz would show an increase in memory use of around 1MB per minute. iDetect was monitoring a FritzBox, so no SSH. When I also enabled (a second) iDectect that was using SSH the memory increased a bit quicker. Might be the ssh, the additional Domoticz 'hardware' of the addition python plugin. That still needs te be tested, but I don't have the time this week.

Also wrote a little testprogram using the exact same SSH approach as the plugin and left it running for a few hours and a poll interval of 5 seconds. It showed no noticeable memory increase.

I have changed the plugin back to the original and I'm coming up with the same conclusion. The memory still runs up, see below:

Schermafbeelding 2023-12-15 205759.jpg (78.25 KiB) Viewed 8761 times

[EscApe]

There seems to be a memory leak issue in the plug-in framework that might have been fixed by a recent commit. I don’t know if it is related, but it’s worth trying the latest Domoticz beta.
I will try it myself, but not right now.

[hjzwiers]

I was having troubles again and I took the initiative to run your beta, no code errors any more but I do get these errors:

2023-12-31 15:45:32.134 Error: iDetect: Trying to remove a unit that does not exist:#Override unit:1
2023-12-31 15:45:32.147 Error: iDetect: Error while trying to import tracker module:catching classes that do not inherit from BaseException is not allowed
2023-12-31 15:45:32.150 Error: iDetect: Error while trying to import tracker module:catching classes that do not inherit from BaseException is not allowed
2023-12-31 15:45:32.153 Error: iDetect: Error while trying to import tracker module:catching classes that do not inherit from BaseException is not allowed

Any clue what this could be?

[EscApe]

I can think of two things to check:
1: make sure you remove all(!) devices that where created by the old version. The beta will create new devices for you.
2: restarting (updating settings) the plug-in leads to some strange behavior. You will have to restart Domoticz to restart the plug-in. This cannot be solved in the plug-in, but seems to be an issue with the plug-in framework for every plug-in that uses certain python modules.

[hjzwiers]

I had to switch off the plug-in due to that Domoticz crashes after about an hour, don't know what causes it. I am currently running the latest beta.

[rensbr]

I had to switch off the plug-in due to that Domoticz crashes after about an hour, don't know what causes it. I am currently running the latest beta.

Latest beta of domoticz or plugin? But you are probably also facing the memory issue I have with the beta plug in.

[hjzwiers]

I am running Domoticz Beta, and the new Beta of the iDetect.

Looks that way (same issue as you had with the memory overflow)

[EscApe]

I cannot figure out what is causing the memory leak when using the parallel-ssh module, so the beta (in GitHub) is now using paramiko again.

Parallel-ssh caught my eye as an alternative because Paramiko was a pain to include in a docker image. But when using the standard Domoticz docker image and a customstart.sh file to install paramiko (see https://www.domoticz.com/wiki/Docker) it is easy enough.

If you want to help figure out why parallel-ssh is causing a memory leak when used in my Domoticz plugin then you can re-enable it by copying ssh_tracker.py.use-pssh.py over ssh_tracker.py in the trackers directory.

Unrelated but good news: Plugins like iDetect can be restarted again in Domoticz 2024.1 Thanks to the Domoticz developer that fixed the framework!

[hjzwiers]

Lots of errors again, start up from clean install 2024.1:

2024-01-04 19:19:38.772 Error: iDetect: Call to function 'onStart' failed, exception details:
2024-01-04 19:19:38.772 Error: iDetect: Traceback (most recent call last):
2024-01-04 19:19:38.772 Error: iDetect: File "/home/hjz/domoticz/plugins/iDetect/plugin.py", line 431, in onStart
2024-01-04 19:19:38.772 Error: iDetect: _plugin.onStart()
2024-01-04 19:19:38.772 Error: iDetect: File "/home/hjz/domoticz/plugins/iDetect/plugin.py", line 262, in onStart
2024-01-04 19:19:38.772 Error: iDetect: if get_domoticz_status(self.OVERRIDE_UNIT):
2024-01-04 19:19:38.772 Error: iDetect: File "/home/hjz/domoticz/plugins/iDetect/plugin.py", line 181, in get_domoticz_status
2024-01-04 19:19:38.772 Error: iDetect: if Devices[unit].nValue == True:
2024-01-04 19:19:38.772 Error: iDetect: KeyError: 255

2024-01-04 19:39:08.100 Status: iDetect: Skip this hearbeat ... system is still preparing

Constantly getting this message. The keypass is not arranged, that would be taken car of. Anyway I will wait for the new version to come out

[rensbr]

I cannot figure out what is causing the memory leak when using the parallel-ssh module, so the beta (in GitHub) is now using paramiko again.

Parallel-ssh caught my eye as an alternative because Paramiko was a pain to include in a docker image. But when using the standard Domoticz docker image and a customstart.sh file to install paramiko (see https://www.domoticz.com/wiki/Docker) it is easy enough.

If you want to help figure out why parallel-ssh is causing a memory leak when used in my Domoticz plugin then you can re-enable it by copying ssh_tracker.py.use-pssh.py over ssh_tracker.py in the trackers directory.

Unrelated but good news: Plugins like iDetect can be restarted again in Domoticz 2024.1 Thanks to the Domoticz developer that fixed the framework!

What is the difference now between your beta plug-in and the normal one?

And what is you experience with the memory leak in the 2024 version?

[EscApe]

Lots of errors again, start up from clean install 2024.1:

2024-01-04 19:19:38.772 Error: iDetect: Call to function 'onStart' failed, exception details:
2024-01-04 19:19:38.772 Error: iDetect: Traceback (most recent call last):
2024-01-04 19:19:38.772 Error: iDetect: File "/home/hjz/domoticz/plugins/iDetect/plugin.py", line 431, in onStart
2024-01-04 19:19:38.772 Error: iDetect: _plugin.onStart()
2024-01-04 19:19:38.772 Error: iDetect: File "/home/hjz/domoticz/plugins/iDetect/plugin.py", line 262, in onStart
2024-01-04 19:19:38.772 Error: iDetect: if get_domoticz_status(self.OVERRIDE_UNIT):
2024-01-04 19:19:38.772 Error: iDetect: File "/home/hjz/domoticz/plugins/iDetect/plugin.py", line 181, in get_domoticz_status
2024-01-04 19:19:38.772 Error: iDetect: if Devices[unit].nValue == True:
2024-01-04 19:19:38.772 Error: iDetect: KeyError: 255

2024-01-04 19:39:08.100 Status: iDetect: Skip this hearbeat ... system is still preparing

Constantly getting this message. The keypass is not arranged, that would be taken car of. Anyway I will wait for the new version to come out

That is actually just one error. It comes down to a non existing "Override" device. If you look at the complete log you probably also see this message: "iDetect: (iDetect) Unit creation failed, Domoticz settings prevent accepting new devices."

The plugin framework behavior has changed. Plugins can now only create devices if "Accept new Hardware Devices" is enabled in the Domoticz settings, which makes sense.

[EscApe]

I cannot figure out what is causing the memory leak when using the parallel-ssh module, so the beta (in GitHub) is now using paramiko again.

Parallel-ssh caught my eye as an alternative because Paramiko was a pain to include in a docker image. But when using the standard Domoticz docker image and a customstart.sh file to install paramiko (see https://www.domoticz.com/wiki/Docker) it is easy enough.

If you want to help figure out why parallel-ssh is causing a memory leak when used in my Domoticz plugin then you can re-enable it by copying ssh_tracker.py.use-pssh.py over ssh_tracker.py in the trackers directory.

Unrelated but good news: Plugins like iDetect can be restarted again in Domoticz 2024.1 Thanks to the Domoticz developer that fixed the framework!

What is the difference now between your beta plug-in and the normal one?

And what is you experience with the memory leak in the 2024 version?

The beta uses the extended (DomoticzEx) framework.The previous version uses the legacy framework, which is limited by the maximum number of units (255).
The beta version only loads python modules that are required by the trackers that are actually in use, while the previous version will try and load every module for every kind of tracker (and displays a warning message for the ones that are missing).

I have not measured the exact difference between 2024.1 and the 2023 Domoticz version, but both show a (slight) increase in memory usage on an otherwise empty Domoticz setup in a docker container. Even with an empty plugin (not iDetect). The memory leak when using parallel-ssh was horrible. In a test setup that would poll every 3 seconds(!) I would see a 0,5MB/minute rise in memory usage when using parallel-ssh . The same setup using paramiko showed 0.02MB/minute. A completely empty python plugin would still show 0.01MB/minute, so I don't think there is much more I can do about it. Memory leaking in the python plugin framework seems te be an ongoing discussion.
Maybe other python or Domoticz versions can improve the situation.

[hjzwiers]

Where can I find those settings, I checked the settings tab, but I cannot find it. I have another plugin which is working fine so I was not aware.

I have the switch under the first tab set to accept new hardware devices ...

[rensbr]

I cannot figure out what is causing the memory leak when using the parallel-ssh module, so the beta (in GitHub) is now using paramiko again.

Parallel-ssh caught my eye as an alternative because Paramiko was a pain to include in a docker image. But when using the standard Domoticz docker image and a customstart.sh file to install paramiko (see https://www.domoticz.com/wiki/Docker) it is easy enough.

If you want to help figure out why parallel-ssh is causing a memory leak when used in my Domoticz plugin then you can re-enable it by copying ssh_tracker.py.use-pssh.py over ssh_tracker.py in the trackers directory.

Unrelated but good news: Plugins like iDetect can be restarted again in Domoticz 2024.1 Thanks to the Domoticz developer that fixed the framework!

What is the difference now between your beta plug-in and the normal one?

And what is you experience with the memory leak in the 2024 version?

The beta uses the extended (DomoticzEx) framework.The previous version uses the legacy framework, which is limited by the maximum number of units (255).
The beta version only loads python modules that are required by the trackers that are actually in use, while the previous version will try and load every module for every kind of tracker (and displays a warning message for the ones that are missing).

I have not measured the exact difference between 2024.1 and the 2023 Domoticz version, but both show a (slight) increase in memory usage on an otherwise empty Domoticz setup in a docker container. Even with an empty plugin (not iDetect). The memory leak when using parallel-ssh was horrible. In a test setup that would poll every 3 seconds(!) I would see a 0,5MB/minute rise in memory usage when using parallel-ssh . The same setup using paramiko showed 0.02MB/minute. A completely empty python plugin would still show 0.01MB/minute, so I don't think there is much more I can do about it. Memory leaking in the python plugin framework seems te be an ongoing discussion.
Maybe other python or Domoticz versions can improve the situation.

Thanks for the conqinues support you are giving. I have updated Domoticz to the latest version now. The plugin is still the 'original' one. Let's wait and see how that works for now.

[EscApe]

Where can I find those settings, I checked the settings tab, but I cannot find it. I have another plugin which is working fine so I was not aware.

I have the switch under the first tab set to accept new hardware devices ...

Looks like you found the setting, since you are confirming it is set to on

I can reproduce the issue on a new install. Seems to be a bug in Domoticz and every plugin will have this problem, but it will only arise with a newly created database.
The plugin framework was not modified to support a database change that was introduced by (probably) 2024.1. I have raised an issue with Domoticz on github.
https://github.com/domoticz/domoticz/is ... 2067161344

In the meanwhile it should work with existing databases that have been upgraded to 2024.1. At least, it works on my updated database. Comparing the databases I can see that the field in question is now required but was optional in the updated database. The plugin framework does not supply a value for this now required field, so the creation of devices failed on a database that was created by 2024.1.

It is possible this issue existed on previous versions. I have not tried a database created by the 2023 version.

[EscApe]

Update on the bug-report: gizmocuz confirmed the database issue and has already committed a fix to the development branch. By experience (no guarantee) it will probably be in the beta build tomorrow.