Domoticz Memory leak in 2024 beta build 15910

[Kedi]

Your picture is not visible. Everybody should use pictures from the widget Process Usage
Because that is the memory consumption from Domoticz

[waltervl]

Does your dzvents scripts use shellcommand or OpenURL?
When not closed correctly these external resources can stay open....

[Kedi]

When you use shell commands in Linux and they are not properly closed they become zombies or you should see them when you do a 'ps aux'
I have seen none of those.
You can find zombie process by

Code: Select all

ps aux | awk '$8 ~ /^[Zz]/'

If openURL is calling an external program (curl or a like) then the same applies.
If it is done internally, Domoticz should take care of close properly.

[niki_lauda]

I am glad that a few have experienced the same problem.
I have phenomenon since the last few months. 2024.4 also run Mqtt and dzevents

Disabling dzevents for now. Will come back with the results

[Kedi]

Last updates to the dzVents system are from 2 months ago.
If your memory rise is from after that time, then 1 of your scripts added or changed the last 2 months could be the cause.

[niki_lauda]

I am glad that a few have experienced the same problem.
I have phenomenon since the last few months. 2024.4 also run Mqtt and dzevents

Disabling dzevents for now. Will come back with the results

Dzevents is not the problem. Memory usage increased with dzevents switched off


Last night briefly disabling admqtt clearly had a positive effect on memory.
Memory usage went down from 128mb to 49 mb.
Can anyone else test this?

[Kedi]

If I disable MQTT-AD on my RPi-4b test system (some 400 devices), the process usage stays the same.

[ajongen70]

I have disabled the Motherboard Sensors Hardware for a few days now and monitoring the Memory usage of Domoticz with Monit shows a stable usage.

Hypothesis: the Motherboard Sensor Hardware creates a memory leak.

Is there anyway to confirm this? Who is maintaining the Motherboard Sensors code?

PS: I do not have MQTT Auto Discovery Gateway enabled. I do have the MQTT Gateway (without AD) enabled.

[Kedi]

That is NOT the case. I have several systems on RPI's. On every system I have Motherboard sensors enabled.
I have an RPi's (Remote slave) which does NOT suffer from a memory leak.
My 'master' system suffers from a memory leak. Both run the same OS version.
My test system with an other OS does not suffer from memory leak.

[waltervl]

On my test system running latest beta I only see memory leaks of approx 30MB per day if I use python event scripts (not plugins). This is already a long time known bug https://github.com/domoticz/domoticz/issues/5091

I have several dzvents scripts running, some python plugins and some native integrations. This all works stable on my test system until I enable python event scripts.

[Kedi]

I don't have any Python event scripts running on my systems.

[waltervl]

And I don't have MQTT AD....

[JanVlietland]

I ran various tests last week. In my case, the more sensors I enable, the more memory leaks occur. Disabling all scripts does not make a difference. Additionally, the type of hardware (MQTT, Gateway USB, Gateway LAN) does not affect the issue. Disabling MQTT slows down the memory leak the most since it connects to about 700 devices.

I plan to run a second PI with a backup from August 2023, but I haven't found the time yet. Restarting daily at 3:30 AM is a workable workaround for my setup.

[gizmocuz]

There are NO memory leaks in MQTT(-AD)...

But the more devices you have, the more memory it uses. This is logical and throughout the whole system.
All device configuration is stored for example, which takes a bit of memory.
Memory is not an issue these days.

There seems to be a known leak when using Python scripts. This has been there since quite some time.
As I don't use Python plugins (except one), and have limited knowledge of side loading this (and wish to stay that way),
this is not a priority for me to solve and I hope other will do so. As already have been tried a few times.

Also 40 MB memory increase is nothing.
Most systems these days have 2-4GB memory. If you run on a system with 100mb of memory I suggest upgrading this.

I have systems running MQTT for months without memory issues.

There are also other forum posts related to memory leaks as well as on Github, and no need to start another one.

And before 'just saying' the memory should come from 'this and that' hardware, I suggest having a look at the code to actually support this nonsense.
Show me the malloc's and new's and other dangerous memory allocations being used (if at all) without being freed.

[waltervl]

There seems to be a known leak when using Python scripts. This has been there since quite some time.
As I don't use Python plugins (except one), and have limited knowledge of side loading this (and wish to stay that way),
this is not a priority for me to solve and I hope other will do so. As already have been tried a few times.

There is an issue with Python Event scripts (on Linux, not windows) as you mention, not with Python Plugins.
https://github.com/domoticz/domoticz/issues/5091

I am still able to reproduce this. But I do not use Python Event scripts but dzvents

[gizmocuz]

Yes there are memory issues throughout the whole Python implementation.
This includes both events and certainly plugins.
There are even example python plugins made and posted on Github that proves this.

[lost]

Most systems these days have 2-4GB memory. If you run on a system with 100mb of memory I suggest upgrading this.
I have systems running MQTT for months without memory issues.

Well, that's my current uptime even if, this time, Domoticz was indeed restarted on January upgrade (otherwise, this would not have been done):

Code: Select all

uptime
09:27:52 up 226 days, 17:55,  1 user,  load average: 0.07, 0.07, 0.05
 
cat /proc/meminfo | grep -i avail
MemAvailable:     482620 kB

So more than 7 months uptime for whole system and almost 3 months without domoticz restart: So that's for sure stable!

I use a PI3B with 1GB, so ~50% used. When I started with Domoticz at the end of 2016 this was 10% but since then I added some IP cams processing (custom service running aside Domoticz) that use some tmpfs in ram to minimize uSD use/wear. So most increase is inked to this.

Only one python plugin (zigbee4domoticz but only 3 zigbee devices compares to 10's zwave) that did not changed figures.

I have 3 python based services (not plugins) for cams/presence-detection/fixed-phone spammers filtering that are only linked to Domoticz using http/json API or generating/feeding custom pages. So out of plugin system that was a bit new at the time, but can observe after restarting the most hungry one
(managing cams) after some changes/additions, memory use drops immediately for ~10% then steadily rise for 6/8 hours and then remains stable to current ~50% use.

So python garbage collector looks very-very long to kick-in, even on a sane system running like a Swiss clock... Probably not very helpful mem management behavior to understand a leak RC that may have taken place hours before. And maybe misleading people that panic before the kind of python interpreter dirty ratio is reached? Don't know if this can be tuned.

Anyway, Domoticz can still run on a system with a few 100's MB with a few care (minimal Linux system/No DE needed if headless...) IMO. 1GB is still large even with some side video/image (basic) processing.

[JanVlietland]

An update that hopefully shines some light on the matter:

I have extracted the Domoticz dir from the backup (dd -> disk image) of july 2023 (2023.2) and installed that version on the RPI and manually copy pasted all scripts.
Result: no memory leak.

Then updated Domoticz to the latest version (2024.4), via the Domoticz site.
Result: no memory leak.

Then installed the latest version of domoticz (with curl) and imported the database.
Result: memory leak

My conclusion is that my Domoticz database causes the issue. So for me problem solved. Hopefully the above will help others to find the root-cause.

ps: I noticed with the clean installs that MQTT and Mysensors autodiscovery is not working if lots of devices are used. But that's another story and will post in that thread.

[Kedi]

My conclusion is that my Domoticz database causes the issue. So for me problem solved. Hopefully the above will help others to find the root-cause.

I don't understand your conclusion. How can the database be the cause, and how did you solve the problem?

[JanVlietland]

With a clean install and importing the data created memory leaks, while the present install with an older version of the database got rid of the memory leak.

Have you tried to export the database, mv domoticz domoticz.bak, reinstall with curl and then imported an older version of your database?