Domoticz Memory leak in 2024 beta build 15910

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!

Moderators: leecollings, remb0

Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post

Please mark your topic as Solved when the problem is solved.
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

With:
Version: 2024.4 (build 15910)
Build Hash: 0f2fcff99
Compile Date: 2024-02-10 12:56:24
dzVents Version: 3.1.8

I still experience memory leaks.
Screenshot_20240304_092023.png
Screenshot_20240304_092023.png (143.31 KiB) Viewed 1783 times
First used the release 2024.4 version, now using the latest beta. No difference.

Current workaround is restarting every night with a cronjob.

Some info:
Screenshot_20240304_092419.png
Screenshot_20240304_092419.png (65.99 KiB) Viewed 1783 times
In total Domoticz is connected to approx 800 devices, and a mix of 24 blockly, lua and dzvents scripts.

Before reverting to an old version I appreciate thoughts and comments.
User avatar
waltervl
Posts: 5148
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Domoticz Memory leak

Post by waltervl »

You cannot revert back to an older version, at least I would not advise it. If so you have to go back to an old database too.
You are showing memory usage of the machine. Is memory usage of the Domoticz process showing the same issue (see motherboard sensor device Process Usage)? If not it is another process on your machine that requires memory.
It seems you are not using Python plugins so it should be in the scripts somewhere.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
User avatar
waltervl
Posts: 5148
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Domoticz Memory leak

Post by waltervl »

You are using a 2024.beta, did it happen with the previous beta too?
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
Kedi
Posts: 536
Joined: Monday 20 March 2023 14:41
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: Somewhere in NL
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by Kedi »

For the same database version you could go back to 15 January 2024, database version changed that day to 167.
Logic will get you from A to B. Imagination will take you everywhere.
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

Thanks your thoughs and comments.

I did not use a previous beta. Previously I used the release versions. It started last with the update in december. But in the meantime I also created additional scripts such as a (gas) degreesdays script.

Good point about the system mem. However I am certain it is the domoticz process, since restarting solves the issue. For example. This is the current system mem
Screenshot_20240304_213847.png
Screenshot_20240304_213847.png (129.35 KiB) Viewed 1685 times
This is the domoticz mem usage:
htop.png
htop.png (130.48 KiB) Viewed 1685 times
After: sudo service domoticz.sh restart
htop new.png
htop new.png (165.99 KiB) Viewed 1685 times

Is there a way to quickly determine the memory usage of the Domoticz subprocesses?

Many thanks again for your support.
User avatar
waltervl
Posts: 5148
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by waltervl »

Perhaps someone knows a better way to debug this but I would start with disabling the degreeday script for a day and see if the memory leak stays.

Do you restart only the Domoticz process or the complete system? You have a device called "process usage", please check that to be sure it is Domoticz (probably is)
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
Kedi
Posts: 536
Joined: Monday 20 March 2023 14:41
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: Somewhere in NL
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by Kedi »

You could try to look for the culprit with programs like gdb and pmap, but that is a tedious process.
I had the same kind of issue at the end of december / start of januari.
I my case I found out that it had probably to do with a system update in mid of december, I did not investigate which program was the 'bad guy'
But it might have to do with 'python3-urllib3'
I have a btrfs system and make snaphots every hour/day, so I just put the system back to before that update, and my problem was solved.
Logic will get you from A to B. Imagination will take you everywhere.
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

I only restart domoticz with: sudo service domoticz.sh restart. I am 100% sure it is domoticz process
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

@Kedi, I am afraid that I have to rebuild the system from scratch and step by step add configuration.
Last edited by JanVlietland on Wednesday 06 March 2024 0:07, edited 1 time in total.
User avatar
waltervl
Posts: 5148
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by waltervl »

I have seen situations that scripts running external commands that are not correctly closed create memory resource issues.

So check the scripts that run external commands.
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

Yes I will start with that part.
Kangooroo42
Posts: 8
Joined: Wednesday 23 October 2019 8:41
Target OS: Windows
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by Kangooroo42 »

Hello i am running domoticz 2024.4 and i have the same problem... did you find where it comes from ?
(running domoticz on windows 11)
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

I did some testing last weeks. The leak correlates with enabled hardware, not with any script. More enabled hardware = more memory leaking away.

Herewith a image.
memory leaks.png
memory leaks.png (285.95 KiB) Viewed 1406 times
Yesterday evening I disabled most of the hardware. At 3:30 the daily restart occurs. At 9:00 I disabled all hardware, except the PI-sensors. 11:30 I started to enabled hardware again.

Since I have in total about 800 devices I need to restart demoticz daily as after a day of two the OOM-killers kills the Domoticz process.
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

Today I installed the latest beta. Same issue persists.

Is it really that my instance is the only one that has memory leaks? If I start with a brand new install the same issue occurs. Cannot believe that I'm the only one having this issue.
Kedi
Posts: 536
Joined: Monday 20 March 2023 14:41
Target OS: Raspberry Pi / ODroid
Domoticz version:
Location: Somewhere in NL
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by Kedi »

No you are not the only one. I also have a small memory increase every day until I get an unresponsive Domoticz after a few weeks.
Still have not figured out what the cause is, but keep looking.
Logic will get you from A to B. Imagination will take you everywhere.
JanVlietland
Posts: 24
Joined: Wednesday 17 June 2020 22:55
Target OS: Raspberry Pi / ODroid
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by JanVlietland »

This is my setup:
Raspberry Pi reference 2023-05-03
Raspbian GNU/Linux 11 (bullseye)
Linux pi 5.10.103-v7l+ #1529 SMP Tue Mar 8 12:24:00 GMT 2022 armv7l GNU/Linux
Raspberry Pi 4 Model B Rev 1.5
Domoticz: Version: 2024.4 (build 15910)
Build Hash: 0f2fcff99
Compile Date: 2024-02-10 12:56:24
dzVents Version: 3.1.8
Python Version: 3.9.2 (default, Mar 12 2021, 04:06:34) [GCC 10.2.1 20210110]

Maybe comparing setups with memory-leaks and without leaks helps finding the root-cause.

Anybody else (not) having the same issue?
ajongen70
Posts: 10
Joined: Sunday 25 February 2024 0:28
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.3
Location: Netherlands
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by ajongen70 »

I am experiencing the simular behaviour. Running 2024.3 on a RPI Zero (made the build myself). I am now thinking of letting monit restart domoticz when it passes 75Mb of memory ( if total memory > 75.0 MB for 5 cycles then restart ). System info:

Version: 2024.3
Build Hash: 3503953dd
Compile Date: 2024-01-24 08:40:07
dzVents Version: 3.1.8
Python Version: 3.9.2 (default, Mar 12 2021, 04:06:34) [GCC 10.2.1 20210110]
Uptime: 14 Days, 6 Hours, 52 Minutes, 50 Seconds

Expecting a crash of domoticz in the coming hours....
Test.jpg.jpeg
Test.jpg.jpeg (256.19 KiB) Viewed 1324 times
User avatar
waltervl
Posts: 5148
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by waltervl »

Process memory usage of 100MB should not crash your system. What other processes are running on your system?
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
ajongen70
Posts: 10
Joined: Sunday 25 February 2024 0:28
Target OS: Raspberry Pi / ODroid
Domoticz version: 2024.3
Location: Netherlands
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by ajongen70 »

There are other processes running on the system (otmonitor, nodered, mosquito MQTT, monit) but they all keep running. Only domoticz crashes and is restarted by monit every once in a while after the used memory of the domoticz process has increased to around 75 - 85 Mb. I did not have this behaviour with previous 2023.x installations.
Schermafbeelding 2024-03-21 201020.png
Schermafbeelding 2024-03-21 201020.png (240.4 KiB) Viewed 1247 times
Kangooroo42
Posts: 8
Joined: Wednesday 23 October 2019 8:41
Target OS: Windows
Domoticz version:
Contact:

Re: Domoticz Memory leak in 2024 beta build 15910

Post by Kangooroo42 »

When I desactivate Dzvents scripts in domoticz parameters, memory stop increasing...(!) but none of my dzvents scripts seems wrong :\
Attachments
dzvents.JPG
dzvents.JPG (25.73 KiB) Viewed 1193 times
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest