Docker container memory is growing

On various Hardware and OS systems: pi / windows / routers / nas, etc

Moderator: leecollings

Post Reply
Gravityz
Posts: 583
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Docker container memory is growing

Post by Gravityz »

Hello,

i switched from Jadahl synology to docker.
everything seems to run great but i notice that the memory ,that the container is using, is growing rapidly

it started around 100MB but now 5 days later is is over 600MB.
This is strange because i mapped the opt/domoticz/userdata to a directory outside the container (on my synology)
The log is also mapped to this location and i use logrotate to avoid big logfiles
anybody any ideas if this growth will stop?
is it perhaps normal because the external storage (DB, backup, log) is recognized as used mem while in fact it is storage on the nas?

domoticz-docker.JPG
domoticz-docker.JPG (41.04 KiB) Viewed 2276 times
domoticz-docker2.JPG
domoticz-docker2.JPG (26.11 KiB) Viewed 2276 times
Last edited by Gravityz on Thursday 09 September 2021 16:32, edited 1 time in total.
User avatar
waltervl
Posts: 5386
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Docker image memory is growing

Post by waltervl »

There are reports of memory leaks in Python plugins using the new extended framework.
Are you using Python plugins?

What beta version of domoticz are you running?
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
EddyG
Posts: 1042
Joined: Monday 02 November 2015 5:54
Target OS: -
Domoticz version:

Re: Docker image memory is growing

Post by EddyG »

Gravityz wrote: Wednesday 08 September 2021 9:26 The log is also mapped to this location and i use logrotate to avoid big logfiles
Just an other question.
How do you logrotate the domoticz log file?
It is outside the container, but to properly use is you should do an "sudo /etc/init.d/domoticz.sh reload" inside the container.
How did you solve that?

And I agree with @waltervl that it might come from Python leaks.
I use a test system in docker without plugins and see no growth.
Gravityz
Posts: 583
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: Docker image memory is growing

Post by Gravityz »

Hi waltervl
as far as ik know i am not using python plugins
i am on 2021.1.13422

is the native buienradar plugin using python underneath.
if it does this might be the problem.
waltervl wrote: Wednesday 08 September 2021 10:06 There are reports of memory leaks in Python plugins using the new extended framework.
Are you using Python plugins?

What beta version of domoticz are you running?
Last edited by Gravityz on Wednesday 08 September 2021 10:49, edited 1 time in total.
Gravityz
Posts: 583
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: Docker image memory is growing

Post by Gravityz »

the logfile is on an external directory on the synology.
So if somebody is in control of the logrotate actions it has to be the synology, not the container(correct me if i am wrong)

i put a file dockerdomoticz in etc/logrotate.d on my nas

Code: Select all

/volume1/docker/domoticz/domoticz.log {
        su root root
	daily
        size 1M
        rotate 4
        notifempty
        compress
        compresscmd /usr/bin/xz
        compressext .xz
        compressoptions -3
        delaycompress
        missingok
	copytruncate
        nocreate
}
then i initiated the proces by
logrotate -f /etc/logrotate.d/dockerdomoticz
domoticz-logrotate.JPG
domoticz-logrotate.JPG (24.94 KiB) Viewed 2263 times
so are you telling me i need to initiate the process from inside the container instead of the nas.
what will change then since it is obviously working


EddyG wrote: Wednesday 08 September 2021 10:19
Gravityz wrote: Wednesday 08 September 2021 9:26 The log is also mapped to this location and i use logrotate to avoid big logfiles
Just an other question.
How do you logrotate the domoticz log file?
It is outside the container, but to properly use is you should do an "sudo /etc/init.d/domoticz.sh reload" inside the container.
How did you solve that?

And I agree with @waltervl that it might come from Python leaks.
I use a test system in docker without plugins and see no growth.
EddyG
Posts: 1042
Joined: Monday 02 November 2015 5:54
Target OS: -
Domoticz version:

Re: Docker image memory is growing

Post by EddyG »

If I remember correctly that Domoticz held the logfile open during the whole session. That was creating a problem with logrotate.
That is why you had to use the reload parameter.
But that might be solved if Domoticz open / writes / closes the log file every time.
Normally logrotate renames and processes the old logfile and creates a new empty logfile, and if Domoticz does not close the logfile then it has the wrong filehandle.
Gravityz
Posts: 583
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: Docker image memory is growing

Post by Gravityz »

I remember that one and i think it was fixed in the jadahl package.
the problem then was that the original file was set to zero but was not accepted so it kept on fillin the same file which got bigger and bigger.
looking at the logfile now it seems to be small.

but indeed if the domoticz container still thinks it is that old file with the old size then the problem might be related

this was the thread about that issue
the issue was fixed on the normal build and also the jadahl build, maybe putting things in docker creates another side effect
maybe i need to play around with the logrotate options to see if i can change things
copytruncate and nocreate were used to solve a problem so maybee the solutions lies here.

https://github.com/domoticz/domoticz/is ... -439677392

another possibility might be that the mapped directory is seen as memory from the container like i said.
because auto backup is used this will create lot's of backups.
i stopped and started the container and the mem value is low (38MB)
if i am right the growth will stop once all the auto backups are in place (67 in total) but growth will also slow down onece all the daily and houtly backups are in place

EddyG wrote: Wednesday 08 September 2021 19:41 If I remember correctly that Domoticz held the logfile open during the whole session. That was creating a problem with logrotate.
That is why you had to use the reload parameter.
But that might be solved if Domoticz open / writes / closes the log file every time.
Normally logrotate renames and processes the old logfile and creates a new empty logfile, and if Domoticz does not close the logfile then it has the wrong filehandle.
EddyG
Posts: 1042
Joined: Monday 02 November 2015 5:54
Target OS: -
Domoticz version:

Re: Docker image memory is growing

Post by EddyG »

So it might be better if the backup directory is also placed outside the container.
I use my own backup script via cron (every 5 minutes in production), so no problem here.
User avatar
waltervl
Posts: 5386
Joined: Monday 28 January 2019 18:48
Target OS: Linux
Domoticz version: 2024.7
Location: NL
Contact:

Re: Docker image memory is growing

Post by waltervl »

I am still wondering why someone would need a domoticz docker install in a production environment. I see it only usable for incompatible environments like some Synology NAS where building the application is a disaster and Docker is running.
For a simple user too many changes to be done and to think about in the Docker configuration. With a compatible device like an RPi the normal application is an easy install and update (including automated backup of complete environment for beta).
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
Gravityz
Posts: 583
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: Docker image memory is growing

Post by Gravityz »

everything is placed outside the container, also the backup directory

i did take a closer look and when the ram of the docker container increases the ram of the synology stays the same.

so i think the ram usage is not only the usage of real ram but it also thinks the /opt/domticz/userdata capacity(which is mapped to volume1/docker/domoticz) is in ram while in fact it is not inside the container but on an external directory

t would be nice if somebody can confirm this for us.

it is definately not the log file because that one is growing in KB's while the ram usage is growing in MB's

@waltervl :agreed but i have a synology so docker is a way to make things easy
Gravityz
Posts: 583
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: Docker container memory is growing

Post by Gravityz »

did some testing and i think i am right about the docker container thinking it is using ram but in fact it is using storage

so docker domoticz container ram usage increased 97MB after 2 days and rising
total usage however only increased 18MB (after memory scrubbing it will be even less)
moment 1
docker container ram 71MB
total synology ram consumed 711MB
ramgebruik1.JPG
ramgebruik1.JPG (53.3 KiB) Viewed 2175 times


moment2
docker container ram 168MB
total synology ram consumed 729MB
ramgebruik2.JPG
ramgebruik2.JPG (51.1 KiB) Viewed 2175 times
Gravityz
Posts: 583
Joined: Wednesday 16 December 2015 19:13
Target OS: NAS (Synology & others)
Domoticz version: 2022.2
Location: Netherlands
Contact:

Re: Docker container memory is growing

Post by Gravityz »

and the usage after 12 days

definately not a ram issue but storage which is recognized by the container as ram
i have 6GB of ram so after approx 53 days the ram usage of the container will approach this 6GB.
let's wait and see

ramgebruik3.JPG
ramgebruik3.JPG (50.98 KiB) Viewed 2147 times
mvdp
Posts: 4
Joined: Sunday 07 October 2018 17:21
Target OS: Linux
Domoticz version:
Contact:

Re: Docker container memory is growing

Post by mvdp »

I can confirm this observation.
I'm running domoticz (version 2023.2) in docker and once every week the docker process is oomed (out-of-memory stopped by the kernel):

Code: Select all

[Wed Oct  2 08:55:59 2024] Out of memory: Kill process 20609 (domoticz) score 331 or sacrifice child
[Wed Oct  2 08:55:59 2024] Killed process 14543 (ls) total-vm:3316kB, anon-rss:80kB, file-rss:0kB
[Wed Oct  2 08:56:37 2024] synoraidtool invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Docker restarts but I have a manual job to create symlinks for the usb devices (the synology links don't show up in docker) so it is a bit annoying.
User avatar
Marathon2010
Posts: 27
Joined: Tuesday 02 February 2021 23:02
Target OS: NAS (Synology & others)
Domoticz version: 2024.7
Location: Netherlands
Contact:

Re: Docker container memory is growing

Post by Marathon2010 »

Running Domoticz 2024.7 under Container Manager (successor of Docker on DSM 7x) on NAS Synology. My RAM and usage shown below. And running stable for long periods without hampering.
Container Manager.jpg
Container Manager.jpg (93.29 KiB) Viewed 810 times
Synology NAS DS920+ DSM7 (Docker) | Integrations: Audio, Heating, Gas, Electricity, Water, Alarm, Car, Camera, Doorbell, Garage | SW: Tasmota, DzVents, Pushbullet | Plugins: GPSLogger, BatteryLevel, Hyundai KIA
mvdp
Posts: 4
Joined: Sunday 07 October 2018 17:21
Target OS: Linux
Domoticz version:
Contact:

Re: Docker container memory is growing

Post by mvdp »

I upgraded to 2024.7 and it keeps running. But with this version the web ui isn’t responsive after a couple of hours.


Edit: The last versions of domoticz had a deadlock problem. When this was fixed a beta version (Version: 2024.7 (build 16286)) of domoticz has been running for weeks without issues.
Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests