Page 1 of 1
Docker container memory is growing
Posted: Wednesday 08 September 2021 9:26
by Gravityz
Hello,
i switched from Jadahl synology to docker.
everything seems to run great but i notice that the memory ,that the container is using, is growing rapidly
it started around 100MB but now 5 days later is is over 600MB.
This is strange because i mapped the opt/domoticz/userdata to a directory outside the container (on my synology)
The log is also mapped to this location and i use logrotate to avoid big logfiles
anybody any ideas if this growth will stop?
is it perhaps normal because the external storage (DB, backup, log) is recognized as used mem while in fact it is storage on the nas?
- domoticz-docker.JPG (41.04 KiB) Viewed 3003 times
- domoticz-docker2.JPG (26.11 KiB) Viewed 3003 times
Re: Docker image memory is growing
Posted: Wednesday 08 September 2021 10:06
by waltervl
There are reports of memory leaks in Python plugins using the new extended framework.
Are you using Python plugins?
What beta version of domoticz are you running?
Re: Docker image memory is growing
Posted: Wednesday 08 September 2021 10:19
by EddyG
Gravityz wrote: ↑Wednesday 08 September 2021 9:26
The log is also mapped to this location and i use logrotate to avoid big logfiles
Just an other question.
How do you logrotate the domoticz log file?
It is outside the container, but to properly use is you should do an "sudo /etc/init.d/domoticz.sh reload" inside the container.
How did you solve that?
And I agree with @waltervl that it might come from Python leaks.
I use a test system in docker without plugins and see no growth.
Re: Docker image memory is growing
Posted: Wednesday 08 September 2021 10:22
by Gravityz
Hi waltervl
as far as ik know i am not using python plugins
i am on 2021.1.13422
is the native buienradar plugin using python underneath.
if it does this might be the problem.
waltervl wrote: ↑Wednesday 08 September 2021 10:06
There are reports of memory leaks in Python plugins using the new extended framework.
Are you using Python plugins?
What beta version of domoticz are you running?
Re: Docker image memory is growing
Posted: Wednesday 08 September 2021 10:30
by Gravityz
the logfile is on an external directory on the synology.
So if somebody is in control of the logrotate actions it has to be the synology, not the container(correct me if i am wrong)
i put a file dockerdomoticz in etc/logrotate.d on my nas
Code: Select all
/volume1/docker/domoticz/domoticz.log {
su root root
daily
size 1M
rotate 4
notifempty
compress
compresscmd /usr/bin/xz
compressext .xz
compressoptions -3
delaycompress
missingok
copytruncate
nocreate
}
then i initiated the proces by
logrotate -f /etc/logrotate.d/dockerdomoticz
- domoticz-logrotate.JPG (24.94 KiB) Viewed 2990 times
so are you telling me i need to initiate the process from inside the container instead of the nas.
what will change then since it is obviously working
EddyG wrote: ↑Wednesday 08 September 2021 10:19
Gravityz wrote: ↑Wednesday 08 September 2021 9:26
The log is also mapped to this location and i use logrotate to avoid big logfiles
Just an other question.
How do you logrotate the domoticz log file?
It is outside the container, but to properly use is you should do an "sudo /etc/init.d/domoticz.sh reload" inside the container.
How did you solve that?
And I agree with @waltervl that it might come from Python leaks.
I use a test system in docker without plugins and see no growth.
Re: Docker image memory is growing
Posted: Wednesday 08 September 2021 19:41
by EddyG
If I remember correctly that Domoticz held the logfile open during the whole session. That was creating a problem with logrotate.
That is why you had to use the reload parameter.
But that might be solved if Domoticz open / writes / closes the log file every time.
Normally logrotate renames and processes the old logfile and creates a new empty logfile, and if Domoticz does not close the logfile then it has the wrong filehandle.
Re: Docker image memory is growing
Posted: Thursday 09 September 2021 8:59
by Gravityz
I remember that one and i think it was fixed in the jadahl package.
the problem then was that the original file was set to zero but was not accepted so it kept on fillin the same file which got bigger and bigger.
looking at the logfile now it seems to be small.
but indeed if the domoticz container still thinks it is that old file with the old size then the problem might be related
this was the thread about that issue
the issue was fixed on the normal build and also the jadahl build, maybe putting things in docker creates another side effect
maybe i need to play around with the logrotate options to see if i can change things
copytruncate and nocreate were used to solve a problem so maybee the solutions lies here.
https://github.com/domoticz/domoticz/is ... -439677392
another possibility might be that the mapped directory is seen as memory from the container like i said.
because auto backup is used this will create lot's of backups.
i stopped and started the container and the mem value is low (38MB)
if i am right the growth will stop once all the auto backups are in place (67 in total) but growth will also slow down onece all the daily and houtly backups are in place
EddyG wrote: ↑Wednesday 08 September 2021 19:41
If I remember correctly that Domoticz held the logfile open during the whole session. That was creating a problem with logrotate.
That is why you had to use the reload parameter.
But that might be solved if Domoticz open / writes / closes the log file every time.
Normally logrotate renames and processes the old logfile and creates a new empty logfile, and if Domoticz does not close the logfile then it has the wrong filehandle.
Re: Docker image memory is growing
Posted: Thursday 09 September 2021 13:08
by EddyG
So it might be better if the backup directory is also placed outside the container.
I use my own backup script via cron (every 5 minutes in production), so no problem here.
Re: Docker image memory is growing
Posted: Thursday 09 September 2021 13:25
by waltervl
I am still wondering why someone would need a domoticz docker install in a production environment. I see it only usable for incompatible environments like some Synology NAS where building the application is a disaster and Docker is running.
For a simple user too many changes to be done and to think about in the Docker configuration. With a compatible device like an RPi the normal application is an easy install and update (including automated backup of complete environment for beta).
Re: Docker image memory is growing
Posted: Thursday 09 September 2021 13:27
by Gravityz
everything is placed outside the container, also the backup directory
i did take a closer look and when the ram of the docker container increases the ram of the synology stays the same.
so i think the ram usage is not only the usage of real ram but it also thinks the /opt/domticz/userdata capacity(which is mapped to volume1/docker/domoticz) is in ram while in fact it is not inside the container but on an external directory
t would be nice if somebody can confirm this for us.
it is definately not the log file because that one is growing in KB's while the ram usage is growing in MB's
@waltervl :agreed but i have a synology so docker is a way to make things easy
Re: Docker container memory is growing
Posted: Friday 10 September 2021 15:24
by Gravityz
did some testing and i think i am right about the docker container thinking it is using ram but in fact it is using storage
so docker domoticz container ram usage increased 97MB after 2 days and rising
total usage however only increased 18MB (after memory scrubbing it will be even less)
moment 1
docker container ram 71MB
total synology ram consumed 711MB
- ramgebruik1.JPG (53.3 KiB) Viewed 2902 times
moment2
docker container ram 168MB
total synology ram consumed 729MB
- ramgebruik2.JPG (51.1 KiB) Viewed 2902 times
Re: Docker container memory is growing
Posted: Monday 20 September 2021 19:29
by Gravityz
and the usage after 12 days
definately not a ram issue but storage which is recognized by the container as ram
i have 6GB of ram so after approx 53 days the ram usage of the container will approach this 6GB.
let's wait and see
- ramgebruik3.JPG (50.98 KiB) Viewed 2874 times
Re: Docker container memory is growing
Posted: Wednesday 02 October 2024 9:20
by mvdp
I can confirm this observation.
I'm running domoticz (version 2023.2) in docker and once every week the docker process is oomed (out-of-memory stopped by the kernel):
Code: Select all
[Wed Oct 2 08:55:59 2024] Out of memory: Kill process 20609 (domoticz) score 331 or sacrifice child
[Wed Oct 2 08:55:59 2024] Killed process 14543 (ls) total-vm:3316kB, anon-rss:80kB, file-rss:0kB
[Wed Oct 2 08:56:37 2024] synoraidtool invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Docker restarts but I have a manual job to create symlinks for the usb devices (the synology links don't show up in docker) so it is a bit annoying.
Re: Docker container memory is growing
Posted: Wednesday 02 October 2024 11:20
by Marathon2010
Running Domoticz 2024.7 under Container Manager (successor of Docker on DSM 7x) on NAS Synology. My RAM and usage shown below. And running stable for long periods without hampering.
- Container Manager.jpg (93.29 KiB) Viewed 1537 times
Re: Docker container memory is growing
Posted: Thursday 10 October 2024 7:43
by mvdp
I upgraded to 2024.7 and it keeps running. But with this version the web ui isn’t responsive after a couple of hours.
Edit: The last versions of domoticz had a deadlock problem. When this was fixed a beta version (Version: 2024.7 (build 16286)) of domoticz has been running for weeks without issues.