Docker container memory is growing
Moderator: leecollings
-
- Posts: 583
- Joined: Wednesday 16 December 2015 19:13
- Target OS: NAS (Synology & others)
- Domoticz version: 2022.2
- Location: Netherlands
- Contact:
Docker container memory is growing
Hello,
i switched from Jadahl synology to docker.
everything seems to run great but i notice that the memory ,that the container is using, is growing rapidly
it started around 100MB but now 5 days later is is over 600MB.
This is strange because i mapped the opt/domoticz/userdata to a directory outside the container (on my synology)
The log is also mapped to this location and i use logrotate to avoid big logfiles
anybody any ideas if this growth will stop?
is it perhaps normal because the external storage (DB, backup, log) is recognized as used mem while in fact it is storage on the nas?
i switched from Jadahl synology to docker.
everything seems to run great but i notice that the memory ,that the container is using, is growing rapidly
it started around 100MB but now 5 days later is is over 600MB.
This is strange because i mapped the opt/domoticz/userdata to a directory outside the container (on my synology)
The log is also mapped to this location and i use logrotate to avoid big logfiles
anybody any ideas if this growth will stop?
is it perhaps normal because the external storage (DB, backup, log) is recognized as used mem while in fact it is storage on the nas?
Last edited by Gravityz on Thursday 09 September 2021 16:32, edited 1 time in total.
- waltervl
- Posts: 5386
- Joined: Monday 28 January 2019 18:48
- Target OS: Linux
- Domoticz version: 2024.7
- Location: NL
- Contact:
Re: Docker image memory is growing
There are reports of memory leaks in Python plugins using the new extended framework.
Are you using Python plugins?
What beta version of domoticz are you running?
Are you using Python plugins?
What beta version of domoticz are you running?
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
Re: Docker image memory is growing
Just an other question.
How do you logrotate the domoticz log file?
It is outside the container, but to properly use is you should do an "sudo /etc/init.d/domoticz.sh reload" inside the container.
How did you solve that?
And I agree with @waltervl that it might come from Python leaks.
I use a test system in docker without plugins and see no growth.
-
- Posts: 583
- Joined: Wednesday 16 December 2015 19:13
- Target OS: NAS (Synology & others)
- Domoticz version: 2022.2
- Location: Netherlands
- Contact:
Re: Docker image memory is growing
Hi waltervl
as far as ik know i am not using python plugins
i am on 2021.1.13422
is the native buienradar plugin using python underneath.
if it does this might be the problem.
as far as ik know i am not using python plugins
i am on 2021.1.13422
is the native buienradar plugin using python underneath.
if it does this might be the problem.
Last edited by Gravityz on Wednesday 08 September 2021 10:49, edited 1 time in total.
-
- Posts: 583
- Joined: Wednesday 16 December 2015 19:13
- Target OS: NAS (Synology & others)
- Domoticz version: 2022.2
- Location: Netherlands
- Contact:
Re: Docker image memory is growing
the logfile is on an external directory on the synology.
So if somebody is in control of the logrotate actions it has to be the synology, not the container(correct me if i am wrong)
i put a file dockerdomoticz in etc/logrotate.d on my nas
then i initiated the proces by
logrotate -f /etc/logrotate.d/dockerdomoticz so are you telling me i need to initiate the process from inside the container instead of the nas.
what will change then since it is obviously working
So if somebody is in control of the logrotate actions it has to be the synology, not the container(correct me if i am wrong)
i put a file dockerdomoticz in etc/logrotate.d on my nas
Code: Select all
/volume1/docker/domoticz/domoticz.log {
su root root
daily
size 1M
rotate 4
notifempty
compress
compresscmd /usr/bin/xz
compressext .xz
compressoptions -3
delaycompress
missingok
copytruncate
nocreate
}
logrotate -f /etc/logrotate.d/dockerdomoticz so are you telling me i need to initiate the process from inside the container instead of the nas.
what will change then since it is obviously working
EddyG wrote: ↑Wednesday 08 September 2021 10:19Just an other question.
How do you logrotate the domoticz log file?
It is outside the container, but to properly use is you should do an "sudo /etc/init.d/domoticz.sh reload" inside the container.
How did you solve that?
And I agree with @waltervl that it might come from Python leaks.
I use a test system in docker without plugins and see no growth.
Re: Docker image memory is growing
If I remember correctly that Domoticz held the logfile open during the whole session. That was creating a problem with logrotate.
That is why you had to use the reload parameter.
But that might be solved if Domoticz open / writes / closes the log file every time.
Normally logrotate renames and processes the old logfile and creates a new empty logfile, and if Domoticz does not close the logfile then it has the wrong filehandle.
That is why you had to use the reload parameter.
But that might be solved if Domoticz open / writes / closes the log file every time.
Normally logrotate renames and processes the old logfile and creates a new empty logfile, and if Domoticz does not close the logfile then it has the wrong filehandle.
-
- Posts: 583
- Joined: Wednesday 16 December 2015 19:13
- Target OS: NAS (Synology & others)
- Domoticz version: 2022.2
- Location: Netherlands
- Contact:
Re: Docker image memory is growing
I remember that one and i think it was fixed in the jadahl package.
the problem then was that the original file was set to zero but was not accepted so it kept on fillin the same file which got bigger and bigger.
looking at the logfile now it seems to be small.
but indeed if the domoticz container still thinks it is that old file with the old size then the problem might be related
this was the thread about that issue
the issue was fixed on the normal build and also the jadahl build, maybe putting things in docker creates another side effect
maybe i need to play around with the logrotate options to see if i can change things
copytruncate and nocreate were used to solve a problem so maybee the solutions lies here.
https://github.com/domoticz/domoticz/is ... -439677392
another possibility might be that the mapped directory is seen as memory from the container like i said.
because auto backup is used this will create lot's of backups.
i stopped and started the container and the mem value is low (38MB)
if i am right the growth will stop once all the auto backups are in place (67 in total) but growth will also slow down onece all the daily and houtly backups are in place
the problem then was that the original file was set to zero but was not accepted so it kept on fillin the same file which got bigger and bigger.
looking at the logfile now it seems to be small.
but indeed if the domoticz container still thinks it is that old file with the old size then the problem might be related
this was the thread about that issue
the issue was fixed on the normal build and also the jadahl build, maybe putting things in docker creates another side effect
maybe i need to play around with the logrotate options to see if i can change things
copytruncate and nocreate were used to solve a problem so maybee the solutions lies here.
https://github.com/domoticz/domoticz/is ... -439677392
another possibility might be that the mapped directory is seen as memory from the container like i said.
because auto backup is used this will create lot's of backups.
i stopped and started the container and the mem value is low (38MB)
if i am right the growth will stop once all the auto backups are in place (67 in total) but growth will also slow down onece all the daily and houtly backups are in place
EddyG wrote: ↑Wednesday 08 September 2021 19:41 If I remember correctly that Domoticz held the logfile open during the whole session. That was creating a problem with logrotate.
That is why you had to use the reload parameter.
But that might be solved if Domoticz open / writes / closes the log file every time.
Normally logrotate renames and processes the old logfile and creates a new empty logfile, and if Domoticz does not close the logfile then it has the wrong filehandle.
Re: Docker image memory is growing
So it might be better if the backup directory is also placed outside the container.
I use my own backup script via cron (every 5 minutes in production), so no problem here.
I use my own backup script via cron (every 5 minutes in production), so no problem here.
- waltervl
- Posts: 5386
- Joined: Monday 28 January 2019 18:48
- Target OS: Linux
- Domoticz version: 2024.7
- Location: NL
- Contact:
Re: Docker image memory is growing
I am still wondering why someone would need a domoticz docker install in a production environment. I see it only usable for incompatible environments like some Synology NAS where building the application is a disaster and Docker is running.
For a simple user too many changes to be done and to think about in the Docker configuration. With a compatible device like an RPi the normal application is an easy install and update (including automated backup of complete environment for beta).
For a simple user too many changes to be done and to think about in the Docker configuration. With a compatible device like an RPi the normal application is an easy install and update (including automated backup of complete environment for beta).
Domoticz running on Udoo X86 (on Ubuntu)
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
Devices/plugins: ZigbeeforDomoticz (with Xiaomi, Ikea, Tuya devices), Nefit Easy, Midea Airco, Omnik Solar, Goodwe Solar
-
- Posts: 583
- Joined: Wednesday 16 December 2015 19:13
- Target OS: NAS (Synology & others)
- Domoticz version: 2022.2
- Location: Netherlands
- Contact:
Re: Docker image memory is growing
everything is placed outside the container, also the backup directory
i did take a closer look and when the ram of the docker container increases the ram of the synology stays the same.
so i think the ram usage is not only the usage of real ram but it also thinks the /opt/domticz/userdata capacity(which is mapped to volume1/docker/domoticz) is in ram while in fact it is not inside the container but on an external directory
t would be nice if somebody can confirm this for us.
it is definately not the log file because that one is growing in KB's while the ram usage is growing in MB's
@waltervl :agreed but i have a synology so docker is a way to make things easy
i did take a closer look and when the ram of the docker container increases the ram of the synology stays the same.
so i think the ram usage is not only the usage of real ram but it also thinks the /opt/domticz/userdata capacity(which is mapped to volume1/docker/domoticz) is in ram while in fact it is not inside the container but on an external directory
t would be nice if somebody can confirm this for us.
it is definately not the log file because that one is growing in KB's while the ram usage is growing in MB's
@waltervl :agreed but i have a synology so docker is a way to make things easy
-
- Posts: 583
- Joined: Wednesday 16 December 2015 19:13
- Target OS: NAS (Synology & others)
- Domoticz version: 2022.2
- Location: Netherlands
- Contact:
Re: Docker container memory is growing
did some testing and i think i am right about the docker container thinking it is using ram but in fact it is using storage
so docker domoticz container ram usage increased 97MB after 2 days and rising
total usage however only increased 18MB (after memory scrubbing it will be even less)
moment 1
docker container ram 71MB
total synology ram consumed 711MB
moment2
docker container ram 168MB
total synology ram consumed 729MB
so docker domoticz container ram usage increased 97MB after 2 days and rising
total usage however only increased 18MB (after memory scrubbing it will be even less)
moment 1
docker container ram 71MB
total synology ram consumed 711MB
moment2
docker container ram 168MB
total synology ram consumed 729MB
-
- Posts: 583
- Joined: Wednesday 16 December 2015 19:13
- Target OS: NAS (Synology & others)
- Domoticz version: 2022.2
- Location: Netherlands
- Contact:
Re: Docker container memory is growing
and the usage after 12 days
definately not a ram issue but storage which is recognized by the container as ram
i have 6GB of ram so after approx 53 days the ram usage of the container will approach this 6GB.
let's wait and see
definately not a ram issue but storage which is recognized by the container as ram
i have 6GB of ram so after approx 53 days the ram usage of the container will approach this 6GB.
let's wait and see
Re: Docker container memory is growing
I can confirm this observation.
I'm running domoticz (version 2023.2) in docker and once every week the docker process is oomed (out-of-memory stopped by the kernel):
Docker restarts but I have a manual job to create symlinks for the usb devices (the synology links don't show up in docker) so it is a bit annoying.
I'm running domoticz (version 2023.2) in docker and once every week the docker process is oomed (out-of-memory stopped by the kernel):
Code: Select all
[Wed Oct 2 08:55:59 2024] Out of memory: Kill process 20609 (domoticz) score 331 or sacrifice child
[Wed Oct 2 08:55:59 2024] Killed process 14543 (ls) total-vm:3316kB, anon-rss:80kB, file-rss:0kB
[Wed Oct 2 08:56:37 2024] synoraidtool invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
- Marathon2010
- Posts: 27
- Joined: Tuesday 02 February 2021 23:02
- Target OS: NAS (Synology & others)
- Domoticz version: 2024.7
- Location: Netherlands
- Contact:
Re: Docker container memory is growing
Running Domoticz 2024.7 under Container Manager (successor of Docker on DSM 7x) on NAS Synology. My RAM and usage shown below. And running stable for long periods without hampering.
Synology NAS DS920+ DSM7 (Docker) | Integrations: Audio, Heating, Gas, Electricity, Water, Alarm, Car, Camera, Doorbell, Garage | SW: Tasmota, DzVents, Pushbullet | Plugins: GPSLogger, BatteryLevel, Hyundai KIA
Re: Docker container memory is growing
I upgraded to 2024.7 and it keeps running. But with this version the web ui isn’t responsive after a couple of hours.
Edit: The last versions of domoticz had a deadlock problem. When this was fixed a beta version (Version: 2024.7 (build 16286)) of domoticz has been running for weeks without issues.
Edit: The last versions of domoticz had a deadlock problem. When this was fixed a beta version (Version: 2024.7 (build 16286)) of domoticz has been running for weeks without issues.
Who is online
Users browsing this forum: No registered users and 0 guests