Log2Ram uses max amount assigned memory

njlammerts · Post by **njlammerts** » Thursday 07 March 2024 10:31

On my Pi-4 + Domoticz on SD card i use Log2Ram as a cache for log files protecting to much read/write to this SD card. When i first started with Pi and Domoticz I had crashes after several months "blowing up" my SD due to much read/write actions.
Since the use of Log2Ram I had not have any SD card problems anymore. I make a SD card copy once a year to a new SD card to prevent card damage.

Since some time I notice on 1 Pi-4/Domoticz configuration that the reserved Log2Ram memory is at 100% use after 2-5 days. Below picture is after 2 day of cleaning up /var/log/ of all log files and do a reboot (first stopped Domoticz and Log2Ram service) On other Pi/Domoticz (I have 4 in total) the used memory is 17, 70, 11 % and the last reboot was over more than a month ago.

: Schermafbeelding 2024-03-07 om 10.26.41.png (142.94 KiB) Viewed 729 times

On the "problem" Pi I have about 400 Zigbee devices listed (125 used) and some errors in the log. I am in the proces of setting up a new Pi4 with external 2.5" SSD with and installing Mosquito, Zigbee2MQTT and Domoticz on/with Docker, managed by Portainer (not successful until now)

I am looking for a "quick fix" by resetting the logfile's once a week to prevent the memory to run full.

Has anyone an idea on how to clean up/ free up the reserved Log2Ram memory?

Kedi · Post by **Kedi** » Thursday 07 March 2024 11:15

Do a

Code: Select all

ls -al /var/log/

and look for what is taking up all the space, and take action on that file(s) by logrotate.
And the same of course for any subdirectory.

njlammerts · Post by **njlammerts** » Thursday 07 March 2024 12:20

Below the result

: Schermafbeelding 2024-03-07 om 12.13.48.png (166.07 KiB) Viewed 716 times

: Schermafbeelding 2024-03-07 om 12.19.13.png (152.95 KiB) Viewed 716 times

Kedi · Post by **Kedi** » Thursday 07 March 2024 15:41

So 'daemon.log' and 'syslog' are too big.
It looks like you have logrotate installed, check with

Code: Select all

sudo systemctl status logrotate.service
sudo systemctl status logrotate.timer

Both should be enabled.
Also look in the dir '/var/log/journal' there might be some big files too.
In '/etc/logrotate.d' there should be a file called 'rsyslog' or 'syslog'
This is my file:

Code: Select all

/var/log/syslog
/var/log/auth.log
{
    rotate 1
    daily
    missingok
    notifempty
    compress
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/msmtp.log
/var/log/daemon.log
/var/log/kern.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
    rotate 1
    weekly
    missingok
    notifempty
    compress
    sharedscripts
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Most important parts are 'rotate 1' and 'daily / weekly'
With that you control how many rotated files are created and if it is done daily or weekly.

Kedi · Post by **Kedi** » Thursday 07 March 2024 16:33

If the files in the 'journal' directory are too big you can change the journal settings in the file '/etc/systemd/journald.conf
Change these settings by editing and removing the '#' as an example:
SystemMaxUse=100M
SystemMaxFileSize=100M
MaxRetentionSec=1week

After the conf change restart the journal service with.

Code: Select all

sudo service systemd-journald restart

njlammerts · Post by **njlammerts** » Thursday 07 March 2024 16:55

hereby the results of commands, including var/log/journal result

: Schermafbeelding 2024-03-07 om 16.53.29.png (171.77 KiB) Viewed 694 times

content of /etc/logrotate.d/rsyslog

/var/log/syslog
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
/usr/lib/rsyslog/rsyslog-rotate
endscript
}

Kedi · Post by **Kedi** » Thursday 07 March 2024 17:00

You missed output from

Code: Select all

sudo systemctl status logrotate.timer

And to see the size of files you should do

Code: Select all

ls -al /var/log/journal

Kedi · Post by **Kedi** » Thursday 07 March 2024 17:02

change rotate from 4 to 1 and weekly to daily or use my '/etc/logrotate.d/rsyslog'

njlammerts · Post by **njlammerts** » Thursday 07 March 2024 20:13

i Changend 4 to 1 and weekly to daily

requested output:

: Schermafbeelding 2024-03-07 om 20.12.03.png (189.92 KiB) Viewed 672 times

Kedi · Post by **Kedi** » Thursday 07 March 2024 20:41

Logrotate.timer is enabled, so logrotate will work.
Now you have to wait until logrotate is run, normally that will be somewhere after midnight.
Your content of /var/log/journal looks good,but depending what is in the dir f95f.........
You could now manually delete those big files and reboot, just to be sure that the /var/log will not reach full before logrotate runs.

njlammerts · Post by **njlammerts** » Thursday 07 March 2024 21:00

in the f95f......... are the following files

system.journal
user-1000.journal

Kedi · Post by **Kedi** » Friday 08 March 2024 9:24

Again it is about how big they are (the '-al' does that).
You could use the settings I suggested.

SystemMaxUse=100M
SystemMaxFileSize=100M
MaxRetentionSec=1week

Did the logrotate help?

njlammerts · Post by **njlammerts** » Friday 08 March 2024 9:42

Kedi · Post by **Kedi** » Friday 08 March 2024 11:45

Again just a 'ls' command does not show file size, only a 'ls -al' command shows the filesize and date.
Put long data in a spoiler tag.
Did you change the changes I proposed in the '/etc/systemd/journald.conf' file?
and do a

Code: Select all

 sudo journalctl --disk-usage

Then you can see how much space is used.

Before a reboot you could also clean dir: /var/hdd.log/journal because otherwise that dir is copied to /var/log/journal at every reboot.

njlammerts · Post by **njlammerts** » Friday 08 March 2024 13:06

started this morning with 50% in use and now, 4h later it has 74% in use

pi@Pi245:/var/log/journal/f95f24a098d047ecb6f4402309e1d1e8 $ ls -al

total 32768
drwxr-sr-x 2 root systemd-journal 80 Mar 8 09:43 .
drwxr-sr-x 3 root systemd-journal 60 Feb 23 14:17 ..
-rw-r----- 1 root systemd-journal 8388608 Mar 8 12:17 system.journal
-rw-r-----+ 1 root systemd-journal 25165824 Mar 8 13:00 user-1000.journal

also cleaned /var/hdd.log/journal/f95f24a098d047ecb6f4402309e1d1e8

below (had already changed it) setting of /etc/systemd/journald.conf

This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
SystemMaxUse=100M
#SystemKeepFree=
SystemMaxFileSize=100M
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
MaxRetentionSec=1week
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
#ReadKMsg=yes
#Audit=no

Kedi · Post by **Kedi** » Friday 08 March 2024 15:18

Find which file(s) grow the fastest.
See which are currently the biggest by

Code: Select all

sudo du -h /var/log

It can't be that difficult to find the file(s) that grow fast and should be corrected by some sort.
Did the logrotate run and are the files daemon.log and syslog now small?

njlammerts · Post by **njlammerts** » Friday 08 March 2024 15:46

This was at +- 13:00h. After ls -la I did a rm *.* in this directory, so it was empty
logrotatie should run this night, I check tomorrow

pi@Pi245:/var/hdd.log/journal/f95f24a098d047ecb6f4402309e1d1e8 $ ls -la
total 32820
drwxr-sr-x+ 2 root systemd-journal 45056 Mar 8 09:43 .
drwxr-sr-x+ 3 root systemd-journal 4096 Feb 23 14:17 ..
-rw-r-----+ 1 root systemd-journal 8388608 Mar 8 13:09 system.journal
-rw-r-----+ 1 root systemd-journal 25165824 Mar 8 13:09 user-1000.journal

below at 15:45h

pi@Pi245:/var/log/journal/f95f24a098d047ecb6f4402309e1d1e8 $ ls -la
total 57344
drwxr-sr-x+ 2 root systemd-journal 120 Mar 8 13:09 .
drwxr-sr-x+ 3 root systemd-journal 60 Feb 23 14:17 ..
-rw-r----- 1 root systemd-journal 8388608 Mar 8 13:09 [email protected]~
-rw-r-----+ 1 root systemd-journal 8388608 Mar 8 15:17 system.journal
-rw-r----- 1 root systemd-journal 25165824 Mar 8 13:09 [email protected]~
-rw-r-----+ 1 root systemd-journal 16777216 Mar 8 15:38 user-1000.journal

deamon and syslog are not present. it seems files in f95f..... are the ones growing now

pi@Pi245:/var/log $ ls -la
total 472
drwxr-xr-x 7 root root 240 Mar 8 13:37 .
drwxr-xr-x 12 root root 4096 May 28 2023 ..
drwxr-xr-x 2 root root 220 Feb 23 14:17 apt
-rw-rw---- 1 root utmp 0 Mar 8 09:36 btmp
-rw-r----- 1 root adm 947 Mar 8 13:09 debug
-rw-r--r-- 1 root root 2664 May 28 2023 faillog
drwxr-sr-x+ 3 root systemd-journal 60 Feb 23 14:17 journal
-rw-rw-r-- 1 root utmp 292292 Mar 8 13:10 lastlog
drwxr-xr-x 2 mosquitto root 60 Mar 4 13:41 mosquitto
drwx------ 2 root root 40 May 3 2023 private
drwxr-xr-x 3 root root 60 Feb 23 14:17 runit
-rw-rw-r-- 1 root utmp 173952 Mar 8 13:10 wtmp

jvdz · Post by **jvdz** » Friday 08 March 2024 15:57

Here is a nice webpage that tells you what can be done with the journals. did you perform the command suggested by:

Kedi wrote: ↑Friday 08 March 2024 11:45 Did you change the changes I proposed in the '/etc/systemd/journald.conf' file?
and do a
CODE: SELECT ALL

sudo journalctl --disk-usage
Then you can see how much space is used.

anyways, look at the linked page and do some checking or even set a max for the journals.

njlammerts · Post by **njlammerts** » Saturday 09 March 2024 12:43

sudo nano /etc/systemd/journald.conf

# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=10000
SystemMaxUse=100M
#SystemKeepFree=
SystemMaxFileSize=100M
#SystemMaxFiles=100
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
MaxRetentionSec=1week
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K
#ReadKMsg=yes
#Audit=no

pi@Pi245:~ $ sudo journalctl --disk-usage
Archived and active journals take up 96.0M in the file system.

pi@Pi245:~ $ journalctl --verify
2c9008: Data object references invalid entry at 35207f8
File corruption detected at /var/log/journal/f95f24a098d047ecb6f4402309e1d1e8/user-1000.journal:3520560 (of 58720256 bytes, 94%).
FAIL: /var/log/journal/f95f24a098d047ecb6f4402309e1d1e8/user-1000.journal (Bad message)
PASS: /var/log/journal/f95f24a098d047ecb6f4402309e1d1e8/system.journal
pi@Pi245:~ $

Kedi · Post by **Kedi** » Saturday 09 March 2024 13:51

Do a

Code: Select all

sudo systemctl stop systemd-journald.service

Remove the journal files (don't remove the journal dir),
Perhaps lower the 100M in the /etc/systemd/journald.conf to 50M and do

Code: Select all

sudo systemctl start systemd-journald.service

Did the logrotate do its job?

Log2Ram uses max amount assigned memory

Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Re: Log2Ram uses max amount assigned memory

Who is online