High write access rate on SD Card

[YellowSky]

Hi everyone,
i wonder that Domoticz is the main process which writes on my filesystem (a SD card).
The results are found with iotop command
In half hour, it writes more than 50 000 Ko of data by several threads
- MainWorker
- MainWorkerRxMsg
- WebServer_8080

Is there a way to reduces this?
Currently, i use log2ram to store the log of domoticz.

Have you got any idea to reduce writing ?
Thank you.

[willemd]

https://www.domoticz.com/wiki/Moving_Lo ... _RAM_Drive

[YellowSky]

https://www.domoticz.com/wiki/Moving_Lo ... _RAM_Drive

It says "/home/pi/domoticz/domoticz.db-wal !! This will no longer work for domoticz version V4.11753 and newer" I use 2022 version.

[willemd]

Yes. I know and am not sure about what that means. See my question here:
viewtopic.php?p=292569#p292569
Never answered .....
I should have mentioned, sorry.


I still did implement it and it still seams to works. The only problem I found is when domoticz is shut down it does not properly clean up those files like it does it the default setup and then a restore becomes tricky.

Alternatively, you might want to look at this post:
viewtopic.php?p=292439#p292439

[lost]

Have you got any idea to reduce writing ?

In /etc/sysctl.conf, you may add/tune:

Code: Select all

vm.dirty_ratio=60
vm.dirty_expire_centisecs=120000

This'll allow much more write cache&merge at the expense of up to last 2mn data loss in case of sudden power fail. My PI is powered through a battery buffer so I avoid this risk.

Quick& easy, without even needing to put logs in a tmpfs...

[YellowSky]

Thank you for your answers.
For the etc/sysctl.conf: if i use log2ram is it usefull? This command requires a reboot?

For the moving of file of domoticz database
what does mean

a restore becomes tricky.

[willemd]

what does mean : a restore becomes tricky.

I mean I have not fully figured out the logic then. If I restore an older database while the wal and shm files still exists, it seems to roll forward to the new state anyway (but is it complete?). If I remvoe the wal and shm files first it seems to do a proper restore of the old state. But I am not totally sure. That's why I used the word "tricky'.

[YellowSky]

Ok, thank you.
I put the db in log2ram directory (var/log) and it works since two days even with reboot.
It remains to check when there is a power outage.

[lost]

It remains to check when there is a power outage.

You'll loose all logs since last sync to storage, so up to 24h with default sync periodicity.

That's why I stick to virtual memory management tuning (using sysfs entries for tuning & sysctl.conf when a good compromise is found after some tests) that are base Linux feature, thus system wide + easier to setup. Being a kernel feature, "controlled crash" scenario can be handled better (immediate data sync) than using an external tool.

Lots of tuning "knobs are available:
https://docs.kernel.org/admin-guide/sysctl/vm.html

Remains the power-fail but in my settings I loose 2mn max + this is unlikely as my system is powered through a LiPo battery buffer.

That's light setup, being 5Vdc, and lasting longer compared to a 220V based UPS with much more conversion/storage losses: 220Vac->Usually 12Vdc copper based (!!) battery->220Vac->220Vac/5Vdc-USB->PI, opposed to 220Vac->DC LiPo 5V(+12/19V usually, so can also back the internet/fiber router power)->PI. As a bonus, that's also cheaper (purchase+usage)

[YellowSky]

In my case, i've just bought an UPS with 5V output (10W) and everything works fine during the power outtage of my town this morning...
To be honest the tuning of the virtual memoryis not clear for me...
Moreover, due to large database (50 Mo) and frequent writing, i think (but i'am maybe wrong) that the ram disk is required, no?