problem with crash SD card

[reza]

hi guys.
first sorry for weak in english.
how resolve the problem of sd crash?
i won't buy a ssd hard (external) because this is Costly and It takes up a lot of space near raspberrypi .

before, i used version 3.5877 and Moving Log and Temporary Files To a RAM Drive so my problem is resolve . but now this source version (3.5877) can not compile on new raspberrypi OS or raspberry4 or 5. i have error about cmake version or libboost version and i can not compile version 3.5877 on new OS.
also new version of domoticz can not support "Moving Log and Temporary Files To a RAM Drive" so again i have problem with crash SD card.
is there any sulution for my problem?
is there a way that compile version 3.5877 on new OS ? or compile new version on new OS with Moving Log and Temporary Files To a RAM Drive ? or delete Log and Temporary files from source in general ( i dont need log)

thanks .

[waltervl]

Good luck trying to build a system from 8(!) years ago. Perhaps just update the system to latest Domoticz?

[reza]

Good luck trying to build a system from 8(!) years ago. Perhaps just update the system to latest Domoticz?

i want just resolve my problem with crash SD. in rpi4 or 5 i can not "Moving Log and Temporary Files To a RAM Drive" and i have crash sd

[waltervl]

in rpi4 or 5 i can not "Moving Log and Temporary Files To a RAM Drive"

Why not?

[reza]

in rpi4 or 5 i can not "Moving Log and Temporary Files To a RAM Drive"

Why not?

because in wiki told :
/home/pi/domoticz/domoticz.db-wal !! This will no longer work for domoticz version V4.11753 and newer
/home/pi/domoticz/domoticz.db-shm !! due to an update of the internal sqlite version

this means i can not move log and Temporary Files To a RAM Drive for domoticz version V4.11753 and newer, is this true?

[waltervl]

Indeed the db-wal ans shm cannot be moved to a RAM, but that does not mean that you cannot create a RAM drive and put the rest of the Domoticz temp files in RAM.

You could also invest some time in implementing a Domoticz backup solution so when the SD fails you have an easy method to revert: https://www.domoticz.com/wiki/Daily_bac ... ternal_hdd

Or invest some money in an M2 adapter and M2 SSD: https://www.sossolutions.nl/pcie-to-m-2 ... berry-pi-5
They are not that big anymore and also not that costly anymore.

[reza]

Indeed the db-wal ans shm cannot be moved to a RAM, but that does not mean that you cannot create a RAM drive and put the rest of the Domoticz temp files in RAM.

so when i cannot move db-wal and shm so i have SD crash soon even if i had a RAM drive? isthis true?
why there is not any way for move log and temprary to RAM drive in new version ? this was a excellent solution for crash SD.

[Kedi]

Or you use an industrial SD-card, they last for years. Sandisk / Western Digital have those.
B.t.w. same price as an small SSD-drive.

Or you put the complete domoticz directory on a ramdrive and rsync that domoticz directory to an USB-stick every XX minutes.
rsnapshot is a perfect program for that.
I use rsnapshot on every Raspberry Pi I have.

[reza]

Or you use an industrial SD-card, they last for years. Sandisk / Western Digital have those.
B.t.w. same price as an small SSD-drive.

Or you put the complete domoticz directory on a ramdrive and rsync that domoticz directory to an USB-stick every XX minutes.
rsnapshot is a perfect program for that.
I use rsnapshot on every Raspberry Pi I have.

how do it?
thank you

[lost]

Or you use an industrial SD-card, they last for years. Sandisk / Western Digital have those.

I used some in the past (maybe from Transcend? Can't for sure remember) and they only last a bit longer.
Something else made a difference: A1/A2 (better IO/s for A2) labelled uSD that targets storage additions for smartphone/tablet use case.

This is also 24/7 usage compared to traditional memory card use-case (swich on a camera, take a few photos/videos that'll trigger 99% sequential access, switch-off): Those are better for random access & being always on allows some background processing needed by static wear levelling. So you're no more on the heavy wear hot-spot creation induced by former SD management resulting mostly from non sequential access with dynamic wear levelling combination. Also, internal erase counters (flash sector erase is the operation that induce wear) used by storage cards controllers FW are manages in controller ram so they can only be managed over an uptime (lost when off), so for almost always off former use case, really nothing could be done to efficiently handle the problem.

=> To sum-up, Ax SD cards internal FW management is closer to a real SSD (still lack discard) and you're still on consumer line product prices.

Also, even without moving logs/DB to some tmpfs in ram (+with a PI3 you have only 1GB & you have to keep some for system run!), Linux sysfs provides some knobs to tune VMM (virtual memory management)/OS caching/write merges/commit times that can be used, especially if the system runs with a backup battery (as caching more means more to loose in case of power fail, can be up to several minutes instead of typically 5s with defaults settings).

They can be tested directly writing to sysfs & made done at reboot automatically by adding settings to /etc/sysctl.conf

For instance I have these additions:

Code: Select all

vm.dirty_ratio=60
vm.dirty_expire_centisecs=60000

See https://docs.kernel.org/admin-guide/sysctl/vm.html for all available "knobs", searching over the ones that makes sense will bring lots of advice/use-cases on the net.

[reza]

Or you use an industrial SD-card, they last for years. Sandisk / Western Digital have those.

See https://docs.kernel.org/admin-guide/sysctl/vm.html for all available "knobs", searching over the ones that makes sense will bring lots of advice/use-cases on the net.

thank you.
What is the approximate lifespan of a 300 GB SSD hard drive for domoticz?

[mgugu]

Why not using a complete read-only filesystem ? This works fine on latest OS. Check https://learn.adafruit.com/read-only-ra ... i/overview.
The change is simply done and reverted with raspi-config.
I have tested on a Pi2 bookworm with domoticz, nodejs and nodered (not dockered). This works fine but every change done in apps will not of course survive to a reboot, so the change has to be done at the very latest step of development.
Please note that docker cannot be used in read only because it already uses an overlay filesystem.

[EddyG]

This works fine but every change done in apps will not of course survive to a reboot, so the change has to be done at the very latest step of development.

Isn't there the problem?

I use these micro sd-cards https://www.westerndigital.com/products ... ED-032G-XI
Some of them are in use for more then 7 years on a Domoticz system with no crash so far.

I will post in a separate thread a solution for putting the complete Domoticz directory in a Ramdrive and still can reboot the system without missing information in Domoticz from sensors, of course excluded by the time the reboot takes place. It also survives crashes.
May take some days before I can post it.

[EddyG]

What is the approximate lifespan of a 300 GB SSD hard drive for domoticz?

I use 2 pieces of 250Gb SSD in raid on my main Domoticz system for approx. 4-5 years now, without any failure.
Those are used very intense with Raspberry pi4 b processor usage of 12-13% with peak writes to the SSD of 13 Gb per day

[reza]

Why not using a complete read-only filesystem ? This works fine on latest OS. Check https://learn.adafruit.com/read-only-ra ... i/overview.
The change is simply done and reverted with raspi-config.
I have tested on a Pi2 bookworm with domoticz, nodejs and nodered (not dockered). This works fine but every change done in apps will not of course survive to a reboot, so the change has to be done at the very latest step of development.
Please note that docker cannot be used in read only because it already uses an overlay filesystem.

thank you for answer.
do you tested it? because in this topic told "this read-only setup won’t always save you"
so if i do compile domoticz on my RPI and and build devices and events and etc... then change my RPI to read only , so my problem is resolve and my SD dont crash?
How many years does my SD card work safely?

[reza]

I will post in a separate thread a solution for putting the complete Domoticz directory in a Ramdrive and still can reboot the system without missing information in Domoticz from sensors, of course excluded by the time the reboot takes place. It also survives crashes.
May take some days before I can post it.

I look forward to your post impatiently. Please put it in the wiki and Inform here as well. If all the directories in new domoticz version can be placed in a RAM drive (as in the old versions), there will be no more problems with the SD card.

[mgugu]

thank you for answer.
do you tested it? because in this topic told "this read-only setup won’t always save you"
so if i do compile domoticz on my RPI and and build devices and events and etc... then change my RPI to read only , so my problem is resolve and my SD dont crash?
How many years does my SD card work safely?

Yes as written above, this function has been tested on off the shelves applications; domoticz, nodejs and nodered. Everything works perfectly. If you need some reboot survivance, you can imagine a daily backup (to NAS for example) of domoticz dB or nodered flows or …. Really simple things. You can also imagine an auto-restore from backup when pi rebooted.
Only updating applications will require RW, but the switching RO<-> RW take only the time for a reboot.

[gizmocuz]

As mentioned, you need at least a Class 10 SD card. Those should not give you troubles.
But you might also be thinking about making backups... One in every few months make a backup of your Domoticz folder to another machine (or USB disk)
7 years is a long time
You could also try to read this SD card via another system, if you are using Windows, try looking for a ext4 reader
Why would you need to store the log? As first user, I don't store the logs... Everything is also logged inside Domoticz itself (for example when you click on the log for a switch)
There are startup commands that you can use to store the log to the /tmp folder (-log file_path), or start Domoticz and disable the database wal mode (-dbase_disable_wal_mode)
Anyway, time to update to the latest Domoticz?

[reza]

As mentioned, you need at least a Class 10 SD card. Those should not give you troubles.
But you might also be thinking about making backups... One in every few months make a backup of your Domoticz folder to another machine (or USB disk)
7 years is a long time
You could also try to read this SD card via another system, if you are using Windows, try looking for a ext4 reader
Why would you need to store the log? As first user, I don't store the logs... Everything is also logged inside Domoticz itself (for example when you click on the log for a switch)
There are startup commands that you can use to store the log to the /tmp folder (-log file_path), or start Domoticz and disable the database wal mode (-dbase_disable_wal_mode)
Anyway, time to update to the latest Domoticz?

thank you for answer.
How to stop write database and log to prevent SD card crash? I don't need log information(SD card health is important to me). I dont know any things about the information of the database and whether it is necessary to write it into the SD card or not.Can you guide me further? i am beginner.
What commands should I enter so that after adding all my devices and events, nothing will be written on the SD card and the lifespan of the SD card will increase?
I don't even need to update. I have a series of equipment at home that are fixed and unchanged, and I only need to control them through the phone. some relais , some sensors , some events. I need a momentary command of the sensor to relay without save in log . run events and change it some time .

[gizmocuz]

By default, Domoticz does not store its log in the database. It does store history of switches, temperature etc etc
This is not much information and not many writes.
In Domoticz->Settings->Log set the log to a minimum of 1 day
And again, use a classs 10 SD card and you should not have issues.
Also, when using a raspberry pi version 2 or 1, this was causing a lot of SD card corruption. This was duo it's design.
You mentioned you have a 4 or 5, you should have no problems.

You can also boot the raspberry pi from network or use "iscsi boot", or you mount a NFS volume and run domoticz from this folder... there are endless options