Read only Raspberry Pi OS to prevent corruption Topic is solved

[lost]

A USB Stick is build to get read/write lot of times. SDCard is build to store

Both share the same MLC NAND flash chips and very similar controllers to use them as usual storage (i.e a machine to read/write LBAs).

=>Don't globally expect better results from most consumer grade USB keys than uSD.

The only way to get something reliable would be to use industrial grade eUSB. But they are designed to be plugged directly on a PCB connector, not a usual external USB one & Raspberry don't exhibits a USB port signals on it's IO pins.
Industrial grade uSD exists, but they are hard to find.

All theses industrial grade devices are provided with a choice of underlying NAND technologies:

SLC: The single level old but best for lifetime (usually 100k erase cycles supported) for write intensive loads. But that may be 100€ for 8 GB... So more that 10x cost/bit increase!
MLC: Consummer grade multi-level (i.e. a cell is able to discriminate several charge loads using comparators, thus able to store several bits/cell. If you can check 3 levels, you can store 4 states thus 2 bits/cell) usually given for 3k erase cycles only (multiple level comparison => less margin and as charge tend to leak more and more with erase cycle count...). These are for read intensive only tasks, only difference with consumer grade products is in the controller firmware: Usually better wear leveling algorithms + better power loss tolerance.
pSLC: Probably the best bargain, it's MLC chips (to benefit from consumer grade chip volume price) used in SLC mode. Given for 30k erase cycles with a cost/bit "only" doubled.

These underlying technologies makes a difference, not the form factor.

A reseller willing to sell such well chosen devices (for instance Swissbit S46u pSLC based series) for home automation users would IMO be welcome.

[toreandre]

I always see a lot of complaining about the Sd cards, but my guess is that its the hardware (RPI) or software (os/applications) that is to blame for the corruptions.

I use SD cards for a lot of different things at home and we use SD cards in a lot of our applications at work. The only time there is a lot of errors its usually software related bugs that cause this. I have never had a corrupt SD card before i got my RPi, now all my cards has errors.

It makes no sense that normal SD cards cant handle the small amount of data a domoticz install generates.

At home i have 4 RPis (2 mediacenters, 1 master domoticz and one slave), the bad sd cards seems to hit quite random and is not related to heavy use, huge amounts of data or heavy load, at one time my main domoticz install had been running fine 150 + days (only OSMC (debian jessie), domoticz and mqtt) but crashed after a controlled reboot.

[Egregius]

Seems that a Domoticz system has a lot of writes.
Since I'm running on Proxmox I can see how much data is written to the disk...

[toreandre]

Seems that a Domoticz system has a lot of writes.
Since I'm running on Proxmox I can see how much data is written to the disk...

Thats insane! Over how long timespan is this data written?

[Egregius]

About 40GB in 3 days 1 hour...

[toreandre]

About 40GB in 3 days 1 hour...

That is crazy amounts, if this is the case with standard domoticz installs its no wonder why the SD cards is breaking.
That data amount is the same as streaming SD (standard definition) quality video for the same amount of time.
And its aprox 500mb every hour.

Does anyone else have read/write statistics?
How can i check this on Debian Jessie?

[lost]

I always see a lot of complaining about the Sd cards, but my guess is that its the hardware (RPI) or software (os/applications) that is to blame for the corruptions.

I use SD cards for a lot of different things at home and we use SD cards in a lot of our applications at work. The only time there is a lot of errors its usually software related bugs that cause this. I have never had a corrupt SD card before i got my RPi, now all my cards has errors.

It makes no sense that normal SD cards cant handle the small amount of data a domoticz install generates.

At home i have 4 RPis (2 mediacenters, 1 master domoticz and one slave), the bad sd cards seems to hit quite random and is not related to heavy use, huge amounts of data or heavy load, at one time my main domoticz install had been running fine 150 + days (only OSMC (debian jessie), domoticz and mqtt) but crashed after a controlled reboot.

I also use this kind of devices at work and had to select some for telecom equipments: That's quite simple, every of them had problems.

-Some could be seen during selection tests (power fail resiliency, wear levelling algo that was supposed to be static but, in fact, dynamic thus generating "hot-spots" of quickly wearing areas because of a manufacturer FW config error...).

-Some were seen after months of use (but thankfully not already in the field): I have in mind a bad type "cast" (SCSI read/write_10 commands have starting LBA + consecutive LBAs nb to get as parameters, this last one being a 16 bit unsigned integer: u16... correctly used on OS/File-System side... but incorrectly retrieved as a signed s16 on storage FW side) inside a firmware, leading to problems with big files with few fragmentation. FS wanted to read a file fragment exceeding 16MB in one read_10 command for instance, allowed by SCSI command set on a device usung 512b LBA size that is supposed able to handle up to 32MB. Over 16MB LBAs nb was seen as negative (heavier bit seen as a sign bit) by the storage FW, hanging it immediately!
As it's not an heavy probability event to get such big file fragments, especially on an embedded system, this kind of problem is very difficult to spot.
Another tricky pb was a hot restart path that remained active (lack of interrupt masking) after a USB reset while it's working data was reset: Result was silent random corruptions of an internal page size (8kB on this device): You think there is something wrong with your file system at restart until one day, corruption affect start of storage wiping partition table. As it's a zone that is untouched after partitioning and only read (then cached) once at early boot steps during FS bring-up, you start to suspect some device FW problems...

-Others that may be due to the use-case: For instance, real-time Linux patches that allow applicative SW to interrupt the kernel. When it falls between command and data phases in the usb-storage subsystem, I've seen devices with some sort of timetout leading to a USB disconnect before a reboot (storage no more available).

Only last one may be considered an OS problem and not a 100% one. All others were device problems. And not really some consumer grade ones.

=> When you experiment storage problems, especially flash based ones much more complicated to manage, don't blame OS/FS etc... especially on a heavily used OS in the server market: Consider it's really reliable & all bugs are quickly spotted on a world of workloads. Most such problems ARE storage device problems, the kind bugs you may search for weeks, manage to reproduce with better probability than your workload designing specific tests to be able to put the manufacturer nose right in he's mess because they will always say there is no problem until you prove them there is!

Your last problem really looks like an heavy wearing one affecting device data only used at boot. Unused during half a year, you just didn't see the corruption (data retention time reduced due to wearing) until you needed the data. This highly probable on a consumer grade device that does not do static wear leveling but dynamic (only areas with data changes are levelled: Thus some static boot/OS data read once at boot then remaining in DDR during uptime are most subject to data loss), as it's more simple than periodivally reading device in the background to spot corruptions, with ECC & while still correctable, to be able to copy the data before aggravation up to be uncorrectable.

[lost]

About 40GB in 3 days 1 hour...

That is crazy amounts, if this is the case with standard domoticz installs its no wonder why the SD cards is breaking.
That data amount is the same as streaming SD (standard definition) quality video for the same amount of time.
And its aprox 500mb every hour.

Does anyone else have read/write statistics?
How can i check this on Debian Jessie?

If you have an ext4 FS, there is an interesting feature that can be monitored from it's sysfs:
https://www.kernel.org/doc/Documentatio ... fs-fs-ext4

See /sys/fs/ext4/<disk>/lifetime_write_kbytes
And /sys/fs/ext4/<disk>/session_write_kbytes

Respectfully: Data written since FS creation (i.e last format) & since last mount (thus since last boot for a fixed storage partition, especially a root one), in kBytes.

On my PI3 reinstalled (stretch) about month ago & upgraded/restarted 11 days ago, using SD (=> mmcblk):

Code: Select all

mount | grep mmc
/dev/mmcblk0p2 on / type ext4 (rw,noatime,commit=20,data=ordered)
/dev/mmcblk0p3 on /local type ext4 (rw,relatime,commit=20,data=ordered)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)

So, root partition (/) is mmcblk0p2, thus:

Code: Select all

cat /sys/fs/ext4/mmcblk0p2/*_write_kbytes 
48815773
21231256

=> 46GB written since last format (including raspbian stretch + domoticz install) & 20GB during last 11 days.

Mounts are using relatime, Domoticz log is active but on a tmpfs in ram (and a cron job does a "truncate -s0 /tmp/domoticz.txt" once a week as file is not logrotated nor can be erase because Domoticz continously handle it)... Well, could do the same on /var/log but I like to have log history after reboot to be able to figure out problems.

Raspbian install is also the small one (headless, thus no desktop env installed: Only accessed using ssh, the only level of "comfort" is having X11 frame buffer installed : xvfb. Thus allowing to install graphical editors like nedit & use them through ssh using X11 forwarding(+on the fly compression: ssh -XC <IP>).

That's about 1.8GB/day, with a few optimisations already done. Problem is that is the FS figures, due to write-amplification phenomenon (mostly due to the NAND sector-erase-only/bit-write nature of flash components) write figures on the media may be much more, especially for small file loads (logs appending...).

On a device not creating hot-spots thanks to good wear leveling algos & data retention monitoring, this may not be a problem. For a 32GB device, quite large for the use, that should be tens of years use. But on a consumer grade device this may not be true & a few hot-spots may reduce lifetime to 1 or 2 years.

[toreandre]

So i have this:

osmc@osmc:~$ cat /sys/fs/ext4/mmcblk0p2/*_write_kbytes
361108273
2895844

So over 300GB written in 150 days (last format)?

There must be something seriously wrong somewhere?

[toreandre]

I always see a lot of complaining about the Sd cards, but my guess is that its the hardware (RPI) or software (os/applications) that is to blame for the corruptions.

I use SD cards for a lot of different things at home and we use SD cards in a lot of our applications at work. The only time there is a lot of errors its usually software related bugs that cause this. I have never had a corrupt SD card before i got my RPi, now all my cards has errors.

It makes no sense that normal SD cards cant handle the small amount of data a domoticz install generates.

At home i have 4 RPis (2 mediacenters, 1 master domoticz and one slave), the bad sd cards seems to hit quite random and is not related to heavy use, huge amounts of data or heavy load, at one time my main domoticz install had been running fine 150 + days (only OSMC (debian jessie), domoticz and mqtt) but crashed after a controlled reboot.

I also use this kind of devices at work and had to select some for telecom equipments: That's quite simple, every of them had problems.

-Some could be seen during selection tests (power fail resiliency, wear levelling algo that was supposed to be static but, in fact, dynamic thus generating "hot-spots" of quickly wearing areas because of a manufacturer FW config error...).

-Some were seen after months of use (but thankfully not already in the field): I have in mind a bad type "cast" (SCSI read/write_10 commands have starting LBA + consecutive LBAs nb to get as parameters, this last one being a 16 bit unsigned integer: u16... correctly used on OS/File-System side... but incorrectly retrieved as a signed s16 on storage FW side) inside a firmware, leading to problems with big files with few fragmentation. FS wanted to read a file fragment exceeding 16MB in one read_10 command for instance, allowed by SCSI command set on a device usung 512b LBA size that is supposed able to handle up to 32MB. Over 16MB LBAs nb was seen as negative (heavier bit seen as a sign bit) by the storage FW, hanging it immediately!
As it's not an heavy probability event to get such big file fragments, especially on an embedded system, this kind of problem is very difficult to spot.
Another tricky pb was a hot restart path that remained active (lack of interrupt masking) after a USB reset while it's working data was reset: Result was silent random corruptions of an internal page size (8kB on this device): You think there is something wrong with your file system at restart until one day, corruption affect start of storage wiping partition table. As it's a zone that is untouched after partitioning and only read (then cached) once at early boot steps during FS bring-up, you start to suspect some device FW problems...

-Others that may be due to the use-case: For instance, real-time Linux patches that allow applicative SW to interrupt the kernel. When it falls between command and data phases in the usb-storage subsystem, I've seen devices with some sort of timetout leading to a USB disconnect before a reboot (storage no more available).

Only last one may be considered an OS problem and not a 100% one. All others were device problems. And not really some consumer grade ones.

=> When you experiment storage problems, especially flash based ones much more complicated to manage, don't blame OS/FS etc... especially on a heavily used OS in the server market: Consider it's really reliable & all bugs are quickly spotted on a world of workloads. Most such problems ARE storage device problems, the kind bugs you may search for weeks, manage to reproduce with better probability than your workload designing specific tests to be able to put the manufacturer nose right in he's mess because they will always say there is no problem until you prove them there is!

Your last problem really looks like an heavy wearing one affecting device data only used at boot. Unused during half a year, you just didn't see the corruption (data retention time reduced due to wearing) until you needed the data. This highly probable on a consumer grade device that does not do static wear leveling but dynamic (only areas with data changes are levelled: Thus some static boot/OS data read once at boot then remaining in DDR during uptime are most subject to data loss), as it's more simple than periodivally reading device in the background to spot corruptions, with ECC & while still correctable, to be able to copy the data before aggravation up to be uncorrectable.

I cant say that this is the case for our products, we have very low failure rate on the SD cards, when we do we always have meetings with the manufactor and almost everytime the problem is software or hardware (device not card) related. At home i have cameras that overwrite the sd card when it is full and two of the sd cards has to over 4 years old now.
I realy belive that this is an issue with bad firmware/software or hardware, in my experience the errors show up more frequently when you do a lot of read/write to the cards. If you leave the system alone you are less likely to experience errors.

[lost]

So i have this:

osmc@osmc:~$ cat /sys/fs/ext4/mmcblk0p2/*_write_kbytes
361108273
2895844

So over 300GB written in 150 days (last format)?

There must be something seriously wrong somewhere?

344GB/150, that's around 2.3GB/day, not so more than my own figures.
This should not be a problem with a correctly managed device.

Consider my figures, on a 32GB uSD. MLC NAND support 3k erase cycles. Considering it's FW flash translation layer induced write amplification, maybe only 1k from the FS that use this storage.

Even with the ability to fully erase (then write) 32GB 1000 times, this would be 31TBW (Tera-Byte Written) endurance. At 1.8GB/day, it's 48 years of use!

But only valid if the device is correctly managed. Otherwise, if only dynamic data are wear-leveled on a system that (after install) mostly write logs and DB data, only a small subset of the full size of the device will be concerned, maybe a few 100's of MB.

If WL only operates on, let's say 1GB, you get only around 1TBW endurance on these "hot-spot" areas... and at 1.8GB/day, that's only 500 days. Maybe less if you write a bit more & temperature is a bit high in the PI enclosure.

My first uSD failed just under a year of use... Had to restore saved image (then DB) 2 times in a month after hanging in IO errors. When I wanted to get last DB on this failing uSD to be able to restore on my new install/uSD, had to try mounting several times because even the partition table could not always be read.

Conclusion: Consumer grade NAND storage management, especially for removable devices, is crap! Because if it wasn't, these write figures would not lead to catastrophic failures in the year or so.

I bought a Kingston one that is supposed to be industrial, at least on the temp range. Not sure on the FW side.

Could not easily find some true industrial brands like Swissbit (S46u, pSLC or S45u, MLC but good FW) easily: Vendor that does not only sell to manufacturers, able to send quantities of 1 at an affordable price (including post/taxes) etc...

[Egregius]

See /sys/fs/ext4/<disk>/lifetime_write_kbytes

Thank you for that!
Writing some script to monitor that, dump the data in a sql database and see if I can make some improvements on my systems

[lost]

See /sys/fs/ext4/<disk>/lifetime_write_kbytes

Thank you for that!
Writing some script to monitor that, dump the data in a sql database and see if I can make some improvements on my systems

That's a nice feature of Ext4 to store these infos in the FS metadata. If you have ext3 but using the ext4 driver in compatibility mode, this also works.

At work I was logging differences using a cron triggered every 24h, so I had write amounts on a day per day basis: Very useful to be able to see the evolution (and compare SW versions, to be able to see quickly if someone start writing too much even sparsely during the day, which could reduce product life estimates).

A lot of ext sysfs tunables can also be used (can be set at boot from a RCs script for instance when a test configuration looks OK for you) to improve media behaviour & lifetime (especially on a flash based media where it's advised to merge writes as much as possible). But as it's some caching "knobs", that's always a "how much data am I ready to loose on a power fail" compromise!