truecrypt - Avoid or detect silent data corruption - Good practices? BtrFS?

Question

1) The goal

I want to preserve the integrity of the data on my storage media, I am currently installing from scratch my personal computers and my new home server (for self-hosted services) so I think this is the right time to ask the right questions to go with the right systems and practices currently in use in 2021.

2) My habits

This reflection is based on an observation:

• I scan all my paper documents to keep them for a lifetime
• I keep all the documents I already receive in digital format for a lifetime
• I keep all souvenir digital photos / videos for a lifetime

I have backup drives (I store at least one in a different location).

As I was initially a Windows user, I had used the FAT32 and NTFS file system, but for 5 years now I have completely switched to linux first with the ext3 or ext4 file system and very recently, I switched to btrfs (for time-shifting and COW functionality). I initially stored the data without encrypting it but soon used truecrypt (7.1a) file containers.

I have been doing this for over 10 years. But unfortunately, sometimes when I have to take an old document or a photo, I find that it got corrupted silently. For example, the start of the image is good, the rest appears in gray.

Many origins can be the cause, hardware failure, power failure or cut, software failure, cosmic particles ... Fortunately I have never had an encryption container failure.

My goal is not to avoid these corruption 100%, but to detect them early enough to retrieve an uncorrupted version before erasing / replacing the backup.

3) Questions

• I have read that the btrfs file system is capable of detecting data corruption. How does it works:
  • I don't want to manage a raid or another system with multiple drives, is this still possible?
  • Do we need software to detect corruption or is it fully integrated with the file system itself?
  • Will corruption be detected on the fly or do we need to make manual action?
  • How to be sure that the backup with good data will not be erased by a corrupted version of the same data?
  • Is it possible to automate recovery from the last backup when corruption is detected?
  • I am aware that corruption can occur before being stored on disk (using non-ECC ram for example). But in case of copying from one drive to another is it possible to ensure the integrity of the data of the destination without manually running a comparison tool?
  • How to avoid silent corruption on external backup drives?
  • I use SSD disks like internal mechanical disks or external USB disks, do they have to be configured differently?
• I want to store my data encrypted:
  • Can I still use trucrypt (7.1a) / veracrypt with file containers?
  • Is it safe or is there a better way?
  • Veracrypt does not offer a btrfs file system, there seems to be workarounds https://erikugel.wordpress.com/2014/02/25/truecrypt-btrfs/ https://unix.stackexchange.com/questions/621887/how-to-shrink-an-encrypted-btrfs-partition, is it sure?

Thanks for reading, sorry for my bad English skill.

question from:https://stackoverflow.com/questions/65864694/avoid-or-detect-silent-data-corruption-good-practices-btrfs

Answer 1

I have some partial answers for you, but I don't think this is the proper place to ask this kind of questions, so that's why I think you got no feedback so far. For instance I've been recently directed at https://unix.stackexchange.com for a BTRFS related question.