Page 1 of 1

What do you do about bit rot? [MS Windows Users]

Posted: September 8th, 2020, 6:16 pm
by Julian
"Bit rot" as in bits getting silently flipped over the years and decades thus causing files to become corrupt without the user knowing.

I'm interested in whether people (home users) do worry about that as a problem and if so what do they do to protect themselves against it? Is it really an issue at all?

- Julian

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 8th, 2020, 7:05 pm
by PrincessB
(home users)


Photographs aside, everything important I've ever written or stored would fit onto a smallish memory stick.

I also upgrade on a relatively frequent basis, so I have less concern about a hard drive than I do with the operating system throwing an bigger spanner in the works than a few letters changed in a word document.

For those more concerned, perhaps a server grade processor with associated motherboard using ECC memory would be a start.

Keeping all data on a very redundant RAID array should find flipped bits and correct them as a normal part of its function. It's quite interesting seeing how many drives you might need. Wikipedia has a decent explanation but three drives is a good starting point.

You'll also need a UPS in case the power goes mid write.

Personally, I think half of the problems are software and the other half from shonky builds using sub-standard components (especially the power supply) or just old equipment. Add in dust build up and some machines (mainly laptops) designed to make cleaning the cooling system a task for an expert and you're just asking for trouble.

As a final point, even with a full sized PC, the cooling solutions are usually so poorly designed they are asking for trouble. The venerable PC case was designed for components that drew a few 10s of watts, not the space heater components that you can buy these days. Some of the Intel chips draw 250W under full load and Nvidia's (about to be be released) top end graphics card need 3x8pin power connectors to supply the 350W of power that it requires.

Many are using water cooling in an attempt to move all of that heat and most of these folk seem to forget that watercooling systems tend to gunk up over time and if they don't keep the rest of the motherboard cool they will just run in to problems by cooking the memory and hard drives.

B.

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 8th, 2020, 7:32 pm
by Infrasonic
Data

https://www.howtogeek.com/356473/how-to ... y-forever/
You may very well be the king of backing up your data regularly, but archiving your data is a completely different ball game. Here’s how you can archive your digital files and keep them around for generations to come.


https://www.youtube.com/watch?v=pekgrP-v5O0&t=151s
Explaining Computers
M-Disc is a more robust form of writable DVD and Blu-Ray media for making longer-lasting archives and backups. Here I look at M-Disc hardware, test compatitbility, and perform an intensive data retention test to see if M-Discs really do last longer than traditional optical media.

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 8th, 2020, 10:33 pm
by Urbandreamer
I don't worry about it.

That said, there are many simple solutions if you feel strongly enough to put in even minimal effort.

Over the years I have lost quite a lot of stuff, but never through bit rot. I've lost e-books, because the early provider went bust. Mp3's, because I didn't take care and sync'ed the directory wrongly. I've had hard disk crashes (yes the head impacted the media). In the 80's I accidentally typed del * .bak instead of del *.bak and deleated my entire source code, before version control! Man, that was horid. We managed to reconstruct most of it using Norton tools.

Bit rot does exist. However it honestly isn't something I feel a need to worry about. I regard hardware faults as more likely. I do run automatic backups of the stuff that I'd like to survive such an event. This DOES NOT avoid bit rot. That is unless you keep historic backups. In simple terms a cosmic ray could flip a bit and then you overwrite the backup with the faulty file.

BTW, why do you ask in terms of MS Windows? Bit rot, as you describe it, isn't really an OS issue. Though I do recognise that the term has been used in the past related to Windows needing reinstalling. Often due to out of date and no longer required DLL's etc using resources.

I fear that I don't have a high opinion of that OS, but let's not blame it for the statistical risk of a cosmic ray corrupting data.

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 9th, 2020, 4:21 pm
by Julian
Urbandreamer wrote:BTW, why do you ask in terms of MS Windows? Bit rot, as you describe it, isn't really an OS issue.

Because if anyone suggested any software solutions, such as "I don't worry but I occasionally run <some-file-check-command> to check the integrity of my data" I wanted people to know that I am running Windows (10) hence if <some-file-check-command> happened to be a Mac or Linux command then it is not going to be applicable to my setup.

I am currently using dual ReFS formatted drives in a mirrored Storage Spaces pool as my D: (user data) drive but after the last Windows 10 update Storage Spaces seems to be having a brainstorm and is showing my pool as 100% full whereas the Explorer "This PC" view shows my D: drive as only about half full which is what it really is. I spent a few hours yesterday trying to find a solution to the Storage Spaces issue but couldn't find anything and I'm now wondering whether I should adopt a "Keep It Simple Stupid" strategy and simply migrate my user data to a single NTFS formatted drive on which I run Chkdsk every now and then. If Chkdsk finds any errors I have 2 versioned backups (Crashplan and Carbonite) from which I could extract version 1 of any corrupted file which takes me as far back as 2012.

I do quite like knowing that ReFS is bit scrubbing in the background when idle but I suppose if I'm honest why do I need to find out immediately if an old file gets corrupted by a bit flip? I only really care if for some reason I need to look at an electricity bill from 1998 (or some other very old file) at which point I presume that when Windows tries to read a file containing a bit-flip the hard drive controller will return a CRC error(*) that will cause Windows to display a cannot-read-file error message and at that point I can retrieve the oldest version I have from either Crashplan or Carbonite which hopefully will be pre and bit-flips. I might additionally back up to some HDs as the links Infrasonic posted suggest.

Maybe the KISS solution is best, simply move all my data onto a single NTFS drive/partition and run chkdsk on it every now and then although with about 350GB of data that might take a while. I can always experiment and maybe set something up to do a chkdsk once a month overnight depending on how long it does take but I wanted to see what other Windows users do before changing my setup.

(*) as long as the corruption isn't such that the corrupted data still generates the same CRC, or both the data and the CRC are corrupted such that they are still in sync with each other

- Julian

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 9th, 2020, 4:37 pm
by Infrasonic
ReFS...https://arstechnica.com/gadgets/2017/08 ... new%20ones.

In the Fall Creators Update, Microsoft is removing the ability to create volumes using its new ReFS file system from Windows 10 Pro. Existing volumes will continue to work, but Pro will no longer be able to create new ones.


That's from 2017.
I'd feel a bit nervous about still using it if I had issues and there's no support for the particular Windows flavour I'm on. (I'm presuming you're not on W10 Pro for Workstations.)

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 9th, 2020, 5:28 pm
by Julian
Infrasonic wrote:ReFS...https://arstechnica.com/gadgets/2017/08 ... new%20ones.

In the Fall Creators Update, Microsoft is removing the ability to create volumes using its new ReFS file system from Windows 10 Pro. Existing volumes will continue to work, but Pro will no longer be able to create new ones.


That's from 2017.
I'd feel a bit nervous about still using it if I had issues and there's no support for the particular Windows flavour I'm on. (I'm presuming you're not on W10 Pro for Workstations.)

Thanks for the warning but I am on Win Pro for Workstations specifically because Microsoft annoyingly discontinued bundling it with regular Windows Pro. I first adopted ReFS pre 2017 when it was bundled with regular Windows Pro but when I came to upgrade in late 2018 I discovered the info you pointed out above. I could still have moved my data drives from my old machine as-is since Windows Pro can read ReFS partitions, it just can't create them, but I thought it best to play safe so I bought a Windows Pro for Workstations upgrade. I most definitely agree with that Arstechnica article though. Pushing ReFS ever further up the price points as opposed to making it a mainstream option for at least Windows Pro users (as it was before) is disappointing.

- Julian

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 9th, 2020, 5:59 pm
by Infrasonic
I wonder if they did it to make things easier for MS?
Pro for WS, server, enterprise etc. would generally have technically aware users or a sysadmin to sort issues.

Maybe they were getting too many support issues from W10 Pro users without the technical know how to resolve ReFS/pools issues?
You can get W10 pro machines from PC World, I can't imagine many of those buyers are going to be techie whizz kids.

Re: What do you do about bit rot? [MS Windows Users]

Posted: September 9th, 2020, 9:15 pm
by xeny
Julian wrote:I'm interested in whether people (home users) do worry about that as a problem and if so what do they do to protect themselves against it? Is it really an issue at all?



My day job is pretty data heavy, and I've encountered files developing corruption there, on "proper" hardware, but we're well over a PB of data at this point, so at some point the odds catch up with you.

At home I keep a master copy on a NAS and another pair of copies on two separate external drives, which are updated alternatingly.

Whenever I'm updating one of the copies, I diff the entire directory tree with Winmerge which will do a bitwise recursive comparison (between three directory trees if you're that way inclined). Comparisons take some time, but that is the computer's time, which is cheap.