New uses for Par

[mkruer]

Background: With the current size of drives, things like bitrot and/or silent data corruption are becoming more problematic, right now SATA hard drives state the error rate to be one uncorrected bit read error in every 1014 bits or 1 unrecoverable error for each 12.5TB written, and we are now seeing 12+ TB prosumer drives.

Enter HomeNAS: With NAS operating systems like FreeNAS and UNraid, we are seeing prosumer users save more and more data for long term archival and use as a media server and fileserver. Both these NAS operating systems suggest using two parity drives (FreeNAS, RaidZ2 or UNraid P+Q parity). to prevent data loss (one drive) and data corruption (2 drives) Giving up 2 drive makes a lot of sense for larger arrays as with each add disk it reduces the overall cost for redundancy

It is with smaller arrays say 4-8 drives that I think Parchive can find a place to grow. With today’s hardware it is very rare to have a catastrophic drive failure most drives will experience data degradation long before compete failure. Parchive was designed to recover missing data with assumption that most of the data was good and you only need recovery blocks for the portion of the data that was bad.

With the NAS operating systems, I could see a plugin developed where a user would be able to set a recovery percentage of something like 5-20% of the drive and then have Par3/4 scan the file system and create a recovery sets in the background. I don’t think you want to create a recover set across the entire systems, but one per directory might make sense.

Just a thought, I am glad to see Par being developed again. I still have DVDs that I used par to fill up the slack space and be able to recover the data even though the DVD now have errors on them. This is one of the reasons why I am ironically moving back to a NAS.

[zvezdochiot]

@mkruer say:

I still have DVDs that I used par to fill up the slack space and be able to recover the data even though the DVD now have errors on them. This is one of the reasons why I am ironically moving back to a NAS.

https://github.com/onechip/cdrparity

See also:

• https://github.com/fridex/rscode-correction
• https://github.com/FS-make-simple/hamming-coder

[mkruer]

@zvezdochiot Not quite sure what is being suggested. I used QuickPar to create the parity in the slack space of DVD in the past. I was just suggesting that as more people start moving to HDD for long term online/nearline storage/backup and as part of storage/backup server and that most of those setups probably will not be setup with n+2 parity to correct against bitrot, silent errors or localized sector failures, Par3/4 might find a "sweetspot" by addressing n+1 + Par.

[brenthuisman]

@mkruer

With the NAS operating systems, I could see a plugin developed where a user would be able to set a recovery percentage of something like 5-20% of the drive and then have Par3/4 scan the file system and create a recovery sets in the background. I don’t think you want to create a recover set across the entire systems, but one per directory might make sense.

This is roughly the idea behind https://github.com/brenthuisman/par2deep (a par2 per file, to enable a living tree).

What I'd really like is par2 integrated into filesystems (ZFS, btrfs). It's more space efficient compared to setting ncopies/redundancy to 2.

[mdnahas]

Thanks for the comments.

I'm working on a design for Par3 and I've had the same thoughts. Basically, filling some of the free space on a disk with redundant data for a snapshot of some files on the drive. Then admins can just run a process every night to update the redundant data and, if a file gets damaged, the admin can regenerate the file.

It's a difficult usage to make work. We'll see. I'm also trying to make sure you can put redundant data inside a file (e.g., a file that works as a ZIP file and contains data to repair itself!). And other usages.

[mkruer]

@brenthuisman

In a perfect world I agree, the real bottleneck I fear would be how PAR works. If RAID5 is a vertical stripe across multiple drives, and RAID6 would be a diagonal stripe, PAR is more of a horizontal stripe. (bad analogy.) Ideally, I would say we would want to have the whole drive PARed, in blocks, and the PAR would be used to create the redundancy. However, with this approach, beyond the matrices calculations the other limiting factor would be the requirement to read across the whole drive and here is lies the real challenge.

Backing up on a per file or per directory seems to be a decent compromise.

[desertwitch]

Not sure if of interest to you (or anyone) still, but I've turned (part of) these ideas into a standalone program depending on par2cmdline (which is used for PAR2 under the hood): https://github.com/desertwitch/par2cron

Mostly to cater to users of the Unraid operating system (where this exists as a plugin now). 😉
Thank you to the par2cmdline developers for keeping the PAR2 spirit alive... it's of great use!