Let's break Synology btrfs

Linux Discussion Linux Servers
1

Hey everyone,

Within the next few hours I’ll be receiving a Synology DS1019+. This is one of Synology’s higher-end models, with five drive bays by default, support for adding more, and all the bells and whistles that you can get at the high-end of their consumer gear. I won’t tell you what I paid for it though. :stuck_out_tongue:

For the first couple weeks of the NAS’ life, I’ll be performing stress tests on the disks that I’ve bought to go with it. I won’t be able to use the NAS for its intended purpose while that’s going on.

After listening to Noah rap on btrfs and how it’s not production-ready, yet reading Synology’s documentation saying it is… I’m intrigued. Synology’s statement is “[…] it is known that Btrfs RAID is unstable and not suitable for production environments. For this reason, Synology chose Linux RAID over Btrfs RAID” (What is the RAID implementation for Btrfs File System on Synology NAS?).

From that, we can see that they’re using Linux’s Device Mapper RAID instead of btrfs native RAID. When btrfs detects a checksum error in a block, then, some Synology software must be used to correct that error!

I’d like to put this to the test.

So, using a few older 320GB drives, I’m going to stress out btrfs. I’ll try to write files to the array, then tamper with the original copy and read it back. Later I’ll try to simulate a disk that’s absolutely trashed – randomly flipping bits past its RAID header but in its data section – see if that causes any problems.

Do you have any other ideas for the tests that I could perform on the system? Ways that you think I could break the RAID?

2 Likes
2

I don’t have any feedback on the testing other than to say that it’s a cool idea. My DS412+ has been in service since August of 2014 with an array of four WD Red 2 TB NAS drives using a Synology Hybrid Raid volume with 1 drive fault tolerance for a total of 5.4 TB. It has been flawless in that time and continues to run beautifully. I have no idea how long it will keep going but I have more than gotten my money’s worth out of this system and will almost certainly buy another one when it’s time to replace it.

1 Like
3

I’ve been using BTRFS for 1.5 years now, and I’ve never had a problem with it. I used it in 2 home-made NAS builds. Note that I was not using RAID (each had 3 BTRFS disks, one being a backup of the other two). These NAS builds were not Synology or Qnap (Note: Qnap also trusts BTRFS). It was the advent of Synology and Qnap including and thereby vetting BTRFS in their products which made me think to myself "ahhh, now BTRFS must be finally production grade, at least for simpler RAIDs".

If you’d like to try to break BTRFS, here’s a gruelling stress test. Copy a bunch of tiny files (say, several GB of ebooks) into the NAS, and halfway through, simulate a power outage by holding down the power button and causing a hard shutdown.

Then after booting back up, read the logs (eg. Syslog). Look for filesystem errors. Maybe do a filesystem check of BTRFS with the RAID array offline. Are there any unrecoverable filesystem errors, such that a fresh reformatting is the only way to get back to a clean filesystem again?

Anyone who has any doubts about BTRFS, this is what I say to you: fine, don’t use RAID then. You still get journalling, very respectable performance, and filesystem snapshotting, which you can automate with snapper (or timeshift, a great GUI tool if you are on Linux Mint, and opted for BTRFS during the install).

I personally feel RAID is overrated, and the majority of users out there don’t really need it, just like the majority of us don’t need a turbo-charged car; a non-turbo-charged car is sufficient in most cases. I say get your backup story arrow straight (which is plenty challenging enough for most people) before you go messing around with RAID.

The linux kernel web page stating current status of BTRFS stability warns not to use RAID56 with BTRFS. RAID levels 0, 1, and 10 (much more commonly used) are said to be stable.

Phoronix also did some recent, nice RAID performance tests on BTRFS, pitting it against EXT4 and XFS (but not ZFS).

1 Like
4

I second Eric’s sentiment. I have a DS1512+ that I’ve also been using for years. At one point I even added the 5 bay expander so I have the potential capacity of 10 drives. I’m also using Synology Hybrid RAID and use 1 disk drive fault tolerance, and have had no problems with it, and that’s running a variety of hard drive’s in terms of size and brand. I backup multiple systems to the NAS, and even with multiple drives that have gone bad at different times, I’ve been able to replace those drives and rebuild the RAID.

1 Like
5

Well, I tried… but I didn’t break it. You can read more on my blog, I couldn’t break Synology SHR+btrfs (yet).

1 Like
6

I read your article, thanks for all that torture testing. The Synology fared really well, I would say. You’ve also left a good procedure for anyone else to follow who might want to torture test stock BTRFS RAID, on a non-Synology build.

7

There’s a fairly active Sub-Reddit on BTRFS here.

There’s also a Synology Sub-Reddit, with a thread called “BTRFS: Tell me your horror stories”.

8

Here’s a somewhat similar BTRFS torture test done, from August 2019. Compares with ZFS and mdadm as well.

And here’s a review of BTRFS which I felt was well presented, called “Five Years of BTRFS”, where he used BTRFS in preference over ZFS. Note: He shied away from BTRFS RAID 6, and uses BTRFS RAID 10 and BTRFS RAID 1 now.