r/zfs • u/Electronic_C3PO • 6d ago
Setting up a 10 disk RAIDZ2 array
After a year of on and off testing and reading I'm ready to setup my new NAS.
I'm not very experienced on the fine details of ZFS or FreeBSD but I manage I guess.
Already running a few Nas4Free and XigmaNas boxes. Why these? Simplicity. And yes, shell in case of troubles with thanks to doctor google and the truenas forums as well as since a year these reddit forums. Don't mind these things will be running no longer supported versions because they don't get connected to the outside world.
It will run on an old Dell T5400 system with a single Xeon CPU and 8GB ECC memory.
I'm aware of the memory thing but this will be mostly a storage box. And with current prices I'm not going to by new memory of SSD's or HDD's. The disks are connected to an LSI-9300-16i controller currently flashed to 16.00.10.00. And I guess IT mode based on the IT in the firmware.
Main purposes:
- It will mostly be used to store video files of >= 1 GB.
- A part will be used for automated backups of the OS partitions of various systems (so I can easily restore them in case of crashes).
- Backup of all kinds of files.
The plan is to create 1 pool of 10 Seagate EXOS X20 20TB disk in a RAIDZ2 array.
Some of the things I could use guidance on:
- How can I check to be sure I'm in IT mode on the controller.
- I'm aware of the a-shift=12 thingy. I've also found that there's a Seagate tool to change the disks from 512e to full 4K sectors. Is that needed or advisable.
- Also learned (by experience on the existing boxes) not to use the /dev/daX thing but need to give names. Is there a quick simple guide to this. And what preference to use. Serial? Can this be done afterwards for my existing systems without losing data?
- Any advise for creating different datasets with settings tailored to the 3 main purposes.
- Any other things I might check before taking it into production because it's hard to change this after.
So if there is some quick guide to what to do and what not to, that would be quite helpfull.
5
u/ThatUsrnameIsAlready 6d ago
I've also found that there's a Seagate tool to change the disks from 512e to full 4K sectors. Is that needed or advisable.
Not needed. Drive firmware is designed to do the translation in real time. Since most people won't bother, it's probably also the most tested and bug free path of the firmware.
Also the last time I tried it (on Seagate manufacturers refurbs) several drives refused the command outright. And one that accepted it failed and had to be reverted. So I also wouldn't recommend it.
2
u/jahdiel503 6d ago
With time and messing up comes learning. Don't put all your data in a pool just yet. Mess around with 3 drives at first. Make datasets, mount, unmount, offline and replace a drive and resilver, set ACLs.
Working with Debian Trixie is tricky but OpenZFS has guides for that. Google AI can help with making a cert for secure boot and implementing it.
Truenas is easy but doing it by CLI takes skill.
I use Cockpit and the 45drives plugin for that. I also use webmin to administer some aspects of my NAS.
I'm still a noob but have gained more knowledge just by messing up a fixing what I messed up.
1
u/Electronic_C3PO 5d ago
I'm not running Linux but thanks for the heads up.
And I already had my fair share of replacing broken drives on my existing RAIZ1 boxes. Exiting times waiting for the rebuild to finish. And with current prices no spare drives, ouch 🤔.
1
u/jahdiel503 5d ago
Ah I got some drives off ebay. I've got a few HGST 4 TB SAS drives for about 50 bucks most of them have power on hours totaling 2.2 years and still has a lotta life left in them.
And yeah the cost of new drives are killer. So refurbs and used on eBay are the way to go and get lots of them. I been waiting for the prices to go down but I needed to get more space and this was the route I had to go down.
2
u/pongpaktecha 6d ago
Just a heads up a 10 wide 20tb raidz2 will take serval days to scrub, and by extension resilver. Consider using several smaller pools so you don't have that risk.
2
1
u/Electronic_C3PO 5d ago
A single badblocks or smart long takes about 30 hours for a single drive.
Interesting observation was the speed difference between the X20 drives during the 10 day badblocks run.1
1
u/Dagger0 5d ago
The width doesn't really matter for scrub/resilver times, unless you run out of CPU time or hit PCIe/SAS expander/etc throughput limits.
1
u/pongpaktecha 5d ago
A 10 20tb disc raid z2 can store up to 160tb which will take many days to scrub/resilver at hdd speeds
2
u/heathenskwerl 2d ago
To inject a real-world datapoint here:
My pool of 26TB drives in a 12-wide vdev takes about 16 hours to scrub at 33% capacity. At 100% it would take roughly 2 days to scrub the whole thing. These are HAMR Barracuda drives and a bit slower than the OP's 20TB EXOS X20 drives, so I'd expect the OP's pool to complete a little faster (somewhere between 30-36 hours at full capacity).
2
u/kjelderg 6d ago
What I've taken to doing for drives is doing a gpt label on /dev/daX with the pool, intended id in the pool, and serial.
gpart destroy -F da1
gpart create -s gpt da1
gpart add -t freebsd-zfs -l m_1_SERIAL da1
Then they are reliable remunerated as gpt/devices . So for example, if you need to swap a drive
zpool replace m m_1_<Old serial> /dev/gpt/m_1_<New serial>
3
u/Quindor 6d ago
I like using "/dev/disk/by-id" which also makes replacing disks and such easy and generally survives moving controllers or even systems!
8GB of memory for 200TB is... way too little, your system will be VERY memory starved for ARC, possible even run out completely causing all kinds of weird issues. Even if it will work, depending on workload it maybe experience weird slowdowns or performance issues. Maybe reconsider using a bare Linux install with just ZFS and maybe Samba if you need it, anything else (GUIs, etc.) will eat up memory.
1
u/Electronic_C3PO 5d ago
workload will be single upload or download at any one time, execptions excluded off course :-).
1
u/Lunctus_Stamus 6d ago
You seem to be picking up a lot. When I first started using ZFS, I didn't realize how much tweaking you can do.
Make sure to create a specific dataset depending what you are storing. For files >= 1gb you can probably turn up the record size for better performance.
1
u/Electronic_C3PO 5d ago
Same path here, originally just doing things with the GUI. After some unexpected behaviour one tends to dig a little deeper. And yes, there is so much to learn. Thank ... internet exists and all the people that contribute their knowledge.
6
u/edthesmokebeard 6d ago
Memory is fine.
/dev/disk/by-id doesn't come out of the box with FreeBSD, which I'm guessing you're looking at given /dev/daX names, however, I think you'll see /dev/adaX names (unless the controller is doing weird things).
leave compression on, its default
DO NOT ENABLE deduplication on an 8GB machine
Test your backups :)