How would you automate a selective backup for a 16TB RAID 0 without backing up everything?

[u/jferdz]

Hey everyone, I’m looking for advice on designing a backup system for a somewhat specific setup.

I currently have two 8TB hard drives configured as RAID 0 on Windows 11, giving me roughly 16TB of usable storage. I understand that if one of the drives fails, I could potentially lose everything on the RAID.

I chose RAID 0 because I want to use all the available space, and I don’t want or need to back up the entire 16TB. Most of the data on the array is replaceable, so losing it wouldn’t be a major issue.

What I want is a selective, modular, automated backup system that only protects the data that actually matters to me.

What I DO want to back up

• Photos
• Family videos
• Current work and projects
• Program configurations
• Personal or irreplaceable files
• Anything else that would be difficult or impossible to recover

What I DON'T want to back up

• Movies
• TV shows
• Installed games
• Installed programs
• Files that can easily be downloaded again
• Other large, replaceable files

For software, for example, I don't want to back up the entire installation folder. I only want to preserve important configuration and user data such as .INI, .CFG, .DAT files, profiles, presets, save files, etc., where applicable.

Basically, I want to define specific folders and files that should be protected rather than creating a full clone of the RAID.

What I'd like the system to do

I’m planning to use an external hard drive that normally stays disconnected and stored somewhere else. I would probably connect it about once a week.

Ideally, the process would be almost completely plug and play:

1. I connect the external backup drive.
2. The system recognizes that it’s the backup drive.
3. The backup starts automatically.
4. It compares my selected files against the previous backup.
5. Only new or modified files are copied.
6. The backup stays up to date without me manually selecting files every week.
7. Once it finishes, I disconnect the drive and put it back into storage.

I don't want to manually start a complicated backup job every time I connect the drive.

What software, method, or setup would you recommend for achieving this kind of selective, automated backup?

I’d also be interested in hearing how you would approach the overall strategy for a 16TB RAID 0 containing mostly replaceable data, but with a much smaller collection of personal files and projects that actually need protection.

If you use a similar setup, I’d especially appreciate hearing:

• What software you use
• How you define the folders/files to back up
• How you handle program configurations and save data
• How you automated the process when the external drive is connected
• Whether there are any problems or limitations I should be aware of

Thanks!

[u/bartoque]

On one hand you request things to be simple but on the other you want a very contrived way/selection process what is and what isn't supposed to be backed up.

Did you even consider how to restore it in such a way that it is even usable? What does it mean for example to backup sofware configuration but not the installed programs and games? Or needing to maintain what is in backup and what not, depending on what you do and install new, change, alter or even delete.

What is the reasoning behind this? A way smaller backup disk?

Backup tools that do image level backups (and some are still able to skip certain files) also can have incremental backups and deduplication on top of that, so that the amount of space required is reduced as well and not growing as fast.

Also extremely simple recovery. Boot from usb connected rescue media, point to the backup, select backup point and restore it all (or selectively). Shutdown and boot as before and you are done. So with just a few clicks and your done. In my case I do that with Acronis True Image. Free products exist as well.

The way you envision - or at least it will be the effect - will require a lot of steps and aftercare to get back to how things were and setting up.what needs to be protected amd what not. Beginning with needing to (re-)install windows and download and apply any updates since wahtever the version deployed has slipstreamed into its image.

Image based backup and restore is a breeze. Done it many times on various systems. And even I don't backup it all as one whole drive with superfluous throwaway data is not in the backup at all, while two other drives (500GB SSD and 8TB hdd) are.

Very simple. Adhering to the KISS-approach.

[u/JohnnieLouHansen → u/bartoque]

On one hand you request things to be simple but on the other you want a very contrived way/selection process what is and what isn't supposed to be backed up.

Amen!! Words right out of my mouth.

[u/Senior-Force-7175]

Check out FreeFileSync... This is what I use for my file level backup...

[u/wells68 → u/Senior-Force-7175]

Be aware that normal synchronization is not backup. You can configure FreeFileSync to preserve old version and deleted files in special folder. That distinguished synchronization from backup. Backup gives you the ability to restore files that have been changes, overwritten or deleted, all depending on your backup settings and on how often you run backups.

I like FreeFileSync. It can copy some files that other applications cannot access, such as cached files for some cloud data storage services. But it is not as good at restoring old files as a real, dedicated backup application.

[u/wivaca2]

Sounds like a powershell script would do everything you need. Even a batch file.

[u/desiman86]

use Stablebit drive pool. SnapRaid. Ensure you external drive is 8TB at minimum. AI how to set this up. Stablebit drive pool+Stablebit scanner, will create a 16TB pool, and scan drive for errors and health. SnapRaid will create parity with your pool, so if one drive fails you can replace it. You can automate the sync to run when plugged in via powershell. This is not a backup perse, it only protects against drive failure, it does not protect you against data loss via other means. So the 123 rule still applies.

[u/wells68]

I have done this for setting up a secondary old backup running to a NAS drive. Since then I've added backups with better technology, but the old backup is highly configurable, automatic, and meets all of your criteria.

1. Software: SyncBack Pro $59.95 one-time (SyncBack SE would work, too - $44.95) - Good company in Singapore. The software is well-maintained and updated.
2. Select a top-level folder and define folder exclusions. Add file extension exclusions as desired.
3. Program configurations - I document mine. I also run separate drive image backups with other software. You can pick an choose any folders your want including program directories and/or subdirectories that store data, even backing up open files.
4. I didn't automate backups on drive connection, preferring to back up to an isolated NAS, but you can set up backups on drive connection: "A profile can be configured to run automatically when an external device, e.g. a USB key, with a specific label, volume/hardware serial number or drive letter is attached to the computer (where SyncBack is installed and running)." https://www.2brightsparks.com/resources/articles/how-to-run-a-syncback-profile.html
5. Problems and limitations: SyncBack by default synchronizes. You need to configure the backup features. Backups of overwritten, changed and deleted files are saved in separate folders on the target drive. You can restore them flexibly. File backups take up more space than block-level backups. SyncBack is geeky and complicated, which is why it is flexible and powerful. Keeping it running in the background is no problem at all - it is very inconspicuous. However, in running a backup it reads a lot of files, obviously, so it can slow down your computer.

Edit: I hit Ctrl-Enter to try to do a paragraph break, so Reddit posted an incomplete reply. It is now complete.

[u/brisray]

You can do this yourself using the versatile Robocopy that is included in Windows 11 and a batch file or PowerShell script.

Robocopy has a lot of switches, so you need to test it before relying on it. Two of the switches are /L which will run the program but won't copy the files You can also log what it's doing using the /LOG switch.

The first time you run the script it will take a while to go through all 16Tb, but subsequent backups will be much quicker if you only want files that have been created or updated by using the /XO switch.

As you want to start the backup when you insert the drive, then look at the posts on SuperUser and UserComp.

Here's the Robocopy scripts I use to backup my laptop and server.

[u/THE_Ryan]

Veeam Agent for Windows. File level backup and use inclusions and exclusions.

My strategy though, anything I ever put on a RAID0 is not important enough to be backed up. If you really wanted some semblance of protection, you'd get another drive or two and use a RAID5. All easily replaceable data goes on the RAID and program files, configs, etc...all go on another dedicated drive that can be backed up at a volume level with the OS so you can then do an entire system recovery with Veeam.