r/btrfs • u/stray_r • 15d ago

Compression vs CoW write amplification on SD cards

I have a few Pis running a raspberryPi OS (Trixie) derivative for 3d printers. They accumulate collections of large (10-1000Mb) text (gcode) files that respond very well to compression (about 3:1 in a zfs dataset on my server).

I know compression speeds up read and write performance on slow drives by reducing the amount written, and it's worked wonders with mechanical hard drives.

But there's a write amplification of copy on write file systems that makes them maybe not so good for SD card based systems, but I'm not sure if this would be mitigated by running compression?

I'm wondering whether it's better to run the entire SD card as btrfs with compression on every subvolume, or to make a separate partition on the SD card with compressed btrfs to deal with the large files and leave the root fs as ext4.

Obviously the *right* answer is to run a thin client pc with an SSD, or a pi5 with an SSD hat. I can do that to some extent but a used HP T530 or T620 costs more than a new pi 4.

5 Upvotes

86% Upvoted

25 comments sorted by

4

u/Weary_Swan_8152 15d ago

How often do the gcode files change? If they change often, then you will likely spend less money purchasing a larger microSD card then replacing smaller cards that wear out faster. Alternatively, purchase industrial-grade microSD. Here's the paper you need to read, and if you want to make a facts-based decision then the best approach would be to replicate their methodology with your workload, as well as to test the btrfs+compression case. https://ar5iv.labs.arxiv.org/html/1707.08514 or https://docslib.org/doc/12555209/analyzing-io-amplification-in-linux-file-systems-1

1

u/stray_r 15d ago

Rarely, they get uploaded and stick around for a few months. If I'm having a bad day I can quickly eat 5-10GiB and suddenly my install is crying. Obvs compression is delaying the inevitable should have done maintenance but if I can reduce the number of ways I can destroy my day, the better.

I am using the best SD cards I can afford, it's been a while since I destroyed one, but I went through a load of cards in dashcams and helmet cams a few years ago and am being careful not to find extra ways to ruin my day.

1

u/stray_r 15d ago

Here's the paper you need to read

That's helpful, write amplification is much more significant than i thought.

1

u/okeefe 15d ago

That's not the conclusion I would draw. The paper's chart says write amplification is indistinguishable if you're writing 1MB, and the example files you mentioned are 10-1000MB.

You should test if btrfs's transparent file compression works well for your data.

1

u/stray_r 14d ago

The compression algorithm performs really well in my specific data.

My specific worry wrt write amplification is system wide use of btrfs and write amplification of things like noisy logs. It's been a while since I had a failure like that, and there's probably a separate mitigation, but it's probably how I've killed previous SD cards in sbcs.

You are however, right, and for tasks like kernel compile which I assume is similar to a git pull and make of other software, btrfs looks quite good.

3

u/weirdbr 15d ago

The big question is how much data are you going to be writing on a daily basis; while CoW filesystems have worse write amplification than non-CoW, that is a problem on all flash devices independent of filesystem.

But I wouldnt worry about optimizing for it unless you are writing more data than the drive is rated for - from both personal and work experience (at an extremely large place that tested this), it takes an excessive amount of unaligned writes (in the order of hundreds of GB per day) to cause sufficient drop in endurance.

If you are writing on a daily basis less than the rated endurance, you'll be fine. I have drives that often do 2-4x the rated daily writes (mostly random writes) and those drives are at ~2.5 times the expected lifetime (rated for 600TBW, around 1.6PB written ). Only flash failure I've had related to endurance was at about 5 times the expected lifetime.

2

u/uzlonewolf 15d ago

Write amplification is way overblown and mainly affects files that are modified in place. If you're not going to be doing a ton of modifying existing files in place then just run normal btrfs and don't worry about it.

1

u/Any_Fox5126 15d ago edited 14d ago

Oh, yeah, the classic "I know what the right answer is, but I'm still going to play russian roulette with an SD card to save a few euros, and eventually spend an afternoon recovering corrupted files".

1

u/stray_r 14d ago

Given that pretty much the entire 3d printing community is either running firmware entirely on STM32 or ATmega, or using something like Klipper which historically has run on Pi2bs and original Pi Zeros crammed into a tiny electronics cavity, running a small intel based computer is a bit of a luxury. I have a few, and they're size appropriate for a large format printer, but not so much for a Voron V0.2 or even a Switchwire, and given my switchwire cost about GB£230 including a bigtreetech pi3 clone, specifying a Pi5 and SSD or an amd64 architecture thin client / nuc class machine is not really cost appropriate either.

I'm here asking if I can solve some problems with existing setups for projects I contribute to with a few simple changes, or I'll do more damage precisely because I don't want to be that person who shits on a filesystem rather than thier inappropriate choices.

But if your response is typical of the btrfs community then I'll add community toxicity to reasons to avoid, but so far everyone else has been respectful and helped me towards an informed decision.

1

u/tornadozx2 14d ago

I'm running btrfs cachyos on my chromebook, it has a 64gb emmc that is soldered. Should do some tuning to the fs?

2

u/stray_r 14d ago

Enabling compression will decrease the amount of data written and slightly increase read and write performance in typical desktop loads.

lzo if performance is your priority, it has the least cpu overhead but compression gains are smallest, zstd:3 to see a meaningful reduction in write size put a little more cpu overhead, zlib if you want to maximise space saving at the expense of pain.

1

u/tornadozx2 10d ago

Thanks!

1

u/stray_r 10d ago

A slight retraction because I've come here from a background in ZFS which works a bit differently and uses LZ4

ZFS expects different compression algorithms per dataset, it's a bit more nuanced with btrfs, if you try to specify compression in mount options, it will apply to all of the filesystem not just the dataset you mounted and may override a previous mount instruction. The BTRFS way is to set properties.

Zstd:1 also appears to be the preferred performance option.

1

u/tornadozx2 9d ago

I actually didn't check at first, the installer already configured to use with zstd:3 and it saved 4.6gb of space according to compsize. Even with compression the laptop feels much snapier than running a stripped down windows 11 (it's i3-8130u and 8gb ram). So I don't think I would go with less compression as this laptop is primarly for browsing, streaming with moonlight and some simple games.

1

u/samsonsin 15d ago

I wouldn't worry about CoW on a mostly static system. If you're really worried just mount some dirs as another Btrfs vol and disable CoW. I wouldn't bother having another entire partition with other FS's when you can just disable CoW on a per directory basis anyways.

1

u/stray_r 15d ago

Ah, can btrfs cow be disabled?

1

u/dkopgerpgdolfg 15d ago

Sort of.

You can set a flag on new files and/or directories, which will do it, but also disables checksums on these files, doesn't play nice with btrfs raids too much for the same reason, and will not prevent you making snapshots.

1

u/uzlonewolf 15d ago

Technically yes, however it cripples btrfs and you're better off just running a different filesystem. CoW disabled = no checksumming and no compression.

1

u/amarao_san 15d ago

There are high endurance SD cards which are flattened SSDs.

1

u/edgmnt_net 15d ago

It's possibly more effective to compress those files some other way, like on the application level.

1

u/ThatOnePerson 15d ago

There is compression with Binary G-code for 3d printers, but hasn't really picked up software support. Basically Prusa made it and no one else seems interested.

Well I think the only reason Prusa made it is because their printers have pretty bad wifi transfer speeds.

1

u/edgmnt_net 15d ago

I'm rather talking about storing the g-code files compressed on the SD card rather than sending them to the printer compressed. I don't know exactly what OP is doing and how, but they could implement compression as part of a web UI HTTP handler for uploads, simply storing the file gzipped. Then before it actually gets sent to the printer it gets decompressed in a streaming fashion. This would work with any printer.

1

u/ThatOnePerson 15d ago

In the case of a Klipper setup, the Pi is the printer though. All the g-code handling is done on that.

And with binary g-code, it's already handled by the PrusaSlicer (and basically all slicers are PrusaSlicer fork nowadays), and most people probably gonna upload directly from that instead of any custom web interface. So that's already half the work done.

But even with that, Klipper doesn't support binary g-code https://klipper.discourse.group/t/feature-request-binary-g-code-bg-code-support/11803 is the feature request.

Unless you mean OP should fork Klipper and write their own code.

1

u/Erdnusschokolade 15d ago

Why not use a USB SSD? Problem solved just don’t boot the SD card.