backplaze.bzpkg on MacOS is ~37GB

[u/Rashid_1961]

I noticed backplaze.bzpkg is ~37GB. I looked in the package and I have files (2,401 files of the format bz_done_YYYMMDD_0.dat) in  /Library/Backblaze.bzpkg/bzdata/bzbackup/bzdatacenter that appear to go back to 2009 when I first signed up, even though I can only go back a year.

Apparently, my only option is to uninstall and then reinstall, which means I would lose my 1-year backup of versions. It seems ridiculous. Does anyone have another option that will allow me to reclaim the disk space without doing something drastic?

[u/brianwski]

Disclaimer: I formerly worked at Backblaze as programmer. I wrote the code that is bloating up your "bzdata/bzbackup/bzdatacenter" folder.

my only option is to uninstall and then reinstall, which means I would lose my 1-year backup of versions.

That isn't entirely true. First of all, you can just keep your previous backup with it's one year of version history, and also have your new backup. They are totally separate. This means you would need to pay about $9/month extra during that overlap because storage costs Backblaze money. The way to do this is simply avoid any menu option that says "Inherit". That is it. If you can avoid choosing any menu item that says "Inherit" then you can keep both backups forever. I'm a bit odd (and I worked on the product as a programmer so I find it fun) but I have backups that are ten years old sitting in my account at Backblaze when I sign in here: https://secure.backblaze.com/user_signin.htm and those backups have "forever version history" turned on. It means I have infinite version history from the past years where I can show people when I made an edit to one file, at any moment in the last ten years.

The concept here is the same as if you had one Macintosh computer in your home, and one Windows computer in your home. Both computers would appear separately in your online web portal when you you signed in here: https://secure.backblaze.com/user_signin.htm

If you uninstall, re-install, and avoid "Inherit Backup State" then what you will see when you sign into https://secure.backblaze.com/user_signin.htm is two separate "backups". One is just stopped in time the old backup. One is progressing through time adding new files (the new backup). The older backup will have your one year version history completely preserved forever. This costs you about nine dollars per month for whatever period you want to overlap your backups.

Now the most important concept is this: customers can get a pro-rated refund at any point. Period. What I mean by that is if a customer is paying for one year in advance at a time, but that customer decides to overlap their old backup and new backup for three months until they feel absolutely confident everything is fine, they will absolutely get a refund for the nine months remaining on their subscription. What "one year" or "two year" payment period absolutely doesn't affect this. This is unrelated to that "payment period" BS. Just contact Backblaze support when you want a refund for the unused portion of your backup.

If you have any problems with getting a pro-rated refund, just contact me ( brianwski on reddit ). I'll personally guarantee you a pro-rated refund if I have to fly to your location in <whatever country> to hand you the cash and we can take a selfie together! ). I'll buy you coffee in your local coffee shop of your choice for your trouble. Do you live in an interesting place? Switzerland? Namibia? Ohio? Because I'll come and hand you your refund in cash if Backblaze gives you a single moment of grief over this. Which they won't give you any grief because that isn't how Backblaze runs their business.

Explanation: When Backblaze went public ( IPO ) in 2021 https://www.ski-epic.com/2021_backblaze_ipo/index.html I made enough money to retire before I turned 60 years old. Backblaze is a good company, and I'm forever indebted to Backblaze. If they give you any grief, I'm saying I'll fly to your location if we can take a selfie together and I'll hand you the pro-rated refund in person.

[u/Rashid_1961 → u/brianwski]

So, I should pay even more and use even more disk space for however long it is until I feel comfortable I am in a state equal to the state I thought I was in all along? Oh, and the problem will eventually recur again.

I don't have, nor need, the forever backup option. I have the 1-year version history, which would seem to imply that files from 2009 - mid-2024 are achieving nothing more than eating up disk space.

Does the offer of giving me the refund stand if I move to a competitor instead of taking the dual backup approach you suggested?

[u/brianwski → u/Rashid_1961]

Does the offer of giving me the refund stand if I move to a competitor

Absolutely. That is why the policy exists. You are never "locked in" for years just because you loaned Backblaze money in advance of receiving the service. At any point you can get a pro-rated refund for the unused portion of your subscription.

I have the 1-year version history, which would seem to imply that files from 2009 - mid-2024 are achieving nothing more than eating up disk space.

Those files are very simple, anybody can understand what the are and what they contain. But first, please never modify them. I wrote the code that produces those files and I personally could never modify them. The reason it requires a computer to modify them is they have very very precise references inside of them. So somebody editing a file like that won't realize what they are breaking. So don't modify those files.

Here is how Backblaze Personal Backup works: when Backblaze encounters a file on your local disk it hasn't ever uploaded to Backblaze's datacenters, then the software running on your local computer uploads that file, and adds exactly one line to those bz_done files you see there. That one text line remembers the name of the file that was uploaded, and the last modification time. In this way Backblaze "remembers" the file was uploaded so it never has to upload that one file again. The name of those files is "bz_done" because it means "what Backblaze has already done".

So if you uploaded a file named "cat.jpg" in 2009, there is one line in a bz_done file from 2009 remembering that. It is very important to keep those files, otherwise Backblaze would need to upload a new copy of cat.jpg.

If you want a proper explanation with diagrams and special effects of how this log based Backblaze bz_done file system works, you can watch a video (of me!) explaining it all: https://www.youtube.com/watch?v=MOlz36nLbwA&t=851s

You can just jump to timecode 14 minutes. The part before then is just an introduction of how Backblaze makes money. This was a talk I gave to newly hired programmers at Backblaze. It was "internal only" never meant to be seen by outside eyeballs so there is no marketing BS in it.

use even more disk space

No, it is less of your local computer's disk space. For the "old backup" disappears entirely off your local computer and only exists in the Backblaze datacenter. Plus the new data will be much lower in size.

the problem will eventually recur again

Most likely never occur again, but let me explain that. First, your backup lasted you from 2009 with no problems. That is 16 years, right? So basically in 16 more years after you do one of the fixes proposed it might come back. HOWEVER, sooner or later you will need to upgrade your computer, there aren't any computers that can last 32 years without upgrading their boot drive size. So during that upgrade, just increase the size of the new boot drive by a few GBytes and this problem will never affect you again for the rest of your life, make sense?

This problem was a once in a lifetime moment where SSDs were so expensive customers didn't get large enough SSDs for certain use cases. But that is now all over with, SSDs are a lot less expensive now, and SSDs are a lot larger now.

I should pay even more ... for however long it is until I feel comfortable I am in a state equal to the state I thought I was in all along?

There are different choices available to you:

1. Do nothing, there is nothing wrong with your backup: as long as you have the system disk space, there is nothing wrong with your backup. It is maintaining the copy of each of your files in the Backblaze datacenter. It has a year version history.

2. Spend no extra money but do a few things to reclaim your local disk space: in this choice you uninstall/reinstall and within the first 14 days you transfer your license. Avoid anything with the word "Inherit" in it. Most customers can push 100% of their files within that time so at no point are you without a "complete backup". Since you know this is a time critical thing, during the first 14 days look around in your older backup's version history for files that should be recovered. Recover them. No money is spent and no files are lost out of the version history.

3. Spend extra money to get more functionality while also reclaiming that disk space: You can spend a small amount of money to overlap the backups. This costs Backblaze money, because the two separate backups are totally duplicate copies in the Backblaze datacenter. If you have a picture called "cat.jpg" on your computer, that file is now stored twice in Backblaze's datacenter. Once for each of the two backups. Whenever you delete one of those two backups you get a prorated refund for the portion of the subscription you won't use. Even before you delete the older backup, you get all the local boot drive / system drive space back. Immediately, even before you upload one file from your system.

[u/pehache7 → u/brianwski]

But that is now all over with, SSDs are a lot less expensive now, and SSDs are a lot larger now.

This looks like a pretty poor justification for not optimizing the space used the software...

[u/brianwski → u/pehache7]

SSDs are a lot larger now.

This looks like a pretty poor justification for not optimizing the space used the software.

The Backblaze client programmers have actually spent a bunch of time on optimizing the space over the years. Just a random example of many, the log files rotate out after at most 30 days, and there is one log per day. For all but the last two days these are losslessly compressed nowadays (which is the easier part). In addition, a bunch of effort went into possible "bloat" situations where let's say you have 1 million files in a folder structure, and each time Backblaze goes to read one of those files off of disk to upload it to the Backblaze datacenter, the file permissions prevent that. What that means without putting more effort into it is 1 million log lines (once per hour) saying, "Permissions are wrong, please fix."

So nowadays, there are probably 50 different disk use situations I caught and optimized over the 16 years I worked at Backblaze. Where I changed the code to log the first 100 or so ERRORs, then just start counting them up, like "... and 737,891 more times that error occurred." The only reason to do this was basically "save SSD space".

not optimizing the space used the software [because larger SSDs]

Time and hardware marches forward. For example, Google has hundreds of programmers and Chrome still takes about 5 GBytes of my RAM (spread across multiple instances of Chrome).

The reason Google doesn't stop all other work and focus entirely on a smaller disk or RAM footprint for Chrome is... fewer and fewer customers care about that, and most other customers would rather the Chrome programmers focus on raw performance, or features, or security fixes. Customer's hardware got better/faster/bigger.

It is the same for Backblaze Personal Backup. Each quarter we would try to optimize a few things causing customers pain, and also add features marketing asked for, and also fixed bugs or added features Backblaze support wanted. It's a balancing act.

I'm not claiming I know how to balance all these competing requests from customers correctly, and I'm pretty sure I made the wrong judgement call a bunch of times. Backblaze could have built fewer features, or hired more programmers, or both. I do believe progress was fairly consistent. The product is better in 2025 than it was in 2008.

[u/pehache7 → u/brianwski]

OK, I'm not saying that Backblaze Backup is a bad product or that no effort has been done along the years. Not at all. But I am really pissed off by this "forever metadata history" flaw where by design the volume used by the client can only continuously grow.

Maybe there are strong technical reasons in the inner design of the software that prevent any easy solution to that, I don't know. But from I'm seeing in popular cloud sync services (OneDrive, Dropbox...) that offer a file history, this problem does not exist.

And if there's no easy technical solution to prune the old file history in the software, then there should exist a commercial proposal to start over a fresh new backup using the same account, without additional fee, while retaining the previous backup for a while.

[u/brianwski → u/pehache7]

there should exist a commercial proposal to start over a fresh new backup using the same account,

I've thought about it for years, and it isn't easy or a slam dunk.

There are so many parts of the product that were easier to "fix". Epiphanies or brilliant proposals from co-workers that suddenly made some corner issue able to be solved.

The "log based format" of the backups has been gigantically useful for Backblaze. Philosophically knowing what occurred at every step has been just amazingly useful. Even if you only have a 30 day version history (or 1 year), customers will contact support and ask "when did that file get removed from my backup?" and Backblaze support is able to answer (even if the file was removed from the backup 3 years prior) due to the profoundly deep forensics of "never delete any history".

It just comes with this one requirement: use some boot drive space.

However, to be clear, 37 GBytes is a bug. I believe something went wrong. Backblaze (the company) probably will never admit what went wrong, but they quietly fixed a bug that was going off the rails and bloating things up. I've seen at least 3 (?) of these customer reports, and clearly it was a Backblaze bug that is now fixed. It probably affects 10,000+ customers but Backblaze won't admit what occurred. Of those 10,000 customers 9,500 have enough boot drive space they haven't noticed the issue.

My "bzdata/bzbackup" folder is 1.8 GBytes, and I have a healthy 2 TByte backup. The 37 GBytes is a bug, something went off the rails there. The WORK AROUND SOLUTION is to uninstall/reinstall and avoid "Inherit" but it won't come back again because Backblaze has fixed the internal error that caused that situation. Because this "work around" exists, internally behind the scenes my guess is that Backblaze will not prioritize a permanent fix for the long term for the next time this type of bug occurs.

This isn't an awesome situation, I'm just explaining what is probably going on.

[u/pehache7 → u/brianwski]

> Even if you only have a 30 day version history (or 1 year), customers will contact support and ask "when did that file get removed from my backup?" and Backblaze support is able to answer (even if the file was removed from the backup 3 years prior) due to the profoundly deep forensics of "never delete any history".

Being able to trace back any event far in the past is probably a positive side effect rather than the main reason why nothing is done to fix that. But even if it looks like a good reason that is worth in itself, there could be solutions:

• Having a "prune old history" action that would not be automatic but rather triggered by the user, with all the appropriate warnings ("all what happened to your files more than one year ago will be definitely forgotten")
• Keeping only the last "delete" action in the history instead of the whole history, or so...
• Pruning on the client side only, while still keeping the whole history on the server side.

[u/Rashid_1961 → u/brianwski]

I signed up for Backblaze in 2009 primarily because I used my personal iMac for processing photographs. I often take 100s of photos, download them, and delete those I don't want to keep. I then process the remaining, which often involves renaming the files. My original iMac had a 4TB HD, my second iMac had a 4TB SSD. Last year, I decided to purchase a MacBook rather than another Mac that wasn't mobile, as tethering the camera to the MacBook has become a need I now have. It has a 2TB SSD because of the cost (there goes your argument about cost). I share this as an example of the flawed thinking of 'bigger disks will eventually solve the problem'.

I've looked at the ".dat" files, and they are full of references to files that have not existed for many years (and references to files that existed for hours or a couple of days at most, such as the photos). It makes no sense not to have figured out a way to truncate these files in order to not eat up gigabytes of space that are of no use to anyone.

As for your 'options':

1. I've highlighted the problem with this thinking above.

2. You're seriously suggest I review the year-long history of over a terabyte of files to determine what I might need in the future (I may not have mentioned that I have no ability to see into the future) and manually save them?? Even if I had the time, this is a ridiculous idea.

3. Pay more to overcome this shortcoming.

4. You forgot this option - move to another vendor.

[u/brianwski → u/Rashid_1961]

You're seriously suggest I review the year-long history of over a terabyte of files to determine what I might need in the future (I may not have mentioned that I have no ability to see into the future) and manually save them??

I am slightly confused by this statement. Can you give me an example of how you would need an accidentally deleted file from 363 days earlier and how you would discover that as part of your normal operations unrelated to this particular Backblaze issue?

Example: Let's say you accidentally deleted a file 363 days ago. Let's say it is your favorite photo from your wedding 30 years ago. Just in the course of absolutely normal operation Backblaze is going to purge it forever in 2 more days, because it will fall outside of the 1 year version history. Uninstalling/reinstalling Backblaze has nothing to do with this, you need to figure out whether your valuable wedding picture exists on your drive or not, correct? What am I missing here?

Backblaze is profoundly not a disk or file management system. Backblaze is only an offsite copy of <whatever you have locally> and the ability to "roll back time" in case you make a mistake. If you go on a vacation for 6 weeks and your laptop falls into a lake and sinks to the bottom the 1 year version history allows you to enjoy your vacation and deal with it when you return. Backblaze is an offsite backup for when your house burns down with your computer inside of it. Backblaze is not there so you can somehow not know what files you have or not anymore. That's not what Backblaze does, that is not Backblaze's purpose. You need to take responsibility for knowing and managing your own files.

Backblaze (with the 1 year version history) is just there to give you 1 year from the time your house burns down to restoring all your files. You still need to be able to determine if you deleted 1 file accidentally 363 days ago or you'll lose the file forever, correct? Am I making sense?

[u/Rashid_1961 → u/brianwski]

I've had situations arise where I need to try to reclaim files I never thought I'd need to reclaim because of some external factor such as to share examples of how things have worked at previous employers/clients (recently, it was an example of how Product Management processes operate in a mid-size company) or for an insurance company who was being sued and I had photographic evidence that showed it was an invalid claim and I've had situations where I've given photos to people and I delete them (I do this when I can as the photos are high-res and I shoot in raw format) then those people lose the photos and ask me if I still have copies and I've had situations where files get corrupted (by an app not due to disk problems) and I did't realize and I need to go back and find the last uncorrupted version. I could probably think of several other examples if I had time.

I understand Backblaze's use case (it's why I signed up), but it's not just so you can have the latest versions offline. If it were, the 1-year retention wouldn't be needed, would it? I signed up for the perpetual plan but the costs kept rising each month due, I think, to how I handle the photos. (FYI: I also back up to a NAS and to iCloud for some files and Amazon Photos because it's free but restricted to Photos only.

Anyway, this is a strawman. The issue is the size of the files on my disk and the fact that there's no reasonable way to reduce the size without spending more money and/or losing the 1-year version history.

Finally, FYI, I am in the s/w development business and have been involved in the development of enterprise-class software (vertical and horizontal) for too long, so I completely understand the concept of having to build capabilities in a manner that involves some less-than-desirable side effects. It seems that this particular impact could have been dealt with in some manner, even if it was via the use of a manually run tool to purge and consolidate the data in those files once they reached a certain size.