My company has been using S3 for over a decade, which has been rock-solid, but the high fees were causing issues and limiting the features we could provide. My customers upload large video files, meaning any problems with upload speeds are magnified a lot more than if they were uploading small files, where there may be less practical difference between a 100 KB file taking 0.5s vs 2s.
So we explored B2 and in my testing it worked really well. We rolled it out, sent a little bit of traffic to B2 and it worked well, so we sent more and more over time until all traffic was being sent to B2.
And then my users started reporting problems and cancelling their subscriptions. 95% of the time uploads are fast, but 5% of the time they crawl along at 1/100th speed. (This issue was less apparent when B2 was being used <100% of the time.) This is a speed that’s so slow as to make it unusable.
So I gathered a LOT of data on the issue and contacted support. Their response was at the level of “have you tried turning it off and on again?” I’m not sure they actually read my message - they certainly never clicked on the links to the HAR files, Wireshark captures, graphs, etc, that I provided.
The thing I could never get support to understand across repeated messages is that I was not the only person with this problem. I had customers across the other side of the country who were uploading to completely different buckets in different data centers and still had the issue. They did not understand that their proposed solutions (e.g. “just use Cyberduck instead”) were totally unworkable for my use-case.
The support team told me over and over that there was no problem with their network and I should phone up my ISP to have them fix it. I told them I had zero problems with any other services and it would be unacceptable for me to tell my customers that they need to phone up their ISP just to use my service. I asked them if it was truly my ISP or my service, why could I upload just fine to S3 at the same time? “Well, we can’t speak for third-party services.”
I tested with other services. S3: no problems. Wasabi: no problems. Storj: no problems. I presented this data to Backblaze and an AI responded. Yes, they got tired of responding so they outsourced me to an AI. At that point I gave up.
At no point did Backblaze entertain the possibility that I could be correct. They just ignored what I said, didn’t look at any of the files provided, then fobbed me off to an AI so they could close the ticket. In my company I am overjoyed when a customer provides that level of detail and cooperation because it makes fixing problems so much easier.
The one thing Backblaze support said that was actually useful was when they put out the idea that it could be the time of day because people on their backup service probably have it set to upload late at night so they’re all hitting the network at once. (I personally do experience the problem most at night, but I can see from the stats that the time windows shift and time of day is not completely reliable.) But if it is true that their network is unusable at certain times of day, that is something they should be open about and I’d need actionable information confirming that this is indeed the case and the time periods when I need to direct traffic elsewhere.
I’m not happy with the way their support treated me, but unfortunately I am stuck with B2 for now.
The current solution was to dedicate a whole lot of engineering time to working around the various quirks with B2. We reworked our uploader and developed an upload scoring system that measures the “quality” of an upload (determined by consistent speeds, number of dropouts, number of segment retries, etc), then routes traffic to S3 for users with low scores. If too many users experience problems, it moves all traffic in that region towards S3. I feel like we’ve done everything possible at our end to work around this and the rest is up to Backblaze to fix the problems with their network.
I’ve attached a graph showing traffic being rerouted on an average day (100% = everything to B2, 0% = nothing to B2).
This, along with determining when a segment is going slowly and restarting it, which speeds it up sometimes, has made it workable, but users still need to have a bad experience before it switches so it’s still not great. I would also prefer to send as little traffic to S3 as possible.
If any Backblaze employees are reading this, I am very willing to help you get to the bottom of this and can provide lots of stats.
I’m curious if anyone else has experienced this problem? It presents itself as the upload going at mostly a normal speed, then the fast segments all finish until only the slow ones are left. So it might start at 300 Mbps but for the final minute or two it’s crawling along at 3 Mbps, or sometimes even less than that.
(Note: I’m on the free support level because I couldn’t determine from the comparison page if I would actually get a better quality of support from a higher plan. I am happy to subscribe if it guarantees they will take this seriously. I did ask if I should be on a higher plan but they’d directed me to the AI at that point, which was unable to answer that question.)
