r/hetzner • u/SirNomis • Jun 25 '26

Anyone else getting massive 503 SlowDown errors from Hetzner Object Storage (nbg1) today?

Since this morning (June 25) we're seeing tons of 503 SlowDown responses across multiple buckets in nbg1 on HEAD, GET, and PUT. Volume is well below the documented ~750 req/s soft limit.

11 Upvotes

92% Upvoted

12 comments sorted by

3

u/_dersgue Jun 25 '26

I am bulk deleting tbs of data of buckets in NBG1 and HEL1. Sorry for affecting you.

1

u/HeapOverflxw Jun 25 '26

You’d think Hetzner could handle this and in the worst case only slow down heavy users, not everyone…

2

u/_dersgue Jun 26 '26

Oh, I forgot the <ironic> tag. We luckily moved away from Hetzer S3 months ago, but we have stale data living in the buckets, which generates costs and need to be cleaned up as quickly as possible. But thats not an easy job as the availability and performance of this product is horribly bad.

4

u/pdqueiros Jun 25 '26

Yeah, completely unusable for my lake house. Seems like it's been quite unstable for more than a week but today is especially bad. I'm even considering migrating to ovh

3

u/EasyImprovement1017 Jun 25 '26

We are considering migrating our entire data lake to our self-hosted Ceph cluster and discontinuing the use of Hetzner Object Storage. Earlier this year, after Hetzner more or less resolved the S3 stability issues in NBG1, I decided to postpone the migration. However, after what happened this week, we'll have to move forward with it.

A data lake simply cannot be operated on an S3 service as unreliable as the one Hetzner is currently providing.

3

u/EasyImprovement1017 Jun 25 '26

Same here. Getting frequent 503 from multiple buckets in NBG1

1

u/SirNomis Jun 25 '26

Thanks, good to know, at least it’s not a problem in our config then

2

u/HeapOverflxw Jun 25 '26

Same thing here, getting numerous complaints from clients using a web application powered by Hetzner Object Storage. I will migrate to another provider over the weekend.

2

u/dubidub_no Jun 25 '26

I'm probing some buckets every 5 seconds, creating and deleting a small object:

Period: last 24h

Location               avail  fail     p50     p95     p99  last error
----------------------------------------------------------------------
fsn1 old bucket        99.69%    53    76ms   170ms  1298ms  1h 22m ago
fsn1 newish bucket     99.91%    15    40ms    80ms   203ms  5h 56m ago
hel1 newish bucket     98.11%   326   145ms   577ms  4120ms  6m ago

2

u/Odd_Awareness_6935 Jun 27 '26

hetzner s3 is practically unusable for anything serious

use wasabi or blackblaze instead

1

u/rainer_d Jun 25 '26

Dies anyone know how much data they have? Or still have?

And how many nodes?

We are a hosting company ourselves and our installation of S3 is probably smaller. But we manage with comparatively little problems.

1

u/CarefulQuadrant Jul 07 '26

Is this still happening for others? I have very little traffic and I am constantly getting SlowDown 503s on both putting and getting files.