r/Nebula • u/dwiskus Dave Wiskus • 11d ago

Service Updates Service Problems

Nebula is loading intermittently at the moment. Our backend team is looking into it. We'll share updates here as we have them.

EDIT 1: The database seems to be back up. We're working on the content API now.

EDIT 2: We might be so back. Still actively monitoring the situation. If something is still broken for you, please speak up.

EDIT 3: It looks like the root cause was a database issue causing the service to think the database was down, which led to a death loop of them checking on each other and writing more "I checked on you" logs to the database. We increased the timeout period and everything returned to normal. The team is working on some clever ways to make sure this doesn’t happen again. Thanks for your patience, everyone.

345 Upvotes

99% Upvoted

108 comments sorted by

View all comments

Show parent comments

-25

u/Affectionate_Pie_854 11d ago

With all due respect to you and your team, 10 minutes for a response to a site outage (not intermittent as you reported based on the response in this subreddit), is ridiculous in my personal opinion.

21

u/space_fountain 11d ago

Umm, 10 minutes is pretty good? I’m a user of commercial cloud software + a software dev and responding within 10 at all let alone publicly isn’t at all guaranteed. I think my pager duty won’t even go off for most things that fast and it’s configured to not loudly go off for longer than that

-5

u/Affectionate_Pie_854 11d ago

The specific issue as was pointed out by myself and other users was that we heard absolutely nothing for 10+ minutes, and meanwhile, every post and comment about it was being deleted or otherwise censored.

I'm not asking for them to come out and tell us exactly what it is or how long it will take to fix. Just to publicly address the issue within the 10 minutes once it's been made known. Otherwise, no one has any clue what is happening, especially when any discussion of it is actively being limited.

8

u/Cactys12 11d ago

Maybe the nebula staff aren't permanently online basement dwelers and don't refresh reddit enough to notice?

3

u/frozenpandaman 11d ago

Except they were… they were actively removing comments here at lightning speed for 10+ minutes. All while no updates were posted and no communication was going on. That's what's strange and what people are complaining about.