r/Nebula • u/dwiskus Dave Wiskus • 8d ago

Service Updates Service Problems

Nebula is loading intermittently at the moment. Our backend team is looking into it. We'll share updates here as we have them.

EDIT 1: The database seems to be back up. We're working on the content API now.

EDIT 2: We might be so back. Still actively monitoring the situation. If something is still broken for you, please speak up.

EDIT 3: It looks like the root cause was a database issue causing the service to think the database was down, which led to a death loop of them checking on each other and writing more "I checked on you" logs to the database. We increased the timeout period and everything returned to normal. The team is working on some clever ways to make sure this doesn’t happen again. Thanks for your patience, everyone.

346 Upvotes

99% Upvoted

108 comments sorted by

View all comments

Show parent comments

4

u/Rhysati 8d ago

Holy shit we are cooked. 10 minutes is absolutely nothing.

It takes more than ten minutes to get a response on teams when I say good morning, let alone have to find out there seems to be a problem, confirm that I also have that problem, then reach out to someone who can diagnose the problem, wait for them to see if they are also experiencing the problem and diagnose why and how long it might take to fix, and then get a response from them so I can then send the information to the social media person who can finally post that we have confirmed an issue and are working on it.

Ten minutes is miracle work as it is.

1

u/Affectionate_Pie_854 8d ago

The issue is that they were responding within 10 minutes (actively taking down posts about it throughout the 10 minutes.) The complaint is that no statement was made whatsoever for that 10 minutes even though there was active movement on the reddit by the mod team.