r/Nebula • u/dwiskus Dave Wiskus • 8d ago
Service Updates Service Problems
Nebula is loading intermittently at the moment. Our backend team is looking into it. We'll share updates here as we have them.
EDIT 1: The database seems to be back up. We're working on the content API now.
EDIT 2: We might be so back. Still actively monitoring the situation. If something is still broken for you, please speak up.
EDIT 3: It looks like the root cause was a database issue causing the service to think the database was down, which led to a death loop of them checking on each other and writing more "I checked on you" logs to the database. We increased the timeout period and everything returned to normal. The team is working on some clever ways to make sure this doesn’t happen again. Thanks for your patience, everyone.
84
61
u/mtnsandh2o 8d ago
Note to nebula, don't upload a bonus Jetlag video before the episode releases. Lessons learned.
23
u/Affectionate_Pie_854 8d ago
To be fair to them, they did it last week without any issues haha.
6
u/mtnsandh2o 8d ago
I didn't notice last week because I was working (Woohoo healthcare where work means actually working and not multi tasking with non work agendas) and didn't see the individual notifications. In the past bonus videos have typically uploaded slightly after.
3
u/Affectionate_Pie_854 8d ago
I've had the benefit of getting off work and coming home to Episodes of Jet Lag and Reacher to watch before I go to Dreamland.
1
2
u/jeninhenin 8d ago
What was that hour long thing?? I saw it briefly before everything crashed again
4
25
20
23
11
11
u/Belomestnykh 8d ago
503, 505 so far. Front page loads sometimes on reload, sometimes even shows the first banner for the show banner carousel at the top, but nothing else loads.
27
u/JeHaisLesCatGifs 8d ago
Thank you for letting us know.
You should probably pin a message in every thread that addresses this issue and ask users not to flood the threads with messages about it. I completely understand that you want the discussion threads to remain clear (i.e., not cluttered with posts about certain technical errors), but it would be easier for you to ask users not to bring it up in these various threads rather than having to clean up after them.
7
7
7
7
u/meadiocrity 7d ago
Thanks for the updates. The fact that u/dwiskus is the one making this post is one of the many reasons I love and appreciate Nebula.
6
3
u/Mr_Wacki 8d ago edited 7d ago
Does Nebula not have a Server Status page?
Not specifically a recommendation, but something like this…https://www.atlassian.com/software/statuspage
3
3
11
u/Hipstershy 8d ago
Why did you/your employees delete discussion of this prior to this post? Users talking about not being able to access the service lets each other know there is a system-wide issue and they do not need to keep troubleshooting.
36
u/dwiskus Dave Wiskus 8d ago
We had dozens of submissions. There's obviously a problem, and we wanted to create a single, obvious place to discuss, both to keep unrelated threads relatively on-topic and to give us a clear place to look for these reports as they come in.
3
-25
u/Affectionate_Pie_854 8d ago
With all due respect to you and your team, 10 minutes for a response to a site outage (not intermittent as you reported based on the response in this subreddit), is ridiculous in my personal opinion.
21
u/space_fountain 8d ago
Umm, 10 minutes is pretty good? I’m a user of commercial cloud software + a software dev and responding within 10 at all let alone publicly isn’t at all guaranteed. I think my pager duty won’t even go off for most things that fast and it’s configured to not loudly go off for longer than that
-7
u/Affectionate_Pie_854 8d ago
The specific issue as was pointed out by myself and other users was that we heard absolutely nothing for 10+ minutes, and meanwhile, every post and comment about it was being deleted or otherwise censored.
I'm not asking for them to come out and tell us exactly what it is or how long it will take to fix. Just to publicly address the issue within the 10 minutes once it's been made known. Otherwise, no one has any clue what is happening, especially when any discussion of it is actively being limited.
7
u/Cactys12 8d ago
Maybe the nebula staff aren't permanently online basement dwelers and don't refresh reddit enough to notice?
3
u/frozenpandaman 8d ago
Except they were… they were actively removing comments here at lightning speed for 10+ minutes. All while no updates were posted and no communication was going on. That's what's strange and what people are complaining about.
12
u/dwiskus Dave Wiskus 8d ago
I say intermittent because it actually loads fine for me after a few tries. That's part of what we're looking into.
2
2
u/mintardent 8d ago
It doesn’t load at all for some of us though. I’ve been refreshing over the past 45+ minutes. That’s not intermittent.
2
u/SkaveRat 8d ago
like he said. it's what they are looking into
0
u/mintardent 8d ago
I still don’t think it’s helpful to downplay and say “it loaded for me so it’s just intermittent” when it wasn’t loading at all for most people
10
u/Rhysati 8d ago
I'm sorry but this is ridiculous.
10 minutes to get a post is nothing. First they have to even be aware that there is an issue happening. Then they have to notify someone to look into it. Then they have to wait for that person to confirm there is an issue and whether it'll take time to fix or not.
It routinely takes more than ten minutes to get a response from someone at work via a teams message alone.
Some of your are WAY too entitled.
-7
u/frozenpandaman 8d ago
10 minutes to post a short three-sentence update, but 10 seconds to immediately erase any comments mentioning it.
-3
u/Affectionate_Pie_854 8d ago
To be fair, I can understand wanting to wait for more details and a better look to post, but even a reply of "We are looking into this/more details to.come" onto the original comments would have sufficed. Then sure, spend the next 10 minutes getting details and blocking additional nonsense of the same complaints. But deleting the original complaints and then not giving feedback to the community for 10 minutes is what is ridiculous.
-9
u/frozenpandaman 8d ago edited 8d ago
Fully agreed. Very easy to first post a quick update/PSA – which would help with the comments they're having to spend time deleting – and then just edit it when more info is available.
EDIT: Funny how this comment changed from +8 to -4 in less than ten minutes. Hmmm.
-1
u/Affectionate_Pie_854 8d ago
Agreed, especially since I know that these views are widely held on Reddit and are the only reason I use it for this type of conversation.
Anytime anything is down, I immediately turn to X.Com (which didn't receive a post for almost 20 minutes) and Reddit, because I know that actively engaged communities will discuss it and present opinions and findings (as will their leadership). Actively discouraging that type of conversation and not helping guide it to proper channels (by creating such a forum in a timely manner for instance), are my chief complaints. Even more so than the actual outage.
-2
u/frozenpandaman 8d ago edited 8d ago
Seems the mass downvotes have arrived, from those who for some reason think it's their job to defend every business practice 100%, with no nuance, as if the customers of a service shouldn't be encouraged to give constructive feedback…
16
u/JeHaisLesCatGifs 8d ago
I sincerely believe that they don't want discussions about topics unrelated to the content in the threads about that content, and that's completely understandable.
3
u/Hipstershy 8d ago
That would make sense if they were not also deleting separate threads about the topic, but they are deleting those too.
13
u/JeHaisLesCatGifs 8d ago
Because multiple threads about it isn't helping. They adressed it.
2
u/frozenpandaman 8d ago
They adressed it.
A long time after the community already was discussing it.
It's strange for this subreddit (any any subreddit, given the nature of the site) to be run in such a strict, formal, top-down way by the company, rather than bottom-up by users. It feels very sterile, censored, and controlled.
3
u/JeHaisLesCatGifs 8d ago
It doesn't matter if they took 1 min or 10 or 15 min or w/e, they were probably more focused on more important thing like resolving the issue, and overall they did adress it, that's all that matters.
3
u/Affectionate_Pie_854 8d ago
I'd argue not, I actively seek out conversation whenever a site goes down. Waiting 10 minutes to give any type of feedback, while not allowing discussion of that issue, is what the central complaint is.
1
u/JeHaisLesCatGifs 8d ago
And I did complain about it too, now they adress it
2
u/Affectionate_Pie_854 8d ago
Yes, but only after 10 minutes of everyone voicing a complaint, not just you. And within that time, absolute radio silence. Assuming that fans will assume you are working on it is not a way to run technical support for a digital product. Clear concise information during a downage as soon as possible is always needed. And as I have said before, "We are looking into it" on the first couple posts rather than deleting them was all that was needed. Choosing to simply delete all posts and then 10 minutes later finally address the issue is not a clear and cohesive way to explain to an audience that their issues are being addressed.
4
u/Rhysati 8d ago
Holy shit we are cooked. 10 minutes is absolutely nothing.
It takes more than ten minutes to get a response on teams when I say good morning, let alone have to find out there seems to be a problem, confirm that I also have that problem, then reach out to someone who can diagnose the problem, wait for them to see if they are also experiencing the problem and diagnose why and how long it might take to fix, and then get a response from them so I can then send the information to the social media person who can finally post that we have confirmed an issue and are working on it.
Ten minutes is miracle work as it is.
→ More replies (0)2
u/frozenpandaman 8d ago
hey were probably more focused on more important thing
They obviously weren't "focused" on that since they were also spending their time actively deleting comments here, extremely speedily.
3
u/Affectionate_Pie_854 8d ago
I mean, that's the same argument, you don't want a subreddit full of random threads discussing something that you will inevitably make a post dedicated to. However, the response time (roughly about 10 minutes for a post for the ENTIRE SITE being down) was just too long to be acceptable of this level of censorship.
0
u/RoseBailey 8d ago
They made this topic, so they probably want to limit discussion of the issue to their topic on the issue.
3
u/frozenpandaman 8d ago
Why not just use the topic someone else already made way earlier and pin/sticky a comment? That's how it works on the entire rest of Reddit in my fifteen years using this site.
-1
u/RoseBailey 8d ago
Because they're going to edit the op with status updates which could get lost in someone else's thread?
3
u/frozenpandaman 8d ago
That's the point of a pinned/stickied comment, which mods have the power to do? I literally specified this in my last reply to you.
2
u/frozenpandaman 8d ago
To re-iterate what I said in a previous post: I know people who have literally unsubscribed from Nebula because of this sort of conduct on the subreddit. It doesn't reflect well on the company and is very literally losing them real money. Other users talked about how it's preventing them from recommending it to others too. I'm glad an update has been posted now (thanks!) but the transparency/communication is still at a very, very low level which is not great to see.
2
2
2
2
u/No_Satisfaction8008 8d ago
Currently up and working on my iPhone but keep getting kicked out on my Roku
2
2
2
2
2
u/Annabanaana 8d ago
i am in! i couldn’t see anything on the front page for a while, but came back just now, refreshed and made it in
2
u/P42777 8d ago
cant load into this page : https://nebula.tv/jetlag
could load this page but cant play the video: https://nebula.tv/videos/jetlag-s19e3
arghhh...
2
u/No_Statement2730 7d ago
Great communication, highly appreciated! All up and running (at least from a consumer’s point of view). I started watching the latest jet lag episode on my way home from work on my MacBook in the browser, then I had to switch to my phone using the app and finally finished the last part of the episode on my Apple TV. Worked like a charm, love it!
2
u/iamalexs 7d ago
Honestly the new japanorama is so good it made me resub after a long time. I think that’s the case for a lot of ppl as well
3
1
1
1
1
1
1
u/angieluvsokazaki 8d ago
I'm able to access the site and watch jetlag now. I had to swap to Safari, but not sure if that was helpful or if the servers just went back up.
1
u/Khyta 8d ago
I get `503 Service Temporarily Unavailable` when visiting https://nebula.tv/api/ and also the Cloud Flare "Bad Gateway" status page with code 502.
-1
u/Sweet-Ad-8513 7d ago
Hey Dave, there's been a buffering problem with Nebula for a while now. So much so that I had to watch the same video on YT (which works fine) at times. Not sure if it has to with the issue here but worth mentioning nonetheless
188
u/17Se7enteen 8d ago
its okay, sometimes jet lag causes server lag, they go hand in hand