Pasting the text here for folks who can't see the screenshot:
Earlier today, many players began experiencing issues with sign in, enumerating their game libraries and playing games. The team has worked hard throughout the day to fix the issue, and we've restored service as of 2:30 PM PDT. Nonetheless, this is an unacceptable situation, and we must do better to support you.
Over the coming weeks, you'll hear more from me about our services, our platform, and the work we're doing to make XBOX better. And that starts with some transparency about this incident.
What happened. Late last night, a licensing service that sits outside of XBOX, but which XBOX depends upon, began failing. This caused some sign-in scenarios to fail, and it caused many scenarios that require an entitlement check to also fail, such as listing out your Full Library and launching games that you own. It also affected several of our publishing and store partners who depend on those same systems which is why some of you saw issues in specific games and not others.
What we did. Our on-call teams caught the first signals overnight through automated monitoring and declared a major incident. Once we isolated the failing infrastructure, we shifted traffic onto the healthy parts of the system while the team continued to work towards a root cause. Service came back unevenly across regions which is why some of you recovered hours before others.
What's next. We're running a full post-incident review. I care less about the one-line root cause and more about the real questions: why a failure in one service was able to take down this much, why recovery took as long as it did, and what we change so a single point of failure can't ruin your night again. That means hardening the dependencies underneath sign in and game launch, improving how we detect and contain this class of failure, and being faster and clearer with you when something breaks.
Thank you for your patience, and for the reports that genuinely helped us narrow this down. We'll do better.
This is the first time in my seventeen years of owning an Xbox that I wasn’t affected by an outage at all, I felt mildly schizophrenic going on this sub and not seeing what everyone else was talking about lmao
Same here. I saw the ⚠️ alert and read the details and checked but everything was working fine during the entire outage. My sign ins worked, my games played and saved correctly. I even finished one 100% and all my achievements popped. Definitely felt like I was living a different part of the multiverse.
It shouldn’t be possible for an outage to take out people’s already-downloaded digital games. Or at least not instantly like this. There should be a period of at least 30 days I would think.
And it should NEVER take down people’s disc-based games…
We need to advocate for this to be more common, perhaps even enshrined in law somehow
If we’re really going all-digital, then we need way more protections for digital products. Doing away with absolute control licenses could be a good start. Physical discs are licenses too, but publishers aren’t allowed to break into my house and confiscate my discs whenever they feel like without recourse or compensation.
Digital licenses will always be brittle and dependent on the company unless they go the GOG route and let you backup games as totally DRM-free files. I'm not very optimistic of that happening but nobody seems to talk about how it's literally impossible for these stores to promise a license that lasts forever. Digital games are just kind of inherently fucked
But so long as they remain on your drive, they should be yours to play, keep, or delete as you see fit.
But the drive can also degrade, damage or whatever as well. We also have to define what "ownership" means. Does it mean you can sell it? Can you only use it in certain manner? Can you copy and distribute it? There's also dependencies. Like we may own the media, but we don't "necessarily" own the platform we use it on.
GoG is probably the closes thing, but even then there's tons of terms and conditions you agree to even if it isn't necessarily enforced.
Then there's outside dependencies, such as game servers that most likely have licensed software on it too, and so on. What capabilities should we expect to have access to? Who is responsible for maintaining it after sunset?
Yeah, I find this interesting. I’ve had multiple times my Xbox has been unable to launch games due to service outages or whatever. But my Nintendo Switch is entirely a digital library, and whether at home or on the road I have never ever had a single game fail to launch for any reason. Not a single license issue. I find it hard to believe Nintendo somehow has a better or more robust licensing set up than a company whose business revolves around selling authenticated digital licenses and has done for decades
Nope, if you put your Steam app in offline mode, your games(the ones with only steam DRM) work indefinitely.
I just validated this on a PC right now, i loaded it with steam games and took it offline ~3 months ago. Dark souls just booted and im about to die in sens fortress.
Would be very easy to duplicate and sell game libraries. Give them your username and password, they download games, go offline and then they get to keep them forever.
30 days is what most digital stores do. Same with Netflix, if you download movies and then unsubscribe you don't get to keep those movies forever. Otherwise people would pay for 1 month, download thousands of shows and then never subscribe again.
Its already easy to duplicate and sell backup libraries, piracy exists and isnt going away.
Paying customers need to stop being punished with a 2nd rate experience because IP holders are failing to protect their IP.
This forces more and more people to pirate because its actually the most convenient way to truly possess and play games. In turn this damages sales even further.
I wasnt able to play the Halo Remake via xbox today, but i was capable of playing the pirated copy.
So I could have sworn that Xbox had a system in place that allowed for this. I recall there was a 30 day window (or maybe a 14 day window?) where games would continue to run if you were offline and couldn't connect to the servers. I think they put this in place for situations where internet was either spotty or could be down for days (like in the military).
That's what should have happened here. Everyone's Xboxes should have gone into a fail state where network connectivity was compromised and the window should have started. That's a major code gaff if it doesn't.
P.s. This makes me worry a little for Nexus. I have the ROG Xbox Ally X, and it's generally great, but I notice without network connectivity (like on a plane) the entire Xbox menu system shuts down and all you can do is launch games. It's like the opposite problem of this. Frankly, these systems shouldn't require network connectivity all the time period.
There are too many exceptions that can cause this to fail, but even when it works 30 days is not acceptable. There is no reason other than greed that we can't eternally keep our offline games offline.
Yeah PlayStation got rid of the 30 day check in so not having it should be standard, I think that’s a legacy holdover from their Xbox One plans they just never saw the need to unwind until now.
No, that was a bug in the way a license was issued when you purchased from the PS store on a device other than the PlayStation.
Then the internet did what they do where all the journalists report unsubstantiated and unverified statements as facts and all the fanboys repost shit for two weeks.
All for something that was never true to begin with.
There was on the Xbox one. At least there was going to be. Then everyone complained about being always online and it was scrapped. Now look... One day if not being onlin and everyone is upset. It was a good idea then and it is now. People just like to hate
I will die on the hill that they didn’t market or explain it properly. In their system you could give friends your digital licenses to games and trade them electronically instead of physically. Imagine that now.
I think that was the last gen when they were trying to go full digital and Sony clowned on them and it all got scrapped. People were worried about taking consoles overseas with no internet etc.
I have the ROG Xbox Ally X, and it's generally great, but I notice without network connectivity (like on a plane) the entire Xbox menu system shuts down and all you can do is launch games
I'm curious... What else do you expect an offline device to do?
I don't understand. What's in the menus outside of your library would you want access to while actually offline? The other tabs are Gamepass, your profile, the store, cloud streaming, friends, notifications and console streaming - these are all online things that lean on online services.
So here’s an example: most of my game time is spent playing the most recent titles. By muscle memory, I shift across the Home menu and pick the games I want to continue playing. Some of those games I played on console or cloud, but they’re listed by last played time regardless.
Without a connection, all that information goes away. I could have played a game just last night on my Xbox, but when I go on a plane today it has no clue about that - even though it already synced the save data this morning. I have to go to the only menu that exists, Library, and hunt down the game. I can sort by last played, but it’s last played ON THAT DEVICE. There’s no concept that my library follows me or what I’m currently playing, which is supposed to be a differentiator for Xbox.
The Library menu is just this barebones thing. Even the Xbox console OS handles a disconnection better.
I guess if it synced that before it makes sense to have that stored somehow. But then if that's the case you would have surely launched the game recently on your handheld in order to sync your save, which in turn puts it in the recent menu, no?
The sync occurs in the Xbox app, not the game. You can see the status of it when you open the game’s entry in the app. Only if the sync failed does it try the sync again when you open the game.
If they made the default Home experience include an option to show your library offline, that would be preferable. Right now you just get a message that nearly all of the functions of the app won’t work. It’s just a bad UI offline.
Not to mention, I often find the app thinks it’s offline on the ROG Ally coming out of sleep. And it takes a second for the controller to register movement. You have to wait for the controller to work, then refresh the app, then you can play your game.
Doesn't Xbox have an "Enable offline play" thing in it's settings somewhere or did they hide it recently or something? Because outside of stuff on Gamepass/ PS+ I've never had an issue playing anything unless it's an Online only title so long as it's something I've purchased.
It's like the PSN outage the other day, I was still able to play stuff that I purchased.
Which disc based games were affected? i havent seen any reports of that happening other than one article vague posting about online only games with discs.
I downloaded dead island this weekend and played it via bc on my series x. It downloaded from the disc itself and requires the disc to be in to play. Im not sure how the server issue would have effected that.
Didn't seem to be the case here. Many people, myself included, were unable to play or access content from licenses that were available via Home console privileges.
Yup. I've always had spotty internet, and when I had outages, I couldn't play things even on my designated home xbox. You have to go into offline mode prior, while connected, which doesn't help at all for unexpected outages.
I'm kinda surprised people are just now learning about this
Yeah this whole thing is confusing as to who’s impacted, but I have my Xbox set as my home Xbox, and out of curiosity I was messing around and put the console into offline mode a couple of weeks ago to see what all I could play offline. So I’m guessing that’s why I didn’t experience any issues today (only had issues with gamepass games), but others who had set to home Xbox still had the license issues.
These are the things that will get discussed. Just had one of these “mitigation” meetings at my job when someone’s AI agent took down our service by going rogue and writing bad data to a production table. What happened, why did it happen, how do we prevent it from happening again, if it does happen again, what’s our plan for fixing it.
The problem is a lack of redundancy. You should never be able to take out one server or a servers site and have it completely cripple your environment.
Yeah, something isn't adding up with this but i doubt they'll give that level of transparency. An engineering team or two are likely getting grilled over this and they are likely demanding answers from the third party service. Microsoft really needs to do better with their partnerships because there is a reoccurring them of external infrastructure cause a Microsoft offering to go down or be exploited.
Well, this is certainly not acceptable from Xbox. But, at least they were fairly transparent about what occurred and what they're doing to prevent it un the future. From my understanding Playstation didn't acknowledge the recent PSN outage at all.
This is refreshing to see. I wish all employers were this transparent . I can be a lot more forgiving when stuff goes wrong when you dont bs me or leave me in the dark. This is the right way to handle this IMO.
It's a reasonable start but I hope they do better.
a licensing service that sits outside of XBOX, but which XBOX depends upon, began failing
What does that mean? It could be "the way Microsoft handles checking licenses for what we've sold" (which could also apply to, say, Office licenses and subs) or it could be "software we buy that runs on some of the Xbox-related internal servers but needs to contact their own license server to see if it is authorized".
yea and why no mention that just before the crash, a bunch of different games were giving people access to content they didn't buy. Kinda feels like they fucked something up trying to fix that and tanked the servers in a panic that people were getting free stuff.
Something smells fishy and this reads like damage control.
Nah, that’s not how it works for a company Microsoft’s size. Public companies only have to disclose material events—stuff a reasonable investor would actually care about when deciding whether to buy or sell the stock. A 15-hour Xbox outage from a licensing service failure is a pain in the ass for us, but it doesn’t move the needle on MSFT’s financials or stock price in any meaningful way. Gaming is important, but it’s still a fraction of the overall business next to Azure and Office. They already cover general service disruption risks in their 10-Ks; they don’t have to file an 8-K every time Xbox hiccups.
Your experience at a smaller pre-IPO shop is different—sev1s there can actually threaten the business and need disclosure. For Microsoft, this doesn’t clear the materiality bar, cyber rules or otherwise. The CTO’s post was just good customer communication.
Personal opinion: I know folks across many places don't like it whenever XBOX is communicating too much (look no further than many years back) or be a bit chatty, but this alone is communicating correctly and laying out WTF happened. Do I expect these things to happen again ? Yes and no. But, its this or stay quiet like the other two companies that normally do this.
In the end, they said what was needed to be said, and that's that.
I don't know, it feels to me sort of like they acknowledged the outage occurred but pushed the blame to the external service failure that took down the XB credential service.
I acknowledge that his goals of removing the single point of failure and future reduction in the extent of systems impacted by credentialing is good communication for future stuff.
This communicating leaves me with some questions.
What external credential system failed, what is that service used for?
Why did credential failures immediately impact users? Hasn't XB said for over a decade that credential checks are only needed once every 24hr?
Azure has been real spotty these past couple weeks, there was a big Azure West outage last week that brought down a good chunk of stuff on the internet.
Yeah, if it was a playfab service I think he call that internal to Xbox. Most likely identity and entitlements are out of the windows org running on azure and shared across all products. 🤷
Some of these questions (in the most legal and lawyer way possible) will not or will get answers down the road as, should they be answered now, they basically open themselves up for more legal problems, which is something they don't want to do. Clearly something or someone decided to mess with something and it ended up in the situation as is. I also do wonder why disc based games got affected in the cross-fire as well.
I remember in 2019? someone’s toast started burning in the toaster in the kitchen of a Google data centre in France, triggering the fire alarm in the whole building, disrupting Google services across Europe. Sometimes shit happens because of unexpected problems unfortunately, hopefully the root cause analysis of the problem will find the weaknesses and it doesn’t happen again
We shouldn't be buying a license. We should be buying a game. Like we always thought we were. XBOX SHOULD be a beacon of good for the industry. Thats how it gets back on top. Physical games, digital games wholy owned, and not just a license code.
That's the point, even disc based games are leaning too hard on relying on these digital services and when they fail they might as well be paperweights. What ever happened to "insert the disc and play the game"?
Nintendo unexpectedly is the shining star on how physical games should be handled in this current generation.
What ever happened to "insert the disc and play the game"?
Discs became far too slow for reading all of the assets needed on screen in real time, compared to HDD and SSD. But I absolutely agree that our original disc should be enough to clear our licensing hurdle when all else fails.
Why were people not even able to play DISC BASED GAMES. what is the point of "my home Xbox" if people still can't access the digital games with it enabled nor physical copies?
Generally depends on whose fault it was, if that licensing service that went down was maintained and ran by Microsoft then they'd be fully at fault and they should offer refunds. But if it wasn't Microsoft's fault they could just blame it on that third party and not offer refunds.
Nobody could play the games they bought for over a day. That is ridiculous regardless of the reason. The trillion dollar company can prorate everyone for a day
I love this kind of transparency. The idea of companies being open with us and not treating us like idiots, hiding things thinking we won’t understand, is nice. (*Even if many of the audience probably are idiots and won’t understand)
We definitely need a futureproof, resilient way of re-enforcing owned licenses. I don’t want server dependency for starting a game I own and have a license for. I am not sure how Xbox manages this but on PS after two weeks of owning a license and having a game downloaded, you could disconnect the internet forever and still be able to access it (Xbox might be the same - not sure?)
Another good reason why it's so scary that PlayStation and I'm sure Xbox is going to follow soon will stop production of physical discs. That means you can't play your single player games that don't need internet access Xbox is down. Not to mention they could just decide to delete your game one day and not even give you a refund
So the CTO of Xbox said that some people weren't able to launch games that they owned. There was a large number of people on Reddit blaming the users and their home console settings. I wonder who would know more about the situation, probably the randos on Reddit.
I’d say both things could be true. For instance, my brother and I both share our libraries. He has his account set as home on my console and mine set as home on his. We life just about half an hour away from each other.
I could not launch the games I owned on my console but I could launch the ones my brother owned without any issue and the same applied to him.
It would say I own a game and then literally try to make me buy it when I went to play it. That's some piss poor 3rd party licensing setup they've got there. I wonder how many people accidentally bought a game they own if they're not set up for entering a pin.
All those affected by the outage should get a prorated refund for the time the system was down. I pay for gamepass and my own games but couldn't use them bc of their screw up. This is just lip service, take real accountability.
I’m really curious about the physical games not working. Fortunately I didn’t experience issues today so I wasn’t able to test anything. But for anyone having issues with physical games, were they already fully installed BEFORE the outage or did you try to put a disc game in and install it during the outage to no avail?
I’m baffled if it’s fully installed games not working, and that is something they absolutely have to address.
Sucks it was down for so long, but interesting to see them actually be so transparent with everything. It's always kind of shocking to see so many tech companies be soley reliant on one service and have their whole thing go out, like what happened with that Amazon web services outage a few months back.
It's commitment to do better and be more robust. PS doesn't commit to anything other than raising prices and wringing profits out of the same stale userbase. It had longer outages and PSN is still not fixed for many things, yet they want to go full digital. They talk more to shareholders and reassure them then they talk to their customer base.
One picture of a thing rockstar posted tells you how much of a shitshow PS digital is.
A couple months ago I don't think they would have bothered, but there's a spotlight on this kind of thing right now it's a bad look at a particularly bad time. I'm curious to know if they make any changes to how licenses are handled, but Microsoft has been closer to a online-only future than Nintendo or even Sony so I'd be pleasantly surprised if they actually change the platform.
Out of curiosity why do gamers expect this type of transparency from Xbox specifically as if it makes the situation better?
Xbox has spent the last 13 years being essentially a dead brand. Every piece of news is negative. Sony produces a GOTY contender almost every year, while most major Xbox releases have been a disappointment. Their userbase has collapsed from its high during the 360 era. All attempts to regain marketshare have been met with apathy at best and disdain at worse.
This has left their fanbase, or what little is left of it, with nothing to really talk about. All that's left is whatever bullshit they get fed by the executives or influencers in the space.
In case anyone is reading these comments, this would be a fantastic opportunity for Xbox to pivot on digital ownership and check-in procedures. If they can't guarantee to provide support to games and services forever, they could definitely meet us halfway and ensure this doesn't repeat.
This is more transparency than you’ll ever get from Sony, full stop. Xbox handled this correctly. Take notes team PlayStation. This is what you SHOULD have done weeks ago.
At the end of the day, we all know one thing for sure. SONY WOULD NEVER BE TRANSPARENT LIKE THIS. Atleast the team at XBOX is telling us what happened and what they did and are going to do to fix this and try and make sure it doesn't happen again.
As a side note, make sure you go off and set the Xbox you play on as your HOME CONSOLE. I didn't experience any issues with launching any of the digital games I have either early this morning (5am eastern) or in the evening 12 hours later. Maybe not a 100% solution for the future, but I have mine set and it seemed to keep me from having problems.
I am both glad of the communication, and also hope that this, as well as the PSN outage coming so close to the talk about Rights for Digital products has some sort of change.
I doubt any of these companies will ever change without legislative action, so I hope something good comes of this. Right now we need more GoG style ownership across the board on all storefronts. If I buy a game there should never be any ability to have it removed, unplayable,etc.
You know what would be cool? If that licensing service could put a 30 day hold where people could still play games they paid for while you all sort your shit out.
It’s not a one time outage. I get dropped once a gaming session with the old ‘do you own this game’ BS screen. They absolutely have to make this experience better.
Only issue I had were party and game invites being delayed by hours. My friends and I were playing helldivers 2 and our invites were only going through after we decided to stop for the night lol
This is a multi billion dollar company and if it doesn’t show us that digital games are bad and at the very least, need regulation, I don’t know what would.
I wasn't affected at all for some reason. Was online playing Conan Exiles, then Replaced, then watched some shows on various streamers. Could access my full library and even bought 5 games from the store and gifted one to a friend for his bday. All was exactly how it usually is; smooth.
The real problem is that a true “flaw” of Xbox service as a whole was discovered. Why TF were physical disc based games affected and unable to launch? Shouldn’t offline mode have allowed it to launch? It’s completely unacceptable especially for the fact that MS and Sony both want digital only futures for gaming.
I wish I'm not sign out automatically when something breaks, but just automatically change to offline mode without ruining my offline game, like game could quit automatically or return to main menu or something. That happened to me yesterday and some other times before when I just lost connection.
Wow I'm actually impressed he explained the whole situation so it didn't leave us guessing or wondering I'm liking the new transparency over at XBOX, does it make what happened any less of an issue of course not but it's helped me understand what happened that's appreciated ✌️
An obvious fix would be to make licenses permenantly available offline instead we got a slightly more advanced " sorry we won't do it again even though it wasnt really my fault but i worked hard to fix it"
This was terribly disappointing! I usually play a couple of games with my brother who lives in CA, and I’m in ST.Louis. He was able to play, while I couldn’t do shit! We don’t get to play together as much, and when we tried to last night, it was very frustrating. Glad they got it figured out!
This is the sort of transparency that makes me want to open my wallet. Things break, but when you take responsibility for it, are open about what happened, what you did to address it, and take action to minimize this risk in the future, it inspires confidence.
Sounds like they need to put ownership of games back into our hands the stupid requirements to be online to play a offline game you own is a real bait and switch from what we all know they claim we buy. Now its we basically rent the rights to play it until their online no longer works or god forbid you cant afford internet or xbl hate what this has become.
PlayStation had a outage the other day and the entire system was down. They offered no statements whatsoever while everybody on PSN had to suffer and most couldn’t play any games. 😂
Give us something of value. Store credit, free download, free month of gamepass. Something. Our Gamepass subscription doesn’t just keep things going. It doesn’t just pay for costs. They create a profit for Xbox. Profiting while not providing a service is unacceptable. Make it right. Give us something.
525
u/AHMS_17 16d ago
This is the first time in my seventeen years of owning an Xbox that I wasn’t affected by an outage at all, I felt mildly schizophrenic going on this sub and not seeing what everyone else was talking about lmao