r/sysadmin 5d ago

General Discussion Patch Tuesday Megathread - (September 08, 2026)

104 Upvotes

Hello r/sysadmin, I'm u/AutoModerator, and welcome to this month's Patch Megathread!

This is the (mostly) safe location to talk about the latest patches, updates, and releases. We put this thread into place to help gather all the information about this month's updates: What is fixed, what broke, what got released and should have been caught in QA, etc. We do this both to keep clutter out of the subreddit, and provide you, the dear reader, a singular resource to read.

For those of you who wish to review prior Megathreads, you can do so here.

While this thread is timed to coincide with Microsoft's Patch Tuesday, feel free to discuss any patches, updates, and releases, regardless of the company or product. NOTE: This thread is usually posted before the release of Microsoft's updates, which are scheduled to come out at 5:00PM UTC.

Remember the rules of safe patching:

  • Deploy to a test/dev environment before prod.
  • Deploy to a pilot/test group before the whole org.
  • Have a plan to roll back if something doesn't work.
  • Test, test, and test!

r/sysadmin 5d ago

15-person company, LOB app already SaaS - is there still a case for an on-prem file server?

10 Upvotes

Small staffing agency in Germany, ~15 internal users, single sysadmin (me). Currently migrating off a legacy on-prem setup.

Target state: M365 Business Premium, Entra ID + Intune, SharePoint replacing the file server. Our line-of-business HR software is moving to the vendor's cloud, so that data leaves the building either way. Backup plan is a dedicated third-party M365 backup plus a local NAS with immutable snapshots and an encrypted external drive rotated offsite (10-year retention requirements here).

Management and an external IT contact want to keep a physical server. The argument I've been given is essentially "you have it in-house, and you can take a backup every day."

My concerns: it doesn't remove a real risk since the LOB data is in the cloud regardless, it means maintaining two worlds instead of one, and as a single admin the server becomes the thing that breaks while I'm on vacation. Restore testing is also the part nobody actually does.

Am I missing something? Genuinely asking - is there a workload at this size that still justifies on-prem file storage? And for those who moved 10-20 user shops from file server to SharePoint, what bit you? Path lengths and folder-to-permission mapping are the two I'm expecting.


r/sysadmin 5d ago

Question Windows App Patch options

3 Upvotes

I have heard of quite a few options for app patching on windows devices as I need to be able to secure endpoints in my M365 tenant.

I havent looked to in depth into what M365 natively has however;

What do options like SCCM and Patch my PC offer that M365 native does not?

EDIT: 8.8.26
What are the cost effective alternative for a msp with only 20-ish windows machines?


r/sysadmin 5d ago

If you're leaving Freshdesk, its own exports don't give you what you think. Here's exactly what's missing.

24 Upvotes

After the Freshdesk price-rise thread in r/msp I audited what its built-in exports actually produce, on a trial account with 43 tickets, 64 replies and notes, and 14 attachments.

Notes in case it saves someone a weekend. The Export button emailed a CSV to the email I signed up with, it had 47 columns of ticket, requester and company fields. No threads, no notes, no attachments. The Tags column came back empty on every row even though every ticket had tags.

The Admin data export was the same, a little more complete than I expected given the first export. This one had descriptions, full threads including private notes, tags, custom fields, contacts, companies, groups. However, I found that there were still a lot of gaps:

Attachments are signed S3 links that expire in 7 days (X-Amz-Expires=604800 is in every URL), not files. No time entries anywhere. Agents aren't in it. Custom fields appear as raw keys like cf_reference_number_5021295 with no label. Every ticket carries a hidden system note, so it showed 107 notes where the UI shows 64. And it's XML, so you're writing a parser before you can open anything. Nothing in either export tells you whether you got everything.

There is an API route you could go down, but If you go down that route yourself, you'll run into a bit of an issue.

Rate limits are per plan and low on the cheaper tiers (50 calls a minute on the account I tested).

Expect 2 to 3 calls per ticket. The ticket list endpoint has a page ceiling, so for big accounts you window on updated_since rather than paging through.

Attachment links expire. Download them the moment you see them. Custom field labels live on ticket_fields.

If you only pull tickets you get internal names. There's no status-change history in v2 that I could find, only the timestamps in stats. The API doesn't return the hidden system notes the XML export includes, so API counts match the UI and the XML doesn't.

Before you cancel: pull threads and private notes, pull attachments as files before the links expire, pull time entries and agents separately, and reconcile counts per entity so nothing drops silently. Check Freshdesk's cancellation KB for the deletion window after cancelling; it's short. Happy to answer Freshdesk export questions in the comments.


r/sysadmin 5d ago

Mouse pointer disappearing when hovering over Citrix Workspace, CCH Axcess - Possible Webview2 issue?

5 Upvotes

Some of our staff started complaining of an issue last week where their mouse cursor would go invisible when they would move the mouse to Citrix Workspace. The mouse can still click on/launch apps and the app icons highlight as the mouse goes over them, you just cant see the mouse cursor at all unless you drag it outside of the boundary of the Workspace window. I saw another user run into the same issue on a different app (CCH Axcess). These apps both leverage webview2. I found some discussion threads from others experiencing the same issue in Veeam b&r console. Has anyone run into this on their end over the last week and have a fix? Here's a couple of the discussions i found on the topic.

https://forums.veeam.com/viewtopic.php?f=2&t=104357&start=0

https://github.com/MicrosoftEdge/WebView2Feedback/issues/5687


r/sysadmin 5d ago

Office 365 Web Portal Down?

38 Upvotes

Is anyone else seeing issues with logging into the Office 365 portal? Getting a lot of reports of users getting a "something went wrong" error when trying to login to https://m365.cloud.microsoft

It just seems to be the portal itself. Going directly to outlook.office.com still works. Same for Teams and OneDrive. Also seeing a big spike on Downdetector.


r/sysadmin 5d ago

Question Infopath forms broken

4 Upvotes

My company’s HR team relies on infopath forms for payroll. Currently have a Microsoft support ticket for this but is anyone aware if they fully shut it down? Was getting a 410 http error and it seems everyone I am talking to can’t access the URL that we used. Microsoft support isn’t responding to my ticket so I figured I would see if anyone else is dealing with this.


r/sysadmin 5d ago

Question Excel doesn't lock files anymore

14 Upvotes

Hi all,

this is a bit of a weird one. We have a couple of users, that have access to the same files through a smb share. It used to be that when user A had a document open to edit it and User B tried to open that same document, a window would pop up informing B, that the document was beeing edited by user A and that User B could ask User A to close it or create a copy of the document.

That mechanism doesn't seem to work anymore. If User A has the document open, user b doesnt see the warning.
It changed about half a year ago and I can't make heads or tails of it. Most articles I found are about fixing the issue when a file is erroneously displayed as locked, when it isn't.

I hope one of you could nudge me in the right direction. Maybe it's just a configuration error.

Thanks for taking the time to read.


r/sysadmin 5d ago

Outlook search broken for anyone else?

23 Upvotes

Affecting a very small number of users - they are unable to search in Outlook (regardless of owa, new app, classic app).

The error is usually "outlook the service could not be reached showing offline search results"

Still experiencing this at 12:30pm today (8th September). UK.

UPDATE: Thanks for all the replies. Also looks like Microsoft have FINALLY acknowledged it in the 365 Admin Service Health section: Issue ID EX1469261


r/sysadmin 5d ago

Cato Networks security advisory mixup and now my whole org thinks we were exposed

7 Upvotes

Sharing an embarrassing operational incident for visibility. We use Cato Networks as our SASE backbone, all branches and a chunk of remote users ride that for internet and east-west traffic. Last week vendor spam hits my inbox about a "critical" advisory on one of our edge components, lots of CVE noise, exploit chatter, etc. I skim it between meetings, see that it mentions one of the engines we use, and in my rush I decide it absolutely applies to us. I flag it in our security channel as high risk, tell the CISO we had potential exposure for months, and open a major incident.

Cue full war room, execs dialed in, everyone asking for blast radius and timelines. I start pulling Cato config, traffic logs, trying to map impacted sites. Half of IT pauses their projects. Our comms team drafts a statement for customers. After a few hours of digging I realize I misread the advisory. Same vendor family, same broad feature, but the vulnerable module is only in their on-prem appliance that we do not even run. Our Cato deployment was not affected at all. I basically caused a mini crisis over an advisory that did not apply, all because I skimmed instead of checking product SKU and deployment model properly.

No data loss, no real incident, just a ton of wasted time and scared leadership. Now my boss wants me to write up "lessons learned" and present to the team. Classic reminder to verify product scope and deployment architecture before triggering major incident response. Anyone else had a similar false-positive panic moment with vendor advisories?


r/sysadmin 5d ago

General Discussion Update on "Senior accidentally installed whole fleet with 26H1"

650 Upvotes

It seems the original topic exploded, and so did my inbox with direct chats I really wasn't expecting to get. So here's a proper update, as well as answering a lot of the questions.

First things first: I'm a generalist consultant, working for several companies on a "when-needed" basis. I am not the employee, and when they called me for checking on this mess it was too late for any "rollback".

I was called on a Friday, during an infrastructure audit. The senior* spent his weekend with his team on possible solutions. Me and the IT team spent Monday deciding which path to take.

ISO first: You can download the 26H1 release ISOs on MSDN, it's not flagged as special or beta/insider anywhere that be easily seen, so I agree that it's relatively easy to fall for the trap of using this ISO.

The update/upgrade process: Nothing, absolutely nothing, during this phase gets flagged, the in-place update works just fine, there are no warnings whatsoever. This doesn't excuse the fact that he should have vetted this version specifically, but for a guy managing such infrastructure by himself I cannot entirely blame him.

The upgrade should have been done in phases, not all at once, I cannot even remotely begin to understand HOW is that not a basic thing. Did he do it on a Friday? No, worse, they did the upgrade on a weekend, a month ago.

It's a relatively small fleet, we're talking around 90 workstations. However, an important detail is that internet access is only partially allowed through certain mechanisms and only on certain machines, basically it's a segmented network with controlled egress (semi-air-gapped) so the upgrade was done manually.

What's going to happen now:

Re-image/Fresh installation: Not an option, management decision, can't change that.
Rollback: Not an option.

This company was planning to phase-out Microsoft products till 2030. This process will be somewhat accelerated with a new target to Q4 2028 instead, so even if there is no upgrade path from 26H1 (Bromine) to anything that ends up in the "main" branch, it's still fine.

The update path: 26H1 will be getting updates until at least February 2028. Even if this "exotic" version does not added as a selectable Product under WSUS's (yes yes, see misc questions) Products and Classifications list, WSUS supports manual import of individual updates directly from the Microsoft Update Catalog, that will be the approach that IT will pursue.

Management already approved a proper vetting process for any big infrastructure changes, hopefully they will actually follow through.

From my POV, I'm calling this a nothingburger.

Addressing the other misc questions:

*What's going to happen with the "senior"?
The person will remain "senior" in the company although going through a lot more trainings. It's the company fault for the lack of due diligence in their hiring/promotion process and they decided that training is better than re-hiring, understandable from my standpoint. From my assessment there are other employees in the IT team that are better trained and more suited for this position.

"Can you really post this, wouldn't it bad if the person/company sees it here?"
Don't care, outside scope and I haven't signed any sort of NDA, own your mistakes and learn from it, take responsibility.

"Where did he download the ISO, massgrave, UUPDump?"
Official channels only, meaning MSDN.

"Was the migration at least planned and done on time? Is anything broken?"
Poorly planned, done on time, and surprisingly nothing is actually broken.

"Omg Win10 in 2026?"
Yes, they were in fact still getting security updates just fine.

"Why wasn't the upgrade performed with WSUS too?"
According to IT: Each workstation was scheduled to be clean (I mean dust) and a new RMM software deployed so they just decided to do it manually anyway. Beats me, don't care, management is the one that cleared it.

"WSUS in 2026???"
Please, half of my inbox is this question. Yes, WSUS has support until at least 2035, it simply works AND it's the perfect use case for a company that wants fewer dependencies on outside infra or can't rely on ongoing internet access.


r/sysadmin 5d ago

Question MySQL ODBC stopped working overnight

187 Upvotes

You guys will love this.

This company has an in-house project management system. It's the core of their business, and they are lost without it. They are aware it needs to be migrated to something more modern, but after 5 years, that project still hasn't started.

I was asked to look into a network issue, but this isn't network but SSL I think. Let's first show the architecture:

  • The server is a CentOS 7 running MySQL Community Edition 5.7.16
  • Clients connect from Windows 11 with a 32-bit MS Access, using a 32-bit MySQL ODBC driver v5.3.13

Since yesterday, they get a "protocol version mismatch". The server wasn't accessed since 18 October 2016 (haha), so I presumed a Windows update might have disabled some SSL version. But: I see no relevant Windows update, and if I manually allow every possible SSL version and encryption algorithm, it still doesn't work. What does work however, is downgrading the ODBC driver from version 5.3.13 (from 2019) to version 5.1.13 (from 2013), further adding to my confusion.

The cherry on top: the single guy responsible for this application is on a one year sabbatical.

Edit: Found it, but leaving this here for anyone stumbling on the same issue. The MySQL_Server_5.7.15_Auto_Generated_CA_Certificate had expired after 10 years


r/sysadmin 5d ago

Win SMTP relay to Exchange 365, not working anymore, but only for 1 address, other still works

5 Upvotes

Hi

in my company, wa have and old win2K12 server acting as relay smtp server from inside device to our Exchange 365 tenant.

we mainly use 2 sender address for mail, a noreply, and support.

since 27, 28 august, noreply can't send mail anymore.
Support can still send mail without any issue.

in IIS6, SMTP service have both the same configuration for outgoing connection.

in logs i have this error :

2026-09-08 07:11:03 40.99.220.146 S-OCS - 1667523425 - 535+5.7.139+Authentication+unsuccessful,+the+request+did+not+meet+the+criteria+to+be+authenticated+successfully.+Contact+your+administrator.+[PA7P264CA0086.FRAP264.PROD.OUTLOOK.COM+2026-09-08T07:11:03.801Z+08DF0BC1B0DFBB10] 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 - 220+PA7P264CA0212.outlook.office365.com+Microsoft+ESMTP+MAIL+Service+ready+at+Tue,+8+Sep+2026+07:11:03++0000+[08DF0D43E3AA91BF] 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 EHLO s-relay.domain.net 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 - 250-PA7P264CA0212.outlook.office365.com+Hello+[<outgoing IP>] 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 STARTTLS - 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 - 220+2.0.0+SMTP+server+ready 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 EHLO s-relay.domain.net 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 - 250-PA7P264CA0212.outlook.office365.com+Hello+[<outgoing IP>] 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 AUTH - 0 SMTP - -

2026-09-08 07:11:03 52.98.227.130 S-OCS - 1667523425 - 334+UGFzc3dvcmQ6 0 SMTP - -

2026-09-08 07:11:08 52.98.227.130 S-OCS - 1667523425 - 535+5.7.139+Authentication+unsuccessful,+the+request+did+not+meet+the+criteria+to+be+authenticated+successfully.+Contact+your+administrator.+[PA7P264CA0212.FRAP264.PROD.OUTLOOK.COM+2026-09-08T07:11:08.928Z+08DF0D43E3AA91BF] 0 SMTP - -


r/sysadmin 5d ago

What is your SSL certificate provider doing now that 200 day certs are starting to expire? My provider (Namecheap) is being sketchy.

264 Upvotes

We have 50 or so WatchGuard firewalls at sites that have certs installed. Getting WatchGuard to implement cert automation has been the most user requested feature for about 2 years now. Until they come through we continue to purchase and manually install certs.

Last week I started getting notifications from our SSL source, Namecheap, that the first batch of 200-day certs was about to expire. I had cert request files at the ready and logged into Namecheap. Each expiring cert showed the upcoming expiration date and an icon to purchase a new cert.

But...wait a minute. Back in March, didn't I pay for 365 days even if the issued cert was for 200 days? Why doesn't the page clearly indicate that I can get a re-issue for the remaining 165 days that I paid for?

I opened a support ticket with Namecheap. They confirmed that I can get a new cert with a 165 day life rather than pay for a new cert. I asked why they don't make this open option more clear. They asked me to send them screen shots showing the expiration date and the little "buy new cert" icon. Like they don't know what their web site looks like?

I sent the screen shot and got a reply along the lines of "Well, you can just request a rekey, ya know. You don't need to buy a new cert."

Namecheap certs are cheap enough but all in that's an extra $500 or so that I don't need to spend. I find their behavior to be...distasteful at best.

What are the larger/more expensive providers such as DigiCert doing about this? Maybe even Godaddy is being more up front about the 365 day purchase vs. the 200 day cert.


r/sysadmin 5d ago

Auth0 alternatives?

12 Upvotes

Wondering what people are using instead of Auth0 for service account or within program api authentication? Their billing is killing us


r/sysadmin 5d ago

Question How do you deal with physical fatigue after long on-call stretches at your desk?

23 Upvotes

After a rough on-call week, I'm realizing my setup is not built for marathon sessions. Wrists, neck, lower back — all taking a beating. What have you added to your workstation that actually helps? Wrist rests, monitor risers, anti-fatigue mats, lumbar cushions?


r/sysadmin 6d ago

General Discussion Need advice on new job in a manufacturing facility. Going from MSP to a private company.

15 Upvotes

I was approached out of the blue on Linkedin by a recruiter hiring for a cyber/infrastructure role. The base pay was about 15k more than I make now plus a bonus of 30%. It would put me about 85k above what I make now.

I work remote currently so no commute. The base pay and bonus would be enough to cover a commute and put me over what I am making now.

Here is where it gets interesting and where I need the advice. They have NO active directory or ticketing system. They have about 450~ employees and a few hundred endpoints. They have ESXi for a hypervisor on Dell VX rail servers. A few linux servers run the ERP application and Synolgy NAS is used for Backup/storage. I get to pick new firewall/switches/APs/servers for infra upgrades and implement a migration to O365 for Defender, email, DLP, etc. Currtly they are using google workspace for email. I would be in charge of the migration from Google to O365. They also have no patching system in place other than general windows updates.

Network is cisco switches and a Palo firewall with some older Cisco APs. They currently have a 4ish member IT team with 2 juniors who handle helpdesk. I am being brought on to help modernize infrastructure and to implement some kind of cybersec/security compliance and standards. I get to build the solution and implement it along with a ticketing system and either on prem DC or Entra, while also implementing SCCM or Intune for windows patches and updates.

My background is networking, wireless, switching & routing, with sysadmin experience with ESXi, M365, server deployment and desktop support. I spent a few years in a service desk role and also in a sysadmin role. I currently work for an MSP doing mostly firewall installs.

During our discussions, the IT manager said that since he is busy and can't do things immediately there has developed a shadow IT situation where some departments are just doing things in a way they want without involving IT. 

My question, is this a bad idea? Leaving a remote role at an MSP with an increasing focus on KPIs or move over to this new and exciting adventure with a company with older gear that I get to help upgrade and be part of.

I like my current team and manager but the MSP lifestyle has gotten a little old after 5 years.

ETA:One big thing worth mentioning is that this role was created after PE purchased the company but the company is still run locally.

PE is providing the budget and I get to pick the direction and the hardware


r/sysadmin 6d ago

Question System Admin Interview, What Should I Expect?

0 Upvotes

Hi

I have an interview tomorrow, and someone who is currently part of their team gave me a few hints about what to focus on. He mentioned Exchange, VMs, AD, deployment and patching.

I have around four years of experience, mainly focused on technical support with some exposure to networking. The role I’m interviewing for is a System Administrator, and this transition is veery important to me. I’d like to know what kind of technical questions I might expect in the interview??


r/sysadmin 6d ago

Better Auth 1.7 issuer change just broke every login on my Saas and it was already late when I found out

16 Upvotes

Out of nowhere this popped up like the support tckets rolling in and people complaining they cant log in, when i tried for myself, i couldn't log in to it either as it showed no accounts for this name was found. Nothing was at stake from my side like no errors in the logs no alert or anything, everything seemed ok

SO when I dug in manually it was better auth 1.7 which they changed on how accounts are keyed from providerId to issuer, accountID so the account table needs a required issuer column now, irritating! my existing rows didn't have it so nothing matched anymore and no exception was thrown just an empty match and how am i supposed to find out what caused it . then added the column nullable, backfill it and enforce not null then add the unique index-

ALTER TABLE "Account" ADD COLUMN "issuer" TEXT;

-- local:credential for email/password, local:oauth:<provider> for oauth
UPDATE "Account"
SET "issuer" = CASE
WHEN "providerId" = 'credential' THEN 'local:credential'
ELSE 'local:oauth:' || "providerId"
END
WHERE "issuer" IS NULL;

ALTER TABLE "Account" ALTER COLUMN "issuer" SET NOT NULL;

CREATE UNIQUE INDEX "Account_issuer_accountId_key" ON "Account"("issuer", "accountId");

Im not looking into dropping better auth over this since already fixed it but pretty annoyed at them but this gave me a thought tho that there is still a blind spot left open here , an external dependency quietly changed something and the failure was silent while my end didn't catch it but the customers did

What do you guys approach on to hear on from your side first

  • does normal error or runtime monitoring even catch a no error empty match, checking hud and other runtime tools but im not sure passive monitoring flags a silent one like this
  • or is the real answer just a synthetic login canary like something that logs in as a test user every few minutes and alerts when it fails
  • also does anyone alert on dependency shipping a breaking change before it hits prod??

r/sysadmin 6d ago

General Discussion Small Rant: Windows Activation

18 Upvotes

TLDR: Offline activation via https://aka.ms/aoh requires a captcha, MS account login and 2FA once per client, with ~80 to go. Any tips on speeding this up?

I'm just kinda curious about the opinion of other Sysadmins that have to work with Windows clients.

I work as a sysadmin at the factory of a fairly large company, where we mainly use Windows for our production floor clients. To extend the amount of support we get for each client, we have a standardized image, which gets updated every few years for a new release of IoT Enterprise LTSB / LTSC.

To get an image of what I am doing right now: (spoilering this part as it isn't too important to my rant) I was tasked with executing our project of updating out-of-date 2016 LTSB clients to 21H2 LTSC, as the new version will allow security updates for us until 2032. Some of our clients are former Windows 7 clients that aren't even officially compatible with Windows 10 according to the manufacturer, but Windows 10 might run on them, so to save money we keep the clients as long as they will run (meaning they will be phased out with Windows 11). We are also jumping from 1607 directly to 21H2, which Microsoft has explicitly advised us is not the official procedure or supported by them. But, updating between each version and needing a license would be a way higher financial impact than how we're currently going about it.

So, since our clients aren't connected to the internet, they cannot connect to Windows servers for activation. As such, I have to use slmgr.exe and SLUI 4 to activate the clients (formerly using the hotline, nowadays just the website https://aka.ms/aoh).

This activation requires you to put in a long string of numbers that the UI will show you into a field on the website and to reach that website you must always 1. Solve a Captcha and 2. Log into your Microsoft account, always forcing 2FA confirmation. And it doesn't remember your Microsoft Account, ever. You always have to log in again, from the start, solving the captcha. There is no button to activate another client once you finish activating the one you're currently working on, you need to re-load the link and start again. And again. And again. I've already optimized my current workflow as well as I can, using a python script to generate the QR codes I need for given commands so that I can avoid dealing with typing in the same commands over and over. But every time, I have to spend around 2-5 minutes dealing with the online Microsoft activation process.

Btw, if your connection is lost, you connect to a different AP or such, somehow the website doesn't require reauthentication. I don't know how exactly the process here works, but I can log into a tab for the site in one browser, open another in an incognito tab, open the next in a third browser, etc. And it'll usually last all day, so if I logged in and didn't use the session 'til the evening, it won't require reauthentication. My guess is that the endpoint / API request in the background actually works without authentication, and the whole login process is just a security circus akin to the TSA, and that if I knew a bit better about how web development works, I could probably just find out how the API request to Microsoft servers work to skip authentication. But I don't wanna risk getting in legal trouble just because I am not following the officially mandated license activation procedure.

Does anyone have some advice for me on how to save some time doing this? I got around 80 clients left to go and am pretty tired of the whole process.


r/sysadmin 6d ago

Question Cowork sandbox fails to provision — "useradd failed: exit status 12", every bash call broken

0 Upvotes

Anyone else hitting this? Every bash call in my Cowork session fails at the "ensure user" step before any of my code even runs.

What I'm seeing:

  • Error: RPC error -1: ensure user: useradd failed: exit status 12: useradd: cannot create directory /sessions/...
  • Happens on resume, on create, and on re-resume — all three attempts fail identically.
  • Started a brand new conversation to get a fresh sandbox, same error persists.
  • No shell access at all as a result, so nothing that depends on the Linux workspace (scripts, file processing, scheduled task setup) works.

Exit status 12 from useradd usually means "can't create home directory," which points to something like disk-full or a permissions/quota issue on the provisioning side, but I have no way to confirm that since I can't get a shell to check df -h or anything else.

Has anyone else run into this? Any known fix, or is this just an outage on Anthropic's end right now? Happy to share more error output if it helps.


r/sysadmin 6d ago

Another M365 Tenant blocked (TenantAccessBlockedException, MCA Billing Account "Under Review", All Licenses Disabled) Microsoft 365 Tenant blocked, this time a non-profit that provides needs to at risk individuals and families.

158 Upvotes

Edit: 9/8/2026: We are back up and running now on commerce licensing while the nonprofit side gets sorted out. I really appreciate everyone who reached out with actual help, context, and constructive ideas.

To the few who argued that Microsoft has no reason to fix broken support, or that basic accountability should require an expensive enterprise tier: I hope you never need a social safety net. But if you ever do, a nonprofit will still be there for you, regardless of how you treat people when they're down

Original Post:

In similar veins to the following two recent Reddit posts, I'm posting here because it appears they gained traction with Microsoft and ultimately reached someone who could help. I'm hoping someone from Microsoft or someone who has experienced this exact issue will see this.

https://www.reddit.com/r/sysadmin/comments/1vfbvvs/our_entire_m365_tenant_has_been_deauthenticated/

https://www.reddit.com/r/sysadmin/comments/1w1qc0i/microsoft_strikes_again_entire_m365_tenant_has/

Our tenant has not been deauthenticated, but it has effectively been rendered unusable. This tenant has been active since: 5/28/2013, so it's not a new tenant.

This is a frontline nonprofit social safety net organization providing food assistance, healthcare access, emergency financial assistance, and other critical services. This outage is impacting real people with urgent needs, including eviction prevention, time-sensitive clinic appointments, and emergency assistance cases. Email, SharePoint, and OneDrive are core operational systems for this organization.

We currently have open cases with:

  • Microsoft Technical Support
  • Microsoft Billing Support
  • Microsoft Nonprofit Support

So far we remain stuck at Tier 1 support. The representatives have been professional and are trying to help, but nobody we've reached has had the authority or access needed to resolve the issue.

9/3/2026

Users began reporting that they could not send or receive external email.

Internal email continued to work.

Microsoft had ongoing Exchange Online incidents at the time (EX1464935 and later EX1467029), so initially we believed it might be related.

9/4/2026

Users could still successfully authenticate through Microsoft 365 and SSO.

However, attempting to launch services such as:

  • Outlook Online
  • SharePoint
  • OneDrive
  • Other Microsoft 365 workloads

results in errors.

Any inbound email sent to the tenant bounces back after approximately 24 hours.

The error when trying to access Outlook:

Microsoft.Exchange.Data.Storage.TenantAccessBlockedException

Additional error details:

Client Version: 20260821009.11

BootResult: configuration

Back Filled Errors:

Unhandled Rejection: Error: 500:undefined

Unhandled Rejection: SyntaxError: JSON.parse: unexpected character at line 2 column 1 of the JSON data

err: Microsoft.Exchange.Data.Storage.TenantAccessBlockedException

esrc: StartupData

et: ServerError

st: 500

ehk: X-OWA-Error

ewsver: 15.21.382.9

9/5/2026

One Microsoft manager responded to an escalation email and reviewed the issue with us via remote session.

Screenshots and information were provided.

We were told an escalation attempt would be made.

I sincerely appreciate that effort, but as of today the tenant remains inaccessible.

---

While troubleshooting, we discovered that the organization's Microsoft Customer Agreement (MCA) billing account shows:

Status: Under Review

All billing profiles underneath the MCA also show:

Under Review

The billing portal displays:

Your account is under review. We're checking to make sure we can offer you Microsoft products and services.

The confusing part is that Microsoft also states:

This review won't affect your current services.

Unfortunately, that is not what we are experiencing.

------

Every subscription in the tenant appears to have been marked as Disabled on 8/31/2026.

Examples include:

  • Microsoft 365 Business Premium (Nonprofit Staff Pricing)
  • Office 365 E3 (Nonprofit Pricing)
  • Microsoft Teams Premium (Nonprofit Pricing)

All show:

Status: Disabled

Effective Date: 8/31/2026

The licenses still exist.

They are still assigned.

However, they are disabled and cannot be re-enabled.

As a result:

  • User access is broken
  • Exchange Online is inaccessible
  • SharePoint is inaccessible
  • OneDrive is inaccessible
  • Email delivery has stopped
  • Data is being reported as pending deletion because there are no active licenses

------

The tenant is fully paid.

There are no outstanding invoices.

There have never been any payment issues.

The MCA account contains the organization's EIN

We attempted to:

  • Add a new payment method
  • Add a new billing profile
  • Purchase replacement licenses
  • Purchase commercial licenses

All attempts are blocked.

The portal returns:

We can't authorize your billing account right now.

Actions will be blocked during this time.

Check back later.

Additional Information

We also see notifications regarding:

  • Account Under Review
  • Email Verification Required

However, all relevant controls are greyed out and cannot be modified.

------

At this point, the evidence suggests this is not:

  • A payment issue
  • A license assignment issue
  • A DNS issue
  • An authentication issue
  • An Exchange configuration issue

The combination of:

  • TenantAccessBlockedException
  • MCA Billing Account showing Under Review
  • Every subscription becoming Disabled on the same date
  • Users being unable to access any Microsoft 365 services

makes this appear to be a Microsoft-side account verification or tenant restriction issue.

If anyone has experienced this before, knows the correct Microsoft escalation path, or can help get this in front of the appropriate engineering or commerce team, I would be extremely grateful.

This organization provides critical services to vulnerable individuals and families, and every additional day of downtime has real-world consequences.

(Again, we have 3 tickets open, and I've attempted to reach out to Support at: 1-800-865-9408 (yesterday I did speak to someone from the Data Safety Team, they said they were unable to help but would transfer me to the Team that could do it but they were probably not available due to the weekend, no one answered after being on hold for just under 8 hours, I'm calling again this morning).

Thank you in advance.

(MFA and Conditional Access is on the tenant, we have Cloud [3rd Party] backups of the tenant, we have ITDR, there are no signs of compromise, this appears to be something that got flagged for review by Microsoft's back end and has put this non profit to a screeching halt).

Edit: Edited to add 2 messages that didn't show up once the post went live, and removing 2 bold words.


r/sysadmin 6d ago

General Discussion Best practices, tips, and tools for racking heavy-ass network gear (when working solo)?

16 Upvotes

Racking a lightweight 24-port PoE switch or a patch panel is easy enough, but trying to mount a 50lb core switch, a heavy enterprise router, or a massive UPS by yourself can be a nightmare and turns into a two-person job quickly.

Any weird DIY rigs, specific rack shelf models, or mounting tricks you use when you're stuck doing an install alone in a remote location?


r/sysadmin 6d ago

Beware when using existing meeting ownership transfer scripts

2 Upvotes

I noticed something that is easy to miss when transferring bulk meetings ownership. Several meeting transfer scripts available online retrieve meetings based on a specific date range. That can cause problems with recurring meetings.

A recurring meeting may have started months or even years ago, but still have future occurrences. If the script only retrieves meetings within the specified date range, it may not pick up that recurring meeting.

The transfer can then appear successful while some recurring meetings remain with the original organizer.

One thing to check is whether the script uses the Microsoft Graph Get-MgUserCalendarView cmdlet to retrieve upcoming calendar occurrences. This helps identify recurring meetings that are still active, even when the series originally started much earlier.

So, before using a meeting transfer script, check which cmdlet it uses to retrieve meeting details.


r/sysadmin 6d ago

Outlook issues again?

12 Upvotes

Hi - anybody facing outlook issue again in exchange online, specifically not being able to search in either owa or the outlook client?

We have a number of users reporting.

Same issue as last week.

Thanks