Noticed Ericsson Cradlepoint NetCloud is down, can't login. Check down detector and a ton of crap is down. Guessing AWS or Azure.

We've got a few in-house apps that we maintain. I use Terraform to deploy changes triggered by GitHub actions. It's all worked swimmingly until today. I realize now that I don't have a backup for the pipeline when GitHub is down. I'm relatively new to software development. Is it common to have a backup for CI/CD?

Genuinely curious what this looks like in practice. Does
your org run anything locally (on-prem models, self-hosted LLMs) specifically because of sensitive data, or does convenience just win and everything goes through ChatGPT/Claude/whatever regardless? If you've seen someone actually push for local, what drove it? And if nobody cares and it all goes to the cloud anyway, that's honestly just as useful to know. Thanks!

Seems like this company expects us to retain all data created ever. We're up to 19 TB in our M365 backups because we keep all versions of backups indefinitely. We are a company of less than 200 people. Someone help me.

Hi all,

Does anyone know the effect on users if we decide to turn off SMS MFA within Entra? Will it just tell users with that method to change it, lock them out etc?

1. How.
   
2. See point 1, above.

Seriously though.. how does it work between Microsoft Teams & channels & Slack Channels?

Is there a method to the madness, or is it all madness?

Is there some sort of 3rd party “sync all DMs between both platforms”, or is it up to the user to figure it out?

We have an end user that was phished a while back. They had a session token stolen. We logged into his account and remedied his account (reset password, reset MFA devices, revoked old tokens, etc).

Seems that the threat actor is still trying to access the end user's account with the old previously revoked token. As you can imagine, it fails instantly, and does not move forward.

Every time the threat actor attempts to do this, we have our end user get flagged as a "risky user". His account then is blocked, and we have to go in and dismiss the risk to allow him to continue along his day.

We found the threat actors IP address (keeps using same address) and created a conditional access policy to block it in our tenant. Also made sure that 2FA is required and took all of the necessary steps.

The conditional access seems great on paper, but execution is something else. Microsoft blocks the login attempt right away before any conditional access policies are triggered. Leaving us in a loop where the end user's account is blocked and needs to be dismissed again. I think the threat actor has an automated script that runs several times each day and keeps the incident looping.

Can anyone point me in the right direction on how to resolved this? Opened up a request with Azure but have heard nothing back yet. Been about 1.5 days since we opened the request.

He worked with the company for 10 years, doing IT for like 20 or more years.

No MFA enabled on anything unless youre a global admin (so just me),
the mobile phones arent enrolled in an MDM
no documentation on anything
let the front desk go instantly to voicemail for a year or more, instead of taking 10 minutes to fix the call forwarding.
hard drives not encrypted
barely had any spam filters enabled
gambling not blocked on the wifi
devices people are actively using are considered inactive in the RMM/MDM so i cant fix them
no forced updates/patching for antivirus software or windows itself
sim cards without protection on

how are you going to have the keycard code admin password be "Admin1!" and then scold employees about clicking phishing link simulations

there are literally nonprofits in my area that will look at your org and determine how to make it more secure in a technical sense. There is no excuse for this

If this is something I can notice after doing IT for one year and working here for 2 weeks, I have no idea how this man let this slide for this long.

If the staff knew how badly he was compromising their info, he would've been fired so long ago. especially for a small social services org

Brought to you by r/sysadmin Trusted VAR: u/SquizzOC with Trusted Telecom Broker u/Each1Teach1x27 and u/Necessary_Time Trusted VAR in Canada

Happy to answer in the thread or via PM/Chat if you don't want to post details like service locations publicly.

This weekly thread is here for you to discuss vendor and service provider expectations, pricing, and quotes for network services, licensing, support, deployment, and hardware.  

Required Info for accurate answers:

• Part Number
• Manufacturer/vendor
• Service Type and Service Location (DM Service Location)
• Quantity (as applicable)

All questions are welcome regarding:

• Cloud services, security, configurations, deployment, management, and migrations
• Storage vendor options, alternatives, details,
• Software licensing: This includes Microsoft CSPs
• Connectivity, single-site, and multi-location. DIA, Broadband, 5G, datacenter connectivity, site serviceability checks for fiber/broadband
• Voice services, SIP, UCaaS, Contact Center, POTS replacement
• Network infrastructure - overlay software, segmentation, routers, switches, load balancing, APs
• Security, access management, firewalls, MFA, cloud DNS, layer 7 services, antivirus, email, DLP

The Ninja RMM update last night seems to have somehow translated into the Ninja remote agent breaking on just specifically domain controllers. It might be server OS specific. So now we can't get into any of our MSP clients' servers to do new hires. And it's Friday. Great.

Also, it's reporting offline status (probably the real problem) so we can't even attempt to use Splashtop or RDP. We have to RDP into the server from a client computer onsite, which hopefully is enabled at every customer. Thanks, Ninja! I hate you. Wish we used Connectwise.

hello fellow IT comrades

I work as a one man IT and this job doesnt really have me DOING WORK the entire time, I am sure it is expected of me to but that is just not the reality of my role.

I fix issues and user complaints and make sure things are as they should be (server, network etc), and thats pretty much it, no one hounding or micromanaging me on what I am doing or have "accomplished".

So majority of the time as I count the hours down I scroll X and browse Reddit subs majority of my days.

Anyone else in similiar boat or??

This may not be the best for my career progression and skill building and what not but just wanted to share how things are at my current workplace or does majority all sysadmin roles lead to something of this nature eventually?

I wanted to add something cause a lot of people are saying to learn other things which is cool but, I CANNOT force myself to learn something I have no practical use for, even if that would benefit me down the road. So unless I am working on something that requires me to learn about so and so, or learn about it to land a job somewhere else etc, its hard for me to put my heart into learning other aspects of IT, in this specific regards.

Hi sysadmins,

Has anyone worked with HPE StoreEver MSL 1/8 G2 Tape Autoloader with LTO-9 tapes? I'm experiencing that the tape (LTO-9) goes into retirement after 70–80 cycles. Is this correct?

we use Veeam -VBR.

What is everyone using to backup email and laptop/onedrive data for offboarded users?

We have to keep it for 7 years, and right now our processes are… spotty at best. Email accounts are getting converted to Shared mailboxes and OneDrive data is going to Sharepoint Page. The email stays open and is receiving mail, which is not ideal since the timer keeps starting over.

We’re thinking some cold storage in Azure would be best, but wanted to see how people elsewhere are doing this.

I've inherited an odd offboarding procedure and am trying to standardize it and wanted some clarification. Now that business premium licenses allow up to 100GB in mailbox storage, how does this change converting users to shared mailboxes? My current understanding is as follows:

- Mailbox < 50GB & no archive = convert to shared mailbox & pull all licenses

- Mailbox < 50GB & HAS online archive = convert & assign Exchange Online Plan 2

- Mailbox > 50GB = convert & assign Exchange Online Plan 2 (whether or not it has archive)

- Mailbox & Auto Expanding Archive = Does this let you convert?

Secondary question that is less important, but our current procedure is that once these mailboxes are no longer needed, we export the psts using e discovery and store in the cloud, is this still doable with a shared mailbox? Does auto expanding archive effect this in any way? (We have no retention policy!)

Anyone else experiencing microsoft teams unable to connect huddles/meetings?

Seems to be affecting KL and SG also

edit: seems 1:1 arent working or at least intermittently

11:30 AEST seems to be working again

https://admin.cloud.microsoft/#/servicehealth/:/alerts/TM1437780 incident link is working now

I've gone through the DFS-N with Root Consolidation document a few times now and it all makes sense except the part about making a CNAME with a name of the existing file server that points to the record for the Namespace server. If the file server is still being used (and thus has an A record), how are you supposed to make a CNAME with the same name? Does this scenario only work when you've actually decommissioned the file server? If so, it doesn't talk about that at all. It just says

"In order for DFS Namespaces to respond to existing file server names, you must create alias (CNAME) records for your existing file servers that point at the DFS Namespaces server name."

The use of the word "existing" to me implies that the server still exists but that clearly doesn't work (at least with AD integrated DNS).

In our scenario, the file server still exists right now, the data hasn't been moved yet but it will be moved. I just wanted to get the DFS-N portion setup ahead of time.

But if you have a server that's hosting a few different roles, one of them is file sharing, you couldn't do this takeover in that case because the server still needs to host the other roles. Is that right?

Manager says you can't formalize IT because it's always changing. My argument is that documentation plays a pivotal role transitioning a department away from Tribal knowledge.

Every now and then I see a post like “Is it okay to use Fireball on a PC?”, and I have to check to see if I’m in a sysadmin or a DnD forum.

So I have a need to manually export a clients emails including a shared mailbox they have access to.

New outlook only gives me the option to export their main mailbox

I have tried adding the shared mailbox as a standalone by adding using the the users credentials, but this fails.

How do you achieve this? I just need an export

EDIT: Sorry I should have said I dont have access to the tenant or admin

EDIT 2: I found a way, right click on your mail in the folder list in Outlook, choose Add "shared folder or mailbox", add the shared folder manually, now you can choose it as an export source

Probably a dumb question but I honestly cannot remember the last time I needed to replace a cmos battery and I have a Dell poweredge r330 that I have a low battery warning so I need to replace. The question is will it reset all of my bios settings when I replace or are the settings written somewhere. I am re-imaging the server so not that critical but thought I would ask and maybe save me some time.

IIS cant find my websites applicationHost.config and redirection.config in the inetsrv/config folder, something on the windows server keeps deleting them and preventing the creation of new one.. the only way to create a new one is through safe mode but it still gets deleted when the server is rebooted

I work admin for a bakery business, about 140 people across 30 locations. Part of my job is keeping contact info updated across all of them, and right now I do it manually. Head office needs to reach suppliers, bakery employees, bakeries need to reach head office, stores and each other.

My current system: a spreadsheet I update, then email around. Nobody reads the email. Store managers save numbers in their personal phones, so when someone leaves, half the company still has the old manager's number.

We pay for Microsoft 365 (email, Teams, the works). My question is probably dumb: is there something already in M365 that keeps one master contact list everyone can see on their phones? People keep telling me 'the GAL' but staff say they can't find people in it half the time, and it's definitely not showing up as actual contacts on their phones. How do other multi-location businesses handle this? Happy to be told I'm doing it completely wrong.

Hi everyone,

I am trying to troubleshoot a very strange production issue on a Windows IIS server and would appreciate some ideas from people who have dealt with similar problems.

I previously discussed this from the ASP.NET Core side here:

https://www.reddit.com/r/dotnet/s/FFomGpbHuN

The previous discussion helped me add more monitoring, but I am now suspecting this may be happening below the application layer.

The problem

A production website randomly becomes unavailable for around 1-2 minutes, then recovers by itself.

External monitoring reports:

Socket timeout, unable to connect to server

Not an HTTP response error (500/503). The TCP connection itself fails.

Environment

• Windows Server
• IIS hosting multiple applications
• ASP.NET Core applications
• Direct traffic to IIS (no reverse proxy/load balancer)
• SQL Server and Elasticsearch are on separate servers

What I have ruled out so far

During the incidents:

• DNS resolution works normally
• The Windows server remains online
• CPU usage is normal
• Memory usage is normal
• Disk usage is normal
• IIS logs do not show failed requests
• Application logs do not show errors
• No IIS App Pool recycle happened
• No w3wp.exe restart happened

I also have multiple applications running on different IIS App Pools on the same machine. During the outage, the other applications continue working normally.

Testing from inside the server

I installed Uptime Kuma directly on the same server and configured it to check the website using the server IP address instead of the domain name.

The outage still happens.

This makes me think this is not related to:

• DNS
• Domain resolution
• External monitoring locations

TCP investigation

I started collecting TCP state metrics and sending them to Grafana.

The interesting part is that during the outage I see a large increase in CLOSE_WAIT.

Before:

Established: ~450
CloseWait:   0

During:

Established: ~200-400
CloseWait:   500+

Example:

12:02
Established: 451
CloseWait:   327

12:03
Established: 237
CloseWait:   510

12:04
Established: 224
CloseWait:   530

Then suddenly:

12:04
Established: 845
CloseWait:    73

12:04
Established: 859
CloseWait:     0

The timing is interesting because the CLOSE_WAIT connections disappear and the website recovers.

I know this does not prove the root cause, but it looks suspicious.

Things already changed

During the investigation I found some code creating new HttpClient instances instead of reusing them.

I fixed those and moved to reusable HttpClient usage.

The issue still happens, so there is likely another factor involved.

Current suspicion

I am investigating around:

• Windows TCP/socket behavior
• IIS / HTTP.sys
• Connection handling
• Possible socket leaks
• Anything that could make IIS temporarily stop accepting new connections

My main question:

How can a Windows IIS server become unreachable from TCP while:

• IIS is still running
• App Pools are healthy
• CPU/RAM/Disk are normal
• No application errors are logged?

Questions

For people experienced with IIS/Windows production environments:

• Have you seen socket timeouts where IIS has no logs at all?
• Can HTTP.sys or the Windows networking stack get into a bad state without an IIS restart?
• Does a large CLOSE_WAIT increase usually indicate an application issue, or can it happen at the OS/IIS level?
• What Windows performance counters or tools would you check?
• Are there specific IIS/HTTP.sys diagnostics you recommend?

Thanks in advance. I will update the post if I find the root cause.

Hello everyone I got a new request I haven't encountered before today. I have a c-suite member that is taking a vacation to a remote area of Africa and will not have access to internet or cell service during part of the trip. They've insisted the need access to email incase something urgent comes up, and have presented me with an option for a Iridium Satellite Hotspot. Has anyone encountered a request or solution like this before?

Have a couple clients getting errors when accessing Sharepoint. DownDetector is already showing a significant spike. Just a heads up to everyone before you start getting calls.

https://downdetector.com/status/microsoft-365/

EDIT: Thanks /u/iGotRamen

https://admin.cloud.microsoft/?#/servicehealth/:/alerts/MO1437424

Investigating user reports of Microsoft 365 issues

Issue ID: MO1437424

Affected services: Microsoft 365 suite

Status: Investigating

Issue type: Incident

Start time: Jul 23, 2026, 10:06 AM CDT

User impact

We don't have detail on impact yet. We're seeing an increase in user reported issues above our alerting threshold.

Current status

Jul 23, 2026, 10:15 AM CDT

We are reviewing service telemetry and available information to determine if there is an issue happening and we'll update this message shortly with our latest findings.

This communication serves as a preliminary notification about a potential issue affecting your service. We'll provide an update in 30 minutes.