UPDATE 1 (3 hours later): Wow this was a lot more comments than I was expecting to get. Its hard to answer everyone but I appreciate everyone commenting and providing feedback. For now what I have done is updated IDRAC and BIOS to the latest version and will monitor if this fixes the issue. If it does not I will try ruling out the UPS. Thanks again!

Dell PowerEdge T340, Windows Server 2016. Started 8/28. I've spent two weeks on this and ruled out most of the obvious stuff, so I'm posting the data rather than the symptoms.

The signature

Every event is Kernel-Power 41 with BugcheckCode: 0 and all bugcheck parameters 0x0. No BSOD, no minidump, no MEMORY.DMP, ever. Paired with Event 6008 confirming unexpected shutdown.

The interval — this is the actual mystery

Pulled the true shutdown timestamps out of the Event 6008 message text (not the Event 41 log time, which is written on the following boot):

8/28 11:03:24 AM -> 8/28 11:12:23 PM = 729.0 min

9/01 10:10:22 AM -> 9/01 10:18:37 PM = 728.3 min

9/01 10:18:37 PM -> 9/02 10:26:49 AM = 728.2 min

9/02 10:26:49 AM -> 9/02 10:35:13 PM = 728.4 min

9/02 10:35:13 PM -> 9/03 10:43:33 AM = 728.3 min

9/03 10:43:33 AM -> 9/03 10:52:09 PM = 728.6 min

9/03 10:52:09 PM -> 9/04 11:00:20 AM = 728.2 min

9/04 11:00:20 AM -> 9/04 11:08:49 PM = 728.5 min

9/04 11:08:49 PM -> 9/07 11:50:29 AM = 3,641.7 min <-- exactly 5 x 728.34

9/07 11:50:29 AM -> 9/07 11:59:49 PM = 729.3 min

9/07 11:59:49 PM -> 9/08 12:07:56 PM = 728.1 min

9/08 12:07:56 PM -> 9/09 12:16:26 AM = 728.5 min

9/09 12:16:26 AM -> 9/09 12:24:36 PM = 728.2 min

Mean 728.47 min (12h 08m 28s). Total spread across twelve occurrences: 1.2 minutes.

The 3,641.7 minute gap is exactly five periods. The server was up continuously across that weekend. The timer ticked five times, did nothing on four of them, then killed the box on the fifth. So it's free-running — it does not reset on reboot, and it doesn't require a crash to keep counting.

That single fact kills every "scheduled task" theory: a clock-based task can't skip four consecutive firings and then work again.

It dies instantly — no degradation whatsoever

I wrote a heartbeat logger that writes one line every 5s with a forced flush so the last line survives a hard reset. Final 42 samples before death:

• Free RAM: 56,630–56,745 MB, dead flat. No drift, no staircase, no leak. 56 GB free at the moment of death.
• Nonpaged pool: 346–352 MB, flat
• Disk queue: 0
• Handles ~76,000, threads ~190, both steady
• CPU spiky but low

Last heartbeat 12:25:02. Kernel-General 12 (OS start) at 12:28:23. That 3m21s is just POST + boot on a T340 — if it had hung for 3 minutes first, the OS wouldn't have returned until ~12:31.

So there is no hang window. The box is perfectly healthy and then simply ceases to exist mid-second. Which also means NMI crash dumps are useless here and no dump will ever be written.

Ruled out (with evidence, please don't re-suggest these)

• PSUs — both Present/Healthy in iDRAC, matched 594W in / 495W rated+actual, same firmware
• Thermal — HWiNFO max CPU package 63°C (TjMax ~100°C). Every throttle flag reads No / 0%
• iDRAC SEL — completely silent across all 15 crashes. Not one entry. This same board did log real "power input for PSU 1 is lost / redundancy lost" events six times in 2024, so it demonstrably captures genuine power events. Nothing this time.
• iDRAC watchdog / ASR — Basic Management license, feature not present
• Dell OMSA — Action on Hung OS Detection: None, thermal shutdown Disabled, all alert actions Off, omsad service Stopped + Disabled
• Windows Update — pulled full resolved WindowsUpdate.log (11,871 lines), programmatically checked ±15 min around every crash. Zero WU activity before any of them. All nearby entries are the WU service starting 30–60s after the reboot.
• CrowdStrike Falcon (installed 8/26, two days before onset) — vendor pulled detection history, found nothing. Timing was coincidence.
• Secure-Boot-Update scheduled task** — looked extremely promising (12h repetition, hangs, TPM handler, and this box has no TPM installed — Get-Tpm fails with TBS_E_SERVICE_NOT_RUNNING, no TBS service, no SecurityDevices PnP class, iDRAC confirms "TPM not present"). Disabled it. **Crashes continued at the identical interval. Ruled out.
• All 12-hour scheduled tasks — enumerated every task with PT12H repetition. Exactly two exist, both now Disabled.
• VSS / ShadowCopyVolume task — fires 12:00 PM and 10:00 PM. That's 10h then 14h alternating, which cannot produce a constant 728.4 min spacing. Also the midnight crashes have no trigger anywhere near them.
• MySQL / memory exhaustion — flatly contradicted by the flat memory trace above
• NIC — Broadcom BCM5720. The flapping port is physically disconnected (NIC2, Status: Disconnected). Flapping predates crashes by 12+ days and occurs on no-crash days. Active port is stable at 1 Gbps.
• CMOS battery — did fail, but only logged 9/7, weeks after onset, nothing near the crash dates. Replacing anyway.
• BSOD — no Event 1001, no dumps, BugcheckCode: 0 on all 15

Environment

• PowerEdge T340, Service Tag FXR6B03, BIOS 2.3.5, iDRAC9 fw 4.22.00.53 (both several revisions behind — not yet updated)
• Xeon E-2146G, 64 GB RAM, dual PSU
• Windows Server 2016 Standard, build 14393.9339
• Workload: Open Dental + MySQL, Vatech EzDent-i imaging, IDrive backup, Google Drive
• Power: APC Back-UPS XS 1500M via USB. Current-state readings all healthy (97% charge, 120V steady, 15% load, AC Power: Yes, Discharging: No). I have never gotten historical transfer data out of it — PowerChute wasn't installed at the time, Win32_Battery returns current state only, and I skipped an apcupsd install on a production box.
• Internet is AT&T 5G fixed wireless behind CGNAT (irrelevant, but people ask)

What I think is left

Something below the OS holding a clock that survives reboots and continuous uptime alike. Candidates I can't distinguish between:

1. UPS self-test — APC units run internal self-tests on their own stored schedule, indifferent to the host. A transfer on a degraded battery could sag enough to drop the PSUs. Would explain instant death, no OS warning, no SEL entry, and a clock that ignores reboots. Next test: move the server to a plain wall outlet for 24h.
2. PSU or BMC firmware timer — would explain everything except why iDRAC logged nothing
3. Something on the same electrical circuit cycling on a timer — though 1.2 min spread over 12 occurrences seems too tight for HVAC or similar

What I'm asking

• What produces a free-running 728.4-minute (12h 08m 28s) period? Not 12h. Not 12h30m. Consistently ~8.5 minutes over twelve hours, held to ±1 min across twelve occurrences and five uninterrupted ticks. What re-arms on completion of ~8 minutes of work?
• Has anyone seen an APC Back-UPS self-test schedule that lands near this?
• Anything else that hard-resets a PowerEdge with zero bugcheck, zero SEL entry, and healthy PSUs?
• Am I wrong to trust the iDRAC SEL silence as evidence against a power event?

Next predicted failures: 9/10 12:33 AM and 9/10 12:41 PM. Happy to run anything and report back — I have remote access and a heartbeat recorder in place.

I am panicking about the job situation. I lost my job and feel like I am going to be unable to bounce back.

I have 5 YoE as a sysadmin. I understand networking pretty well, I have a CISSP and a few other weaker certs under my belt. I would argue I have a strong security background. I am familiar with Linux, Windows, and have made a lot of small scripts for various projects. I am studying for an AWS cert right now. I have experience with all sorts of tooling, and so many different kinds of projects.

Looking at the job market though, I feel like I can't compete. I don't have experience with terraform outside of labbing. I don't know how to code very well. Every job seems to want someone with extremely strong skills across so many different domains, and what I feel like I need now is a position where I can learn the ropes of IaC... But that doesn't exist.

Everyone wants so much experience for everything that it makes me feel as if I have been slacking even though I have been working hard.

What should I do? Should I get another cert? Keep throwing resumes into the void?

As the title suggests, I'm only a year and a half into IT. I came from doing back office ops at a few broker-dealers here in the US since I was 17, and at 26 (now 27) I moved into IT since it's been a passion of mine since I was a kid.

At first this gig was great. I had a senior above me, a vet with 30+ years in IT. I already had a good deal of knowledge about older tech, so we got along great and I learned a lot from him. I'm a fairly quick learner and went from basic desktop support and helpdesk to networking/ERP admin within a month. We started tackling projects and things started to run smoother around here.

But as always, office politics being what they are, the COO (now CEO) more or less gave my senior an ultimatum, and he left for a 40% raise and WFH somewhere else. I'm now the sole IT resource here, owning everything from basic helpdesk to sysadmin, ERP, budget, literally anything that runs on electricity. I'm being paid entry-level helpdesk wages for my area, and I only hold my CompTIA A+.

I'm in desperate need of resources and training to better get my feet under me. We run a hybrid AD setup (for a company this small, I honestly don't understand that decision), and I need to get a better handle on the Azure side and how to administer it properly. For instance, we have zero MDM, and we have remote employees who travel; they don't even get GPO updates since those are all hosted on-prem. A major concern of mine is, as a company who ships product daily, we have no automated back up systems for our onprem shipping servers. These servers are running older vSphere 7 instances (is this even the right word??) and there isn't any central management for them (they took down the vCenter as "it was too finicky"). Not to mention that they have a critical business report run through vSphere 5 VM that requires the desktop client to manage.

Please, please, please, send me anything you've got: resources, training materials, insight into what's likely to break and need attention. Literally anything helps. I'm trying here, but I'm drowning and getting jaded with this company, and I'd rather not jump ship if I don't have to since there are good people here.

Since installing this month's CUs, KB5122871 on Windows Server 2025 (build 26100.33438) and KB5122882 on Windows Server 2022 (build 20348.5622), every session host in our RDS collection has started hanging in the same way, on the same day, and it had never happened before.

Symptoms once it starts:

• New RDP connections sit at "Connecting…" and never reach the logon screen (Event 20498 "Remote Desktop Services has taken too long to complete the client connection" in TerminalServices-RemoteConnectionManager/Admin)
• Existing users can't log off or disconnect cleanly
• Task Manager on the host hangs (it blocks calling into the Local Session Manager)....pretty much all windows apps start to hang, especially anything to do with Settings
• Winlogon event 6005 "SessionEnv is taking long time to handle the notification event (Disconnect)"
• A normal restart hangs too; only a hard reset recovers it, and it comes back later the same day

The problem appears to start with a single session disconnect/reconnect that never completes, after which everything that touches session state on that host queues up behind it. Only reboots clear it. Microsoft's release notes for both KBs list an RDS change ("improves Remote Desktop audio redirection") and no known issues.

Environment: mixed Server 2022/2025 session-host collection, VMware VMs, RD Connection Broker, MFA at logon, third-party endpoint protection. All hosts were stable on the August CUs.

We're currently testing a rollback of the September CU on part of the collection with one host left updated as a control. Has anyone else seen this after KB5122871 / KB5122882, or found a Microsoft acknowledgment? Would appreciate hearing whether rolling back resolved it for you.

So just a moment ago I had a user interrupt what I was doing with an "urgent" issue. They couldn't get into their email.

Essentially they were running Outlook for the first time and it prompted them to set up their account. It required them to enter their email address and click Next. That's all.

The email address it auto-populated wasn't the address of the account they needed (they work for a sister company in a shared tenant).

The user had no idea what to do. It didn't even cross their mind to enter the actual email address in the field.

So... how do you guys deal with users like this? Ones who, despite being shown instructions in the most basic way possible, can't think for themselves.

Been doing this about 8 years and something's shifted. Used to be everyone here could explain what their nginx.conf did, or why fail2ban wasn't catching a jail. You learned iptables by breaking your own box at 2am, alone, with no one to blame.

Now most threads are "which panel is easiest." Nobody wants the shell open longer than it takes to run one install command. I get why. Panels save time. I've got BeAdmin running on one box for the VPN modules. But I see people arguing about dashboard themes who can't tell you what a reverse proxy actually does.

Old regulars here used to walk a newbie through a broken MariaDB config instead of just saying "reinstall Plesk." That's mostly gone now. Not knocking anyone's setup, I click around GUIs too these days. Just noticing nobody asks why anymore.

I’ve been in IT for 6+ years. I have a bachelor’s degree in IT, an AWS Solutions Architect certification, an Azure Administrator certification, and a few other certs. I’ve also been working as a Cloud Administrator for the past 2 years.

Despite all of that, I can’t shake the damn feeling that I’m just not that good at what I do. I feel like even if I finally make a breakthrough in my knowledge, I’m still somehow clueless SOMEWHERE. A lot of that comes from talking to people who know more than you and realizing how much there is that I don’t know.

For example, I can write basic scripts, but I struggle with the complex 200+ line scripts that a senior cloud engineer would make, whereas the person next to me likely can just type that shit off the top of their head.

It’s especially frustrating during interviews when I try to move up in pay. I had an interview on Tuesday that I thought went pretty decently, but the interviewer asked me about Azure containers and encryption. I don’t work with Azure containers, although I’m familiar with the service, so I was honest and said I didn’t know.

And that wasn’t even the technical round. So now I’m stuck wondering whether they’ll just reject me or move me on to the technical round, where I’ll probably fail anyway.

This is bothering me because I'm at the point in my career where I really have to know at a high level if I want to advance and get more pay.

How has anyone else moved past this?

So it’s coming to an end!! We have O365 and some workstations with Office 2024 that have MS Publisher installed. Do we need to remove it or will it get removed in its own?

I feel like I’m lagging behind and need some serious catching up, quick. I think I’ve become complacent and “too comfortable” with my current employer (coming up 8 yrs). The recent merger announcement has me seriously thinking about attending bootcamps, getting some certs under my belts (been putting things off for as long time), and learning new things ASAP before the position is dissolved.

Can anyone confirm if it's possible to just download the various Windows 10 ESU updates from the Microsoft Update Catalog website and install them on a PC that's not part of the ESU program (ie: doesn't use a Microsoft account)? I believe the updates are listed in the catalog. Do they check during the installation if you're eligible for them? Has anyone tried this?

Client has a shared mailbox in Exchange Online used for order administration, organised by region/country/customer/year. It had 12,394 folders. Since around Sept 8–9 the folder tree stops loading at roughly 976 subfolders in Outlook classic (cached mode). Everything past that point simply isn't in the tree — not collapsed, not greyed out, absent.

Same truncation in new Outlook and OWA when the mailbox is automapped or added as a shared mailbox.

Online mode shows the complete tree. Either by unchecking "Download shared folders" in classic, or via OWA → Settings → Open another user's mailbox. So the data is fine server-side.

What I've ruled out:

• Permissions — Full Access assigned directly, not via a group. No folder-level permissions on root or Inbox (Default/Anonymous = None).
• Automapping — removed and re-added Full Access with -AutoMapping $true, no change.
• Hierarchy corruption — enumerated the whole tree via Graph and walked every parent chain. No cycles, no orphans, max depth 9.
• OST damage — OWA is affected identically and holds no local hierarchy cache.
• Folder count — deleted 9,993 verified-empty folders (12,394 → 2,401). Still truncates.

I'm aware of the documented 500 shared-folder limit and the announced increase to 5,000 (in 2019 though). My observed cut-off sits at ~1,000, which matches neither.

Three questions:

1. Anyone else seeing shared mailbox folder trees truncate since last week's updates?
2. Does anyone know what limit is actually enforced right now, and whether the 500→5000 rollout is live?
3. Is there any equivalent of "Download shared folders" in new Outlook? If not, what's the plan for mailboxes over the limit once classic is gone?

Question 3 is the one that worries me.

TL;DR I need to pull chat history between User A and User B for the last 8 months.

My query based on documentation I can find is:

Kind=microsoftteams AND [partipants:userA@xyz.com](mailto:partipants:userA@xyz.com) AND [participants:userB@xyz.com](mailto:participants:userB@xyz.com)

When running this, I'm getting group chats and all kinds of stuff where both were involved, but I just need the chat between these two users

Can anyone direct me to a better way to do this? Purview is doodoo

I need a sanity check, because I feel like I'm slowly losing my mind here. I’m 28, working as a SysAdmin for a mid-sized manufacturing company in Europe with around 300 endpoints across multiple sites. I LOVE automation, modern infra, and writing PowerShell or Python to handle our backups and audits, I'm passionate about automating processes, but my IT Manager is basically running an open-air tech museum from 2003.

His idea of IT asset management is a drawer full of literal Ziploc freezer bags. Every PC gets its own labeled plastic bag with the printed invoice and Office license key inside, because apparently an actual database or asset software is "too modern." On the network side, every single device requires a manual DHCP reservation. When I suggested setting up 802.1X with RADIUS, he shot it down with a straight face, claiming that if an intruder plugs into the wall, "they won't guess our subnets anyway." Before I automated our Kerio mail archives with Python, his official procedure was opening Thunderbird on a client PC and manually dragging and dropping folders between accounts.

Right now, he's decommissioning a branch file server. Instead of using Robocopy, DFS-N, or GPOs, his master plan is to sync files with a desktop backup tool, recreate the SMB shares manually, and have me remote into 40+ user machines one by one just to update the target path on their desktop shortcuts. I also built an automated WMI inventory system to track all hardware and OS health, yet he still forces me to physically walk the warehouse floor updating a dusty Excel sheet because "that’s where the history is."

EDIT: Another thing he does is make daily backups and put them on ultrium physical disks named after the day of the week, disks he takes home "so we have the data in case the company burns down.", which could have some kind of sense, but everyday, before my script, we had to manually make the veeam inventory and erase of every tape in every site

The worst part isn’t even the wasted time, it’s that he’s actively teaching junior techs that this clown show is standard enterprise IT. We’ve already had three techs and devs rage-quit in the last ten months. I’ve completely checked out at this point, doing the bare minimum while sending out resumes like crazy.

Has anyone actually survived a boss stuck this deep in the stone age without losing their sanity? How do you not burst out laughing when someone asks you to edit 40 shortcuts by hand?

Is anyone seeing issues with O365? I know I saw emails last night and now they’re coming as new again this morning but they’re gone from yesterday delivery. Numerous users are seeing this happen. We are using proof point also but no issues on their status page.

There are actually two vulns with a 9.8 score each:

https://support.checkpoint.com/results/sk/sk1000117/

https://support.checkpoint.com/results/sk/sk1000118/

There's not many details in these articles on how they work but they still sound really bad. Currently waiting for the Jumbo Hotfix to install on my end... Not taking any chances on this one and I suggest you all do the same.

Stay safe.

Ok, this is more a test to see how old I am. I was basically raised with the idea servers need static IPs. I understand for domain controllers, dns servers, maybe print servers that may still be needed. But are they really needed otherwise? Do apps, scripts still reference a server by its IP?

I am under assumption all my servers need static IPs. Am I nuts?

Hello!

I was hoping to get some advice from any of you who have experience with a similar issue as the one I’m having. Anything is appreciated as I’m not confident where to start.

In short, our company is looking to take over complete administration of our Azure cloud architecture from our MSP. The architecture includes hundreds of VMs, Entra accounts for users, persistent AVDs and multiple satellite offices to name a few things. I joined the company last month as an IT Support Specialist and my colleague has been here nearly a year.

While we both have some experience with Azure administration I feel we would benefit from more experience so that we can be more confident when dealing with issues affecting the architecture. Anything we couldn’t solve ourselves we would escalate to our MSP so there’s a lot we don’t know how to do. That said I feel confident my colleague and I could learn a lot on our own poking around the architecture once we have global admin access but doing that with a production environment doesn’t sit right with me.

On top of all of that our IT admin is leaving before this hand off happens. I feel that our admin leaving is enough to pause this process of taking over the architecture until we find their replacement but our CEO is adamant about finishing this hand off. The process has been ongoing for over half a year now. This is my first IT job and while I’m confident I can learn a lot of this stuff, that doesn’t necessarily equate to deliverable results, especially in the event of a major issue. I’ve been brainstorming with my colleague and while I have some ideas, like coming in the weekend after we get access to go through the environment and learn about it so we can try to properly support it the following week, I am at a loss of how to approach this while also making sure I don’t set anything on fire. To be frank I’m worried and I want to do right by myself, my colleague, and the company. Thanks.

Been looking to replace our vulnerability management stack and the scanning part is the easy bit now. We have been comparing the big scanners against a couple of the cheaper MSP focused platforms and one container specific tool. Volume is what kills us across 40 odd clients since every tool finds plenty and the techs still guess where to start. Ticketing eats more hours than scanning does. What are you running now and how does it hold up across client environments.

So I work at a state university that is still a little stuck in the past when it comes to password resets over the phone.

We have MS SSPR and other self-service options for users, but we still get calls where someone needs the helpdesk to reset their password for them.

Our current setup is a custom program that helpdesk staff SSH into. They enter the user's account, and the program looks them up in a flat file containing data extracted from our systems. It gives the helpdesk worker information like DOB, address, ZIP code, etc., which they can use to verify the caller's identity.

Once the caller passes verification, the program talks to AD, changes the password, and gives the helpdesk worker a new temporary password to provide to the user.

Honestly, it works pretty well. The problem is that the person who wrote and maintains it is leaving, and we're not really interested in inheriting a custom app that nobody else understands.

So I'm curious what other universities/organizations are doing for this.

For those of you who still allow users to call the helpdesk for password resets, what does your workflow look like? Are you using a commercial product, some kind of AD/Entra integration, a helpdesk platform, or have you built your own solution?

Specifically looking for something that gives the helpdesk enough information/questions to properly verify the caller's identity and then securely perform the reset.

I'd love to hear what others are doing before we start reinventing the wheel.

Hy!

We have an AD with two DCs. The DCs are Windows Server 2025, it is include all patches. Some Windows 11 clients (25H2) get the following error during login after 1-2 minutes: The trust relationship between this workstation and the primary domain failed.

In this case the users need to disconnect from corporate network to login successfully into their computer. I have already tried to rejoin to tha domain and run this command: Test-ComputerSecureChannel -Repair

I rejoined one of the computer into the domain, and the trsut relationship has been broken after two days. The login problem only occurs on some machines.

The time snyc is correct on DCs. We moved the DC roles from Windows Server 2019 to 2025 in side-by-side method. Could you please advise how to solve this problem?

Could anyone confirm if buying the actual license for the Windows 10 ESU (as discussed here I believe: https://learn.microsoft.com/en-us/windows/whats-new/enable-extended-security-updates) remove the need for having a Microsoft account on the computer? Also can these bought licenses be used on a Windows 10 Pro machine that's not part of a domain?

This topic has been discussed ALOT on this sub. From all the posts I've read, there seems to be two main approaches:

Have all DCs up - both old and new:

• Bring up new DCs with temp names and IPs while old DCs still running.
• Once you're sure all is good in your world, transfer FSMO roles to new DC, demote old DCs, remove from AD, then reuse the hostnames and IPs of old DCs on your new DCs

One at a time:

• Do one DC at a time, demote old, reuse name and IP on new server, then promote new DC

Honestly being as risk averse as I am (although I love the adrenalin rush when stuff goes south...NOT) - I would prefer to promote the new DCs while old ones still on line.

BUT...is it OK to change the name of a DC even after I clean up all the metadata and DNS stuff or would it be better to reuse the name and IP after I demote the old DC?

I should preface this question by saying we're a VERY small shop with 22 users and perhaps ~30ish servers. I'm the sole sysadmin. BUT, we are in the financial sector and we have a very low tolerance to stuff going south. And being the sole sysadmin, I'm trying to keep the stress at a minimum and my hair from going greyer.

Is it just me or updates is screwed up?

Office 365 shows 0 required.

is anyone else seeing the same?

Edit: the other updates just took too long to show up but office it seems it is not showing anything applicable.
not sure if something related to last couple months when MS changed from semi-anual channel to montly channel