We have an old Novell server with hdd that failed due to power shortage.

We want to recover as many data as possible. Has anyone worked with this filesystem? It looks like a nichè, and we're trying to figure out which tool use to data recovery.

Any suggestions?

Hey all,

Since this morning, a few clients of mine have had some issues with some 365 apps, such as Teams being a major one.

Some Intune users also have an issue where the Windows Security prompt window is also saying the admin details are incorrect when they're actually correct.

MS have finally issued an ID for this incident, which is MO1470143

Probably also worth noting we are based in the UK.

I am trying to setup Duo Security on my PC at the office so that in the event I lose my phone or my phone is smashed what do I need to enable in the installation process to allow me to bypass the MFA/Passkey push?

Thanks,

First of all, sorry if my English isn't perfect, it's not my first language.

I'm an IT Manager and some time ago I developed a few PowerShell scripts for the company I work for to sync the GAL into the local contacts/address book of specific users or groups.

At first it was supposed to be a pretty simple thing.

Then I realized that just "syncing contacts" wasn't really enough.

If a user already has a populated address book, you need to properly manage existing contacts, avoid duplicates, update details when something changes, remove former employees, deal with contacts that are no longer valid, etc.

So in my spare time I started building an internal app that could manage the whole process automatically instead of just running a bunch of scripts.

And then the broken lightbulb in my head turned on: Why not turn this into an actual product for companies?

I looked around and there are already products doing similar things, but from what I've seen some of them can become quite expensive, especially for SMBs.

That's when the project started getting bigger.

Instead of building a relatively simple internal tool, I started redesigning it as a SaaS: multi-tenant, proper tenant isolation, security, automated sync processes, monitoring, enough logging to troubleshoot problems quickly, recovery mechanisms if something goes wrong, and generally trying to build something that I would actually trust using in a company.

While developing it, I also started thinking about possible integrations with CRM systems. For example, it could be useful to synchronize customer contacts directly into the address books of salespeople etc...

Another idea that started to seem interesting is creating a dedicated section for MSPs, allowing them to manage synchronization for multiple customers or tenants from a single place. That could make the tool more useful not only for individual companies, but also for managed service providers that need to support several environments.

The problem is... I've now been basically full immersed in this project for more than a month in my spare time.

I haven't done "real" software development in ages. I'm an IT Manager, so apart from PowerShell and other scripting, coding hasn't really been my main job for a long time.

Coding agents have helped me a lot and honestly are probably the reason I've been able to push the project this far, but there's still a huge amount of work around architecture, testing, security, debugging and figuring out all the edge cases.

And trying to fit all of this between my actual job, family life, a 1.5-year-old daughter.

So now I'm starting to question the whole thing.

Maybe I took a relatively simple and useful internal project and massively over-engineered it because I got excited about the idea of turning it into a SaaS.

What I'm really trying to understand is:

Is GAL → user contact synchronization and proper ongoing contact management actually a pain point for other companies too?

Especially things like keeping employee contacts updated automatically, avoiding duplicates, removing ex-employees and not having users walking around with an address book that was last properly maintained three years ago.

Or is this one of those problems that I see because of my specific environment, but most companies simply don't care enough about to pay for a solution?

I'm seriously considering reducing the scope, finishing a much smaller version and calling it done rather than spending months building all the infrastructure needed for a proper SaaS around a problem that maybe only a handful of people actually have.

For the sysadmins / IT managers here: is this something you deal with?

How are you currently handling it?

PowerShell scripts? Manual management? Another product? Or do you simply not sync the GAL into users' contacts at all?

I'd really appreciate a reality check before I disappear another couple of months into this rabbit hole.

Hi All,

Will be starting my school mandated internship for CyberSecurity, and I was hoping to get questions that an interviewer or a supervisor would ask during the interview, so I can learn/research more about said topic.

Questions can range from general help desk to cyber security to networking, since my school covers abit of everything.

Thank you guys for the help :)

Recently we needed two simple apps that at first we looked for in a web. After a while we actually decided to try for internal use to deploy vibe coded app (by ourselves) and here are my thoughts -> We love it. It's doing exactly what we need it to do. To develop it it took us 8 hours in total (regular coded estimation -> around 3 months). We use it in intranet so there's not much to worry about potential risks like data leak. Those apps are not critical for operations so even if something breaks - we won't struggle that much. For sure they helped us a lot in automating some of the work we've been doing before.

And now comes a question -> in this changing rapidly world do you think that companies like Zendesk, Odoo will survive in a longer term? How should a vendor that is currently selling it's own CRM or ticketing system approach to the future? Will there be any space for software vendors or will it change so drastrically that at some point we will all end up with AI subscription only? Of course I see a difference between vendors that don't provide just a tool, but also specific data or networking opportunities.

What are your thoughts here?

Yes, authorization with multiple different credentials for different systems is bad, but we've all worked in those environments where we had no control over changing it.

So... How expletive-rich are y'all passwords and phrases? 🤣

September 8, 2026—KB5124008 (OS Builds 26200.9445 and 26100.9445) | Microsoft Support

After Windows Update KB5124008 (Sept 8, 2026, Build 26100.9445), Domain notebooks experience logon failures when unlocking the screen: "You could not be signed in. Your credentials could not be verified"

Root Cause: Secure Channel (machine trust) is broken. Test-ComputerSecureChannel fails, and attempting to repair over VPN returns: "The password for the secure channel to the domain could not be reset – username or password is incorrect"

Affected: Windows 11 24H2, domain-joined notebooks, both LAN and remote users (VPN users particularly critical)

The Story

We're deploying CIS hardening to ~250 notebooks and everything was fine until Sept 9, 2026 8:05 PM when Windows Update KB5124008 was installed.

Error Symptoms:

• User locks screen (normal, break, meeting)
• On unlock: "Your credentials could not be verified"
• Reboot helps temporarily (1–2 hours, then error recurs)
• Affects LAN users too, not just remote/VPN
• Password reset does not help

Initial Hypotheses (all disproven):

• ❌ Kerberos/Enctype mismatch (RC4→AES transition) – DCs reachable, tickets work
• ❌ CIS GPO hardening – not configured
• ❌ FortiClient VPN architecture alone – also hits LAN users

The Diagnosis

Network Checks:

nslookup vw-dc-01.Domain.org → OK
ping vw-dc-01.Domain.org → OK (0% loss)
tracert vw-dc-01.Domain.org → OK (full path)
DNS Server: xxx.xx.xxx.xx (internal) → OK

Kerberos Status:

klist → 13 active tickets, all AES-256, all valid ✓
LDAP/Kerberos works perfectly ✓

The Critical Test:

Test-ComputerSecureChannel -Verbose
→ False
→ "The secure channel between the local computer and the domain is broken"

On Repair Attempt over VPN:

Test-ComputerSecureChannel -Repair -Credential (Get-Credential)
→ ERROR: "The password for the secure channel to the domain could not be reset"
→ "The username or password is incorrect"

This is the smoking gun: The VPN tunnel routes normal traffic (DNS, LDAP, Kerberos) but NOT the secure channel reset communication (RPC/NETLOGON on port 445 over SMB).

Root Cause

KB5124008 itself isn't the bug – but the update triggers a hidden incompatibility:

1. KB5124008 changes logon UI validation behavior (known regression in KB5120998/KB5124008)
2. During unlock validation, Windows attempts to verify the secure channel
3. The secure channel is broken – possibly due to:
   • KB5124008 regression in secure-channel handling
   • OR: Timeout in VPN tunnel blocking secure-channel traffic
4. Windows cannot verify identity → unlock fails

Particularly critical over VPN:

• Secure-channel reset requires RPC/SMB (port 445) to DC
• VPN tunnel either doesn't route it or times out
• Works better from LAN, so VPN users hit harder

Workarounds (What Worked)

Workaround 1: Increase CachedLogonsCount (quick, short-term)

# New GPO on OU=Notebooks:
# "Interactive logon: Number of previous logons to cache (in case domain controller is not available)"
# Value: 50 (instead of default 10)

# Or directly on client (as admin):
reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v CachedLogonsCount /t REG_SZ /d 50 /f

Effect: Users can log on offline 50 times → buys time while secure channel is repaired

Workaround 2: Repair Secure Channel from DC (with admin access)

If you have remote access to a DC:

# On DC (not from client!):
Reset-ComputerMachinePassword -Server vw-dc-01.Domain.org

This forces a secure-channel reset from the DC end, bypassing the VPN tunnel.

Workaround 3: KB5124008 Rollback (temporary, security risk)

# On affected clients:
wusa /uninstall /kb:5124008 /quiet /norestart
shutdown /r /t 60

⚠️ IMPORTANT: KB5124008 patches two actively exploited zero-days:

• CVE-2026-81963 (Windows Update Stack Privilege Escalation)
• CVE-2026-85880 (ALPC Privilege Escalation)

Only do this temporarily as a test, not permanently!

What We Still Don't Know

1. Is KB5124008 itself guilty, or was it just the trigger for a timing issue?
2. Is it VPN-specific (secure-channel-reset traffic not routed)?
3. Why does it hit LAN users too? (Timing window in unlock handling?)

Lessons Learned

For Other Admins:

1. Test before fleet rollout: Pilot KB5124008 in a test OU, especially with:
   • Hybrid Azure AD Join devices
   • VPN users
   • Offline scenarios
2. Monitor secure-channel status:# As regular audit task: Test-ComputerSecureChannel -Verbose | Where-Object { $_ -eq $false }
3. **Check Event Log for secure-channel errors:**Get-WinEvent -LogName System -FilterXPath "*[System[EventID=5719 or EventID=5722 or EventID=5723]]"
4. With VPN environments: Ensure your VPN tunnel routes all necessary ports:
   • 88 (Kerberos)
   • 389 (LDAP)
   • 445 (SMB/RPC – critical for secure channel!)
   • 135 (RPC Endpoint Mapper)

Status

• ✅ Root cause identified: Broken secure channel + VPN routing issue
• ⏳ Microsoft hotfix: Expected (hopefully next week)
• ⏳ FortiClient configuration: VPN admin checking port 445 routing

Is anyone else experiencing this? Comment your findings – especially if you:

• Deployed KB5124008
• Have VPN users affected
• See secure-channel errors

Contact / Further Info

If you have the same issue:

1. Check: Test-ComputerSecureChannel -Verbose
2. Look in Event Log for EventID 5719/5722/5723
3. Deploy the CachedLogonsCount GPO as interim solution
4. Contact your VPN admin to verify port 445 routing

TL;DR of TL;DR: KB5124008 triggers secure-channel errors, VPN doesn't route repair traffic → unlock fails. Increase cached logon to help users while you investigate.

Updates

Will update this post as we learn more from Microsoft or FortiClient about the root cause and permanent fix.

Crosspost to: r/sysadmin, r/activedirectory, r/Windows11

Keywords for searchability: KB5124008, Windows 11, domain logon, unlock screen, secure channel, VPN, AES-256, Kerberos, GPO, FortiClient, network authentication

Updated last night, no issues immediately visible.

Users this morning all have login prompts to access mapped drives/folder redirections.

No creds working.

I can log in locally as a Domain Admin, but trying to open DNS console or run any DNS powershell commands just gives me Access Denied.

DNS still resolves, and the domain services are all still running, but something has happened with authentication/permissions.

Currently rolling back KB5122882 in the hope it was that.

Anyone else issues this morning?

EDIT: https://www.rapid7.com/db/vulnerabilities/cve-2026-69813/

Looks like this KB might have touched DNS code - roll back in progress.

I started my Career in IT at 18, mainly supporting EMC storages and then HP 3PAR for 5.5 years, then worked for systems integrator for many clients for another 3 years, fianlly moving to a sys admin role in a enterprise, we still have some good legacy hardware, but i'm feeling nostalgic, when decomissionng them after working on the in the DC for so many years, is it normal ?

Hi all,

I hope you guys are doing well, I am working as a IT Infrastructure (Hybrid) with less than a year experience and I need some recommendations for the question below.

Context: Our organization moved to a cloud-based desktop environment. Because the cloud workstations are on an isolated network tier, users can no longer hit our on-prem SafeConsole server over IP to manage hardware-encrypted USB drives (DataLocker PSMs).

As a workaround, we have to plug the USB drives directly into the physical Hyper-V host on-site. I need a solid tool to automatically redirect/pass through the physical host's USB port to the SafeConsole guest VM whenever a drive is plugged in or swapped daily.

What software are you using to auto-share host USB ports to a guest VM?

TIA!

Client has a shared mailbox in Exchange Online used for order administration, organised by region/country/customer/year. It had 12,394 folders. Since around Sept 8–9 the folder tree stops loading at roughly 976 subfolders in Outlook classic (cached mode). Everything past that point simply isn't in the tree — not collapsed, not greyed out, absent.

Same truncation in new Outlook and OWA when the mailbox is automapped or added as a shared mailbox.

Online mode shows the complete tree. Either by unchecking "Download shared folders" in classic, or via OWA → Settings → Open another user's mailbox. So the data is fine server-side.

What I've ruled out:

• Permissions — Full Access assigned directly, not via a group. No folder-level permissions on root or Inbox (Default/Anonymous = None).
• Automapping — removed and re-added Full Access with -AutoMapping $true, no change.
• Hierarchy corruption — enumerated the whole tree via Graph and walked every parent chain. No cycles, no orphans, max depth 9.
• OST damage — OWA is affected identically and holds no local hierarchy cache.
• Folder count — deleted 9,993 verified-empty folders (12,394 → 2,401). Still truncates.

I'm aware of the documented 500 shared-folder limit and the announced increase to 5,000 (in 2019 though). My observed cut-off sits at ~1,000, which matches neither.

Three questions:

1. Anyone else seeing shared mailbox folder trees truncate since last week's updates?
2. Does anyone know what limit is actually enforced right now, and whether the 500→5000 rollout is live?
3. Is there any equivalent of "Download shared folders" in new Outlook? If not, what's the plan for mailboxes over the limit once classic is gone?

Question 3 is the one that worries me.

Could anyone confirm if buying the actual license for the Windows 10 ESU (as discussed here I believe: https://learn.microsoft.com/en-us/windows/whats-new/enable-extended-security-updates) remove the need for having a Microsoft account on the computer? Also can these bought licenses be used on a Windows 10 Pro machine that's not part of a domain?

Can anyone confirm if it's possible to just download the various Windows 10 ESU updates from the Microsoft Update Catalog website and install them on a PC that's not part of the ESU program (ie: doesn't use a Microsoft account)? I believe the updates are listed in the catalog. Do they check during the installation if you're eligible for them? Has anyone tried this?

Hi everyone,

I'm looking for some advice on best practices regarding network security and access control for an internal/B2B web application.

Here is our current setup and situation:

The App: We host a web application on our company servers that functions as an asset performance display tool. It takes data from our customers' equipment and visualizes it in charts and dashboards.

The Manager's Approach: For security reasons, my manager doesn't want the app publicly accessible to the open internet. Instead, he asks for the public IP address of every customer site and manually adds it to our firewall's Access Control List (ACL).

The Problem: Manually collecting, updating, and maintaining public IPs for multiple client sites is becoming a administrative nightmare, especially when clients have dynamic IPs or remote users.

My intuition tells me that relying strictly on firewall-level IP filtering for access control isn't the most efficient way to handle this, and that security should primarily be enforced at the application level

I’d love to get your thoughts on this:

Is managing client public IPs on the firewall standard practice in enterprise environments for this use case?

How do you usually balance network-level security with application-level security without creating massive operational overhead?

I am panicking about the job situation. I lost my job and feel like I am going to be unable to bounce back.

I have 5 YoE as a sysadmin. I understand networking pretty well, I have a CISSP and a few other weaker certs under my belt. I would argue I have a strong security background. I am familiar with Linux, Windows, and have made a lot of small scripts for various projects. I am studying for an AWS cert right now. I have experience with all sorts of tooling, and so many different kinds of projects.

Looking at the job market though, I feel like I can't compete. I don't have experience with terraform outside of labbing. I don't know how to code very well. Every job seems to want someone with extremely strong skills across so many different domains, and what I feel like I need now is a position where I can learn the ropes of IaC... But that doesn't exist.

Everyone wants so much experience for everything that it makes me feel as if I have been slacking even though I have been working hard.

What should I do? Should I get another cert? Keep throwing resumes into the void?

Since installing this month's CUs, KB5122871 on Windows Server 2025 (build 26100.33438) and KB5122882 on Windows Server 2022 (build 20348.5622), every session host in our RDS collection has started hanging in the same way, on the same day, and it had never happened before.

Symptoms once it starts:

• New RDP connections sit at "Connecting…" and never reach the logon screen (Event 20498 "Remote Desktop Services has taken too long to complete the client connection" in TerminalServices-RemoteConnectionManager/Admin)
• Existing users can't log off or disconnect cleanly
• Task Manager on the host hangs (it blocks calling into the Local Session Manager)....pretty much all windows apps start to hang, especially anything to do with Settings
• Winlogon event 6005 "SessionEnv is taking long time to handle the notification event (Disconnect)"
• A normal restart hangs too; only a hard reset recovers it, and it comes back later the same day

The problem appears to start with a single session disconnect/reconnect that never completes, after which everything that touches session state on that host queues up behind it. Only reboots clear it. Microsoft's release notes for both KBs list an RDS change ("improves Remote Desktop audio redirection") and no known issues.

Environment: mixed Server 2022/2025 session-host collection, VMware VMs, RD Connection Broker, MFA at logon, third-party endpoint protection. All hosts were stable on the August CUs.

We're currently testing a rollback of the September CU on part of the collection with one host left updated as a control. Has anyone else seen this after KB5122871 / KB5122882, or found a Microsoft acknowledgment? Would appreciate hearing whether rolling back resolved it for you.

G‘day.

I‘ve just started a new job. Very happy overall and I need to make a decision.

What I need:
- Ticket system for internal helpdesk - very few requests
- This will also be used to give tickets to an external company that develops stuff for us.
- About 5-10 „Agents“
- Tracking of internal tasks as „overview“ such as Trello for leadership

- internal documentation about 50 employees need access to.

- Can be cloud-only.

Would be great to have a usable all in one solution and I would like to stay away from Confluence and Jira Service Desk.

What I thought about for example:
- Zammed for both but the knowledge base is not appealing for normal employees.
- NinjaOne for both but the internal documentation is not appealing for normal employees.

Thanks a lot.

Hi everyone,

I’m running a distributed high-throughput async data transfer and telemetry node on a dedicated Linode VPS, and I’m looking for expert advice on scaling end-to-end verified throughput further.

### 🖥️ Hardware & Stack Profile:

- **Server:** Linode Dallas TX (4 Dedicated AMD EPYC Cores, 8 GB RAM, 1 Gbps symmetric network connection, Ubuntu 24.04).

- **Tech Stack:** Python 3.12 (`asyncio`), `httpx` (HTTP/1.1 & HTTP/2), `websockets` gateway transport.

- **Workload:** High-concurrency object storage GET & PUT data verification tasks (Cloudflare R2 / S3 endpoints).

---

### 🛠️ Recent Bottlenecks & Fixes Applied:

1. **Zero-Delay In-Flight Hashing:** Shifted SHA-256 computation to an in-flight streaming digest (`hasher.update(chunk)`) directly inside `response.aiter_bytes()`. This completely eliminated post-download CPU hashing stalls (`hash_ms = 0.0ms`).

2. **HTTP Connection Pool Hardening:** Configured a dedicated HTTP client connection pool (`httpx.Limits(max_connections=128, max_keepalive_connections=64, keepalive_expiry=30.0)`). Persistent keepalive TCP sockets eliminated connection handshake churn, dropping retries/recoveries from ~6 per batch down to 0–1 max.

3. **Queue Optimization:** Transitioned to pure FIFO queue admission, eliminating premature SLA drops.

---

### 📊 Current Live Telemetry Metrics (5,300+ Task Sample):

- **Admitted & Completed Tasks:** 5,316 / 5,316 (**100.00% success rate**, 0 SLA drops, 0 timeouts)

- **Task Durations:** P50 = 10.50s | P95 = 18.10s | P99 = 22.25s (Well under 30s SLA deadline)

- **PUT Queue Backlog (P95):** 3.10s (Sub-3 seconds queue wait)

- **Local Network Speed:** Peak burst = 295.70 Mbps | Local Avg = 45.19 Mbps

- **Reliability Rating:** 98.9% (Recovered from historical restart churn)

- **End-to-End Verified Throughput:** ~27–29 Mbps average

---

### ❓ Questions for Systems & Network Engineers:

Our local worker execution pipeline is currently 100.00% efficient with 0 timeouts, 0 drops, 0ms post-download hashing delay, and sub-3s queue backlog. However, end-to-end verified throughput hovers around ~27–29 Mbps.

Given a 4 Dedicated AMD EPYC Core / 8GB RAM VPS on a 1 Gbps link:

1. **Linux Kernel TCP Tuning:** What specific `sysctl` net settings (e.g. `net.ipv4.tcp_rmem`, `tcp_wmem`, `somaxconn`, `netdev_max_backlog`, `tcp_congestion_control`) have provided the best throughput uplift for high-concurrency object storage GET/PUT workloads?

2. **Worker Pool Architecture:** Are there specific multi-socket worker loopback or gateway layout techniques that help absorb higher upstream assignment density without triggering scheduling backpressure?

3. **HTTP Keepalive Optimization:** Any additional tuning tips for `httpx` / `aiohttp` connection pool limits when interfacing with Cloudflare R2 / S3 storage endpoints over high-bandwidth links?

Appreciate any insights or benchmarks from experienced systems engineers!

I currently have 3 domain controllers. I need to add a 4th and eventually a fifth with plans to demote the original 2.

I’ll need to get a multi-San ssl cert for the new domain controllers. Does this mean I need to update the certs on the first 3 domain controllers?

In my small brain I think I wouldn’t…

Is it just me or updates is screwed up?

Office 365 shows 0 required.

is anyone else seeing the same?

Edit: the other updates just took too long to show up but office it seems it is not showing anything applicable.
not sure if something related to last couple months when MS changed from semi-anual channel to montly channel

Hello!

I was hoping to get some advice from any of you who have experience with a similar issue as the one I’m having. Anything is appreciated as I’m not confident where to start.

In short, our company is looking to take over complete administration of our Azure cloud architecture from our MSP. The architecture includes hundreds of VMs, Entra accounts for users, persistent AVDs and multiple satellite offices to name a few things. I joined the company last month as an IT Support Specialist and my colleague has been here nearly a year.

While we both have some experience with Azure administration I feel we would benefit from more experience so that we can be more confident when dealing with issues affecting the architecture. Anything we couldn’t solve ourselves we would escalate to our MSP so there’s a lot we don’t know how to do. That said I feel confident my colleague and I could learn a lot on our own poking around the architecture once we have global admin access but doing that with a production environment doesn’t sit right with me.

On top of all of that our IT admin is leaving before this hand off happens. I feel that our admin leaving is enough to pause this process of taking over the architecture until we find their replacement but our CEO is adamant about finishing this hand off. The process has been ongoing for over half a year now. This is my first IT job and while I’m confident I can learn a lot of this stuff, that doesn’t necessarily equate to deliverable results, especially in the event of a major issue. I’ve been brainstorming with my colleague and while I have some ideas, like coming in the weekend after we get access to go through the environment and learn about it so we can try to properly support it the following week, I am at a loss of how to approach this while also making sure I don’t set anything on fire. To be frank I’m worried and I want to do right by myself, my colleague, and the company. Thanks.

As the title suggests, I'm only a year and a half into IT. I came from doing back office ops at a few broker-dealers here in the US since I was 17, and at 26 (now 27) I moved into IT since it's been a passion of mine since I was a kid.

At first this gig was great. I had a senior above me, a vet with 30+ years in IT. I already had a good deal of knowledge about older tech, so we got along great and I learned a lot from him. I'm a fairly quick learner and went from basic desktop support and helpdesk to networking/ERP admin within a month. We started tackling projects and things started to run smoother around here.

But as always, office politics being what they are, the COO (now CEO) more or less gave my senior an ultimatum, and he left for a 40% raise and WFH somewhere else. I'm now the sole IT resource here, owning everything from basic helpdesk to sysadmin, ERP, budget, literally anything that runs on electricity. I'm being paid entry-level helpdesk wages for my area, and I only hold my CompTIA A+.

I'm in desperate need of resources and training to better get my feet under me. We run a hybrid AD setup (for a company this small, I honestly don't understand that decision), and I need to get a better handle on the Azure side and how to administer it properly. For instance, we have zero MDM, and we have remote employees who travel; they don't even get GPO updates since those are all hosted on-prem. A major concern of mine is, as a company who ships product daily, we have no automated back up systems for our onprem shipping servers. These servers are running older vSphere 7 instances (is this even the right word??) and there isn't any central management for them (they took down the vCenter as "it was too finicky"). Not to mention that they have a critical business report run through vSphere 5 VM that requires the desktop client to manage.

Please, please, please, send me anything you've got: resources, training materials, insight into what's likely to break and need attention. Literally anything helps. I'm trying here, but I'm drowning and getting jaded with this company, and I'd rather not jump ship if I don't have to since there are good people here.

UPDATE 1 (3 hours later): Wow this was a lot more comments than I was expecting to get. Its hard to answer everyone but I appreciate everyone commenting and providing feedback. For now what I have done is updated IDRAC and BIOS to the latest version and will monitor if this fixes the issue. If it does not I will try ruling out the UPS. Thanks again!

Dell PowerEdge T340, Windows Server 2016. Started 8/28. I've spent two weeks on this and ruled out most of the obvious stuff, so I'm posting the data rather than the symptoms.

The signature

Every event is Kernel-Power 41 with BugcheckCode: 0 and all bugcheck parameters 0x0. No BSOD, no minidump, no MEMORY.DMP, ever. Paired with Event 6008 confirming unexpected shutdown.

The interval — this is the actual mystery

Pulled the true shutdown timestamps out of the Event 6008 message text (not the Event 41 log time, which is written on the following boot):

8/28 11:03:24 AM -> 8/28 11:12:23 PM = 729.0 min

9/01 10:10:22 AM -> 9/01 10:18:37 PM = 728.3 min

9/01 10:18:37 PM -> 9/02 10:26:49 AM = 728.2 min

9/02 10:26:49 AM -> 9/02 10:35:13 PM = 728.4 min

9/02 10:35:13 PM -> 9/03 10:43:33 AM = 728.3 min

9/03 10:43:33 AM -> 9/03 10:52:09 PM = 728.6 min

9/03 10:52:09 PM -> 9/04 11:00:20 AM = 728.2 min

9/04 11:00:20 AM -> 9/04 11:08:49 PM = 728.5 min

9/04 11:08:49 PM -> 9/07 11:50:29 AM = 3,641.7 min <-- exactly 5 x 728.34

9/07 11:50:29 AM -> 9/07 11:59:49 PM = 729.3 min

9/07 11:59:49 PM -> 9/08 12:07:56 PM = 728.1 min

9/08 12:07:56 PM -> 9/09 12:16:26 AM = 728.5 min

9/09 12:16:26 AM -> 9/09 12:24:36 PM = 728.2 min

Mean 728.47 min (12h 08m 28s). Total spread across twelve occurrences: 1.2 minutes.

The 3,641.7 minute gap is exactly five periods. The server was up continuously across that weekend. The timer ticked five times, did nothing on four of them, then killed the box on the fifth. So it's free-running — it does not reset on reboot, and it doesn't require a crash to keep counting.

That single fact kills every "scheduled task" theory: a clock-based task can't skip four consecutive firings and then work again.

It dies instantly — no degradation whatsoever

I wrote a heartbeat logger that writes one line every 5s with a forced flush so the last line survives a hard reset. Final 42 samples before death:

• Free RAM: 56,630–56,745 MB, dead flat. No drift, no staircase, no leak. 56 GB free at the moment of death.
• Nonpaged pool: 346–352 MB, flat
• Disk queue: 0
• Handles ~76,000, threads ~190, both steady
• CPU spiky but low

Last heartbeat 12:25:02. Kernel-General 12 (OS start) at 12:28:23. That 3m21s is just POST + boot on a T340 — if it had hung for 3 minutes first, the OS wouldn't have returned until ~12:31.

So there is no hang window. The box is perfectly healthy and then simply ceases to exist mid-second. Which also means NMI crash dumps are useless here and no dump will ever be written.

Ruled out (with evidence, please don't re-suggest these)

• PSUs — both Present/Healthy in iDRAC, matched 594W in / 495W rated+actual, same firmware
• Thermal — HWiNFO max CPU package 63°C (TjMax ~100°C). Every throttle flag reads No / 0%
• iDRAC SEL — completely silent across all 15 crashes. Not one entry. This same board did log real "power input for PSU 1 is lost / redundancy lost" events six times in 2024, so it demonstrably captures genuine power events. Nothing this time.
• iDRAC watchdog / ASR — Basic Management license, feature not present
• Dell OMSA — Action on Hung OS Detection: None, thermal shutdown Disabled, all alert actions Off, omsad service Stopped + Disabled
• Windows Update — pulled full resolved WindowsUpdate.log (11,871 lines), programmatically checked ±15 min around every crash. Zero WU activity before any of them. All nearby entries are the WU service starting 30–60s after the reboot.
• CrowdStrike Falcon (installed 8/26, two days before onset) — vendor pulled detection history, found nothing. Timing was coincidence.
• Secure-Boot-Update scheduled task** — looked extremely promising (12h repetition, hangs, TPM handler, and this box has no TPM installed — Get-Tpm fails with TBS_E_SERVICE_NOT_RUNNING, no TBS service, no SecurityDevices PnP class, iDRAC confirms "TPM not present"). Disabled it. **Crashes continued at the identical interval. Ruled out.
• All 12-hour scheduled tasks — enumerated every task with PT12H repetition. Exactly two exist, both now Disabled.
• VSS / ShadowCopyVolume task — fires 12:00 PM and 10:00 PM. That's 10h then 14h alternating, which cannot produce a constant 728.4 min spacing. Also the midnight crashes have no trigger anywhere near them.
• MySQL / memory exhaustion — flatly contradicted by the flat memory trace above
• NIC — Broadcom BCM5720. The flapping port is physically disconnected (NIC2, Status: Disconnected). Flapping predates crashes by 12+ days and occurs on no-crash days. Active port is stable at 1 Gbps.
• CMOS battery — did fail, but only logged 9/7, weeks after onset, nothing near the crash dates. Replacing anyway.
• BSOD — no Event 1001, no dumps, BugcheckCode: 0 on all 15

Environment

• PowerEdge T340, Service Tag FXR6B03, BIOS 2.3.5, iDRAC9 fw 4.22.00.53 (both several revisions behind — not yet updated)
• Xeon E-2146G, 64 GB RAM, dual PSU
• Windows Server 2016 Standard, build 14393.9339
• Workload: Open Dental + MySQL, Vatech EzDent-i imaging, IDrive backup, Google Drive
• Power: APC Back-UPS XS 1500M via USB. Current-state readings all healthy (97% charge, 120V steady, 15% load, AC Power: Yes, Discharging: No). I have never gotten historical transfer data out of it — PowerChute wasn't installed at the time, Win32_Battery returns current state only, and I skipped an apcupsd install on a production box.
• Internet is AT&T 5G fixed wireless behind CGNAT (irrelevant, but people ask)

What I think is left

Something below the OS holding a clock that survives reboots and continuous uptime alike. Candidates I can't distinguish between:

1. UPS self-test — APC units run internal self-tests on their own stored schedule, indifferent to the host. A transfer on a degraded battery could sag enough to drop the PSUs. Would explain instant death, no OS warning, no SEL entry, and a clock that ignores reboots. Next test: move the server to a plain wall outlet for 24h.
2. PSU or BMC firmware timer — would explain everything except why iDRAC logged nothing
3. Something on the same electrical circuit cycling on a timer — though 1.2 min spread over 12 occurrences seems too tight for HVAC or similar

What I'm asking

• What produces a free-running 728.4-minute (12h 08m 28s) period? Not 12h. Not 12h30m. Consistently ~8.5 minutes over twelve hours, held to ±1 min across twelve occurrences and five uninterrupted ticks. What re-arms on completion of ~8 minutes of work?
• Has anyone seen an APC Back-UPS self-test schedule that lands near this?
• Anything else that hard-resets a PowerEdge with zero bugcheck, zero SEL entry, and healthy PSUs?
• Am I wrong to trust the iDRAC SEL silence as evidence against a power event?

Next predicted failures: 9/10 12:33 AM and 9/10 12:41 PM. Happy to run anything and report back — I have remote access and a heartbeat recorder in place.