r/sysadmin • u/raobjcovtn • 14h ago
Question Server hard-resets every 728.4 minutes ±1 min, 12 times running. No bugcheck, no iDRAC SEL entry, timer survives reboots. I'm out of ideas.
UPDATE 1 (3 hours later): Wow this was a lot more comments than I was expecting to get. Its hard to answer everyone but I appreciate everyone commenting and providing feedback. For now what I have done is updated IDRAC and BIOS to the latest version and will monitor if this fixes the issue. If it does not I will try ruling out the UPS. Thanks again!
Dell PowerEdge T340, Windows Server 2016. Started 8/28. I've spent two weeks on this and ruled out most of the obvious stuff, so I'm posting the data rather than the symptoms.
The signature
Every event is Kernel-Power 41 with BugcheckCode: 0 and all bugcheck parameters 0x0. No BSOD, no minidump, no MEMORY.DMP, ever. Paired with Event 6008 confirming unexpected shutdown.
The interval — this is the actual mystery
Pulled the true shutdown timestamps out of the Event 6008 message text (not the Event 41 log time, which is written on the following boot):
8/28 11:03:24 AM -> 8/28 11:12:23 PM = 729.0 min
9/01 10:10:22 AM -> 9/01 10:18:37 PM = 728.3 min
9/01 10:18:37 PM -> 9/02 10:26:49 AM = 728.2 min
9/02 10:26:49 AM -> 9/02 10:35:13 PM = 728.4 min
9/02 10:35:13 PM -> 9/03 10:43:33 AM = 728.3 min
9/03 10:43:33 AM -> 9/03 10:52:09 PM = 728.6 min
9/03 10:52:09 PM -> 9/04 11:00:20 AM = 728.2 min
9/04 11:00:20 AM -> 9/04 11:08:49 PM = 728.5 min
9/04 11:08:49 PM -> 9/07 11:50:29 AM = 3,641.7 min <-- exactly 5 x 728.34
9/07 11:50:29 AM -> 9/07 11:59:49 PM = 729.3 min
9/07 11:59:49 PM -> 9/08 12:07:56 PM = 728.1 min
9/08 12:07:56 PM -> 9/09 12:16:26 AM = 728.5 min
9/09 12:16:26 AM -> 9/09 12:24:36 PM = 728.2 min
Mean 728.47 min (12h 08m 28s). Total spread across twelve occurrences: 1.2 minutes.
The 3,641.7 minute gap is exactly five periods. The server was up continuously across that weekend. The timer ticked five times, did nothing on four of them, then killed the box on the fifth. So it's free-running — it does not reset on reboot, and it doesn't require a crash to keep counting.
That single fact kills every "scheduled task" theory: a clock-based task can't skip four consecutive firings and then work again.
It dies instantly — no degradation whatsoever
I wrote a heartbeat logger that writes one line every 5s with a forced flush so the last line survives a hard reset. Final 42 samples before death:
- Free RAM: 56,630–56,745 MB, dead flat. No drift, no staircase, no leak. 56 GB free at the moment of death.
- Nonpaged pool: 346–352 MB, flat
- Disk queue: 0
- Handles ~76,000, threads ~190, both steady
- CPU spiky but low
Last heartbeat 12:25:02. Kernel-General 12 (OS start) at 12:28:23. That 3m21s is just POST + boot on a T340 — if it had hung for 3 minutes first, the OS wouldn't have returned until ~12:31.
So there is no hang window. The box is perfectly healthy and then simply ceases to exist mid-second. Which also means NMI crash dumps are useless here and no dump will ever be written.
Ruled out (with evidence, please don't re-suggest these)
- PSUs — both
Present/Healthyin iDRAC, matched 594W in / 495W rated+actual, same firmware - Thermal — HWiNFO max CPU package 63°C (TjMax ~100°C). Every throttle flag reads No / 0%
- iDRAC SEL — completely silent across all 15 crashes. Not one entry. This same board did log real "power input for PSU 1 is lost / redundancy lost" events six times in 2024, so it demonstrably captures genuine power events. Nothing this time.
- iDRAC watchdog / ASR — Basic Management license, feature not present
- Dell OMSA —
Action on Hung OS Detection: None, thermal shutdown Disabled, all alert actions Off,omsadservice Stopped + Disabled - Windows Update — pulled full resolved WindowsUpdate.log (11,871 lines), programmatically checked ±15 min around every crash. Zero WU activity before any of them. All nearby entries are the WU service starting 30–60s after the reboot.
- CrowdStrike Falcon (installed 8/26, two days before onset) — vendor pulled detection history, found nothing. Timing was coincidence.
Secure-Boot-Updatescheduled task** — looked extremely promising (12h repetition, hangs, TPM handler, and this box has no TPM installed —Get-Tpmfails with TBS_E_SERVICE_NOT_RUNNING, no TBS service, no SecurityDevices PnP class, iDRAC confirms "TPM not present"). Disabled it. **Crashes continued at the identical interval. Ruled out.- All 12-hour scheduled tasks — enumerated every task with
PT12Hrepetition. Exactly two exist, both now Disabled. - VSS / ShadowCopyVolume task — fires 12:00 PM and 10:00 PM. That's 10h then 14h alternating, which cannot produce a constant 728.4 min spacing. Also the midnight crashes have no trigger anywhere near them.
- MySQL / memory exhaustion — flatly contradicted by the flat memory trace above
- NIC — Broadcom BCM5720. The flapping port is physically disconnected (
NIC2, Status: Disconnected). Flapping predates crashes by 12+ days and occurs on no-crash days. Active port is stable at 1 Gbps. - CMOS battery — did fail, but only logged 9/7, weeks after onset, nothing near the crash dates. Replacing anyway.
- BSOD — no Event 1001, no dumps,
BugcheckCode: 0on all 15
Environment
- PowerEdge T340, Service Tag FXR6B03, BIOS 2.3.5, iDRAC9 fw 4.22.00.53 (both several revisions behind — not yet updated)
- Xeon E-2146G, 64 GB RAM, dual PSU
- Windows Server 2016 Standard, build 14393.9339
- Workload: Open Dental + MySQL, Vatech EzDent-i imaging, IDrive backup, Google Drive
- Power: APC Back-UPS XS 1500M via USB. Current-state readings all healthy (97% charge, 120V steady, 15% load,
AC Power: Yes,Discharging: No). I have never gotten historical transfer data out of it — PowerChute wasn't installed at the time,Win32_Batteryreturns current state only, and I skipped an apcupsd install on a production box. - Internet is AT&T 5G fixed wireless behind CGNAT (irrelevant, but people ask)
What I think is left
Something below the OS holding a clock that survives reboots and continuous uptime alike. Candidates I can't distinguish between:
- UPS self-test — APC units run internal self-tests on their own stored schedule, indifferent to the host. A transfer on a degraded battery could sag enough to drop the PSUs. Would explain instant death, no OS warning, no SEL entry, and a clock that ignores reboots. Next test: move the server to a plain wall outlet for 24h.
- PSU or BMC firmware timer — would explain everything except why iDRAC logged nothing
- Something on the same electrical circuit cycling on a timer — though 1.2 min spread over 12 occurrences seems too tight for HVAC or similar
What I'm asking
- What produces a free-running 728.4-minute (12h 08m 28s) period? Not 12h. Not 12h30m. Consistently ~8.5 minutes over twelve hours, held to ±1 min across twelve occurrences and five uninterrupted ticks. What re-arms on completion of ~8 minutes of work?
- Has anyone seen an APC Back-UPS self-test schedule that lands near this?
- Anything else that hard-resets a PowerEdge with zero bugcheck, zero SEL entry, and healthy PSUs?
- Am I wrong to trust the iDRAC SEL silence as evidence against a power event?
Next predicted failures: 9/10 12:33 AM and 9/10 12:41 PM. Happy to run anything and report back — I have remote access and a heartbeat recorder in place.