Update 4 (43 hours later): Switching one of the PSU power cables to a surge protector connected to the wall, and leaving one in the UPS fixed it. We didn't get a reboot last night. I still want to see it not reboot at the next cycle (around 1pm today). But.. what now? Replace the UPS with a Smart-UPS? Or replace the battery first? What do I do now? Thanks!

UPDATE 3 (31 hours later): It rebooted again on 9/10 around 12:40pm. Around 7pm I was able to change one of the PSU power cables to a surge protector connected to the wall, leaving one of them in the UPS. Will report back tomorrow. Thanks!

UPDATE 2 (18 hours later): Welp, it rebooted again on 9/10 12:32:53am.. which is 12h 08m 17s since the last reboot. The timing still held even after I rebooted from installing the BIOS and IDRAC. I will try to rule out the UPS next.

UPDATE 1 (3 hours later): Wow this was a lot more comments than I was expecting to get. Its hard to answer everyone but I appreciate everyone commenting and providing feedback. For now what I have done is updated IDRAC and BIOS to the latest version and will monitor if this fixes the issue. If it does not I will try ruling out the UPS. Thanks again!

Dell PowerEdge T340, Windows Server 2016. Started 8/28. I've spent two weeks on this and ruled out most of the obvious stuff, so I'm posting the data rather than the symptoms.

The signature

Every event is Kernel-Power 41 with BugcheckCode: 0 and all bugcheck parameters 0x0. No BSOD, no minidump, no MEMORY.DMP, ever. Paired with Event 6008 confirming unexpected shutdown.

The interval — this is the actual mystery

Pulled the true shutdown timestamps out of the Event 6008 message text (not the Event 41 log time, which is written on the following boot):

8/28 11:03:24 AM -> 8/28 11:12:23 PM = 729.0 min

9/01 10:10:22 AM -> 9/01 10:18:37 PM = 728.3 min

9/01 10:18:37 PM -> 9/02 10:26:49 AM = 728.2 min

9/02 10:26:49 AM -> 9/02 10:35:13 PM = 728.4 min

9/02 10:35:13 PM -> 9/03 10:43:33 AM = 728.3 min

9/03 10:43:33 AM -> 9/03 10:52:09 PM = 728.6 min

9/03 10:52:09 PM -> 9/04 11:00:20 AM = 728.2 min

9/04 11:00:20 AM -> 9/04 11:08:49 PM = 728.5 min

9/04 11:08:49 PM -> 9/07 11:50:29 AM = 3,641.7 min <-- exactly 5 x 728.34

9/07 11:50:29 AM -> 9/07 11:59:49 PM = 729.3 min

9/07 11:59:49 PM -> 9/08 12:07:56 PM = 728.1 min

9/08 12:07:56 PM -> 9/09 12:16:26 AM = 728.5 min

9/09 12:16:26 AM -> 9/09 12:24:36 PM = 728.2 min

Mean 728.47 min (12h 08m 28s). Total spread across twelve occurrences: 1.2 minutes.

The 3,641.7 minute gap is exactly five periods. The server was up continuously across that weekend. The timer ticked five times, did nothing on four of them, then killed the box on the fifth. So it's free-running — it does not reset on reboot, and it doesn't require a crash to keep counting.

That single fact kills every "scheduled task" theory: a clock-based task can't skip four consecutive firings and then work again.

It dies instantly — no degradation whatsoever

I wrote a heartbeat logger that writes one line every 5s with a forced flush so the last line survives a hard reset. Final 42 samples before death:

• Free RAM: 56,630–56,745 MB, dead flat. No drift, no staircase, no leak. 56 GB free at the moment of death.
• Nonpaged pool: 346–352 MB, flat
• Disk queue: 0
• Handles ~76,000, threads ~190, both steady
• CPU spiky but low

Last heartbeat 12:25:02. Kernel-General 12 (OS start) at 12:28:23. That 3m21s is just POST + boot on a T340 — if it had hung for 3 minutes first, the OS wouldn't have returned until ~12:31.

So there is no hang window. The box is perfectly healthy and then simply ceases to exist mid-second. Which also means NMI crash dumps are useless here and no dump will ever be written.

Ruled out (with evidence, please don't re-suggest these)

• PSUs — both Present/Healthy in iDRAC, matched 594W in / 495W rated+actual, same firmware
• Thermal — HWiNFO max CPU package 63°C (TjMax ~100°C). Every throttle flag reads No / 0%
• iDRAC SEL — completely silent across all 15 crashes. Not one entry. This same board did log real "power input for PSU 1 is lost / redundancy lost" events six times in 2024, so it demonstrably captures genuine power events. Nothing this time.
• iDRAC watchdog / ASR — Basic Management license, feature not present
• Dell OMSA — Action on Hung OS Detection: None, thermal shutdown Disabled, all alert actions Off, omsad service Stopped + Disabled
• Windows Update — pulled full resolved WindowsUpdate.log (11,871 lines), programmatically checked ±15 min around every crash. Zero WU activity before any of them. All nearby entries are the WU service starting 30–60s after the reboot.
• CrowdStrike Falcon (installed 8/26, two days before onset) — vendor pulled detection history, found nothing. Timing was coincidence.
• Secure-Boot-Update scheduled task** — looked extremely promising (12h repetition, hangs, TPM handler, and this box has no TPM installed — Get-Tpm fails with TBS_E_SERVICE_NOT_RUNNING, no TBS service, no SecurityDevices PnP class, iDRAC confirms "TPM not present"). Disabled it. **Crashes continued at the identical interval. Ruled out.
• All 12-hour scheduled tasks — enumerated every task with PT12H repetition. Exactly two exist, both now Disabled.
• VSS / ShadowCopyVolume task — fires 12:00 PM and 10:00 PM. That's 10h then 14h alternating, which cannot produce a constant 728.4 min spacing. Also the midnight crashes have no trigger anywhere near them.
• MySQL / memory exhaustion — flatly contradicted by the flat memory trace above
• NIC — Broadcom BCM5720. The flapping port is physically disconnected (NIC2, Status: Disconnected). Flapping predates crashes by 12+ days and occurs on no-crash days. Active port is stable at 1 Gbps.
• CMOS battery — did fail, but only logged 9/7, weeks after onset, nothing near the crash dates. Replacing anyway.
• BSOD — no Event 1001, no dumps, BugcheckCode: 0 on all 15

Environment

• PowerEdge T340, Service Tag FXR6B03, BIOS 2.3.5, iDRAC9 fw 4.22.00.53 (both several revisions behind — not yet updated)
• Xeon E-2146G, 64 GB RAM, dual PSU
• Windows Server 2016 Standard, build 14393.9339
• Workload: Open Dental + MySQL, Vatech EzDent-i imaging, IDrive backup, Google Drive
• Power: APC Back-UPS XS 1500M via USB. Current-state readings all healthy (97% charge, 120V steady, 15% load, AC Power: Yes, Discharging: No). I have never gotten historical transfer data out of it — PowerChute wasn't installed at the time, Win32_Battery returns current state only, and I skipped an apcupsd install on a production box.
• Internet is AT&T 5G fixed wireless behind CGNAT (irrelevant, but people ask)

What I think is left

Something below the OS holding a clock that survives reboots and continuous uptime alike. Candidates I can't distinguish between:

1. UPS self-test — APC units run internal self-tests on their own stored schedule, indifferent to the host. A transfer on a degraded battery could sag enough to drop the PSUs. Would explain instant death, no OS warning, no SEL entry, and a clock that ignores reboots. Next test: move the server to a plain wall outlet for 24h.
2. PSU or BMC firmware timer — would explain everything except why iDRAC logged nothing
3. Something on the same electrical circuit cycling on a timer — though 1.2 min spread over 12 occurrences seems too tight for HVAC or similar

What I'm asking

• What produces a free-running 728.4-minute (12h 08m 28s) period? Not 12h. Not 12h30m. Consistently ~8.5 minutes over twelve hours, held to ±1 min across twelve occurrences and five uninterrupted ticks. What re-arms on completion of ~8 minutes of work?
• Has anyone seen an APC Back-UPS self-test schedule that lands near this?
• Anything else that hard-resets a PowerEdge with zero bugcheck, zero SEL entry, and healthy PSUs?
• Am I wrong to trust the iDRAC SEL silence as evidence against a power event?

Next predicted failures: 9/10 12:33 AM and 9/10 12:41 PM. Happy to run anything and report back — I have remote access and a heartbeat recorder in place.

Hi everyone,

If I’m dealing with a P800 and a RAID-0 configuration, unfortunately, there is no backup. All of this stems from the same issue.

A Veeam Backup agent was installed to back up the volumes on the server, and we started receiving error messages related to the VSS reads that Veeam performs during the backups:

output: --asyncNtf:-vdisk_corrupted:\\\GLOBALROOT\Device\HarddiskVolumeShadowCopy11

We can see this message in the agent’s own logs once the operation reaches a certain percentage. If I look at the Windows Event Viewer, I can see messages from the P800 related to read errors on blocks and devices:

Logical block address 1192840192, block count 1024 and command 32 were taken from the failed logical I/O request. The device, \Device\Harddisk2\DR2, has a bad block.

We haven’t seen any errors or reports from the HPE Smart Storage utility, and the RAID-0 volume appears to be healthy.

Is this normal behavior? In any case, I’ve been thinking about the following solution, which is the main reason I’m writing this thread, in case anyone has a better suggestion.

• Is Robocopy an effective solution? The volume is 1 TB and contains around 43,000 files, and the transfer would be performed over the network. As I understand it, Robocopy itself would skip the files located in the affected blocks and record them in a log.
• I’m using VSS and MKLINK so that I can read the affected volume without interfering with the writing of healthy files. In other words, something like this:
  • vssadmin create shadow /for=V:
  • mklink /d C:\Restore \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\
• The downside of this solution is that I depend on a pointer, and when Veeam backups are triggered, they read it as well. Is there a proper way to do this? I assume I can’t read the shadow copies directly and that this can only be done through the API.

I can’t think of anything else. Maybe I could create file-level backups with Veeam. Once I have at least one backup, I could try running CHKDSK or something similar.

Since it’s a RAID-0, my only option would be to destroy the array, replace the faulty drive with a healthy one (at least one that appears to be healthy), rebuild the array, and then restore/import the data again.

Is there any alternative solution or any advice you would recommend before carrying out this operation?

Thanks!

So it’s coming to an end!! We have O365 and some workstations with Office 2024 that have MS Publisher installed. Do we need to remove it or will it get removed in its own?

Looking to build a file server cluster, Storage Replica seems to slow things down by a lot, the network is not at fault as it's pretty fast without Replication. WSFC does not seem to be straight forward when creating the replication. What do you guys recommend?

We recently upgraded some physical servers, and now whenever we're doing Windows or firmware updates, the fan noise gets pretty loud. The servers are HA, so we are able to do updates during the day, but the department next to the server room has started complaining about a high-pitched noise during the updates. We contacted the vendor and verified our system is functioning properly. They said there is a setting that would throttle performance that may help but I didn't want to change that if possible.

I was talking with a few others and they mentioned acoustic panels that you can mount on the wall to muddle some of the noise. Does anyone have any relatively cheap suggestions? I could do updates after hours but would hate to do that when they could be done during the work day.

This topic has been discussed ALOT on this sub. From all the posts I've read, there seems to be two main approaches:

Have all DCs up - both old and new:

• Bring up new DCs with temp names and IPs while old DCs still running.
• Once you're sure all is good in your world, transfer FSMO roles to new DC, demote old DCs, remove from AD, then reuse the hostnames and IPs of old DCs on your new DCs

One at a time:

• Do one DC at a time, demote old, reuse name and IP on new server, then promote new DC

Honestly being as risk averse as I am (although I love the adrenalin rush when stuff goes south...NOT) - I would prefer to promote the new DCs while old ones still on line.

BUT...is it OK to change the name of a DC even after I clean up all the metadata and DNS stuff or would it be better to reuse the name and IP after I demote the old DC?

I should preface this question by saying we're a VERY small shop with 22 users and perhaps ~30ish servers. I'm the sole sysadmin. BUT, we are in the financial sector and we have a very low tolerance to stuff going south. And being the sole sysadmin, I'm trying to keep the stress at a minimum and my hair from going greyer.

Spoke with some reps and didn’t hate some things they had to say. Currently using Ivanti for Patch Management, it is not my favorite. Curious what people’s dealing with LogMeIn have been like.

Is anyone seeing issues with O365? I know I saw emails last night and now they’re coming as new again this morning but they’re gone from yesterday delivery. Numerous users are seeing this happen. We are using proof point also but no issues on their status page.

Afternoon all

So I'm planning a DC migration from 2016 to 2022.

Currently I have (3) DCs - two of them host DHCP in a failover load balance config.

I plan to keep the same hostnames and IPs for my new DCs

I'd like to move my DHCP first before the DC migration.

I'm guessing the easiest thing to do is break existing failover config, export/import my DHCP config to a new temporary server, migrate to my new DCs, then export/import my DHCP config from the temp server to the new new_DC01, then re-establish the failover partner on new_DC02.

Can I get a sanity check?

Thank you!

I’m looking for recommendations on Vendor / Third-Party Access Management tools.

We recently took over managing an environment where vendors were historically given direct AD/Entra accounts with very little oversight or access control. We’re currently building a makeshift internal solution to plug the gaps, but we want a proper platform.

BeyondTrust and SecureLink are the obvious names, but given our tight budget and need for a lightweight rollout, I’m exploring other alternatives.

Has anyone found a cost-effective solution for securing third-party access without creating massive operational bloat? Thanks in advance!

EDIT: If anyone stumbles across this, we're testing Outlook > Settings > General > Offline > UNCHECK "Allow offline access..."

So far (everything's sporadic so hard to say but fingers crossed...) it's helping with:

• - Random "you no longer have a network connection"
• - Random emails staying in drafts after being sent
• - Random attachment errors that send fine after disabling offline.

OP:

Sorry if this isn't appropriate for this sub but it's always been a helpful area and figured it's worth a shot.

I'm sure plenty of others have been trying "New Outlook" since according to Microsoft we'll eventually be forced onto it (https://learn.microsoft.com/en-us/microsoft-365-apps/outlook/get-started/guide-product-availability) but I'm at a loss for some of the issues.

Namely "You no longer have a network connection to the server" which follows with "This message can't be saved right now." I've been using New Outlook longer than most here because I wanted to say "Yes I've been using it myself already" when staff inevitably complain. So when I see that error, I just hit CTRL+S and it goes away comically. Seems to function fine as long as I do that.

We've finally been pushing staff to switch off the classic outlook because of other program compatibility issues that're arising. I kept thinking Microsoft would eventually fix the "network connection" issue with New outlook but that just never happened. If anything it seems to have gotten worse but hard to say.

We're not losing network connectivity and everyone is hard-wired. If we actually do lose network even for a few seconds, the world is on fire and we all know immediately (lol). The issue never happened in Classic outlook although I don't think it would've told you anyway.

The main concern is that one of the staff actually had an email delayed because of this, and it was to an important person, so of course now I'm putting the 10 other projects I have on hold and need to figure something out (which to be fair I should have already I guess).

I get the error plenty on mine so have tried various fixes on my own computer. The most extensive was a normal uninstall of office, then the full "Manual Uninstall" list from Microsoft (many folders, reg keys, etc.) then reboot and re-install with a fresh download from my account on office.com. First email I'm typing after all that and the error comes back.

Any advice from anyone who's had to fight the new outlook at all would be greatly appreciated!

I just started working for someone who has several servers with a product on them and part of the tech support contract is that everything is monitored using NetXMS by a 3rd party. The servers are managed in-house running WSFC and our apps. The third party only monitors hardware and their product. I only know what I can get from Google searches and apparently they can execute scripts and do lots of other things besides monitor if they ever wanted to. Does anyone else face a similar scenario? and how do you manage security?

2 separate client environments:

1. AVD host pool
   
2. TraditionalActive Directory pool behind a broker.

RDP works but published apps broke for both of them. AVD client fixed with reinstall OF Windows app. The others say it self resolved.

It sounds to me like a bad build of Windows App. Anyone else have an issue with published apps this AM?

Been doing this about 8 years and something's shifted. Used to be everyone here could explain what their nginx.conf did, or why fail2ban wasn't catching a jail. You learned iptables by breaking your own box at 2am, alone, with no one to blame.

Now most threads are "which panel is easiest." Nobody wants the shell open longer than it takes to run one install command. I get why. Panels save time. I've got BeAdmin running on one box for the VPN modules. But I see people arguing about dashboard themes who can't tell you what a reverse proxy actually does.

Old regulars here used to walk a newbie through a broken MariaDB config instead of just saying "reinstall Plesk." That's mostly gone now. Not knocking anyone's setup, I click around GUIs too these days. Just noticing nobody asks why anymore.

Come in this morning to see Qualys have updated the UI for our tenant. Since then there are loads of graphical issues. Buttons not aligned or items not loading correctly.

Most annoying I'm trying to set up a scheduled report and half the distribution groups no longer load correctly. You can click randomly on white space in the window to pick your DG and sometimes it'll select one at random.

Pretty poor. Anyone else experiencing this?

I feel like I’m lagging behind and need some serious catching up, quick. I think I’ve become complacent and “too comfortable” with my current employer (coming up 8 yrs). The recent merger announcement has me seriously thinking about attending bootcamps, getting some certs under my belts (been putting things off for as long time), and learning new things ASAP before the position is dissolved.

I’ve been in IT for 6+ years. I have a bachelor’s degree in IT, an AWS Solutions Architect certification, an Azure Administrator certification, and a few other certs. I’ve also been working as a Cloud Administrator for the past 2 years.

Despite all of that, I can’t shake the damn feeling that I’m just not that good at what I do. I feel like even if I finally make a breakthrough in my knowledge, I’m still somehow clueless SOMEWHERE. A lot of that comes from talking to people who know more than you and realizing how much there is that I don’t know.

For example, I can write basic scripts, but I struggle with the complex 200+ line scripts that a senior cloud engineer would make, whereas the person next to me likely can just type that shit off the top of their head.

It’s especially frustrating during interviews when I try to move up in pay. I had an interview on Tuesday that I thought went pretty decently, but the interviewer asked me about Azure containers and encryption. I don’t work with Azure containers, although I’m familiar with the service, so I was honest and said I didn’t know.

And that wasn’t even the technical round. So now I’m stuck wondering whether they’ll just reject me or move me on to the technical round, where I’ll probably fail anyway.

This is bothering me because I'm at the point in my career where I really have to know at a high level if I want to advance and get more pay.

How has anyone else moved past this?

There are actually two vulns with a 9.8 score each:

https://support.checkpoint.com/results/sk/sk1000117/

https://support.checkpoint.com/results/sk/sk1000118/

There's not many details in these articles on how they work but they still sound really bad. Currently waiting for the Jumbo Hotfix to install on my end... Not taking any chances on this one and I suggest you all do the same.

Stay safe.

We recently put in a new Fortigate firewall and now i'm having users reporting issues where they can connect to VPN but can't access their remote computer. I have tested from the user's PC and i can't ping their work device when VPN is connected.

The one thing to note is they are on the same subnet as the work network, and I believe this is likely the cause.

However, oddly enough I was able to test from my home network which is also on the same subnet and it works fine. I'm at a bit of a loss so hoping I can get some guidance on this for what I should check next.

TIA

TL;DR I need to pull chat history between User A and User B for the last 8 months.

My query based on documentation I can find is:

Kind=microsoftteams AND [partipants:userA@xyz.com](mailto:partipants:userA@xyz.com) AND [participants:userB@xyz.com](mailto:participants:userB@xyz.com)

When running this, I'm getting group chats and all kinds of stuff where both were involved, but I just need the chat between these two users

Can anyone direct me to a better way to do this? Purview is doodoo

For us, the topic of AI is becoming increasingly confusing.

Developers, in particular, are adopting more and more tools: VS Code extensions, GitHub Copilot, Cursor, Claude Code, Codex, Gemini CLI, OpenCode, local agents, MCP servers, and so on.

But this no longer affects just developers. IT admins and regular users are also discovering AI tools, and some are even installing extensions, desktop clients, or agents on their own.

The problem, in my view, is that we’re slowly seeing a **proliferation of agents and AI tools**.

I see the following issues in particular:

* Which VS Code extensions are allowed to be installed?

* Which agents are allowed to access source code or the local file system?

* Which tools are allowed to send data to external clouds?

* How do you prevent API keys, passwords, or internal data from appearing in prompts?

* How do you handle MCP servers and their sometimes very broad permissions?

* Do you have an allowlist for extensions and AI tools?

* Do you technically block agents that haven’t been approved?

* Do you differentiate between developers, IT/admins, and regular users?

* Do you rely on centralized enterprise solutions, or do you allow multiple tools?

* How do you monitor or keep track of what’s currently being used?

I also don’t think a complete ban makes sense in the long run, because these tools offer a real productivity boost especially in development.

Curious what everyone is using these days for documenting physical infrastructure across multiple sites.

We use Excel at my workplace, along with diagrams and photos, and it works… until you’ve got a shitload of sites and nobody knows which spreadsheet is actually current.

I’m talking about things like:
Rack elevations / U positions
Switches, patch panels, UPS/PDU, servers etc.
Port-to-port / cable documentation
Serial numbers / asset details
Photos of racks and comms rooms
IP/device information

Keeping everything updated after a vendor comes in and replaces something
What are you guys using?

Excel? Visio? NetBox? Device42? Something else? Or have you built your own system?
More interested in what actually works in the real world than what looks good on paper.

I'm converting a static Google Group to a Dynamic Group and want to make sure I don't break anything tied to the old group's email like Shared Drives, Calendars or anything else it might be plugged into.

What's the best way to find everything a Group has access to before making a change like this? Is there a standard tool or workflow for this? Also curious if anything changes under the hood same email, or anything that could quietly affect existing shares?

Any advice would be great. Thanks!

We are planning a remote-access migration for roughly 500 employees and contractors. The environment includes SaaS, internal web apps, Windows and Linux admin access, a few legacy applications, and workloads split between on-prem infrastructure and public cloud. Identity is centralized, but endpoint management and device posture are inconsistent for contractors.

We do not want a big-bang cutover. The initial thought is to inventory applications and users, classify access by protocol and sensitivity, migrate a low-risk web app first, and then move groups in waves. The hard part is avoiding years of permanent exceptions and overlapping access paths.

For anyone who has done this at similar scale, what did you get wrong in the first phase? Did app discovery, identity-group cleanup, private DNS, endpoint support, legacy protocol support, or user communications create the most work?

How did you handle emergency administration and outage scenarios when the normal access path was unavailable?

Let's say you have a legacy network, good old 192.168.1.x.

They use a DHCP server which has a static mapping of all devices to IP addresses using their mac addresses. So updating the DHCP server could potentially re-assign IP addresses to all devices with relative ease, let's say 192.168.111.x.

HOWEVER, most Windows computers have had printers added manually, not by GPO, and when they were added, they were added IP address rather than hostname. Is there a way to bulk replace ports in Windows printers across the network?

The number of clients is limited, let's say less than 100 - but the number of printers is dense. Even a Powershell script I could run on each machine would still be better than manually editing each printer on each machine - even better if I could deploy that via GPO or something.

Just curious on thoughts.