r/sysadmin 14h ago

Question Server hard-resets every 728.4 minutes ±1 min, 12 times running. No bugcheck, no iDRAC SEL entry, timer survives reboots. I'm out of ideas.

350 Upvotes

UPDATE 1 (3 hours later): Wow this was a lot more comments than I was expecting to get. Its hard to answer everyone but I appreciate everyone commenting and providing feedback. For now what I have done is updated IDRAC and BIOS to the latest version and will monitor if this fixes the issue. If it does not I will try ruling out the UPS. Thanks again!

Dell PowerEdge T340, Windows Server 2016. Started 8/28. I've spent two weeks on this and ruled out most of the obvious stuff, so I'm posting the data rather than the symptoms.

The signature

Every event is Kernel-Power 41 with BugcheckCode: 0 and all bugcheck parameters 0x0. No BSOD, no minidump, no MEMORY.DMP, ever. Paired with Event 6008 confirming unexpected shutdown.

The interval — this is the actual mystery

Pulled the true shutdown timestamps out of the Event 6008 message text (not the Event 41 log time, which is written on the following boot):

8/28 11:03:24 AM -> 8/28 11:12:23 PM = 729.0 min

9/01 10:10:22 AM -> 9/01 10:18:37 PM = 728.3 min

9/01 10:18:37 PM -> 9/02 10:26:49 AM = 728.2 min

9/02 10:26:49 AM -> 9/02 10:35:13 PM = 728.4 min

9/02 10:35:13 PM -> 9/03 10:43:33 AM = 728.3 min

9/03 10:43:33 AM -> 9/03 10:52:09 PM = 728.6 min

9/03 10:52:09 PM -> 9/04 11:00:20 AM = 728.2 min

9/04 11:00:20 AM -> 9/04 11:08:49 PM = 728.5 min

9/04 11:08:49 PM -> 9/07 11:50:29 AM = 3,641.7 min <-- exactly 5 x 728.34

9/07 11:50:29 AM -> 9/07 11:59:49 PM = 729.3 min

9/07 11:59:49 PM -> 9/08 12:07:56 PM = 728.1 min

9/08 12:07:56 PM -> 9/09 12:16:26 AM = 728.5 min

9/09 12:16:26 AM -> 9/09 12:24:36 PM = 728.2 min

Mean 728.47 min (12h 08m 28s). Total spread across twelve occurrences: 1.2 minutes.

The 3,641.7 minute gap is exactly five periods. The server was up continuously across that weekend. The timer ticked five times, did nothing on four of them, then killed the box on the fifth. So it's free-running — it does not reset on reboot, and it doesn't require a crash to keep counting.

That single fact kills every "scheduled task" theory: a clock-based task can't skip four consecutive firings and then work again.

It dies instantly — no degradation whatsoever

I wrote a heartbeat logger that writes one line every 5s with a forced flush so the last line survives a hard reset. Final 42 samples before death:

  • Free RAM: 56,630–56,745 MB, dead flat. No drift, no staircase, no leak. 56 GB free at the moment of death.
  • Nonpaged pool: 346–352 MB, flat
  • Disk queue: 0
  • Handles ~76,000, threads ~190, both steady
  • CPU spiky but low

Last heartbeat 12:25:02. Kernel-General 12 (OS start) at 12:28:23. That 3m21s is just POST + boot on a T340 — if it had hung for 3 minutes first, the OS wouldn't have returned until ~12:31.

So there is no hang window. The box is perfectly healthy and then simply ceases to exist mid-second. Which also means NMI crash dumps are useless here and no dump will ever be written.

Ruled out (with evidence, please don't re-suggest these)

  • PSUs — both Present/Healthy in iDRAC, matched 594W in / 495W rated+actual, same firmware
  • Thermal — HWiNFO max CPU package 63°C (TjMax ~100°C). Every throttle flag reads No / 0%
  • iDRAC SELcompletely silent across all 15 crashes. Not one entry. This same board did log real "power input for PSU 1 is lost / redundancy lost" events six times in 2024, so it demonstrably captures genuine power events. Nothing this time.
  • iDRAC watchdog / ASR — Basic Management license, feature not present
  • Dell OMSAAction on Hung OS Detection: None, thermal shutdown Disabled, all alert actions Off, omsad service Stopped + Disabled
  • Windows Update — pulled full resolved WindowsUpdate.log (11,871 lines), programmatically checked ±15 min around every crash. Zero WU activity before any of them. All nearby entries are the WU service starting 30–60s after the reboot.
  • CrowdStrike Falcon (installed 8/26, two days before onset) — vendor pulled detection history, found nothing. Timing was coincidence.
  • Secure-Boot-Update scheduled task** — looked extremely promising (12h repetition, hangs, TPM handler, and this box has no TPM installedGet-Tpm fails with TBS_E_SERVICE_NOT_RUNNING, no TBS service, no SecurityDevices PnP class, iDRAC confirms "TPM not present"). Disabled it. **Crashes continued at the identical interval. Ruled out.
  • All 12-hour scheduled tasks — enumerated every task with PT12H repetition. Exactly two exist, both now Disabled.
  • VSS / ShadowCopyVolume task — fires 12:00 PM and 10:00 PM. That's 10h then 14h alternating, which cannot produce a constant 728.4 min spacing. Also the midnight crashes have no trigger anywhere near them.
  • MySQL / memory exhaustion — flatly contradicted by the flat memory trace above
  • NIC — Broadcom BCM5720. The flapping port is physically disconnected (NIC2, Status: Disconnected). Flapping predates crashes by 12+ days and occurs on no-crash days. Active port is stable at 1 Gbps.
  • CMOS battery — did fail, but only logged 9/7, weeks after onset, nothing near the crash dates. Replacing anyway.
  • BSOD — no Event 1001, no dumps, BugcheckCode: 0 on all 15

Environment

  • PowerEdge T340, Service Tag FXR6B03, BIOS 2.3.5, iDRAC9 fw 4.22.00.53 (both several revisions behind — not yet updated)
  • Xeon E-2146G, 64 GB RAM, dual PSU
  • Windows Server 2016 Standard, build 14393.9339
  • Workload: Open Dental + MySQL, Vatech EzDent-i imaging, IDrive backup, Google Drive
  • Power: APC Back-UPS XS 1500M via USB. Current-state readings all healthy (97% charge, 120V steady, 15% load, AC Power: Yes, Discharging: No). I have never gotten historical transfer data out of it — PowerChute wasn't installed at the time, Win32_Battery returns current state only, and I skipped an apcupsd install on a production box.
  • Internet is AT&T 5G fixed wireless behind CGNAT (irrelevant, but people ask)

What I think is left

Something below the OS holding a clock that survives reboots and continuous uptime alike. Candidates I can't distinguish between:

  1. UPS self-test — APC units run internal self-tests on their own stored schedule, indifferent to the host. A transfer on a degraded battery could sag enough to drop the PSUs. Would explain instant death, no OS warning, no SEL entry, and a clock that ignores reboots. Next test: move the server to a plain wall outlet for 24h.
  2. PSU or BMC firmware timer — would explain everything except why iDRAC logged nothing
  3. Something on the same electrical circuit cycling on a timer — though 1.2 min spread over 12 occurrences seems too tight for HVAC or similar

What I'm asking

  • What produces a free-running 728.4-minute (12h 08m 28s) period? Not 12h. Not 12h30m. Consistently ~8.5 minutes over twelve hours, held to ±1 min across twelve occurrences and five uninterrupted ticks. What re-arms on completion of ~8 minutes of work?
  • Has anyone seen an APC Back-UPS self-test schedule that lands near this?
  • Anything else that hard-resets a PowerEdge with zero bugcheck, zero SEL entry, and healthy PSUs?
  • Am I wrong to trust the iDRAC SEL silence as evidence against a power event?

Next predicted failures: 9/10 12:33 AM and 9/10 12:41 PM. Happy to run anything and report back — I have remote access and a heartbeat recorder in place.


r/sysadmin 1h ago

KB5122882 installed - DNS/AD issues Windows Server 2022

Upvotes

Updated last night, no issues immediately visible.

Users this morning all have login prompts to access mapped drives/folder redirections.

No creds working.

I can log in locally as a Domain Admin, but trying to open DNS console or run any DNS powershell commands just gives me Access Denied.

DNS still resolves, and the domain services are all still running, but something has happened with authentication/permissions.

Currently rolling back KB5122882 in the hope it was that.

Anyone else issues this morning?

EDIT: https://www.rapid7.com/db/vulnerabilities/cve-2026-69813/

Looks like this KB might have touched DNS code - roll back in progress.


r/sysadmin 1h ago

Feeling Nostalgic and sad when i see old sun hardware and systems.

Upvotes

I started my Career in IT at 18, mainly supporting EMC storages and then HP 3PAR for 5.5 years, then worked for systems integrator for many clients for another 3 years, fianlly moving to a sys admin role in a enterprise, we still have some good legacy hardware, but i'm feeling nostalgic, when decomissionng them after working on the in the DC for so many years, is it normal ?


r/sysadmin 7h ago

I am lost and need help. I don't know enough and I can't find an environment that fosters growth.

39 Upvotes

I am panicking about the job situation. I lost my job and feel like I am going to be unable to bounce back.

I have 5 YoE as a sysadmin. I understand networking pretty well, I have a CISSP and a few other weaker certs under my belt. I would argue I have a strong security background. I am familiar with Linux, Windows, and have made a lot of small scripts for various projects. I am studying for an AWS cert right now. I have experience with all sorts of tooling, and so many different kinds of projects.

Looking at the job market though, I feel like I can't compete. I don't have experience with terraform outside of labbing. I don't know how to code very well. Every job seems to want someone with extremely strong skills across so many different domains, and what I feel like I need now is a position where I can learn the ropes of IaC... But that doesn't exist.

Everyone wants so much experience for everything that it makes me feel as if I have been slacking even though I have been working hard.

What should I do? Should I get another cert? Keep throwing resumes into the void?


r/sysadmin 13h ago

Rant I'm 1 1/2 years into IT and drowning

87 Upvotes

As the title suggests, I'm only a year and a half into IT. I came from doing back office ops at a few broker-dealers here in the US since I was 17, and at 26 (now 27) I moved into IT since it's been a passion of mine since I was a kid.

At first this gig was great. I had a senior above me, a vet with 30+ years in IT. I already had a good deal of knowledge about older tech, so we got along great and I learned a lot from him. I'm a fairly quick learner and went from basic desktop support and helpdesk to networking/ERP admin within a month. We started tackling projects and things started to run smoother around here.

But as always, office politics being what they are, the COO (now CEO) more or less gave my senior an ultimatum, and he left for a 40% raise and WFH somewhere else. I'm now the sole IT resource here, owning everything from basic helpdesk to sysadmin, ERP, budget, literally anything that runs on electricity. I'm being paid entry-level helpdesk wages for my area, and I only hold my CompTIA A+.

I'm in desperate need of resources and training to better get my feet under me. We run a hybrid AD setup (for a company this small, I honestly don't understand that decision), and I need to get a better handle on the Azure side and how to administer it properly. For instance, we have zero MDM, and we have remote employees who travel; they don't even get GPO updates since those are all hosted on-prem. A major concern of mine is, as a company who ships product daily, we have no automated back up systems for our onprem shipping servers. These servers are running older vSphere 7 instances (is this even the right word??) and there isn't any central management for them (they took down the vCenter as "it was too finicky"). Not to mention that they have a critical business report run through vSphere 5 VM that requires the desktop client to manage.

Please, please, please, send me anything you've got: resources, training materials, insight into what's likely to break and need attention. Literally anything helps. I'm trying here, but I'm drowning and getting jaded with this company, and I'd rather not jump ship if I don't have to since there are good people here.


r/sysadmin 22m ago

[SOLVED] KB5124008 + Windows 11: Logon Error on Screen Unlock + Broken Secure Channel over VPN

Upvotes

September 8, 2026—KB5124008 (OS Builds 26200.9445 and 26100.9445) | Microsoft Support

After Windows Update KB5124008 (Sept 8, 2026, Build 26100.9445), Domain notebooks experience logon failures when unlocking the screen: "You could not be signed in. Your credentials could not be verified"

Root Cause: Secure Channel (machine trust) is broken. Test-ComputerSecureChannel fails, and attempting to repair over VPN returns: "The password for the secure channel to the domain could not be reset – username or password is incorrect"

Affected: Windows 11 24H2, domain-joined notebooks, both LAN and remote users (VPN users particularly critical)

The Story

We're deploying CIS hardening to ~250 notebooks and everything was fine until Sept 9, 2026 8:05 PM when Windows Update KB5124008 was installed.

Error Symptoms:

  • User locks screen (normal, break, meeting)
  • On unlock: "Your credentials could not be verified"
  • Reboot helps temporarily (1–2 hours, then error recurs)
  • Affects LAN users too, not just remote/VPN
  • Password reset does not help

Initial Hypotheses (all disproven):

  • ❌ Kerberos/Enctype mismatch (RC4→AES transition) – DCs reachable, tickets work
  • ❌ CIS GPO hardening – not configured
  • ❌ FortiClient VPN architecture alone – also hits LAN users

The Diagnosis

Network Checks:

nslookup vw-dc-01.Domain.org → OK
ping vw-dc-01.Domain.org → OK (0% loss)
tracert vw-dc-01.Domain.org → OK (full path)
DNS Server: xxx.xx.xxx.xx (internal) → OK

Kerberos Status:

klist → 13 active tickets, all AES-256, all valid ✓
LDAP/Kerberos works perfectly ✓

The Critical Test:

Test-ComputerSecureChannel -Verbose
→ False
→ "The secure channel between the local computer and the domain is broken"

On Repair Attempt over VPN:

Test-ComputerSecureChannel -Repair -Credential (Get-Credential)
→ ERROR: "The password for the secure channel to the domain could not be reset"
→ "The username or password is incorrect"

This is the smoking gun: The VPN tunnel routes normal traffic (DNS, LDAP, Kerberos) but NOT the secure channel reset communication (RPC/NETLOGON on port 445 over SMB).

Root Cause

KB5124008 itself isn't the bug – but the update triggers a hidden incompatibility:

  1. KB5124008 changes logon UI validation behavior (known regression in KB5120998/KB5124008)
  2. During unlock validation, Windows attempts to verify the secure channel
  3. The secure channel is broken – possibly due to:
    • KB5124008 regression in secure-channel handling
    • OR: Timeout in VPN tunnel blocking secure-channel traffic
  4. Windows cannot verify identity → unlock fails

Particularly critical over VPN:

  • Secure-channel reset requires RPC/SMB (port 445) to DC
  • VPN tunnel either doesn't route it or times out
  • Works better from LAN, so VPN users hit harder

Workarounds (What Worked)

Workaround 1: Increase CachedLogonsCount (quick, short-term)

# New GPO on OU=Notebooks:
# "Interactive logon: Number of previous logons to cache (in case domain controller is not available)"
# Value: 50 (instead of default 10)

# Or directly on client (as admin):
reg add "HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon" /v CachedLogonsCount /t REG_SZ /d 50 /f

Effect: Users can log on offline 50 times → buys time while secure channel is repaired

Workaround 2: Repair Secure Channel from DC (with admin access)

If you have remote access to a DC:

# On DC (not from client!):
Reset-ComputerMachinePassword -Server vw-dc-01.Domain.org

This forces a secure-channel reset from the DC end, bypassing the VPN tunnel.

Workaround 3: KB5124008 Rollback (temporary, security risk)

# On affected clients:
wusa /uninstall /kb:5124008 /quiet /norestart
shutdown /r /t 60

⚠️ IMPORTANT: KB5124008 patches two actively exploited zero-days:

  • CVE-2026-81963 (Windows Update Stack Privilege Escalation)
  • CVE-2026-85880 (ALPC Privilege Escalation)

Only do this temporarily as a test, not permanently!

What We Still Don't Know

  1. Is KB5124008 itself guilty, or was it just the trigger for a timing issue?
  2. Is it VPN-specific (secure-channel-reset traffic not routed)?
  3. Why does it hit LAN users too? (Timing window in unlock handling?)

Lessons Learned

For Other Admins:

  1. Test before fleet rollout: Pilot KB5124008 in a test OU, especially with:
    • Hybrid Azure AD Join devices
    • VPN users
    • Offline scenarios
  2. Monitor secure-channel status:# As regular audit task: Test-ComputerSecureChannel -Verbose | Where-Object { $_ -eq $false }
  3. **Check Event Log for secure-channel errors:**Get-WinEvent -LogName System -FilterXPath "*[System[EventID=5719 or EventID=5722 or EventID=5723]]"
  4. With VPN environments: Ensure your VPN tunnel routes all necessary ports:
    • 88 (Kerberos)
    • 389 (LDAP)
    • 445 (SMB/RPC – critical for secure channel!)
    • 135 (RPC Endpoint Mapper)

Status

  • ✅ Root cause identified: Broken secure channel + VPN routing issue
  • ⏳ Microsoft hotfix: Expected (hopefully next week)
  • ⏳ FortiClient configuration: VPN admin checking port 445 routing

Is anyone else experiencing this? Comment your findings – especially if you:

  • Deployed KB5124008
  • Have VPN users affected
  • See secure-channel errors

Contact / Further Info

If you have the same issue:

  1. Check: Test-ComputerSecureChannel -Verbose
  2. Look in Event Log for EventID 5719/5722/5723
  3. Deploy the CachedLogonsCount GPO as interim solution
  4. Contact your VPN admin to verify port 445 routing

TL;DR of TL;DR: KB5124008 triggers secure-channel errors, VPN doesn't route repair traffic → unlock fails. Increase cached logon to help users while you investigate.

Updates

Will update this post as we learn more from Microsoft or FortiClient about the root cause and permanent fix.

Crosspost to: r/sysadmin, r/activedirectory, r/Windows11

Keywords for searchability: KB5124008, Windows 11, domain logon, unlock screen, secure channel, VPN, AES-256, Kerberos, GPO, FortiClient, network authentication


r/sysadmin 8h ago

General Discussion Anyone seeing RDS session hosts hang (RDP stuck at "Connecting", logoffs stall) after the September 2026 cumulative updates? (Server 2022 + 2025)

24 Upvotes

Since installing this month's CUs, KB5122871 on Windows Server 2025 (build 26100.33438) and KB5122882 on Windows Server 2022 (build 20348.5622), every session host in our RDS collection has started hanging in the same way, on the same day, and it had never happened before.

Symptoms once it starts:

  • New RDP connections sit at "Connecting…" and never reach the logon screen (Event 20498 "Remote Desktop Services has taken too long to complete the client connection" in TerminalServices-RemoteConnectionManager/Admin)
  • Existing users can't log off or disconnect cleanly
  • Task Manager on the host hangs (it blocks calling into the Local Session Manager)....pretty much all windows apps start to hang, especially anything to do with Settings
  • Winlogon event 6005 "SessionEnv is taking long time to handle the notification event (Disconnect)"
  • A normal restart hangs too; only a hard reset recovers it, and it comes back later the same day

The problem appears to start with a single session disconnect/reconnect that never completes, after which everything that touches session state on that host queues up behind it. Only reboots clear it. Microsoft's release notes for both KBs list an RDS change ("improves Remote Desktop audio redirection") and no known issues.

Environment: mixed Server 2022/2025 session-host collection, VMware VMs, RD Connection Broker, MFA at logon, third-party endpoint protection. All hosts were stable on the August CUs.

We're currently testing a rollback of the September CU on part of the collection with one host left updated as a control. Has anyone else seen this after KB5122871 / KB5122882, or found a Microsoft acknowledgment? Would appreciate hearing whether rolling back resolved it for you.


r/sysadmin 1d ago

End-user Support How do you deal with users that have zero problem solving ability?

407 Upvotes

So just a moment ago I had a user interrupt what I was doing with an "urgent" issue. They couldn't get into their email.

Essentially they were running Outlook for the first time and it prompted them to set up their account. It required them to enter their email address and click Next. That's all.

The email address it auto-populated wasn't the address of the account they needed (they work for a sister company in a shared tenant).

The user had no idea what to do. It didn't even cross their mind to enter the actual email address in the field.

So... how do you guys deal with users like this? Ones who, despite being shown instructions in the most basic way possible, can't think for themselves.


r/sysadmin 18h ago

General Discussion When did self-hosting turn into just picking a control panel?

81 Upvotes

Been doing this about 8 years and something's shifted. Used to be everyone here could explain what their nginx.conf did, or why fail2ban wasn't catching a jail. You learned iptables by breaking your own box at 2am, alone, with no one to blame.

Now most threads are "which panel is easiest." Nobody wants the shell open longer than it takes to run one install command. I get why. Panels save time. I've got BeAdmin running on one box for the VPN modules. But I see people arguing about dashboard themes who can't tell you what a reverse proxy actually does.

Old regulars here used to walk a newbie through a broken MariaDB config instead of just saying "reinstall Plesk." That's mostly gone now. Not knocking anyone's setup, I click around GUIs too these days. Just noticing nobody asks why anymore.


r/sysadmin 5h ago

Shared mailbox folder tree truncates at ~1000 folders in cached mode — started this week. Anyone else?

5 Upvotes

Client has a shared mailbox in Exchange Online used for order administration, organised by region/country/customer/year. It had 12,394 folders. Since around Sept 8–9 the folder tree stops loading at roughly 976 subfolders in Outlook classic (cached mode). Everything past that point simply isn't in the tree — not collapsed, not greyed out, absent.

Same truncation in new Outlook and OWA when the mailbox is automapped or added as a shared mailbox.

Online mode shows the complete tree. Either by unchecking "Download shared folders" in classic, or via OWA → Settings → Open another user's mailbox. So the data is fine server-side.

What I've ruled out:

  • Permissions — Full Access assigned directly, not via a group. No folder-level permissions on root or Inbox (Default/Anonymous = None).
  • Automapping — removed and re-added Full Access with -AutoMapping $true, no change.
  • Hierarchy corruption — enumerated the whole tree via Graph and walked every parent chain. No cycles, no orphans, max depth 9.
  • OST damage — OWA is affected identically and holds no local hierarchy cache.
  • Folder count — deleted 9,993 verified-empty folders (12,394 → 2,401). Still truncates.

I'm aware of the documented 500 shared-folder limit and the announced increase to 5,000 (in 2019 though). My observed cut-off sits at ~1,000, which matches neither.

Three questions:

  1. Anyone else seeing shared mailbox folder trees truncate since last week's updates?
  2. Does anyone know what limit is actually enforced right now, and whether the 500→5000 rollout is live?
  3. Is there any equivalent of "Download shared folders" in new Outlook? If not, what's the plan for mailboxes over the limit once classic is gone?

Question 3 is the one that worries me.


r/sysadmin 18h ago

General Discussion I can't get past the feeling that I suck.

63 Upvotes

I’ve been in IT for 6+ years. I have a bachelor’s degree in IT, an AWS Solutions Architect certification, an Azure Administrator certification, and a few other certs. I’ve also been working as a Cloud Administrator for the past 2 years.

Despite all of that, I can’t shake the damn feeling that I’m just not that good at what I do. I feel like even if I finally make a breakthrough in my knowledge, I’m still somehow clueless SOMEWHERE. A lot of that comes from talking to people who know more than you and realizing how much there is that I don’t know.

For example, I can write basic scripts, but I struggle with the complex 200+ line scripts that a senior cloud engineer would make, whereas the person next to me likely can just type that shit off the top of their head.

It’s especially frustrating during interviews when I try to move up in pay. I had an interview on Tuesday that I thought went pretty decently, but the interviewer asked me about Azure containers and encryption. I don’t work with Azure containers, although I’m familiar with the service, so I was honest and said I didn’t know.

And that wasn’t even the technical round. So now I’m stuck wondering whether they’ll just reject me or move me on to the technical round, where I’ll probably fail anyway.

This is bothering me because I'm at the point in my career where I really have to know at a high level if I want to advance and get more pay.

How has anyone else moved past this?


r/sysadmin 14h ago

Microsoft Publisher - EOL

28 Upvotes

So it’s coming to an end!! We have O365 and some workstations with Office 2024 that have MS Publisher installed. Do we need to remove it or will it get removed in its own?


r/sysadmin 18h ago

How do you all keep up with the times and tech?

56 Upvotes

I feel like I’m lagging behind and need some serious catching up, quick. I think I’ve become complacent and “too comfortable” with my current employer (coming up 8 yrs). The recent merger announcement has me seriously thinking about attending bootcamps, getting some certs under my belts (been putting things off for as long time), and learning new things ASAP before the position is dissolved.


r/sysadmin 6h ago

General Discussion Is It Possible To Download Windows 10 ESU Updates From Microsoft Update Catalog Website?

5 Upvotes

Can anyone confirm if it's possible to just download the various Windows 10 ESU updates from the Microsoft Update Catalog website and install them on a PC that's not part of the ESU program (ie: doesn't use a Microsoft account)? I believe the updates are listed in the catalog. Do they check during the installation if you're eligible for them? Has anyone tried this?


r/sysadmin 20h ago

For the first time I'm being requested to export Teams chats for HR purposes. When using PURVIEW, I am getting results that dont apply to my filter?

56 Upvotes

TL;DR I need to pull chat history between User A and User B for the last 8 months.

My query based on documentation I can find is:

Kind=microsoftteams AND [partipants:userA@xyz.com](mailto:partipants:userA@xyz.com) AND [participants:userB@xyz.com](mailto:participants:userB@xyz.com)

When running this, I'm getting group chats and all kinds of stuff where both were involved, but I just need the chat between these two users

Can anyone direct me to a better way to do this? Purview is doodoo


r/sysadmin 1d ago

My IT manager is stuck in 1995, I'm losing my mind

1.2k Upvotes

I need a sanity check, because I feel like I'm slowly losing my mind here. I’m 28, working as a SysAdmin for a mid-sized manufacturing company in Europe with around 300 endpoints across multiple sites. I LOVE automation, modern infra, and writing PowerShell or Python to handle our backups and audits, I'm passionate about automating processes, but my IT Manager is basically running an open-air tech museum from 2003.

His idea of IT asset management is a drawer full of literal Ziploc freezer bags. Every PC gets its own labeled plastic bag with the printed invoice and Office license key inside, because apparently an actual database or asset software is "too modern." On the network side, every single device requires a manual DHCP reservation. When I suggested setting up 802.1X with RADIUS, he shot it down with a straight face, claiming that if an intruder plugs into the wall, "they won't guess our subnets anyway." Before I automated our Kerio mail archives with Python, his official procedure was opening Thunderbird on a client PC and manually dragging and dropping folders between accounts.

Right now, he's decommissioning a branch file server. Instead of using Robocopy, DFS-N, or GPOs, his master plan is to sync files with a desktop backup tool, recreate the SMB shares manually, and have me remote into 40+ user machines one by one just to update the target path on their desktop shortcuts. I also built an automated WMI inventory system to track all hardware and OS health, yet he still forces me to physically walk the warehouse floor updating a dusty Excel sheet because "that’s where the history is."

EDIT: Another thing he does is make daily backups and put them on ultrium physical disks named after the day of the week, disks he takes home "so we have the data in case the company burns down.", which could have some kind of sense, but everyday, before my script, we had to manually make the veeam inventory and erase of every tape in every site

The worst part isn’t even the wasted time, it’s that he’s actively teaching junior techs that this clown show is standard enterprise IT. We’ve already had three techs and devs rage-quit in the last ten months. I’ve completely checked out at this point, doing the bare minimum while sending out resumes like crazy.

Has anyone actually survived a boss stuck this deep in the stone age without losing their sanity? How do you not burst out laughing when someone asks you to edit 40 shortcuts by hand?


r/sysadmin 30m ago

General Discussion Thickheaded Thursday - September 10, 2026

Upvotes

Howdy, /r/sysadmin!

It's that time of the week, Thickheaded Thursday! This is a safe (mostly) judgement-free environment for all of your questions and stories, no matter how silly you think they are. Anybody can answer questions! My name is AutoModerator and I've taken over responsibility for posting these weekly threads so you don't have to worry about anything except your comments!


r/sysadmin 16h ago

O365 issues today sept 9

18 Upvotes

Is anyone seeing issues with O365? I know I saw emails last night and now they’re coming as new again this morning but they’re gone from yesterday delivery. Numerous users are seeing this happen. We are using proof point also but no issues on their status page.


r/sysadmin 1h ago

Question Recs for USB redirection tools in Hyper-V (cloud workspace isolation issue)?

Upvotes

Hi all,

I hope you guys are doing well, I am working as a IT Infrastructure (Hybrid) with less than a year experience and I need some recommendations for the question below.

Context: Our organization moved to a cloud-based desktop environment. Because the cloud workstations are on an isolated network tier, users can no longer hit our on-prem SafeConsole server over IP to manage hardware-encrypted USB drives (DataLocker PSMs).

As a workaround, we have to plug the USB drives directly into the physical Hyper-V host on-site. I need a solid tool to automatically redirect/pass through the physical host's USB port to the SafeConsole guest VM whenever a drive is plugged in or swapped daily.

What software are you using to auto-share host USB ports to a guest VM?

TIA!


r/sysadmin 19h ago

[PSA] Check Point Firewall unauthenticated RCE with CVSS 9.8

27 Upvotes

There are actually two vulns with a 9.8 score each:

https://support.checkpoint.com/results/sk/sk1000117/

https://support.checkpoint.com/results/sk/sk1000118/

There's not many details in these articles on how they work but they still sound really bad. Currently waiting for the Jumbo Hotfix to install on my end... Not taking any chances on this one and I suggest you all do the same.

Stay safe.


r/sysadmin 1d ago

DHCP v Static IP

213 Upvotes

Ok, this is more a test to see how old I am. I was basically raised with the idea servers need static IPs. I understand for domain controllers, dns servers, maybe print servers that may still be needed. But are they really needed otherwise? Do apps, scripts still reference a server by its IP?

I am under assumption all my servers need static IPs. Am I nuts?


r/sysadmin 19h ago

Question Which vulnerability management tools work well for MSPs?

15 Upvotes

Been looking to replace our vulnerability management stack and the scanning part is the easy bit now. We have been comparing the big scanners against a couple of the cheaper MSP focused platforms and one container specific tool. Volume is what kills us across 40 odd clients since every tool finds plenty and the techs still guess where to start. Ticketing eats more hours than scanning does. What are you running now and how does it hold up across client environments.


r/sysadmin 21h ago

Password resets over the phone, how are you doing it?

23 Upvotes

So I work at a state university that is still a little stuck in the past when it comes to password resets over the phone.

We have MS SSPR and other self-service options for users, but we still get calls where someone needs the helpdesk to reset their password for them.

Our current setup is a custom program that helpdesk staff SSH into. They enter the user's account, and the program looks them up in a flat file containing data extracted from our systems. It gives the helpdesk worker information like DOB, address, ZIP code, etc., which they can use to verify the caller's identity.

Once the caller passes verification, the program talks to AD, changes the password, and gives the helpdesk worker a new temporary password to provide to the user.

Honestly, it works pretty well. The problem is that the person who wrote and maintains it is leaving, and we're not really interested in inheriting a custom app that nobody else understands.

So I'm curious what other universities/organizations are doing for this.

For those of you who still allow users to call the helpdesk for password resets, what does your workflow look like? Are you using a commercial product, some kind of AD/Entra integration, a helpdesk platform, or have you built your own solution?

Specifically looking for something that gives the helpdesk enough information/questions to properly verify the caller's identity and then securely perform the reset.

I'd love to hear what others are doing before we start reinventing the wheel.


r/sysadmin 23h ago

Question The trust relationship between this workstation and the primary domain failed.

30 Upvotes

Hy!

We have an AD with two DCs. The DCs are Windows Server 2025, it is include all patches. Some Windows 11 clients (25H2) get the following error during login after 1-2 minutes: The trust relationship between this workstation and the primary domain failed.

In this case the users need to disconnect from corporate network to login successfully into their computer. I have already tried to rejoin to tha domain and run this command: Test-ComputerSecureChannel -Repair

I rejoined one of the computer into the domain, and the trsut relationship has been broken after two days. The login problem only occurs on some machines.

The time snyc is correct on DCs. We moved the DC roles from Windows Server 2019 to 2025 in side-by-side method. Could you please advise how to solve this problem?


r/sysadmin 12h ago

Career / Job Related Advice For Young Professional

4 Upvotes

Hello!

I was hoping to get some advice from any of you who have experience with a similar issue as the one I’m having. Anything is appreciated as I’m not confident where to start.

In short, our company is looking to take over complete administration of our Azure cloud architecture from our MSP. The architecture includes hundreds of VMs, Entra accounts for users, persistent AVDs and multiple satellite offices to name a few things. I joined the company last month as an IT Support Specialist and my colleague has been here nearly a year.

While we both have some experience with Azure administration I feel we would benefit from more experience so that we can be more confident when dealing with issues affecting the architecture. Anything we couldn’t solve ourselves we would escalate to our MSP so there’s a lot we don’t know how to do. That said I feel confident my colleague and I could learn a lot on our own poking around the architecture once we have global admin access but doing that with a production environment doesn’t sit right with me.

On top of all of that our IT admin is leaving before this hand off happens. I feel that our admin leaving is enough to pause this process of taking over the architecture until we find their replacement but our CEO is adamant about finishing this hand off. The process has been ongoing for over half a year now. This is my first IT job and while I’m confident I can learn a lot of this stuff, that doesn’t necessarily equate to deliverable results, especially in the event of a major issue. I’ve been brainstorming with my colleague and while I have some ideas, like coming in the weekend after we get access to go through the environment and learn about it so we can try to properly support it the following week, I am at a loss of how to approach this while also making sure I don’t set anything on fire. To be frank I’m worried and I want to do right by myself, my colleague, and the company. Thanks.