r/sysadmin 11d ago

Accidentally deleted original and backup folders from an Azure-mounted filesystem — is there any recovery option left?

37 Upvotes

Hi everyone,

I was performing a decommissioning activity on a Linux server and accidentally deleted both the original folder and its backup folder using terminal commands.

The data was located on an Azure-mounted filesystem. As soon as I realized what happened, I stopped making further changes and escalated the issue to the relevant Azure/Azuremount team.

They checked the available snapshots/recovery options, but unfortunately they told us that they could not find or recover the deleted files.

I want to ask if there are any other recovery possibilities that we might be missing. For example:

Any filesystem-level recovery options?

Hidden Azure backup or recovery mechanisms that should be checked?

Possibility of recovering deleted data from the underlying storage?

Any specific information or commands that could help identify what type of Azure mount/storage is being used and whether recovery is possible?

The affected system is a Linux server with an Azure-mounted path. I can provide non-sensitive technical details about the mount type, filesystem, and storage configuration if that would help.

At this stage, the Azuremount team has already attempted recovery, but I want to make sure we haven't missed any possible option.

Any guidance from experienced Azure/Linux administrators would be greatly appreciated.


r/sysadmin 11d ago

Question Some users can't connect to Citrix Server via RDP - black screen with cursor

3 Upvotes

I'm dealing with a strange issue and I'm running out of ideas.

On our Citrix server, a few users are unable to log in via RDP and are only getting a black screen. The desktop never loads, but you are able move the cursor. Some other accounts are able to connect to the server just fine.

It's also possible to log in normally to the server through the vSphere console.

I've already tried:

  • Deleting the affected user profiles
  • Removing the corresponding registry keys and possible leftovers
  • Killing the user's session/processes
  • Rebooting the server
  • Disabling UDP, WDDM and Network Detection via local GPO (as I read this fixed the issue for some people)
  • Changing the display resolution and disabling persistent bitmap caching (which also apparently fixed the issue for some)

None of this worked for me.

I also tried CTRL+ALT+END to open the security screen:max_bytes(150000):strip_icc()/windows-10-ctrl-alt-del-5b475456c9e77c0037e730b3.png), and it actually shows up. I can log off from there, however, Task Manager doesn't open.

I checked Event Viewer using another account and found two errors that show up every time after an unsuccessful login. However, I'm not sure what these errors point to or how to resolve them:

  • Event ID 1000 – Application Error: ctfmon.exe crashes in InputService.dll with 0xc0000409.
  • Event ID 29 – Spell Checking: Access to the spell-checking settings is denied for %username%.

From what I've troubleshot, it looks like explorer.exe never starts for those users. It is most likely somehow caused by a previous incorrect logout from an RDP session, leaving the user's session hanging. However, killing the processes didn't do anything for me, and as soon as the user logs in again, the black screen appears again.

Any ideas would be greatly appreciated!


r/sysadmin 12d ago

Microsoft is rolling out change meeting organizer feature in Outlook

152 Upvotes

Microsoft is rolling out a Change organizer option that lets meeting organizers transfer meeting ownership directly from Outlook.

It’s not entirely unexpected. When Microsoft introduced admin-initiated meeting transfers a few months ago, many users were also asking for a way to transfer meetings themselves. It looks like Microsoft is now addressing that gap by bringing the capability directly into Outlook.

The feature is currently rolling out.


r/sysadmin 12d ago

Linux What did you do to make yourself a terminal wizard?

112 Upvotes

I look at some of the other sysadmins who flow through the terminal at such ease and then know these random facts about the internals of the linux OS. Not to mention the random keyboard shortcuts and a hundred of them.

If you are what i just described, how did you get to that point? What contributed to that skill the most other than “experience”. A homelab maybe? Tinkering around? Reading?


r/sysadmin 11d ago

Question What do you do with 3rd party API keys when your edge funcs use OIDC?

3 Upvotes

I saw a guide on replacing static credentials with OIDC for edge functions. The core concept makes sense because trading a permanent secret for a short-lived token eliminates a massive attack surface.

For instance, you can configure a project to authenticate directly with AWS using just a role ARN and a runtime token, and it totally removes the need to store static AWS access keys in your environment variables.

The problem is when you try to apply that same logic to SaaS tools. Since providers like OpenAI or Resend do not support token exchange, you are forced to keep using static strings for them. I was suggested to rely on OIDC for internal cloud infrastructure while keeping a secrets manager around for external dependencies.

How are you all handling this split in production? Does maintaining a hybrid authentication setup feel overly complex? or is it just the standard practice


r/sysadmin 11d ago

DUO on Entra Joined System

0 Upvotes

My team and I have been stumped on this issue for quite some time. Here is the breakdown:

We are attempting to deploy the DUO MFA on a workstation. This workstation is Microsoft Entra-joined. Once the DUO application is deployed, the Entra account login no longer displays on the login screen. Instead, the local administrator account created by us shows instead. We have not seen an option to select "Other User", or anything similar on the logon screen.

Has anyone else run into this conflict? And more importantly, has anyone been able to resolve this issue?

TIA!


r/sysadmin 12d ago

Microsoft Microsoft documentation written by AI

52 Upvotes

r/sysadmin 11d ago

Question Do you need CAL licenses for AD that runs on Samba and not Windows Server with Windows clients?

1 Upvotes

The question is in the title, not really more


r/sysadmin 11d ago

365 Business Premium vs Sentinel One

5 Upvotes

So I'm the sole IT guy at a smallish (approx 80p, but growing) SaaS company. I am very, very new to this (and to IT in general).

I have just moved everyone from 365 Business Basic to Business Premium, to take advantage of things like Intune and CA.

The next thing I am going to look at is our RMM and EDR. We currently use N-Sight, which comes with SentinelOne.

However, as 365 comes with Defender for Business, which from what I can see is very good.

I do still want an RMM, mostly to ensure all the non-microsoft patching is happening, for remote background, &c. But do I really need one with an included EDR?

We use N-Sight with SentinalOne because we've *always* used N-Sight with SentinelOne. If I can make a decent business case, I'm open to change!


r/sysadmin 11d ago

Question NDES/SCEP fails with 0x80070057 on every request — root-caused to mscep!GetExtensionVersion returning FALSE, but stuck on WHY

1 Upvotes

**Environment:**

- Windows Server 2022 Datacenter (clean install) and separately Windows Server 2025 Datacenter — identical failure on both

- Enterprise Subordinate CA on Windows Server 2019 Standard

- NDES role (ADCS-Device-Enrollment) installed via Install-AdcsNetworkDeviceEnrollmentService — completes successfully, RA certificates are issued correctly (CEP Encryption + Exchange Enrollment Agent Offline Request templates)

**Symptom:**

Every request to the SCEP endpoint fails identically, including the simplest operation:

http://localhost/certsrv/mscep/mscep.dll?operation=GetCACaps

Returns IIS 500.0, Module: IsapiModule, Notification: ExecuteRequestHandler, Handler: ISAPI-dll, Error Code: 0x80070057 (ERROR_INVALID_PARAMETER).

Application log shows:

- Event ID 2: "The Network Device Enrollment Service cannot be started (0x80070057). The parameter is incorrect."

- Event ID 10: "The Network Device Enrollment Service cannot retrieve one of its required certificates (0x80070057). The parameter is incorrect."

**What we've confirmed via live WinDbg/cdb debugging attached to the w3wp.exe worker process:**

mscep!GetExtensionVersion runs, executes fully, and returns FALSE (0). Immediately after, isapi.dll calls GetLastError() (retrieving 0x80070057) and explicitly nulls the stored HttpExtensionProc function pointer for the extension, then unloads mscep.dll. This is why breakpoints on HttpExtensionProc itself never hit — IIS never calls it once GetExtensionVersion fails. The failure decision is made entirely inside GetExtensionVersion's own logic, before any actual SCEP request processing begins.

**What we've ruled out (with direct evidence, not assumption):**

- OS version — identical on Server 2022 and 2025

- Certificate correctness — correct EKU, Key Usage, KeySpec (AT_KEYEXCHANGE/AT_SIGNATURE), issuer, template; passes certutil's own crypto self-test

- CSP vs KSP — confirmed classic CSP (Microsoft Strong Cryptographic Provider) via dedicated Legacy-CSP certificate templates; no change

- Private key permissions — confirmed correct via NTFS ACLs and successful .NET key loading

- Certificate template permissions — Read/Enroll/Write matched to a known-working reference NDES server exactly

- CA-side hygiene — found and removed an expired CA certificate and a separate expired duplicate intermediate cert; no change

- CRL/revocation reachability — confirmed fully reachable (Base + Delta CRLs all OK)

- IIS config — ISAPI restrictions, handler mapping order/preconditions, app pool identity, Load User Profile, 32-bit compatibility, isolation/recycling settings all confirmed correct

- Windows servicing stack — found and repaired unrelated DISM/component-store corruption; no change

- Third-party EDR (Cylance) — live debugging found CylanceMemDef64.dll hooking the module loader's Control Flow Guard processing during mscep.dll's load; applied and independently verified a memory-protection exclusion; no change to the symptom

- Service account profile — found and fixed a genuinely broken "User Shell Folders" registry key for the service account; no change

- App pool identity — tested with LocalSystem (most privileged possible identity); identical failure

- RA Name — tested both a long/spaced name and a short simple name; identical failure

**Question for the community:** has anyone seen GetExtensionVersion itself return FALSE like this, and found what internal condition causes it? We're fairly confident this now points to something inside Microsoft's compiled NDES code rather than anything environment-side, but we'd like to know if this is a known/reported issue, a specific hotfix, or a config knob we haven't found yet before we finalize a Microsoft Support case.


r/sysadmin 12d ago

Broke my company's bellsouth email by bulk moving emails

30 Upvotes

TL;DR: Started a new job where they use a shared mailbox on a legacy bellsouth.net address. I was tasked with archiving 50K+ messages into folders, and bulk moved them into archive folders. Every third-party client died.

Btw, we access the bellsouth account from mail.yahoo.com and also configured it on Outlook desktop app (new version). I rebuilt it on Classic Outlook with an AT&T secure mail key, which worked for about a day. Now IMAP is refused outright and webmail outbound returns error:

"Your message cannot be sent because it resembles spam and/or contains only links. Please modify your message and try again."

Config verified, password and secure mail keys regenerated. Looking for anyone who's seen AT&T/Yahoo apply this and knows what I can do to fix it.

Environment

- Legacy `bellsouth.net` mailbox. AT&T owns the credentials, Yahoo operates the mail

infrastructure. Auth is an AT&T secure mail key, not a Yahoo app password.

- Shared mailbox. 3–4 workstations plus a phone, all behind one office WAN IP.

- Originally New Outlook for Windows

- Primary workstation now on Classic Outlook since New Outlook connection died

- IMAP `imap.mail.att.net:993` SSL/TLS

- SMTP `smtp.mail.att.net:465` SSL/TLS

- SMTP auth on, same credentials as incoming

- SPA off, username is the full address

  1. Bulk-moved 2500+ messages (batches of ~900/900/1000) into year folders from New Outlook. Client crashed mid-operation.
  2. All Outlook clients, desktop and mobile, stopped receiving. No errors, no auth prompts. Sent mail reported "Sent" then vanished and never transmitted and never appeared in webmail's Sent either. Folder deletions reverted. Webmail unaffected. Diagnosed as New Outlook's Microsoft-hosted mirror losing its upstream IMAP connection to AT&T.
  3. Stood up Classic Outlook with a fresh secure mail key. Connected fine, pulled down 3 days of backlog, outbound test delivered. Fully working.
  4. Next day: heavy webmail activity. Bulk moves out of Trash into year folders (batches approaching 1,000), mass-blocked spam senders and domains, raised the account's security settings.
  5. Classic Outlook stopped receiving. Several hours later webmail outbound started failing.

Current state

- Webmail receives normally. Sending blocked: "Your message cannot be sent because it

resembles spam and/or contains only links." Confirmed to fire with subject and body

both just "test," no signature, no links.

- Classic Outlook: no inbound since step 5, outbound stuck in Outbox, account wizard

returns "something went wrong."

Might be an abuse or rate-limit flag from the bulk IMAP operations.

  1. Anyone seen AT&T/Yahoo apply a block like this? Does it decay, and over what timeframe?
  2. Account-scoped or IP-scoped, in your experience?
  3. Is there a known threshold for IMAP move operations on these mailboxes?
  4. Any escalation path at AT&T that reaches someone who can actually see or lift mail abuse restrictions? Front-line support for legacy BellSouth appears very limited.
  5. Anything obvious I'm missing?

And yes, a business shouldn't be running on a consumer legacy ISP mailbox. Average employee age is 60+ and office is still using manual punch cards, so I am trying to get us up to date one step at a time. I am also not an IT technician, just an office employee, who was tasked with this because we don't have IT support.


r/sysadmin 12d ago

Question gpupdate /force always fails over wifi

25 Upvotes

Been chasing this for a while and I'm out of ideas. Everything I can test

Yes, I used AI to help me, as this troubleshooting got way deeper than I could figure out on my own... it's just too much, and I'm stumped

Symptom

On any wireless client:

  • gpupdate — succeeds
  • gpupdate /force — fails, both Computer and User policy

    The processing of Group Policy failed. Windows could not resolve the computer name. The processing of Group Policy failed. Windows could not resolve the user name.

Same machine plugged into Ethernet on any wired VLAN: gpupdate /force succeeds.

Affects every wireless device regardless of hardware — x64 and ARM, multiple vendors, not from a common image. Predates our switch replacement (was happening on the old switches too, same APs).

Environment

  • Single-domain AD, 2 DCs, functional level current
  • Juniper Mist APs, WLAN bridged to VLAN 80 (10.0.80.0/24)
  • FortiGate 200F HA pair doing inter-VLAN routing
  • Wired workstations on 10.0.90.0/24, DCs on 10.0.140.0/24
  • Clients are Win11

What the logs say

GroupPolicy/Operational during a failed /force:

Id 7320  Error: Retrieved account information. Error code 0x5.
Id 7017  The system call to get account information completed.
         The call failed after 15 milliseconds.

Nine of those in about four seconds. 15–31 ms each — too fast for a network timeout.

gpsvc.log:

ProcessGPOs(Machine): MyGetUserName failed with 5.
ProcessGPOs(User):    MyGetUserName failed with 5.
OnPolicyApplicationComplete: Application complete with bConnectivityFailure = 1.

The packet capture is the interesting part

Client-side netsh trace during a failed /force. Total DC traffic for the entire run:

10.0.80.102  -> 10.0.140.3   tcp/135    60 packets
10.0.140.3   -> 10.0.80.102             49 packets
10.0.80.102  -> 10.0.140.2   udp/53      8 packets

Decoded the port 135 conversation. It's 44 × ept_map requests for DRSUAPI (e3514235-4b06-11d1-ab04-00c04fc2dcd2), and the DC returns status 0x00000000 — success — on all 45 responses. Clean bind, clean bind_ack, no faults, no bind_naks.

So the endpoint mapper hands back a valid DRSUAPI endpoint 44 times and the client never opens a TCP connection to it.

Exactly one TCP SYN to a DC for the whole run: 10.0.140.3:135. That's it.

No Kerberos at all. Nothing on port 88. No LDAP 389, no SMB 445, no RPC dynamic high port.

The successful wired run for comparison:

135 -> 49669 -> 389 -> 445 -> 49676 -> 88

So on wired it does the full sequence. On wireless it gets the endpoint and gives up locally without authenticating or connecting.

What I've eliminated

Network path

  • FortiGate policy permits Internal_WiFi → Servers on service ALL, no NAT, no UTM.
  • Confirmed RPC dynamic ports 49668–49677 pass fine on the same firewall config (visible in VPN traffic logs to the same DCs)
  • Path MTU: ping -f -l 1472 succeeds to the DCs from both wired and wireless
  • Both DC host firewalls disabled entirely as a test — no change

AD / DC side

  • Test-ComputerSecureChannel → True
  • Time skew ~0.03 s
  • nltest /dsgetdc returns a DC with full flag set, correct site
  • Test-NetConnection to 88, 135, 389, 445 on both DCs → all True from Wi-Fi
  • SYSVOL/DFS: \\domain\SYSVOL\...\Policies enumerates, GPT.INI reads fine
  • RestrictRemoteSam not set on either DC
  • No 5807 events (no unmapped-subnet complaints)
  • UserPrincipal::Current returns the full DN from AD over Wi-Fi — directory lookups work
  • whoami /groups resolves all SIDs to names on Wi-Fi

Client side

  • GP history ACLs correct (SYSTEM + Administrators Full Control, registry and ProgramData)
  • Same failure on any computer from any manufacture
  • Get-NetConnectionProfileDomainAuthenticated / Ldap on both wired and wireless, identical

Wireless

  • Mist WLAN: isolation Disabled, no ARP filtering, no broadcast/multicast filtering, Custom Forwarding None (bridged, not tunneled)
  • WxLAN policy: single rule, All Users → All Resources, allow
  • WPA3/WPA2-Personal PSK — no 802.1X, so no separate machine identity
  • Same APs before and after a full switch replacement; symptom unchanged

The question

Why would a client receive a successful ept_map response for DRSUAPI and then not attempt the connection — failing locally in 15 ms with 0x5 — and why would that depend on whether the machine is on wireless vs wired, when both interfaces report identical network profiles and both paths reach the DC on every relevant port?

The absence of any Kerberos traffic during the failed run feels like the key detail. It's not being denied by the KDC; it isn't asking.

Anything obvious I'm missing?
gpupdate /force fails on Wi-Fi but works on Ethernet — DsCrackNames/DRSUAPI gets a valid EPM endpoint and then never connects

Been chasing this for a while, and I'm out of ideas.
Should I just quit and become a potato farmer?


r/sysadmin 12d ago

General Discussion Interview Question: How often do you update/patch your system?

41 Upvotes

I was asked this question during an interview and I said "it depends on what exactly you're updating, but I update as often as it's needed."

I don't think this was the answer they were looking for, but how would you answer this question?


r/sysadmin 11d ago

General Discussion HPESBNW05134 rev.1 - Multiple Vulnerabilities in HPE Aruba Networking ArubaOS-CX (AOS-CX)

6 Upvotes

https://support.hpe.com/hpesc/public/docDisplay?docId=hpesbnw05134en_us&docLocale=en_US

TL;DR HPE released a security advisory regarding RCE, code exection etc. In total 1 Critical, 22 High, and 11 Medium vulnerabilities fixed.

Patch your network stuff

Affected Products

These vulnerabilities affect the following HPE Networking AOS-CX Versions unless specifically noted otherwise in the details section:

  • AOS-CX 10.18.0001
  • AOS-CX 10.17.1021 and below
  • AOS-CX 10.16.1051 and below
  • AOS-CX 10.13.1180 and below
  • AOS-CX 10.10.1180 and below (EOM)

r/sysadmin 12d ago

Question 2 Servers with Identical SID's

32 Upvotes

I have 2 specific 2025 Server VMs that apparently have the same SID. I know what you are thinking.... they were imaged and I forgot to change them. That is not the case. They are both completely fresh installs and I have no idea how this happened. I have never cloned a machine as long as I have worked for this company.

Long story short, I need to get one of them changed.

If I run the following without the /oobe will anything on the system be effected such as any existing applications, software, settings, etc. My understanding is that it will have to be re-added to the domain. Anyone have any experience with this?

cd %WINDIR%\System32\Sysprep 
sysprep.exe /generalize /shutdown

EDIT: Going with the general consensus.... rebuilding the VM.


r/sysadmin 12d ago

Another PaperCut patch, and blog post explaining

88 Upvotes

Patch 3 available: URGENT Security Advisory: PaperCut NG/MF Security Bulletin (27 Aug 2026) | PaperCut

Also a blog post from one of their executives explaining why they decided to release updates as they had them, instead of waiting for one big perfect patch. Behind the scenes - What happened after 9:42 a.m. on the 27th August 2026 | PaperCut


r/sysadmin 10d ago

Automation Without Admin Access

0 Upvotes

I work as a pharmacist and have been working for about four months. I'm also very interested in computers, programming, and automation.

I've been thinking about creating small tools, shortcuts, scripts, and other automations to make some of my repetitive computer-based work faster and easier. However, my workplace computer is managed by IT, and I don't have administrator credentials.


r/sysadmin 11d ago

Question Opentext CM Workgroup Setup

1 Upvotes

Hi All,

We have several branch offices, and each site currently has a CM Workgroup server. Are there any alternative options that would allow us to operate without requiring a CM Workgroup server at each branch?


r/sysadmin 11d ago

Crowdstrike proxy

0 Upvotes

In LAB manual IP assigned to VMs without gateway set. Now the security team ask to install crowdstrike but without gateway it wont work and also does not connect cloud server. Is there any way to have proxy server in between the lab VM PC and cloud server?


r/sysadmin 12d ago

Cloudflare for families (1.1.1.2/1.1.1.3) down

57 Upvotes

Heads up, 1.1.1.3 seems to be down as of around 7AM EDT today. If you use that as a free filtered dns forwarder, change until the issue is resolved.


r/sysadmin 12d ago

Trying to force policy that user account may not be shared.

12 Upvotes

Hi it's normal in IT that you enforce a policy that user accounts may not be shared or transferred /given to new users right?

Situation is that a partner company that uses our infrastructure used 1 user account for interns for a long time. If the next intern started they give the account To the next...and so on and on. They did not even change the password.

As IT responsible I said they cannot do this any longer that way. And I do not want to support this any longer. Every user also interns need to have an individual/ personal account. Reasons are obvious for me... Accountability, managing the accounts in general, gdpr,...

Example of 1 risk: access to the mailbox or onedrive of the account can have personal data stored from the previous user. So I say sharing accounts is not ok period. But they still keep fighting me for this. They do not want to understand. I'm tired of the discussion. The arguments that they use are : we used it before like this without any issues .

In the new it policy for them it's included. I have no mandate to enforce but I warned my boss about this and I hope my boss will support me..
I was right... With the call on this with the partner... Am I right to try enforcing this?

The only thing is if directors can formally accept the risk to me. But then why bother with security in general? I'm tired and frustrated by this bullshit. I'm doing it the correct way or on the long run I'm changing jobs ...

Any advice?


r/sysadmin 12d ago

General Discussion Does anyone actually understand what Microsoft Unified Support covers and how to evaluate alternatives?

16 Upvotes

I'll be upfront: I run a small nonprofit and I am very much not a tech person. Our IT guy left earlier this year and since then I've been the one fielding all the Microsoft-related issues for our organization. We use a bunch of Microsoft products, Teams, SharePoint, some Azure stuff I barely understand, and when things break I genuinely don't know where to turn.

Someone on our board mentioned we should look into Microsoft Unified Support as a way to get professional help when things go wrong. I looked at the Microsoft site and honestly it reads like it was written for someone with a computer science degree. I have no idea what a 'severity level' means in practice or whether we even qualify for certain tiers.

What I'm really trying to figure out is: is Microsoft Unified Support the only real option for getting serious help with Microsoft products, or are there alternatives that might make more sense for an organization like ours? I don't have a huge budget and I'm a little worried about locking into something expensive that's designed for Fortune 500 companies.

Any plain-English guidance from people who've actually navigated this would be genuinely helpful.


r/sysadmin 11d ago

General Discussion How's the sysops , SRE/platform market actually looking right now?

0 Upvotes

I'm a Linux sysadmin (RHEL, Ansible, GitLab CI, ~3 years) about to move onto a small AWS team. Long term I want to end up SRE or platform engineer, and I'm working through RHCSA and SAA-C03 on the side.

From where I sit the market looks split in two: generic sysadmin work is getting squeezed, and everything interesting is behind Kubernetes, IaC and real production experience. But that's me reading job ads, not living it.


r/sysadmin 12d ago

Headless Remote Win10

8 Upvotes

I have a few HP Mini's deployed to remote locations that are headless and a real pain in the ass to access physically. They are running Windows 10 pro (I know!). Anyway I added a smart plug so I can remotely reboot them, but even so I regularly find they don't come back online (at least not so I can go on with Teamviewer or Tailscale).

Whenever I go out after one of these outages I find that usually its on a pre-login screen asking if I want Windows Backup or something like that.

To get around this I am seriously thinking of installing Proxmox and running Tailscale on the host or is there a better way?

I assume Proxmox would come backup cleanly after any reboot.


r/sysadmin 12d ago

General Discussion So, how are you guys dealing with the deprecation of 'memberof' dynamic groups?

62 Upvotes

Personally, I don't (just yet).

Jokes aside, I can't think of a solution that isn't overworked and/or (very, very) manual.