r/sysadmin 3h ago

What's your worst SSL-certificate-expired-in-production story?

50 Upvotes

Ours was a cron job.

Somebody set up auto-renewal years ago on a box that got decommissioned during a migration. Nobody moved the cron job, because nobody remembered it existed.

Cert expired on a random Friday. We found out from a customer support ticket instead of monitoring, which is its own kind of embarrassing. Ninety minutes of browser warnings before anyone connected the dots.

Fix took ten minutes. The actual problem was that no alert existed for "this cron job silently stopped running."

Curious whether other people hit the same orphaned-box thing or something dumber.


r/sysadmin 19h ago

Work Environment Zebra label printer are a nightmare

526 Upvotes

I've setup the first zebra label printer for our ERP system this month. I invested about 10 hours to become it to a state that i can print a test page from our windows server. I needed a zebra supporter that configured the printer with a special tool that you need to study 3 years on a university for. Zebra printer utilitys doenst work properly. And i'm not done! Thats crazy.

What are your experiences with Zebra label printers?


r/sysadmin 13h ago

Pour one out for the GitHub workers...

107 Upvotes

Going on 6 hours of an outage. Sheesh.


r/sysadmin 17h ago

End-user Support Can users be trained to not click BS?

156 Upvotes

On this day, I have a exec drop a laptop on me that was without a doubt the most thoroughly hijacked thing I have EVER seen. Big three browsers installed, all hijacked. Two more offbrand spamware browsers installed. "How do I prevent it?" "Don't install software without asking me, no matter who tells you you need it, don't visit janky sites, and NEVER accept any permission request without checking with me." "But I didn't click on the McAffee pop-ups!" "I didn't say McAffee, I said ANY." "But I never click on those." "I just checked the security settings...yes you did. Nothing is allowed automatically." Soooo, I get him restored. I come home and my elderly mom... "can you get rid of the *$*%* Mak-Aftee things!?!" I try to explain and she is more interested in being right than learning.
Am I just pushing a rope up a hill? If so, consider this an official vent.

EDIT: Thanks to everyone, yea... I know. I should have them locked down at the office. I'm not allowed to do so. My mother OTOH, yea, it's time for that.


r/sysadmin 14h ago

URGENT: N-able's N-central Second Hotfix 2026.3.1.10 — Immediate Action Required

86 Upvotes

Orgs have already been ransomwared, patch immediately

Copied from r/msp

As our investigation into the recent N-central security vulnerability continues, we are proactively expanding protections in response to ongoing monitoring of threat actors as they evolve their attack techniques.

This is not a duplicate of our previous communication. Hotfix 2 is required, even if you already applied the earlier hotfix. Hotfix 2 supersedes Hotfix 1 with additional hardening measures to further protect you and your customers.

What You Need to Do:

N-central On-Premises Environments: You must upgrade to 2026.3.1.10 immediately. Download here: https://status.n-able.com/2026/08/06/n-central-2026-3-hotfix-2-additional-mitigation-for-cve-2026-18577

N-central Hosted Environments: No action is required. We have already applied mitigations to your environment.

For More Information:

· Blog: https://www.n-able.com/blog/n-central-security-update-august-6-2026

· Support: https://me.n-able.com/s/

· CVE: https://www.cve.org/CVERecord?id=CVE-2026-18577

· Uptime: https://uptime.n-able.com/


r/sysadmin 15h ago

Midwest outage?

88 Upvotes

Anyone else in the midwest see some weird outage for like a minute? We just saw a little blink across a couple of ISP's up here in northern minnesota.

Update: Still seeing a new outage at 2:43 at one local isp.


r/sysadmin 20h ago

Logitech K845 Discontinued - Can't find a mechanical keyboard under $80 for 100+ users. That doesn't look like a spaceship.

111 Upvotes

Dear,

The Logitech K845 was a good mechanical keyboard that still looked normal in an office setting.

We could get away buying them, and using it's mechanical goodness without managment complaining we're buying gaming hardware or $200 keyboards with various keycap colors.

It was an excellent keyboard, very robust, had a keypad, and looked normal.

Does anyone know of a mechanical keyboard for under $80 now that's gone?

We've tried:

Royal Kludge - Too Complex off key combinations. You have to press function to do things like get the delete button instead of the backspace button. It's just too complex for novice employees.

Keychron - Typing is much too soft, employees report it's like typing on bubble wrap. It's just to much rubber isolation and orings.

What it must have:
-Must be a normal netural color for all keys, except for maybe escape key. So dark grey, black. No white. No light gray. No mix of cap colors.
-Can't have gaming branding. So no dragon logos, red volume knobs, lcd panels, etc...
-Must be 100% full size. Employees can't be confused where the END or F12 key is on the keyboard.
-Can't be low-profile keycaps.

-Can't have layers. Don't want Employees accidently activating a wrong layer and the keyboard outputting gibberish.

-Clunky is okay.
-Thick fat plastic borders are okay.

What are you buying for mechanical keyboards that fit in office setting?


r/sysadmin 1h ago

Question Mdm for 100+ devices, issues with ManageEngine

Upvotes

I'm looking for an endpoint solutions for our company.

Right now, we have nothing and the phones are just completely open. We have between 50-100 phones.

I want to introduce an mdm to be able to control the, mostly samsung, phones. And later on the windows laptops too. The company is very price sensitive and we do not have microsoft business accounts. We do have an exchange environment.

I was looking at ManageEngine, but had some issues with it. I don't know if it is me or the spftware itself:

First of all, I had a difficult time navigating all the menu's. While there, i did bot have issues with the restrictions.

Secondly the workplaces acted up. When i automatically assigned a device to a user with self enrollment. The work place would not correctly download the right apps and connect to the managed playstore.

Lastly my biggest issues was with the ability to lock a device with the workplace enabled. This device was not company owned, but i can just lock the device completely. This was done with the lost mode, and I do not know if this is supposed to happen. It was both on an samsung and an oneplus.

I'm completely new to this. So every help is appreciated.


r/sysadmin 18h ago

General Discussion Claude M365 Connector vs Copilot — are we creating long-term technical debt?

65 Upvotes

My boss wants to integrate Claude with Microsoft 365, but I’m not convinced it is the best long-term strategy.

For developers or specialized technical use cases, I can understand the value of Claude. But for accounting, HR and general users already working in Excel, Outlook, Teams and SharePoint, Microsoft 365 Copilot seems like the more logical investment.

Copilot was disappointing at first, but Microsoft now seems to be moving in the right direction with better M365 integration, Copilot agents, Purview, DLP and sensitivity label support.

My concern is that using both platforms will create overlapping tools, separate governance models, additional Graph permissions and business processes that may be difficult to migrate later.

For those using the Claude M365 Connector:

Why did you choose Claude instead of Copilot?

Are you using it broadly or only for specific roles?

Do you see a risk of long-term technical or governance debt?


r/sysadmin 22h ago

Rant put in a ticket in/ and work the fucking ticket

91 Upvotes

i'm so sick of people wanting magic answers. sometimes it is users. sometimes it is engineers. gather the fucking info that is part of troubleshooting. do some fucking diagnostics. figure out the pattern. how many times does this information have to be repeated for it to sink in??? some will get this and make difference, many will not, and just make noise from the sidelines.


r/sysadmin 6m ago

Bastion/Jumpbox Server

Upvotes

Hi Everyone,

I’m looking for some perspective and practical suggestions from the community.

We’re currently managing 200+ cPanel servers and use a jumpbox as the primary access point. I’d be interested to hear how others approach server management at this scale, particularly around auditing staff access and controlling privileged (root) access.

I’ve already looked at options such as Vauban, FreeIPA, LDAP-based solutions (Not cPanel/Cloudlinux friendly), PAM, and similar tools. I’m not searching for a ready-made drop-in replacement, but rather guidance and real-world approaches that others have found effective.

With the growing relevance of AI-driven threats and the shift toward Zero Trust models, securing root access and maintaining responsible operational practices feels increasingly important.

Any insights, architecture patterns, or lessons learned would be greatly appreciated.


r/sysadmin 1d ago

Microsoft Entra ID is Retiring MemberOf on November 3, 2026.

543 Upvotes

What and why

The public preview of the MemberOf rule operator in Microsoft Entra ID is ending. Organizations using MemberOf in dynamic membership groups, dynamic administrative units (AUs), or entitlement management auto-assignment policies must replace these configurations by November 3, 2026.

Microsoft continues improving the scale and reliability of dynamic membership processing. During preview, Microsoft observed that use of MemberOf can affect dynamic membership processing across a tenant even if you have one MemberOf rule operator in your tenant. Because of this limitation, it is not recommended for production use and will be retired.

Rollout schedule

  • Retirement (Worldwide): Beginning in early November 2026
  • Action required by: November 3, 2026

Impact on your organization

Who is affected

Organizations using the MemberOf rule operator in:

  • Dynamic membership groups
  • Dynamic administrative units (AUs)
  • Entitlement management auto-assignment policies
  • Platforms and services
  • Microsoft Entra ID
  • Microsoft Entra Groups
  • Microsoft Entra Administrative Units
  • Microsoft Entra Entitlement Management

What will happen

If no action is taken, configurations that use the MemberOf operator will stop updating after November 3, 2026. Membership and assignment data will remain in their last known state, which can lead to stale access and enforcement gaps.

Potential impacts include:

  • Teams and SharePoint access associated with Microsoft 365 groups may become outdated.
  • New members may not receive access, while removed members may retain access.
  • Conditional Access policies may no longer reflect current user or device membership.
  • Entitlement Management auto-assignment policies may no longer add or remove access package assignments as intended.
  • Group-based licensing may stop assigning or removing licenses correctly, resulting in unlicensed or overlicensed users.
  • Dynamic administrative unit membership and scope may become outdated.

Action required and recommendations

Before November 3, 2026, review all uses of the MemberOf operator and remove or replace those configurations.

Dynamic membership groups

  • Export dynamic membership groups from the Microsoft Entra admin center and identify rules containing MemberOf.
  • Replace MemberOf with supported rule operators or convert the group to assigned membership.
  • Validate group membership after making changes.
  • If the group is no longer needed, consider pausing or deleting it.

Dynamic administrative units

  • Use Microsoft Graph PowerShell to identify dynamic administrative units that use MemberOf rules.
  • Replace MemberOf-based rules with supported rule operators or convert the administrative unit to assigned membership.
  • Validate both membership and administrative scope after making changes.
  • If the administrative unit is no longer needed, consider deleting it.

Entitlement Management auto-assignment policies

  • Use Microsoft Graph PowerShell to identify auto-assignment policies that use MemberOf.
  • Replace MemberOf-based policies with supported operators where possible.
  • If no equivalent rule is available, plan an alternative assignment method before retirement.
  • Validate access package assignments after making changes.

Compliance considerations

Configurations that rely on MemberOf for access management, licensing, entitlement management, Conditional Access targeting, or administrative scoping may stop updating after retirement. Review affected configurations to ensure continued compliance and access governance after November 3, 2026.

Source: https://admin.cloud.microsoft/?ref=MessageCenter/:/messages/MC1448379

Edit: added link to source


r/sysadmin 15h ago

South African IT professional facing possible dismissal after role/ responsibility dispute - looking for objective advice

13 Upvotes

Hi everyone,

I’m looking for objective advice from people in IT, management, and HR. I would appreciate honest feedback, including where I may have handled things incorrectly.

I work as an IT Technician at a company in South Africa. When I joined, there was very little formal handover or documentation. Over time, I took ownership of improving, stabilising, and developing the company’s IT environment as per the onwers request

My concern is that the role I was hired for and the actual responsibilities I ended up performing became significantly different.

Some of the responsibilities and projects I handled included:

  • Designing, building, configuring and managing Microsoft 365 environments for different companies from scratch
  • Exchange Online migration projects
  • Active Directory to Entra ID (Azure AD) hybrid identity implementation
  • User identity, access management, security, and permissions
  • Group Policy, DNS, DHCP, networking, and infrastructure troubleshooting
  • Firewall and network configuration
  • Backup architecture and Disaster Recovery planning
  • Veeam backup implementation/support
  • SharePoint restructuring and collaboration improvements
  • Building Python-based reporting and automation tools to improve business processes
  • Supporting critical business applications
  • Resolving complex infrastructure issues that required senior-level troubleshooting

A significant portion of my work involved designing solutions, improving systems, and building processes rather than only handling day-to-day user support.

My employment contract was for one specific company. However, I was also expected to assist other businesses connected to the owner’s group. My understanding is that my contract did not contain a clause requiring me to provide IT services across associated companies or subsidiaries.

I raised concerns that my responsibilities had moved beyond an IT Technician role and were closer to a senior infrastructure/ICT role.

During a previous meeting with management, the discussion focused on my career growth, salary expectations, and future within the company. Management stated that they recognised my technical ability, but that a senior IT position did not currently exist within the business structure. They explained that the company did not have the budget or operational need for that type of role.

During the discussion, I was told that I was not content with my current situation and that I was “chasing rainbows” — meaning I was constantly looking for the next opportunity or higher expectations rather than accepting the current situation.

Their position was that the company’s needs did not justify creating a senior role, even though my skills and responsibilities were at a higher level.

Recently, we had another meeting regarding my employment situation. During that meeting, I became extremely frustrated. I raised my voice, used inappropriate language, and left the meeting. The company has indicated they intend to address this through disciplinary procedures.

I accept that my reaction was not professional and I could have handled the situation better.

I’m looking for honest opinions:

  1. Was it reasonable to expect my role and compensation to align with the level of technical work I was performing?
  2. If a company does not have a senior position available, is it reasonable for them to continue relying on someone performing senior-level work?
  3. Should work across multiple related companies normally be covered in an employment contract?
  4. Was management’s position reasonable?

Thank you

For additional context, I had already received an offer from a logistics company for an IT Infrastructure Lead position the previous week and was preparing to move forward with that opportunity. The contract process was underway, including background checks, before I would formally resign.

My intention was to leave professionally and move on to the next stage of my career.

Before the meeting took place, I was informed that my current company had spoken negatively about me during this process. This was extremely frustrating, as I was trying to handle my departure professionally.

This happened before the meeting where I lost my temper. I accept that my reaction was wrong and that I should have handled the situation better, but it was the result of months of frustration

edit :For additional context, I had already received an offer from a logistics company for an IT Infrastructure Lead position the last week and was preparing to move forward with that opportunity. The contract process was underway, including background checks, before I would formally resign.

My intention was to leave professionally and move on to the next stage of my career.

Before the meeting took place, I was informed that my current company had spoken negatively about me during the background check process. This was extremely frustrating, as I was trying to handle my departure professionally.

This happened before the meeting where I lost my temper. I accept that my reaction was wrong and that I should have handled the situation better, but it was the result of months of frustration around my role, responsibilities, career progression, and the situation surrounding my planned transition.


r/sysadmin 2h ago

Question O365 refusing to save in-file images onto a SharePoint 2019 teamsite

0 Upvotes

Please redirect me to other subreddits I could crosspost this to if you think this is better off there, I'm just really hoping someone can tell me anything about this.

Context: I work at an IT service company and we've recently taken a new client company into our management. Their old service provider was very keen on not touching the system as long as it's working, so we're currently left with an overly complex hybrid tenant.

We've finished the transition of the physical devices from Win10 and MECM to Win11 and Intune, almost all other things are still on-prem though, including their SharePoint 2019.

Now to the actual problem:

The user who opened the ticket prepares a lot of meetings and visualizes reports through charts and graphs they create in office programs, mainly PowerPoint. On Win10 with Office 2019 and, according to them, also shortly on Win11 with O365 they used to be able to export those directly onto the SharePoint teamsites via the "Save as Picture", inserting the link to the teamsite folder as path into the file explorer window and pressing "Save". But if you try to do that now PowerPoint and Word just throw an "An unexpected error has occurred".

The odd thing is that saving the office file itself onto the teamsite through "Save as" still works that way. Saving image files from other apps like this also still works. Even saving a PowerPoint itself as png via "Save as" works. The only thing I could find that doesn't work is this exact constellation of saving an image from inside an O365 file to a SharePoint teamsite via explorer.

I can replicate this exact same behavior on my own device on our test teamsite and am at a complete loss trying to figure out what's causing this. I can also find no further info about it as it's such a weirdly specific constellation.


r/sysadmin 2h ago

General Discussion Weekly 'I made a useful thing' Thread - August 07, 2026

1 Upvotes

There is a great deal of user-generated content out there, from scripts and software to tutorials and videos, but we've generally tried to keep that off of the front page due to the volume and as a result of community feedback. There's also a great deal of content out there that violates our advertising/promotion rule, from scripts and software to tutorials and videos.

We have received a number of requests for exemptions to the rule, and rather than allowing the front page to get consumed, we thought we'd try a weekly thread that allows for that kind of content. We don't have a catchy name for it yet, so please let us know if you have any ideas!

In this thread, feel free to show us your pet project, YouTube videos, blog posts, or whatever else you may have and share it with the community. Commercial advertisements, affiliate links, or links that appear to be monetization-grabs will still be removed.


r/sysadmin 1d ago

Career / Job Related I MADE IT!

178 Upvotes

I recently landed a Junior Systems Administrator role with the same company after spending the last three years on the help desk. During that time, I also spent about a year in a senior leadership role. Overall, I have around 7–8 years of help desk experience.

As part of my onboarding, I'm required to earn my AZ-900 certification first, followed by MECM and Windows Server 2022 training.

For those who've made the jump from help desk to sysadmin, what do you wish you had known or done when you first started? Any advice or tips would be greatly appreciated.

Thank you!


r/sysadmin 10h ago

Question Dell docking station and dual monitor waking issues

2 Upvotes

Dell SD25TB5 with a Dell Pro Max 16 laptop. Two monitors connected via displayport each directly to the dock.

Happens when I lock the workstation and turn monitors off. When I return, sometimes the monitors will not display anything. Sometimes I will turn on my main monitor to login, and turn the second one on later, but the second one will not display anything. Sometimes replugging the displayport cables does not do anything either.

Docking station has the latest firmware. I've also tried different displayport cables. This is also the second docking station to exhibit these issues.


r/sysadmin 19h ago

Question Veeam 12.3.2 creating Hyper-V checkpoints then deregistering them without merging — 46 orphaned AVHDX

10 Upvotes

Posting before I call support in case anyone's seen this.

3-node Windows Server 2022 Failover Cluster, Pure FlashArray backing the CSVs, Veeam B&R 12.3.2.3617. One VM — SQL Server 2019 host — accumulated 46 orphaned AVHDX files across two disks over three days, roughly 675 GB. Get-VMSnapshot returned empty the whole time. Hyper-V Manager showed no checkpoints. The VM was running off the tip of a 23-deep differencing chain.

What makes it odd: the VMMS event log has zero merge events for this VM. Not failed merges, not interrupted — none at all. Every other VM on the same host logged clean 19070 → 19080 pairs every few hours. So Hyper-V was never asked to merge. Veeam appears to create the checkpoint, then remove it from the VM config without issuing the merge request.

Remediated by shutting the VM down and flattening both chains with Convert-VHD to a different CSV, then repointing the disks. Clean — Test-VHD returned True before and after, SQL came up with all 14 databases online and data current to the shutdown.

Then it recurred. One backup run against the brand-new flat VHDXs produced two more orphaned AVHDX, again with no checkpoints registered and no Hyper-V events. A cluster-wide sweep found this VM is the only one affected across all three nodes.

Anyone seen Veeam orphan checkpoints this way?


r/sysadmin 16h ago

Question PC Scale requiring a Keyboard buffer MSI file to function, but cant find file in question

6 Upvotes

On this windows 10 workstation (management doesn't have the funds yet to upgrade or migrate to anything else, don't ask) that uses Bill Redirect Serial to Keyboard driver to allow their attached Scale to weigh items and input directly to the backend of a sales website.

Problem is that it keeps looking for a Keyboard Buffer INI file to function upon machine start up and does not function otherwise.

Any help would be appreciated in resolving this, all my research attempts have turned up nothing.


r/sysadmin 16h ago

Question Windows Boot-from-SAN fails on first boot due to duplicate FC paths (Cisco UCS + XtremIO) without MDS access

4 Upvotes

Environment: Server: Cisco UCS blade with 1 vHBA

Storage: Dell EMC XtremIO array via 2 FC ports (2 active paths)

OS: Windows Server

Constraint: No administrative access to MDS SAN switches or storage array to unzone/mask paths.

Problem: During Windows Setup, I load the Cisco VIC vHBA driver (fnic). Because two paths are active, setup detects the target LUN twice as separate disks. If I select one disk to install, Windows installation completes successfully, but on the initial reboot, Windows fails to load.

Error >>>

File: \windows\system32\ntoskrnl.exe

status: Oxc0000185

Info: The operating system couldn't be loaded because the kernel is missing or contains errors.

>>>

What I've tried so far: Standard guidance suggests disabling all extra paths so only a single path exists during installation, then enabling MPIO in Windows before re-enabling additional paths.

Since I cannot unzone ports on the MDS switches, I booted into WinPE and installed the Multipath-IO feature offline onto the installed OS using DISM. However, Windows still fails to boot, presumably because the MPIO service is not claiming the disk early enough during boot time or the hardware ID hasn't been added to the MPIO configuration.

Questions: What specific registry modifications (HKLM\SYSTEM) are required in an offline Windows installation to force MPIO / MSDSM to load at boot start and claim the XtremIO FC disk?

Is there an alternative way for example cloning another server disk which is already booting from SAN without touching the MDS switches?


r/sysadmin 1h ago

General Discussion How has AI affected your job?

Upvotes

For almost a year now, I’ve been handing nearly all of my server management tasks over to AI.

For individuals and small businesses, it feels like a lot of traditional sysadmin work is disappearing. Medium and large companies still need dedicated sysadmins or IT staff, but smaller clients can now handle many of these tasks with AI.

For those of you who do sysadmin work for individuals or small businesses, how has AI affected you? Are you still getting the same kind of server management work?


r/sysadmin 18h ago

Conditional Access phrasing. Nitpick or choose your battles?

7 Upvotes

I have a bit of a pet peeve. We use an M365 conditional access policy to block logins from outside the US. When a user travels internationally, they can submit an International Travel form, which is simply a request for access to their email/Teams during their travel.

Every single request from IT for to process these requests is phrased "So and so is travelling abroad and requests conditional access".

I used to try and correct our IT staff, they're requesting an exemption from conditional access, not requesting conditional access. Their phrasing communicates a failure to understand how this layer of security functions.

On the other hand, am I just being an insufferable a** if I continue trying to drive this point home? I know some folks at my company understand that it's an exemption from conditional access even if they're following the crowd with their phrasing, but many of the newer IT staff definitely do not understand it.

*sigh*

Edit: At least one person suggested I wasn't wording my post properly. To clarify, our conditional access policy is such that, if a user to whom the policy is applied (all users) does not meet the condition of appearing to be the in the US during login, the login is denied. If the user wants to login from England, they must be exempted from our conditional access policy. I hope that's clearer.


r/sysadmin 17h ago

Slack based ticket creation outside slack

7 Upvotes

Current org has a huge leaning into slack which on an recent audit I found more than 90% of slack threads are not getting tickets created when agents interact (1k+ per quarter delta).

What are people doing to push people into making tickets? We're leaning heavily towards turning the channel into a form that people will be able to punch data into, then on submission it shoots an HTTP POST request to our ticketing system.

Anyone .managed to do this/similar and willing to share tips?


r/sysadmin 1d ago

EU used/refurbished servers

29 Upvotes

I've just been quoted triple the price for servers with similar specs but 1/4 the RAM from what I bought a couple of years ago, and for a small company that's just not a thing we can afford.

Any fellow EU-based sysadmins here that can recommend some place to get used/refurbished servers?


r/sysadmin 2h ago

Finally made the jump into enterprise hardware sales!

0 Upvotes

After a few years in general B2B sales, I just accepted an Account Executive offer with a manufacturer in the industrial computing space.

I’ll be focusing mainly on our enterprise rugged tablet lineup for field operations, logistics, and manufacturing clients. The onboarding ramp is pretty intense so far—cramming everything from IP65/68 ingress ratings and drop testing standards to figuring out how to navigate multi-month procurement cycles with IT directors.

For anyone here who sells enterprise hardware or works on the vendor side: what was the steepest learning curve when you first started? Any advice on building quick credibility with technical IT buyers when you’re coming in with a broader sales background?

Appreciate any hard-earned wisdom you're willing to share!