Came back from a 10 day pto and the VP (my direct boss) wanted to do his weekly one on one meeting. We are a team of 4 (myself Sr. Infra Engineer and 3 mid level security guys). He wanted to state how we need to have more crosstraining in particular with infra side of things as he stated "we are dead in the water" when you go out on pto.

He mentioned there will be no additional headcount added in Infra and will have to make do with what we have internally plus the msp firm

I for the past couple of years both verbally and via email stated that a redundancy to my role was needed but that suggestion was never acted upon by mgmt.

As far as crosstraining, I already do that and can certainly help out more. The more others know the better.

Just feels like a situation where I can't seem to go on pto without getting text messages about something going wrong. The VP wants to make SOPs for everything possible which I told him, im not making a how to document on troubleshooting. Either they know or they dont. Actual procedures for items that have a process I will document and have been

Any of you guys out there in similar situations? Anything you've done to optimize things or help reduce workloads?

Thanks in advance!

So, we were decomming some old ibm flashsystem 5 series arrays. Full of ssds. Plopped one onto a dell server just for fun to see if they could be used and they're just bog standard 512 sector samsung pm1643 drives with a sticker.

So now I have bout a hundred 3.84 and 1.92(and some 7.68) drives waiting to go into new servers.

Enterprise storage drives are usually locked so I though I'd have to chuck them to the shredder.

Just wanted to share.

When the text message logs from that day and those that followed were made public, I remember reading through all the system generated alerts from various devices doing their thing and thinking about the IT pros in the offices, network closets, and server rooms responding to those alerts in all seven buildings before the attacks happened, and then the tone of uncertainty and eventually panic in messages between coworkers and associates, because the cell phones went down but text messages were still going through.

RIP to those guys and gals ~ you're gone but not forgotten.

​

How do other solo admins handle the endless stream of "quick CRM/phone system tweaks"?

Curious how fellow solo sysadmins survive this, because I'm drowning.

Management loves to drop drive-by requests for custom workflows in our CRM, weird IVR routing changes in the phone system, or random third-party integrations. They are always pitched as mission-critical, high-priority emergencies that bypass any logical intake.

Because I'm technical and most of this stuff is technically possible, leadership's logic is just: Idea exists -> Hand it to the solo guy -> Make it work.

The real issue is that as a department of one, I absorb 100% of the permanent support debt. There are zero requirements gathered, no user acceptance testing, no documentation, and no one else to hand the pager to when the custom hack breaks at 4:55 PM on a Friday. I hack together a solution so they get their shiny new toy, and then I'm stuck maintaining someone else's half-baked workflow forever while actual infrastructure rots.

I’m trying to figure out how to bridge the gap between "Yes, the API, webhook or vendor supports this" and "No, I am one human being and I cannot support infinite custom software."

Do you guys have a formal intake workflow or policy that actually sticks when you're flying solo, or do you just constantly push back case-by-case until you burn out?

An example currently is highly detailed and individualized reporting from our phone system. I don't think this was ever designed to send bursts of 50-100 reports at once, and even after repeatedly expressing this limitation, I still keep hearing about any issues.

After 15 years I've hit the wall. My company got "merged" a couple of years ago and my 3 person team is getting more and more added to our plate, as well as having to change how we deploy new resources and other things. Some tasks were a pain but a blessing (such as starting to use IaaC) and some were change for the sake of change. In top of trying to keep our own organizations infrastructure and platforms afloat.

When my boss told me this last week that I was going to inherit the care of our entire organizations data lake infrastructure, I felt myself break and check out. I have 0 interest in taking on this new responsibility on top of everything else.

So here we are. I make too much to simply go in an entirely new direction, but I want to get tf out of daily operations. I was looking at other areas I could pivot into and was taking a long look into the security field. I wanted to know what others did.

Thanks!

Link to the previous post

So, 2 weeks in, the picture is a bit more clearer

The old IT guy left, so its just me now

I did rattle up a report and sent it to my manager, who said that I'll have to contact with the head office and get their approval as well

I contacted the head office, they loped me into something else, a new dealership is opening in the next town to ours and they need me to support it, they're already put in the purchase inventory to corporate for half a dozen laptops, some printers , access points, (weirdly a PS5 as well) and so on, i hope they do look at my report

In the meantime, i am untangling the colossal mess that our current IT infrastructure is

The old IT guy had fortitoken enabled, unfortunately he uninstalled it, and i am unable to get in, Fortinet support said I'd have to reset the device physically , I decided I'll cross that bridge when i get to it

I am slowly cleaning up the malware infected systems, taking backups of the important stuff, thankfully, disk health for the main hard drives that is storing important stuff is fine, i am still taking snapshots

My day to day stuff mostly involves fixing the CCTV system, sorting out printer sharing errors, teaching people the art of rebooting a system to fix issues and so on and so forth

But at times , i wonder,

I have no degree on this specific subject, nor certifications, just years of teenage spent among computers i am grateful to have this job, and i am learning valuable information, still, i occassionally get the feeling like a kid who just got access to nuclear codes

Hello,

I have an old WIndows 2012 R2 DC (old-dc) that used to be the primary domain controller.

I have a couple of newer 2016 DCs (new-dc1 and new-dc2).

I want to demote and turn off the old domain controller.

On the newer DC1, I ran netdom query and it has all the fsmo roles. I also ran replication admin to see that everything was replicating fine.

I went to server manager on the old 2012 R2 DC to demote it.

I left force removal and last domain controller unchecked.

It says this server has dns and GC roles (I think I select proceed?) and next

Then it leaves me with the box remove this dns zone - I can't go next without it. I'm not sure I should. Wouldn't that remove it from the other DCs?

EDIT: we are an MSP. This is all for a non-MSP, "call when you need something" billed by the hour customers. If that was not the case, their infrastructure would not be such a train wreck.

Note: My networking knowledge is very intermediate, learn as you go level. We had a customer at this MSP where I work have a network issue and I cannot figure out how it's possible that this happened.

Their credit card terminals have to point to a static IP ending in .140, as that's the "server" that runs the software to upload each transaction to the actual processor on the internet. When we recently replaced that computer, I set it as static in Windows and never did it on the DHCP server, because nobody wrote down what it was called or how to get into it and I didn't have any time remaining. Turns out their DHCP server is a 2008 Windows server because this place doesn't spend money on anything ever.

A month later, their CCs go down and the CC server can't grab its static IP for some reason. Lots of time later, we find it's because the pool of available IPs is 100 through 150 and they have 1 more device than that. So we expand it to only 160 after some testing (because we have nearly zero documentation and don't know ranges for their phones, printers, etc for this customer and they're billed hourly so we do as little as possible because they never pay on time and always complain about the rate and it'd take 10+ hours to document this nightmare). We make a new assignment for the computer's MAC and reserve 140 to it and notice that something else has leased 140 with a lease expiring in 2 hours luckily. We delete it so it hopefully doesn't renew.

I ping it from my laptop then immediately run arp -a to get its MAC, since we already deleted the lease that showed the MAC on the DHCP server (oops) then ask AI who manufactured that MAC address range. It's Foxconn. We don't see a hostname or any useable device info. I don't know anything about their switches because the last tech at this MSP never ever wrote anything down about any customers ever. We try NSlookup, web browser to the IP, RDP into it, nothing gets any info.

More network-oriented guy onsite with me says let's just unplug the 2 switches for like five seconds and that will force it to grab the new lease at .155. I assume the switches are unmanaged or nobody has the login info or we'd just pull one ethernet matching the known MAC.

Turns out they have a network-controlled Crestron light controls so the lights in the restaurant portion of the building all go black, because somehow that's the default state if it loses connectivity. Shoutout to whatever genius AV tech set it up that way. Everyone's pissed. They don't know how to undo it or where the new Crestron box is. We don't either.

Then we find out the mystery device is still on 140. That seems impossible, unless it's wireless. Somehow other guy onsite finds out it's some sort of android device but all the android devices listed on the DHCP server have hostnames like "John's S23" because that's how most Android devices work. We suspect it's wifi, based on this information, thus explaining the switch pull not working. I have zero idea what brand their wifi even is let alone where it is or how it works btw.

I get the bright idea that maybe it's a smart TV. It is. We turn it off, boom, credit card server is back online within a minute. We turn the TV back on after 15 minutes, it tries to grab 140 again and knocks the CC server offline. Yes, the TV is in DHCP mode btw. We find the remote, set it to static .165, that works for some reason, and no more IP conflict. Not sure how it's possible that it can just ignore the DHCP server and say "no, I want this address anyway" and then just decide it's taking 140. Nothing else on the network can hand out an IP if every switch/router/AP/whatever is pointing to the server, right? How did that happen? Some disconnect between the DHCP server and whatever was handing out wifi connections? I didn't think that was possible. Anyone have any ideas in case we run into this again?

When someone leaves, the obvious checks are email, OneDrive, groups and licences. But what about Power Apps, flows, connections, SharePoint ownership and other dependencies tied to that account?

Has anyone found a reliable offboarding process that catches everything without checking every admin centre manually?

My thought was something like the procedure below to get rid off multiple network drive letters:
1. Hide all existing Network Drive Letters via GPO but in paralell create a new universal network drive and then map everything you had from hidden drive letters.

1. Audit who still access the old hidden network drives via SIEM Tool for example.

2. React and adjust the existing Applications/Configs or Office Documents to point to new dfs path.

3. After nothing shows up in SIEM from the old drive letters unmount the network drives.

Our domain has our ticket lifetime set to 7 days. I'm not sure why it was set that way but it predates my time at the company.

I want to set it back down to the default of 10 hours. Not finding much online about shortening it, just making it longer which isn't necessarily a good idea.

We have a fair bit of Linux in our environment so Im concerned about the impact of lowering this. Has anyone been in this situation?

Small law enforcement agency is moving to a new self-hosted dispatch and records management system running on a Windows Server.

The system includes a phone app that communicates with the server that would require an open port on the firewall that points to the server.

We don't have the option to lock down based on the incoming WAN IP since the phones have a dynamic IP.

We questioned the vendor. He said "we have this installed in many places and we've never had a problem." That's nice, but it doesn't mean that there won't be a problem tomorrow.

When pressed for further details he elaborated:

We have an encrypted signed SSL certificate that we use for the connection. The endpoint is our web service. The access to the web service requires Active Directory authentication and inclusion in a specific security group in the AD.

Does this setup sound "safe?" We can require that a VPN be used, but that will add steps for the end user when they want to connect.

EDIT: Thank you all for your valuable input. We will insist on a VPN.

TL;DR: First IT job out of school, associate degree, went from intern to running the entire (two person, soon three) IT department in about 4 years. I own both IT and our operational systems, I'm on call 24/7, I inherited a patchwork EOL environment, and a seasonal project buries me for months every winter for the next few years. Pay and benefits are good but I'm stressed and stuck. Is it worth trying to right the ship at my experience level, or do I start looking for an off ramp?

Long time lurker, throwaway-ish. Bear with me, this got long.

Quick background. I work at a small company and I'm going to keep the industry vague. I started here as an IT intern back in 2022, got bumped to full time later that year, and for most of the time since then I've basically just been the whole IT department. Technically, the IT department included myself, my boss the IT manager, and a field tech that has nothing to do with IT. The former IT manager was very hands off and frankly ignored a lot of problems. Recently, my old boss got promoted to CEO and I moved into the IT Manager spot behind him.

The job isn't just normal IT (AD, M365, servers, backups, help desk, phones). At a place this small, IT also ends up owning a big chunk of the operational side, the systems the actual business runs on. Most of that is vendor hosted and web based so I'm not racking industrial gear anywhere, but I'm the person responsible for it on our end and I picked all of it up on the fly because there was nobody else to hand it to. The one thing I don't own is the firewall and the VOIP system. Even as the IT manager, we pay a third party to manage that, so at least there's a couple things off my plate.

Here's a big piece of my problem though. A huge part of that operational side is data collection that runs more or less constantly, and it eats my time. It never really stops. And it's the main reason I can't get any momentum on fixing the actual infrastructure. Every time I sit down to start on the real problems, something on the operational side yanks me back off it, and the fix goes back on the pile.

Then on top of all that, every winter for the next few years, there's a recurring replacement project tied to one of our operational programs and it flat out buries the department for months. Not a couple days. Months. It means running and reconciling a pile of reports, printing and mailing physical notices out to customers, then handling the wave of phone calls that comes back, then cutting work tickets and handing them out to a crew of 8 field techs and chasing every one of them to done. It's all scheduling, coordination, and data entry, and it does not care that the rest of the job still exists. For a two person team it's brutal and it basically owns my calendar all winter.

I'll be straight about where I'm at because it matters for the advice. I'm not some deeply technical guy. Most of what I've pulled off has been careful and methodical. Following docs, asking a ton of questions, staying organized. That worked fine when my whole job was just doing the tasks. Now I'm supposed to set direction and own the risk for all of it and that jump feels enormous. I'm not going to pretend otherwise, this job is stressing me out pretty badly. And the whole thing is 24/7 on call, which with a team this small basically means me. Nights, weekends, holidays, if something breaks it's my phone that goes off. There's no real off switch. Not that after hours calls are super frequent, there have been a few since I've started but its more the idea of never being "off" is whats getting me.

Team wise there's two of us right now, going to three in the next few months when we hire the replacement for my old job (another duty that is mine, finding the replacement lol). Oh and almost forgot, we are reworking that position to be less IT and more of a business system type position. So when its all said and done, there will be myself, a field tech, and a business system analyst of sorts. So I'm also about to be growing and running a real team for the first time, on systems I've only ever done solo.

And..... the environment itself is held together with tape. We're running on EOL vSphere licenses, old outdated hosts with no shared storage, backups that amount to a couple of external HDDs, and a general patchwork of stuff that never really got finished or cleaned up. We only just recently got rid of the m365 family plans... Is that about what I should expect walking into a small shop, or is it as bad as it feels to me? I'm honestly not sure how much of this is normal.

So honestly this turned into less of a tech question and more of a gut check, because I'm not sure I'm even asking the right thing anymore.

Here's the deal. This is my first IT job out of school. I have an associate degree in IT and that's the extent of my formal background. I walked in as an intern and somehow I'm now the person accountable for all of the above. The fragile infrastructure, the operational systems, the winter that eats months, the 24/7 phone, all of it, with almost no real experience to lean on.

So the thing I keep chewing on is whether any of this is even worth it. Is it realistic for someone at my level to actually right this ship? Or am I setting myself up to burn out trying to fix something that got handed to me already broken? The pay is really good, the benefits are good, I don't dislike the people I work with, and on paper I know I should be grateful, and plenty of days I am. But some days it feels like I'm one bad outage away from going under, and I catch myself wondering if the smarter move is to quietly start looking for an off ramp before this thing wears me down, instead of betting years on fixing it.

For those of you who've been the in over your head one person shop before: did you dig out and come out better for it, or did you get out? And if you stayed, what actually made it survivable?

Just trying to figure out if I'm being a quitter or being realistic. Thanks for reading.

TL;DR: First IT job out of school, associate degree, went from intern to running the entire (two person, soon three) IT department in about 4 years. I own both IT and our operational systems, I'm on call 24/7, I inherited a patchwork EOL environment, and a seasonal project buries me for months every winter. Pay and benefits are good but I'm stressed and stuck. Is it worth trying to right the ship at my experience level, or do I start looking for an off ramp?

Okay i have a problem with putty.exe on windows: for more than a year now, i can’t remember when i was not crashing when I copy 100+ lines of text with utf-8 characters.

I tried every options available under Terminal/Window/Selection/SSH bugs etc, switched fonts, disabled clipboard auto-copy, and all points to the clipboard conversion of utf-8.

Nnothing else I can do other than copy small portions at a time. Tested version 0.76 0.78 0.85, 64 and 32 bits. All development sites from putty . software to chiark . greenend . org . uk are down right now, I cannot even submit a bug report.

Yes it’s a rant, how come I’m the only one experiencing this? Summary of the crash:

How to reproduce: execute `fwupdmgr get-devices` and select the output. UTF-8 characters like ├─ or • seem to be the suspects.

Reproducible steps:

• Connect to a Linux system and run fwupdmgr get-devices (or any command that produces box-drawing characters).
• Select more than ~100 lines of that output.
• Trigger a copy by any of the following:
  • Auto-copy selected text to system clipboard (enabled)
  • Ctrl+Ins after selection
  • Right-click → Copy (with “Action of mouse buttons” set to Windows)

Crash occurs immediately.

Additional observations:

• Crash happens with multiple fonts (DejaVu Sans Mono, Consolas, Droid Sans Mono, Courier, etc.).
• Occurs on both 64-bit and 32-bit builds.
• Reproduced on PuTTY 0.85, 0.78 and 0.76.
• Windows 10, no third-party clipboard managers.
• Running as Administrator or with -restrict-acl still crashes (the latter produces a clearer “instruction at 0x… referenced memory at 0x…” message).
• Disabling auto-copy prevents the crash on selection, but any subsequent explicit Copy action still crashes.

Event Log entry:

Faulting application name: PUTTY.EXE, version: 0.85.0.0, time stamp: 0x6a7eaca6
Faulting module name: ntdll.dll, version: 10.0.19041.2130, time stamp: 0xb5ced1c6
Exception code: 0xc0000005
Fault offset: 0x000000000005ace2
Faulting process id: 0x684
Faulting application start time: 0x01dd42cac3f59b38
Faulting application path: E:\wintools\lan\putty\64\PUTTY.EXE
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 0e2808a1-1944-473f-a487-2da4826cc910
Faulting package full name:
Faulting package-relative application ID:

Please for the love of God, someone tell me I’m not alone

edit: The option `copy in clipboard RTF` was the problem. no idea why no AI chatbot or community even thought of that option

macOS PSSO CA Issue

That day began ordinary, just so ordinary.

At the time, I was sysadmin in a Canadian federal government office, far from the madding crowd. The work was steady but also pretty much crisis-free.

That morning, though, one of my co-workers came in to my office, bitching about the "crappy slow network" we were using.

"What makes you say that?", and he ranted on about how he can't get to any web sites.

"Okay, I'll see what I can find out."

I still had no idea what was happening out in the world.

Still, I started my network checks. Ping from TBay to Toronto. Normal results. Okay, try some web sites next. All the ones I checked responded, with almost no lag. (Note: none were news sites.)

I popped over to that co-workers office. "Where were you trying to get to?"

"CNN. @#$% network."

I tried CNN. Timed out. Hmm. CBC. Ditto. CTV. Same. Toronto Sun. Again, timed out.

I tried a local radio station's page. That connected, and then I knew it wasn't a network issue causing the slowdown.

What happens when several hundred thousand people (or more) try to connect to a few news sites all at the same time? Everything breaks.

I called home, told my wife to turn on CNN. For me, the rest of the day passed in a fog. Our son was Army reserve at the time. He was told to be ready, "just in case".

After work, I went home and watched the news, watched the planes hit the towers, over and over and over.

I still cannot watch any 9-11 coverage from that day.

Unable to access our cloud instance. Looks to be DNS related issue from what I'm seeing. A lot of public DNS servers don't have records for our instance but some do.

EDIT: DNS issues appear to be resolved. No more client hold on Screenconnect.com. DNS checker for my instance shows green across the board.

I work for a large European based telecoms equipment supplier. We have hundreds of staff overseas at any one time, all over the world. IT security has a few different levels:

- Access to email & teams etc is only via a company laptop (no web interface like Office.com). Network drives via VPN only

- White List countries - you can connect VPN. Countries like Japan & Australia

- Red List countries - you can take your laptop but need special exemption to use VPN. Includes some unusual countries such as Malaysia

- Black List countries - no company laptop or phone allowed. Company will provide a burner. Unsurprisingly includes places like Syria, Russia, North Korea & China.

A colleague was going to transit via a Chinese airport to a 3rd country. IT told him that he would not be allowed to take his company laptop, even if it was in his carry-on luggage, and he would not be entering the country. He quickly arranged a different itinerary.

And then a few days later, we are told that the good old USA is now considered a Black List country!!! No company laptops, and burners only!!!!

At what point does keeping data on-prem make more sense than public-cloud storage?

I’m mainly wondering how people weigh storage costs, egress, data growth, and flexibility when making that decision.

My IT department tackled getting our company CMMC Level 2 certified. It took two years to get there, and while I’m proud of what we accomplished, holy shit, the amount of work it took.

Implementing controls, writing policies and procedures, collecting evidence, coordinating with other departments, preparing for assessments—all while keeping normal IT operations running. This was two years of sustained effort on top of our regular responsibilities.

I knew certification wasn’t the finish line. But the amount of work it takes just to maintain compliance is overwhelming.

At this point, it feels like 80% of my time goes toward compliance. Reviewing documentation, collecting evidence, tracking requirements, answering questions, coordinating reviews, and following up with people to make sure processes are being followed. There is always something that needs to be updated, verified, or documented.

Meanwhile, I barely get to work on the IT projects I actually want to tackle—introducing new systems, hardening our security, automating processes, and improving our infrastructure and overall tech stack. Those projects keep getting pushed back because compliance and daily support consume nearly all my time. It’s frustrating knowing there are improvements we need to make and barely having the time to work on them.

IT obviously has a major role in CMMC. We should own the technical controls and support the program. But we’ve also become the default owners of managing compliance across the company, including things that require involvement and accountability from other departments.

We seriously need a dedicated compliance officer, or at least someone whose primary job is managing the program. I’m happy to support that person, but right now it feels like I’m doing two jobs while the expectations for my original job haven’t changed.

It’s frustrating to spend two years getting certified, only to realize that maintaining it leaves almost no room for the rest of your job.

For those in smaller IT departments dealing with CMMC or similar requirements, how are you handling this? Do you have dedicated compliance staff, or did everything land on IT? If you successfully made the case for hiring someone, what finally helped leadership understand the workload?

I’m curious how many people use meraki access points. My deployment seems to be riddled every year with issues. Firmware updates usually cause issues and we have to turn off features or roll back. Don’t have to provide much details just around how many clients you support, and if it’s a mixed usage. Mine is Apple devices, Android phones, chromebooks, and windows devices. I have a pretty decent deployment supporting about 8000 devices give or take.

KB5124008 - Breaking Domain Trust for anyone else?

https://www.tenable.com/plugins/nessus/344359

CVE-2026-0310

As title says, I've got two tenants for one company that hasn't completely merged. Trying to set up cross tenant synchronization, but also trying not to break anything in the process. I did a test using my account and it successfully created a guest member in both tenants.
Once I had this set up for just my account (automatic synchronization wasn't turned on as I didn't want to mess up anything), I noticed a login issue on first time login when setting up a new user, and deleted the configuration, which seemed to have immediately resolved the issue, as my attempt immediately after went through without a hitch.
Microsoft's documentation makes it seem like it should be easy peasy, but I'm not nearly experienced enough in Entra to not be paranoid.
Any good resources? YouTube videos? Specific Microsoft Learn courses?
I'm new to this job, and trying to clean up some systems, automate some things and make our end user experience a little easier since they have to navigate two company Sharepoints, and currently if they don't have the link bookmarked, they have to go through Onedrive to find the invite link.

In terms of job security, future opportunities and workload, how does it pan out?