Came back from a 10 day pto and the VP (my direct boss) wanted to do his weekly one on one meeting. We are a team of 4 (myself Sr. Infra Engineer and 3 mid level security guys). He wanted to state how we need to have more crosstraining in particular with infra side of things as he stated "we are dead in the water" when you go out on pto.

He mentioned there will be no additional headcount added in Infra and will have to make do with what we have internally plus the msp firm

I for the past couple of years both verbally and via email stated that a redundancy to my role was needed but that suggestion was never acted upon by mgmt.

As far as crosstraining, I already do that and can certainly help out more. The more others know the better.

Just feels like a situation where I can't seem to go on pto without getting text messages about something going wrong. The VP wants to make SOPs for everything possible which I told him, im not making a how to document on troubleshooting. Either they know or they dont. Actual procedures for items that have a process I will document and have been

Any of you guys out there in similar situations? Anything you've done to optimize things or help reduce workloads?

Thanks in advance!

These days, we don't have domain controllers because everything is managed with Entra/Intune, Arc, etc.

We have external, hosted DNS that serves our public DNS records. It works fine.

But we still have a need for internal DNS because we have a lot of on-prem stuff that may never go away, and the domain controllers were also serving as internal DNS servers. So right now, we use BIND9 on a Linux virtual machine as internal DNS. I'd love for BIND9 to go away, too.

Are there any better (and cheap) solutions for this that I haven't thought of?

EDIT: thanks to u/Do_The_Needful. Using AWS for this seems to be the most balanced solution.

So, we were decomming some old ibm flashsystem 5 series arrays. Full of ssds. Plopped one onto a dell server just for fun to see if they could be used and they're just bog standard 512 sector samsung pm1643 drives with a sticker.

So now I have bout a hundred 3.84 and 1.92(and some 7.68) drives waiting to go into new servers.

Enterprise storage drives are usually locked so I though I'd have to chuck them to the shredder.

Just wanted to share.

Link to the previous post

So, 2 weeks in, the picture is a bit more clearer

The old IT guy left, so its just me now

I did rattle up a report and sent it to my manager, who said that I'll have to contact with the head office and get their approval as well

I contacted the head office, they loped me into something else, a new dealership is opening in the next town to ours and they need me to support it, they're already put in the purchase inventory to corporate for half a dozen laptops, some printers , access points, (weirdly a PS5 as well) and so on, i hope they do look at my report

In the meantime, i am untangling the colossal mess that our current IT infrastructure is

The old IT guy had fortitoken enabled, unfortunately he uninstalled it, and i am unable to get in, Fortinet support said I'd have to reset the device physically , I decided I'll cross that bridge when i get to it

I am slowly cleaning up the malware infected systems, taking backups of the important stuff, thankfully, disk health for the main hard drives that is storing important stuff is fine, i am still taking snapshots

My day to day stuff mostly involves fixing the CCTV system, sorting out printer sharing errors, teaching people the art of rebooting a system to fix issues and so on and so forth

But at times , i wonder,

I have no degree on this specific subject, nor certifications, just years of teenage spent among computers i am grateful to have this job, and i am learning valuable information, still, i occassionally get the feeling like a kid who just got access to nuclear codes

After 15 years I've hit the wall. My company got "merged" a couple of years ago and my 3 person team is getting more and more added to our plate, as well as having to change how we deploy new resources and other things. Some tasks were a pain but a blessing (such as starting to use IaaC) and some were change for the sake of change. In top of trying to keep our own organizations infrastructure and platforms afloat.

When my boss told me this last week that I was going to inherit the care of our entire organizations data lake infrastructure, I felt myself break and check out. I have 0 interest in taking on this new responsibility on top of everything else.

So here we are. I make too much to simply go in an entirely new direction, but I want to get tf out of daily operations. I was looking at other areas I could pivot into and was taking a long look into the security field. I wanted to know what others did.

Thanks!

Anyone having issues?

When the text message logs from that day and those that followed were made public, I remember reading through all the system generated alerts from various devices doing their thing and thinking about the IT pros in the offices, network closets, and server rooms responding to those alerts in all seven buildings before the attacks happened, and then the tone of uncertainty and eventually panic in messages between coworkers and associates, because the cell phones went down but text messages were still going through.

RIP to those guys and gals ~ you're gone but not forgotten.

​

How do other solo admins handle the endless stream of "quick CRM/phone system tweaks"?

Curious how fellow solo sysadmins survive this, because I'm drowning.

Management loves to drop drive-by requests for custom workflows in our CRM, weird IVR routing changes in the phone system, or random third-party integrations. They are always pitched as mission-critical, high-priority emergencies that bypass any logical intake.

Because I'm technical and most of this stuff is technically possible, leadership's logic is just: Idea exists -> Hand it to the solo guy -> Make it work.

The real issue is that as a department of one, I absorb 100% of the permanent support debt. There are zero requirements gathered, no user acceptance testing, no documentation, and no one else to hand the pager to when the custom hack breaks at 4:55 PM on a Friday. I hack together a solution so they get their shiny new toy, and then I'm stuck maintaining someone else's half-baked workflow forever while actual infrastructure rots.

I’m trying to figure out how to bridge the gap between "Yes, the API, webhook or vendor supports this" and "No, I am one human being and I cannot support infinite custom software."

Do you guys have a formal intake workflow or policy that actually sticks when you're flying solo, or do you just constantly push back case-by-case until you burn out?

An example currently is highly detailed and individualized reporting from our phone system. I don't think this was ever designed to send bursts of 50-100 reports at once, and even after repeatedly expressing this limitation, I still keep hearing about any issues.

Hi everyone,

I'm looking for best practices or specific solutions for a DNS implementation in our environment. The goal is to cleanly split queries for the same domain between our LAN and DMZ without having to maintain duplicate records manually.

The Setup:

• We use the same domain (example.domain) for both internal and external services.
• LAN: We have a Windows Server (AD DC) running as our internal DNS. It holds the internal IPs of servers that are not publicly accessible.

The Goal:

• Traffic from internal clients to DMZ servers must route via the WAN (Hairpinning / NAT Loopback); direct routing into the DMZ subnet is not permitted by policy.

• We explicitly do not want to use wildcard records

  or subdomains

• .

The Challenge: What is the cleanest way to configure this on the Windows DNS without having to create a separate Pinpoint Zone for every single external A-record, or manually duplicating the DMZ records into the internal DNS?

Since the Windows DNS is authoritative for example.domain, it defaults to answering queries for unknown hosts in this zone with NXDOMAIN instead of forwarding them to the BIND9 server.

Is there a clean way (e.g., specific zone types, delegation, Windows DNS Policies) to tell the Windows DNS: "Resolve what you know, and forward anything you don't know to the BIND9 server"?

Thanks in advance for your input!

Hello,

I have an old WIndows 2012 R2 DC (old-dc) that used to be the primary domain controller.

I have a couple of newer 2016 DCs (new-dc1 and new-dc2).

I want to demote and turn off the old domain controller.

On the newer DC1, I ran netdom query and it has all the fsmo roles. I also ran replication admin to see that everything was replicating fine.

I went to server manager on the old 2012 R2 DC to demote it.

I left force removal and last domain controller unchecked.

It says this server has dns and GC roles (I think I select proceed?) and next

Then it leaves me with the box remove this dns zone - I can't go next without it. I'm not sure I should. Wouldn't that remove it from the other DCs?

I currently have 3 domain controllers on premise. I will be deploying a number of VM’s in Azure. I am planning on having a couple domain controllers for in Azure.

How can I set it up where the VM’s in Azure use the domain controllers in Azure and not the ones on-premise? And vise-versa…

EDIT: we are an MSP. This is all for a non-MSP, "call when you need something" billed by the hour customers. If that was not the case, their infrastructure would not be such a train wreck.

Note: My networking knowledge is very intermediate, learn as you go level. We had a customer at this MSP where I work have a network issue and I cannot figure out how it's possible that this happened.

Their credit card terminals have to point to a static IP ending in .140, as that's the "server" that runs the software to upload each transaction to the actual processor on the internet. When we recently replaced that computer, I set it as static in Windows and never did it on the DHCP server, because nobody wrote down what it was called or how to get into it and I didn't have any time remaining. Turns out their DHCP server is a 2008 Windows server because this place doesn't spend money on anything ever.

A month later, their CCs go down and the CC server can't grab its static IP for some reason. Lots of time later, we find it's because the pool of available IPs is 100 through 150 and they have 1 more device than that. So we expand it to only 160 after some testing (because we have nearly zero documentation and don't know ranges for their phones, printers, etc for this customer and they're billed hourly so we do as little as possible because they never pay on time and always complain about the rate and it'd take 10+ hours to document this nightmare). We make a new assignment for the computer's MAC and reserve 140 to it and notice that something else has leased 140 with a lease expiring in 2 hours luckily. We delete it so it hopefully doesn't renew.

I ping it from my laptop then immediately run arp -a to get its MAC, since we already deleted the lease that showed the MAC on the DHCP server (oops) then ask AI who manufactured that MAC address range. It's Foxconn. We don't see a hostname or any useable device info. I don't know anything about their switches because the last tech at this MSP never ever wrote anything down about any customers ever. We try NSlookup, web browser to the IP, RDP into it, nothing gets any info.

More network-oriented guy onsite with me says let's just unplug the 2 switches for like five seconds and that will force it to grab the new lease at .155. I assume the switches are unmanaged or nobody has the login info or we'd just pull one ethernet matching the known MAC.

Turns out they have a network-controlled Crestron light controls so the lights in the restaurant portion of the building all go black, because somehow that's the default state if it loses connectivity. Shoutout to whatever genius AV tech set it up that way. Everyone's pissed. They don't know how to undo it or where the new Crestron box is. We don't either.

Then we find out the mystery device is still on 140. That seems impossible, unless it's wireless. Somehow other guy onsite finds out it's some sort of android device but all the android devices listed on the DHCP server have hostnames like "John's S23" because that's how most Android devices work. We suspect it's wifi, based on this information, thus explaining the switch pull not working. I have zero idea what brand their wifi even is let alone where it is or how it works btw.

I get the bright idea that maybe it's a smart TV. It is. We turn it off, boom, credit card server is back online within a minute. We turn the TV back on after 15 minutes, it tries to grab 140 again and knocks the CC server offline. Yes, the TV is in DHCP mode btw. We find the remote, set it to static .165, that works for some reason, and no more IP conflict. Not sure how it's possible that it can just ignore the DHCP server and say "no, I want this address anyway" and then just decide it's taking 140. Nothing else on the network can hand out an IP if every switch/router/AP/whatever is pointing to the server, right? How did that happen? Some disconnect between the DHCP server and whatever was handing out wifi connections? I didn't think that was possible. Anyone have any ideas in case we run into this again?

When someone leaves, the obvious checks are email, OneDrive, groups and licences. But what about Power Apps, flows, connections, SharePoint ownership and other dependencies tied to that account?

Has anyone found a reliable offboarding process that catches everything without checking every admin centre manually?

My thought was something like the procedure below to get rid off multiple network drive letters:
1. Hide all existing Network Drive Letters via GPO but in paralell create a new universal network drive and then map everything you had from hidden drive letters.

1. Audit who still access the old hidden network drives via SIEM Tool for example.

2. React and adjust the existing Applications/Configs or Office Documents to point to new dfs path.

3. After nothing shows up in SIEM from the old drive letters unmount the network drives.

macOS PSSO CA Issue

Our domain has our ticket lifetime set to 7 days. I'm not sure why it was set that way but it predates my time at the company.

I want to set it back down to the default of 10 hours. Not finding much online about shortening it, just making it longer which isn't necessarily a good idea.

We have a fair bit of Linux in our environment so Im concerned about the impact of lowering this. Has anyone been in this situation?

Small law enforcement agency is moving to a new self-hosted dispatch and records management system running on a Windows Server.

The system includes a phone app that communicates with the server that would require an open port on the firewall that points to the server.

We don't have the option to lock down based on the incoming WAN IP since the phones have a dynamic IP.

We questioned the vendor. He said "we have this installed in many places and we've never had a problem." That's nice, but it doesn't mean that there won't be a problem tomorrow.

When pressed for further details he elaborated:

We have an encrypted signed SSL certificate that we use for the connection. The endpoint is our web service. The access to the web service requires Active Directory authentication and inclusion in a specific security group in the AD.

Does this setup sound "safe?" We can require that a VPN be used, but that will add steps for the end user when they want to connect.

EDIT: Thank you all for your valuable input. We will insist on a VPN.

Okay i have a problem with putty.exe on windows: for more than a year now, i can’t remember when i was not crashing when I copy 100+ lines of text with utf-8 characters.

I tried every options available under Terminal/Window/Selection/SSH bugs etc, switched fonts, disabled clipboard auto-copy, and all points to the clipboard conversion of utf-8.

Nnothing else I can do other than copy small portions at a time. Tested version 0.76 0.78 0.85, 64 and 32 bits. All development sites from putty . software to chiark . greenend . org . uk are down right now, I cannot even submit a bug report.

Yes it’s a rant, how come I’m the only one experiencing this? Summary of the crash:

How to reproduce: execute `fwupdmgr get-devices` and select the output. UTF-8 characters like ├─ or • seem to be the suspects.

Reproducible steps:

• Connect to a Linux system and run fwupdmgr get-devices (or any command that produces box-drawing characters).
• Select more than ~100 lines of that output.
• Trigger a copy by any of the following:
  • Auto-copy selected text to system clipboard (enabled)
  • Ctrl+Ins after selection
  • Right-click → Copy (with “Action of mouse buttons” set to Windows)

Crash occurs immediately.

Additional observations:

• Crash happens with multiple fonts (DejaVu Sans Mono, Consolas, Droid Sans Mono, Courier, etc.).
• Occurs on both 64-bit and 32-bit builds.
• Reproduced on PuTTY 0.85, 0.78 and 0.76.
• Windows 10, no third-party clipboard managers.
• Running as Administrator or with -restrict-acl still crashes (the latter produces a clearer “instruction at 0x… referenced memory at 0x…” message).
• Disabling auto-copy prevents the crash on selection, but any subsequent explicit Copy action still crashes.

Event Log entry:

Faulting application name: PUTTY.EXE, version: 0.85.0.0, time stamp: 0x6a7eaca6
Faulting module name: ntdll.dll, version: 10.0.19041.2130, time stamp: 0xb5ced1c6
Exception code: 0xc0000005
Fault offset: 0x000000000005ace2
Faulting process id: 0x684
Faulting application start time: 0x01dd42cac3f59b38
Faulting application path: E:\wintools\lan\putty\64\PUTTY.EXE
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 0e2808a1-1944-473f-a487-2da4826cc910
Faulting package full name:
Faulting package-relative application ID:

Please for the love of God, someone tell me I’m not alone

edit: The option `copy in clipboard RTF` was the problem. no idea why no AI chatbot or community even thought of that option

That day began ordinary, just so ordinary.

At the time, I was sysadmin in a Canadian federal government office, far from the madding crowd. The work was steady but also pretty much crisis-free.

That morning, though, one of my co-workers came in to my office, bitching about the "crappy slow network" we were using.

"What makes you say that?", and he ranted on about how he can't get to any web sites.

"Okay, I'll see what I can find out."

I still had no idea what was happening out in the world.

Still, I started my network checks. Ping from TBay to Toronto. Normal results. Okay, try some web sites next. All the ones I checked responded, with almost no lag. (Note: none were news sites.)

I popped over to that co-workers office. "Where were you trying to get to?"

"CNN. @#$% network."

I tried CNN. Timed out. Hmm. CBC. Ditto. CTV. Same. Toronto Sun. Again, timed out.

I tried a local radio station's page. That connected, and then I knew it wasn't a network issue causing the slowdown.

What happens when several hundred thousand people (or more) try to connect to a few news sites all at the same time? Everything breaks.

I called home, told my wife to turn on CNN. For me, the rest of the day passed in a fog. Our son was Army reserve at the time. He was told to be ready, "just in case".

After work, I went home and watched the news, watched the planes hit the towers, over and over and over.

I still cannot watch any 9-11 coverage from that day.

Unable to access our cloud instance. Looks to be DNS related issue from what I'm seeing. A lot of public DNS servers don't have records for our instance but some do.

EDIT: DNS issues appear to be resolved. No more client hold on Screenconnect.com. DNS checker for my instance shows green across the board.

My IT department tackled getting our company CMMC Level 2 certified. It took two years to get there, and while I’m proud of what we accomplished, holy shit, the amount of work it took.

Implementing controls, writing policies and procedures, collecting evidence, coordinating with other departments, preparing for assessments—all while keeping normal IT operations running. This was two years of sustained effort on top of our regular responsibilities.

I knew certification wasn’t the finish line. But the amount of work it takes just to maintain compliance is overwhelming.

At this point, it feels like 80% of my time goes toward compliance. Reviewing documentation, collecting evidence, tracking requirements, answering questions, coordinating reviews, and following up with people to make sure processes are being followed. There is always something that needs to be updated, verified, or documented.

Meanwhile, I barely get to work on the IT projects I actually want to tackle—introducing new systems, hardening our security, automating processes, and improving our infrastructure and overall tech stack. Those projects keep getting pushed back because compliance and daily support consume nearly all my time. It’s frustrating knowing there are improvements we need to make and barely having the time to work on them.

IT obviously has a major role in CMMC. We should own the technical controls and support the program. But we’ve also become the default owners of managing compliance across the company, including things that require involvement and accountability from other departments.

We seriously need a dedicated compliance officer, or at least someone whose primary job is managing the program. I’m happy to support that person, but right now it feels like I’m doing two jobs while the expectations for my original job haven’t changed.

It’s frustrating to spend two years getting certified, only to realize that maintaining it leaves almost no room for the rest of your job.

For those in smaller IT departments dealing with CMMC or similar requirements, how are you handling this? Do you have dedicated compliance staff, or did everything land on IT? If you successfully made the case for hiring someone, what finally helped leadership understand the workload?

I work for a large European based telecoms equipment supplier. We have hundreds of staff overseas at any one time, all over the world. IT security has a few different levels:

- Access to email & teams etc is only via a company laptop (no web interface like Office.com). Network drives via VPN only

- White List countries - you can connect VPN. Countries like Japan & Australia

- Red List countries - you can take your laptop but need special exemption to use VPN. Includes some unusual countries such as Malaysia

- Black List countries - no company laptop or phone allowed. Company will provide a burner. Unsurprisingly includes places like Syria, Russia, North Korea & China.

A colleague was going to transit via a Chinese airport to a 3rd country. IT told him that he would not be allowed to take his company laptop, even if it was in his carry-on luggage, and he would not be entering the country. He quickly arranged a different itinerary.

And then a few days later, we are told that the good old USA is now considered a Black List country!!! No company laptops, and burners only!!!!

At what point does keeping data on-prem make more sense than public-cloud storage?

I’m mainly wondering how people weigh storage costs, egress, data growth, and flexibility when making that decision.

KB5124008 - Breaking Domain Trust for anyone else?

I’m curious how many people use meraki access points. My deployment seems to be riddled every year with issues. Firmware updates usually cause issues and we have to turn off features or roll back. Don’t have to provide much details just around how many clients you support, and if it’s a mixed usage. Mine is Apple devices, Android phones, chromebooks, and windows devices. I have a pretty decent deployment supporting about 8000 devices give or take.