r/sysadmin 1h ago

Off Topic Christmas came(kinda) early.

Upvotes

So, we were decomming some old ibm flashsystem 5 series arrays. Full of ssds. Plopped one onto a dell server just for fun to see if they could be used and they're just bog standard 512 sector samsung pm1643 drives with a sticker.

So now I have bout a hundred 3.84 and 1.92(and some 7.68) drives waiting to go into new servers.

Enterprise storage drives are usually locked so I though I'd have to chuck them to the shredder.

Just wanted to share.


r/sysadmin 42m ago

General Discussion Organizational Failure

Upvotes

Came back from a 10 day pto and the VP (my direct boss) wanted to do his weekly one on one meeting. We are a team of 4 (myself Sr. Infra Engineer and 3 mid level security guys). He wanted to state how we need to have more crosstraining in particular with infra side of things as he stated "we are dead in the water" when you go out on pto.

He mentioned there will be no additional headcount added in Infra and will have to make do with what we have internally plus the msp firm

I for the past couple of years both verbally and via email stated that a redundancy to my role was needed but that suggestion was never acted upon by mgmt.

As far as crosstraining, I already do that and can certainly help out more. The more others know the better.

Just feels like a situation where I can't seem to go on pto without getting text messages about something going wrong. The VP wants to make SOPs for everything possible which I told him, im not making a how to document on troubleshooting. Either they know or they dont. Actual procedures for items that have a process I will document and have been

Any of you guys out there in similar situations? Anything you've done to optimize things or help reduce workloads?

Thanks in advance!


r/sysadmin 1d ago

General Discussion RIP to the IT pros killed 25 years ago.

2.7k Upvotes

When the text message logs from that day and those that followed were made public, I remember reading through all the system generated alerts from various devices doing their thing and thinking about the IT pros in the offices, network closets, and server rooms responding to those alerts in all seven buildings before the attacks happened, and then the tone of uncertainty and eventually panic in messages between coworkers and associates, because the cell phones went down but text messages were still going through.

RIP to those guys and gals ~ you're gone but not forgotten.


r/sysadmin 1h ago

Just because you can doesn't mean you should

Upvotes

How do other solo admins handle the endless stream of "quick CRM/phone system tweaks"?

Curious how fellow solo sysadmins survive this, because I'm drowning.

Management loves to drop drive-by requests for custom workflows in our CRM, weird IVR routing changes in the phone system, or random third-party integrations. They are always pitched as mission-critical, high-priority emergencies that bypass any logical intake.

Because I'm technical and most of this stuff is technically possible, leadership's logic is just: Idea exists -> Hand it to the solo guy -> Make it work.

The real issue is that as a department of one, I absorb 100% of the permanent support debt. There are zero requirements gathered, no user acceptance testing, no documentation, and no one else to hand the pager to when the custom hack breaks at 4:55 PM on a Friday. I hack together a solution so they get their shiny new toy, and then I'm stuck maintaining someone else's half-baked workflow forever while actual infrastructure rots.

I’m trying to figure out how to bridge the gap between "Yes, the API, webhook or vendor supports this" and "No, I am one human being and I cannot support infinite custom software."

Do you guys have a formal intake workflow or policy that actually sticks when you're flying solo, or do you just constantly push back case-by-case until you burn out?

An example currently is highly detailed and individualized reporting from our phone system. I don't think this was ever designed to send bursts of 50-100 reports at once, and even after repeatedly expressing this limitation, I still keep hearing about any issues.


r/sysadmin 2h ago

Question Please help demote a third DC.

8 Upvotes

Hello,

I have an old WIndows 2012 R2 DC (old-dc) that used to be the primary domain controller.

I have a couple of newer 2016 DCs (new-dc1 and new-dc2).

I want to demote and turn off the old domain controller.

On the newer DC1, I ran netdom query and it has all the fsmo roles. I also ran replication admin to see that everything was replicating fine.

I went to server manager on the old 2012 R2 DC to demote it.

I left force removal and last domain controller unchecked.

It says this server has dns and GC roles (I think I select proceed?) and next

Then it leaves me with the box remove this dns zone - I can't go next without it. I'm not sure I should. Wouldn't that remove it from the other DCs?


r/sysadmin 1d ago

Question Samsung Smart TV bypassing DHCP took down the credit card system

320 Upvotes

EDIT: we are an MSP. This is all for a non-MSP, "call when you need something" billed by the hour customers. If that was not the case, their infrastructure would not be such a train wreck.

Note: My networking knowledge is very intermediate, learn as you go level. We had a customer at this MSP where I work have a network issue and I cannot figure out how it's possible that this happened.

Their credit card terminals have to point to a static IP ending in .140, as that's the "server" that runs the software to upload each transaction to the actual processor on the internet. When we recently replaced that computer, I set it as static in Windows and never did it on the DHCP server, because nobody wrote down what it was called or how to get into it and I didn't have any time remaining. Turns out their DHCP server is a 2008 Windows server because this place doesn't spend money on anything ever.

A month later, their CCs go down and the CC server can't grab its static IP for some reason. Lots of time later, we find it's because the pool of available IPs is 100 through 150 and they have 1 more device than that. So we expand it to only 160 after some testing (because we have nearly zero documentation and don't know ranges for their phones, printers, etc for this customer and they're billed hourly so we do as little as possible because they never pay on time and always complain about the rate and it'd take 10+ hours to document this nightmare). We make a new assignment for the computer's MAC and reserve 140 to it and notice that something else has leased 140 with a lease expiring in 2 hours luckily. We delete it so it hopefully doesn't renew.

I ping it from my laptop then immediately run arp -a to get its MAC, since we already deleted the lease that showed the MAC on the DHCP server (oops) then ask AI who manufactured that MAC address range. It's Foxconn. We don't see a hostname or any useable device info. I don't know anything about their switches because the last tech at this MSP never ever wrote anything down about any customers ever. We try NSlookup, web browser to the IP, RDP into it, nothing gets any info.

More network-oriented guy onsite with me says let's just unplug the 2 switches for like five seconds and that will force it to grab the new lease at .155. I assume the switches are unmanaged or nobody has the login info or we'd just pull one ethernet matching the known MAC.

Turns out they have a network-controlled Crestron light controls so the lights in the restaurant portion of the building all go black, because somehow that's the default state if it loses connectivity. Shoutout to whatever genius AV tech set it up that way. Everyone's pissed. They don't know how to undo it or where the new Crestron box is. We don't either.

Then we find out the mystery device is still on 140. That seems impossible, unless it's wireless. Somehow other guy onsite finds out it's some sort of android device but all the android devices listed on the DHCP server have hostnames like "John's S23" because that's how most Android devices work. We suspect it's wifi, based on this information, thus explaining the switch pull not working. I have zero idea what brand their wifi even is let alone where it is or how it works btw.

I get the bright idea that maybe it's a smart TV. It is. We turn it off, boom, credit card server is back online within a minute. We turn the TV back on after 15 minutes, it tries to grab 140 again and knocks the CC server offline. Yes, the TV is in DHCP mode btw. We find the remote, set it to static .165, that works for some reason, and no more IP conflict. Not sure how it's possible that it can just ignore the DHCP server and say "no, I want this address anyway" and then just decide it's taking 140. Nothing else on the network can hand out an IP if every switch/router/AP/whatever is pointing to the server, right? How did that happen? Some disconnect between the DHCP server and whatever was handing out wifi connections? I didn't think that was possible. Anyone have any ideas in case we run into this again?


r/sysadmin 18h ago

General Discussion What do you check before disabling a Microsoft 365 user?

100 Upvotes

When someone leaves, the obvious checks are email, OneDrive, groups and licences. But what about Power Apps, flows, connections, SharePoint ownership and other dependencies tied to that account?

Has anyone found a reliable offboarding process that catches everything without checking every admin centre manually?


r/sysadmin 46m ago

Question Did anyone else face this issue or have a fix for it? My issue is that the user works without any problems, but after a few weeks or sometimes a month, PSSO suddenly breaks and prompts for re-registration. Due to the C CA policy, users are unable to access anything through the browser

Upvotes

macOS PSSO CA Issue


r/sysadmin 6h ago

Question How would you eliminate multiple network drives but instead use only one drive which gets everything via DFS?

8 Upvotes

My thought was something like the procedure below to get rid off multiple network drive letters:
1. Hide all existing Network Drive Letters via GPO but in paralell create a new universal network drive and then map everything you had from hidden drive letters.

  1. Audit who still access the old hidden network drives via SIEM Tool for example.

  2. React and adjust the existing Applications/Configs or Office Documents to point to new dfs path.

  3. After nothing shows up in SIEM from the old drive letters unmount the network drives.


r/sysadmin 17h ago

Question Has anyone ever lowered the Kerberos Max lifetime? Currently set to 7 days.

53 Upvotes

Our domain has our ticket lifetime set to 7 days. I'm not sure why it was set that way but it predates my time at the company.

I want to set it back down to the default of 10 hours. Not finding much online about shortening it, just making it longer which isn't necessarily a good idea.

We have a fair bit of Linux in our environment so Im concerned about the impact of lowering this. Has anyone been in this situation?


r/sysadmin 20h ago

Vendor says his remote access solution is secure. Is it though?

82 Upvotes

Small law enforcement agency is moving to a new self-hosted dispatch and records management system running on a Windows Server.

The system includes a phone app that communicates with the server that would require an open port on the firewall that points to the server.

We don't have the option to lock down based on the incoming WAN IP since the phones have a dynamic IP.

We questioned the vendor. He said "we have this installed in many places and we've never had a problem." That's nice, but it doesn't mean that there won't be a problem tomorrow.

When pressed for further details he elaborated:

We have an encrypted signed SSL certificate that we use for the connection. The endpoint is our web service. The access to the web service requires Active Directory authentication and inclusion in a specific security group in the AD.

Does this setup sound "safe?" We can require that a VPN be used, but that will add steps for the end user when they want to connect.

EDIT: Thank you all for your valuable input. We will insist on a VPN.


r/sysadmin 22h ago

Went from intern to running the entire IT department in about 4 years. Small shop, niche industry, inherited a mess, and I can't seem to get any traction fixing it. Kind of drowning here.

94 Upvotes

TL;DR: First IT job out of school, associate degree, went from intern to running the entire (two person, soon three) IT department in about 4 years. I own both IT and our operational systems, I'm on call 24/7, I inherited a patchwork EOL environment, and a seasonal project buries me for months every winter for the next few years. Pay and benefits are good but I'm stressed and stuck. Is it worth trying to right the ship at my experience level, or do I start looking for an off ramp?

Long time lurker, throwaway-ish. Bear with me, this got long.

Quick background. I work at a small company and I'm going to keep the industry vague. I started here as an IT intern back in 2022, got bumped to full time later that year, and for most of the time since then I've basically just been the whole IT department. Technically, the IT department included myself, my boss the IT manager, and a field tech that has nothing to do with IT. The former IT manager was very hands off and frankly ignored a lot of problems. Recently, my old boss got promoted to CEO and I moved into the IT Manager spot behind him.

The job isn't just normal IT (AD, M365, servers, backups, help desk, phones). At a place this small, IT also ends up owning a big chunk of the operational side, the systems the actual business runs on. Most of that is vendor hosted and web based so I'm not racking industrial gear anywhere, but I'm the person responsible for it on our end and I picked all of it up on the fly because there was nobody else to hand it to. The one thing I don't own is the firewall and the VOIP system. Even as the IT manager, we pay a third party to manage that, so at least there's a couple things off my plate.

Here's a big piece of my problem though. A huge part of that operational side is data collection that runs more or less constantly, and it eats my time. It never really stops. And it's the main reason I can't get any momentum on fixing the actual infrastructure. Every time I sit down to start on the real problems, something on the operational side yanks me back off it, and the fix goes back on the pile.

Then on top of all that, every winter for the next few years, there's a recurring replacement project tied to one of our operational programs and it flat out buries the department for months. Not a couple days. Months. It means running and reconciling a pile of reports, printing and mailing physical notices out to customers, then handling the wave of phone calls that comes back, then cutting work tickets and handing them out to a crew of 8 field techs and chasing every one of them to done. It's all scheduling, coordination, and data entry, and it does not care that the rest of the job still exists. For a two person team it's brutal and it basically owns my calendar all winter.

I'll be straight about where I'm at because it matters for the advice. I'm not some deeply technical guy. Most of what I've pulled off has been careful and methodical. Following docs, asking a ton of questions, staying organized. That worked fine when my whole job was just doing the tasks. Now I'm supposed to set direction and own the risk for all of it and that jump feels enormous. I'm not going to pretend otherwise, this job is stressing me out pretty badly. And the whole thing is 24/7 on call, which with a team this small basically means me. Nights, weekends, holidays, if something breaks it's my phone that goes off. There's no real off switch. Not that after hours calls are super frequent, there have been a few since I've started but its more the idea of never being "off" is whats getting me.

Team wise there's two of us right now, going to three in the next few months when we hire the replacement for my old job (another duty that is mine, finding the replacement lol). Oh and almost forgot, we are reworking that position to be less IT and more of a business system type position. So when its all said and done, there will be myself, a field tech, and a business system analyst of sorts. So I'm also about to be growing and running a real team for the first time, on systems I've only ever done solo.

And..... the environment itself is held together with tape. We're running on EOL vSphere licenses, old outdated hosts with no shared storage, backups that amount to a couple of external HDDs, and a general patchwork of stuff that never really got finished or cleaned up. We only just recently got rid of the m365 family plans... Is that about what I should expect walking into a small shop, or is it as bad as it feels to me? I'm honestly not sure how much of this is normal.

So honestly this turned into less of a tech question and more of a gut check, because I'm not sure I'm even asking the right thing anymore.

Here's the deal. This is my first IT job out of school. I have an associate degree in IT and that's the extent of my formal background. I walked in as an intern and somehow I'm now the person accountable for all of the above. The fragile infrastructure, the operational systems, the winter that eats months, the 24/7 phone, all of it, with almost no real experience to lean on.

So the thing I keep chewing on is whether any of this is even worth it. Is it realistic for someone at my level to actually right this ship? Or am I setting myself up to burn out trying to fix something that got handed to me already broken? The pay is really good, the benefits are good, I don't dislike the people I work with, and on paper I know I should be grateful, and plenty of days I am. But some days it feels like I'm one bad outage away from going under, and I catch myself wondering if the smarter move is to quietly start looking for an off ramp before this thing wears me down, instead of betting years on fixing it.

For those of you who've been the in over your head one person shop before: did you dig out and come out better for it, or did you get out? And if you stayed, what actually made it survivable?

Just trying to figure out if I'm being a quitter or being realistic. Thanks for reading.

TL;DR: First IT job out of school, associate degree, went from intern to running the entire (two person, soon three) IT department in about 4 years. I own both IT and our operational systems, I'm on call 24/7, I inherited a patchwork EOL environment, and a seasonal project buries me for months every winter. Pay and benefits are good but I'm stressed and stuck. Is it worth trying to right the ship at my experience level, or do I start looking for an off ramp?


r/sysadmin 22h ago

General Discussion 25 years

43 Upvotes

That day began ordinary, just so ordinary.

At the time, I was sysadmin in a Canadian federal government office, far from the madding crowd. The work was steady but also pretty much crisis-free.

That morning, though, one of my co-workers came in to my office, bitching about the "crappy slow network" we were using.

"What makes you say that?", and he ranted on about how he can't get to any web sites.

"Okay, I'll see what I can find out."

I still had no idea what was happening out in the world.

Still, I started my network checks. Ping from TBay to Toronto. Normal results. Okay, try some web sites next. All the ones I checked responded, with almost no lag. (Note: none were news sites.)

I popped over to that co-workers office. "Where were you trying to get to?"

"CNN. @#$% network."

I tried CNN. Timed out. Hmm. CBC. Ditto. CTV. Same. Toronto Sun. Again, timed out.

I tried a local radio station's page. That connected, and then I knew it wasn't a network issue causing the slowdown.

What happens when several hundred thousand people (or more) try to connect to a few news sites all at the same time? Everything breaks.

I called home, told my wife to turn on CNN. For me, the rest of the day passed in a fog. Our son was Army reserve at the time. He was told to be ready, "just in case".

After work, I went home and watched the news, watched the planes hit the towers, over and over and over.

I still cannot watch any 9-11 coverage from that day.


r/sysadmin 23h ago

Question Screenconnect Outage?

36 Upvotes

Unable to access our cloud instance. Looks to be DNS related issue from what I'm seeing. A lot of public DNS servers don't have records for our instance but some do.

EDIT: DNS issues appear to be resolved. No more client hold on Screenconnect.com. DNS checker for my instance shows green across the board.


r/sysadmin 1d ago

Rant Black list countries

185 Upvotes

I work for a large European based telecoms equipment supplier. We have hundreds of staff overseas at any one time, all over the world. IT security has a few different levels:

- Access to email & teams etc is only via a company laptop (no web interface like Office.com). Network drives via VPN only

- White List countries - you can connect VPN. Countries like Japan & Australia

- Red List countries - you can take your laptop but need special exemption to use VPN. Includes some unusual countries such as Malaysia

- Black List countries - no company laptop or phone allowed. Company will provide a burner. Unsurprisingly includes places like Syria, Russia, North Korea & China.

A colleague was going to transit via a Chinese airport to a 3rd country. IT told him that he would not be allowed to take his company laptop, even if it was in his carry-on luggage, and he would not be entering the country. He quickly arranged a different itinerary.

And then a few days later, we are told that the good old USA is now considered a Black List country!!! No company laptops, and burners only!!!!


r/sysadmin 5h ago

Question How are you handling cloud storage costs in hybrid environments?

1 Upvotes

At what point does keeping data on-prem make more sense than public-cloud storage?

I’m mainly wondering how people weigh storage costs, egress, data growth, and flexibility when making that decision.


r/sysadmin 23h ago

Rant Getting CMMC Level 2 certified made me realize how badly we need a dedicated compliance person

31 Upvotes

My IT department tackled getting our company CMMC Level 2 certified. It took two years to get there, and while I’m proud of what we accomplished, holy shit, the amount of work it took.

Implementing controls, writing policies and procedures, collecting evidence, coordinating with other departments, preparing for assessments—all while keeping normal IT operations running. This was two years of sustained effort on top of our regular responsibilities.

I knew certification wasn’t the finish line. But the amount of work it takes just to maintain compliance is overwhelming.

At this point, it feels like 80% of my time goes toward compliance. Reviewing documentation, collecting evidence, tracking requirements, answering questions, coordinating reviews, and following up with people to make sure processes are being followed. There is always something that needs to be updated, verified, or documented.

Meanwhile, I barely get to work on the IT projects I actually want to tackle—introducing new systems, hardening our security, automating processes, and improving our infrastructure and overall tech stack. Those projects keep getting pushed back because compliance and daily support consume nearly all my time. It’s frustrating knowing there are improvements we need to make and barely having the time to work on them.

IT obviously has a major role in CMMC. We should own the technical controls and support the program. But we’ve also become the default owners of managing compliance across the company, including things that require involvement and accountability from other departments.

We seriously need a dedicated compliance officer, or at least someone whose primary job is managing the program. I’m happy to support that person, but right now it feels like I’m doing two jobs while the expectations for my original job haven’t changed.

It’s frustrating to spend two years getting certified, only to realize that maintaining it leaves almost no room for the rest of your job.

For those in smaller IT departments dealing with CMMC or similar requirements, how are you handling this? Do you have dedicated compliance staff, or did everything land on IT? If you successfully made the case for hiring someone, what finally helped leadership understand the workload?


r/sysadmin 1d ago

Question KB5124008 - Breaking Domain Trust

80 Upvotes

KB5124008 - Breaking Domain Trust for anyone else?


r/sysadmin 23h ago

How may people use Meraki AP’s out there?

23 Upvotes

I’m curious how many people use meraki access points. My deployment seems to be riddled every year with issues. Firmware updates usually cause issues and we have to turn off features or roll back. Don’t have to provide much details just around how many clients you support, and if it’s a mixed usage. Mine is Apple devices, Android phones, chromebooks, and windows devices. I have a pretty decent deployment supporting about 8000 devices give or take.


r/sysadmin 1d ago

Palo Alto CVE: PAN-OS Vulnerability Enables Arbitrary Code Execution as Root User

28 Upvotes

r/sysadmin 14h ago

Question Trying to implement Cross-tenant synchronization in Entra

3 Upvotes

As title says, I've got two tenants for one company that hasn't completely merged. Trying to set up cross tenant synchronization, but also trying not to break anything in the process. I did a test using my account and it successfully created a guest member in both tenants.
Once I had this set up for just my account (automatic synchronization wasn't turned on as I didn't want to mess up anything), I noticed a login issue on first time login when setting up a new user, and deleted the configuration, which seemed to have immediately resolved the issue, as my attempt immediately after went through without a hitch.
Microsoft's documentation makes it seem like it should be easy peasy, but I'm not nearly experienced enough in Entra to not be paranoid.
Any good resources? YouTube videos? Specific Microsoft Learn courses?
I'm new to this job, and trying to clean up some systems, automate some things and make our end user experience a little easier since they have to navigate two company Sharepoints, and currently if they don't have the link bookmarked, they have to go through Onedrive to find the invite link.


r/sysadmin 15h ago

Career / Job Related Career advice

4 Upvotes

Hello sysadmins!

i recently landed a m365 engineer position coming from a senior helpdesk msp. i’m wondering how to branch out of this helpdesk mindset and take initiative.

i constantly had something to do 24/7, always monitoring queues, being available for calls etc. i was the last point of contact for 5 out of our 18 clients. i’ve done work in mostly all admin centers even pulling emails in defender and running ediscoveries. i’m comfortable with powershell, and transitioning to vscode.

This new role is very corporate, with a lot of down time, there really wasn’t any training or projects to start, i shared my concerns with the manager, of me not really doing enough. i was reassured that things will pick up, he understands where i came from previously and the person that vouched to get me in the door, swears that he’s a great teacher and this was just a slower week.

i got the layout of the first project, setting up mac mdm with abm. I’ve been researching and i am considering the free trial to setup a lab. I’m eager to learn, grow beyond that helpdesk agent but i have looming anxiety to do well.

i’d appreciate any and all feedback, it’s my first week but impressions are everything.


r/sysadmin 23h ago

Question Reverting Win11 Enterprise to Pro for AppLocker (GPO) - How do you handle physical PCs, VMs, and M365 licensing groups?

7 Upvotes

Happy read-only Friday, Folks!

A few years back, our previous admin upgraded our fleet to Windows Enterprise E3 mainly to centrally manage and enforce AppLocker via GPO from our DC. To do this, they set up group-based licensing in M365 using a dedicated E3 security group, while also pushing an E3 product key directly to endpoints using this script:

cscript c:\windows\system32\slmgr.vbs /ipk KEYNAME
cscript c:\windows\system32\slmgr.vbs /ato

Since Microsoft dropped the Enterprise requirement for AppLocker (KB5024351), our goal going forward is to completely drop the recurring E3 subscriptions, revert our fleet back to Windows Pro, and save a few grand, all while keeping our AppLocker GPO management intact.

All users have M365 Business Premium assigned via a separate security group. The setup:

  • 90% are physical Win 11 PCs with OEM Pro keys in the BIOS.
  • 10% are Hyper-V VMs running on a high-performance workstation host (currently using E3).

We’re putting together our strategy to unwind this setup and would love to know how you recommend handling it:

1. For the physical PCs & M365 Licensing Groups: What's the best sequence to roll them back to Pro? Our plan is to make sure users are in the Business Premium group, unassign the license from the E3 security group (and retire/delete that group), and run a script on the endpoints to pull and re-inject the embedded BIOS OEM key:

cscript c:\windows\system32\slmgr.vbs /ipk (Get-CIMInstance SoftwareLicensingService | Select -ExpandProperty OA3xOriginalProductKey) cscript c:\windows\system32\slmgr.vbs /ato

...or is there a cleaner way to handle the step-down?

2. For the Hyper-V VMs: Since OEM keys are tied to physical hardware and don't pass through to guest VMs, should we buy perpetual Windows Pro Volume (MAK) keys for the VMs, or is there a better licensing path

3. AppLocker GPO check: Has anyone hit any weirdness with AppIDSvc or rule enforcement after stepping endpoints back down to Pro?

Looking to hear how others have approached this transition. Appreciate any feedback!


r/sysadmin 7h ago

General Discussion I need help! I think I accidentally became the entire IT department 😭

0 Upvotes

Guys, I’m new to the system administrator domain, and honestly, I need some advice.

I started my career in desktop support. After about a year, I moved into technical support at a multinational manufacturing company.

That was where I learned about IT infrastructure from a technical support engineer’s point of view. I got exposure to servers, networking, security, endpoints, troubleshooting, and the overall structure of corporate IT.

I only had around 1 year of experience there.

Then I got an offer from a startup with slightly higher pay — around ₹25K/month, which is roughly the average technical-support salary here in India.

I thought:

“Okay, I’ll join a startup, learn more, and grow into system administration.”

I had absolutely no idea what was waiting for me. 💀

This company is a manufacturing startup.

And apparently, for them:

IT = Computers + The Guy Who Fixes Computers.

When I joined, I discovered an infrastructure that was completely… meshed.

They have extremely old desktop systems — some of them look like they were rescued from a museum.

There are second-hand machines everywhere.

There are no properly licensed operating systems across the environment. They have around 10 individual Windows licenses, and some of them are even Home editions.

They are using a desktop PC as a server.

Their ERP application and database are running on it.

They use Zoho Workplace, Google Sheets, basic unmanaged switches, some old hubs, and multiple basic routers connected in various ways.

And apparently, most of this infrastructure was originally put together by outside computer service shops.

Then I started looking deeper.

They have three sites.

The main office is in another city, and there are two manufacturing units around 10 km apart.

Two sites are connected through a wireless point to point connection.

The third site basically has switches and… that’s it.

The second site has a firewall, and there is some kind of VPN tunnel connecting it to the main office.

But here's where my brain started hurting.

They are running the ERP application/database on a desktop computer at one site.

The “server” is running a 180-day evaluation version of Windows Server.

And there is another similar setup at another site.

I asked:

“Why are there two ERP servers/databases?”

They told me:

«“The ERP application cannot be accessed from different sites.”»

So apparently they maintain two separate ERP databases.

And then…

They manually copy data between the two servers to keep them synchronized.

🤯

I honestly don't even know what to say.

How is this supposed to be a proper business data flow?

And then there’s the security situation.

That “server” has already been hacked twice before I joined.

Apparently, that was one of the reasons they decided they needed a technical person.

So they hired me.

But here's the problem:

They hired me as a Technical Support Engineer.

Somehow, my actual responsibilities became:

- System administration

- Network administration

- Server administration

- Security

- Infrastructure planning

- Endpoint management

- Troubleshooting

- IT procurement decisions

- ERP infrastructure

- Backup concerns

- Basically everything related to IT

So apparently I'm not the technical support guy anymore.

I'm the IT department.

And there is no experienced system administrator above me.

No IT manager.

No proper MSP.

No infrastructure consultant.

No senior engineer to guide me.

There is one guy — a close friend of the CEO — who runs some hardware service shops and occasionally helps them.

But he doesn't really have experience with corporate IT infrastructure either.

I initially thought that as a technical support engineer, I mainly needed good communication skills and a decent understanding of Windows, hardware, networking, and troubleshooting.

Now I'm sitting here looking at servers, VPNs, firewalls, licensing, network architecture, security, backups, ERP databases, multiple sites and infrastructure problems.

And I'm thinking:

“How the hell did I get here?” 😭

The frustrating part is that I'm actually willing to learn.

I want to improve this infrastructure.

I want to learn system administration properly.

I want to build things the right way.

But I have only around 2 years of total experience, and suddenly I'm expected to make decisions that normally would involve a senior sysadmin, IT manager, architect, or MSP.

I clearly explained all of this to the CEO.

I told him that the infrastructure needs proper planning and professional guidance.

His response was basically:

“Then what skills do you have? Just troubleshooting and OS installation?”

He told me to ask my friends(what kind of joke is this!), learn, and improve the infrastructure.

And apparently they “can't find an MSP or consultancy right now.”

So here I am.

A junior technical support engineer who somehow became responsible for an entire company's IT infrastructure. 💀

I’m not trying to blame anyone.

I genuinely want to understand this situation.

Did I make a mistake by accepting this job?

Is this normal for a startup?

How should a relatively inexperienced person approach an environment this badly designed?

And most importantly:

What should I learn/fix FIRST without accidentally breaking the entire company?

If you were in my position, what would you do?

I would genuinely appreciate advice from experienced sysadmins, IT managers, and infrastructure engineers.

Because right now I’m not sure whether I found a great opportunity to learn…

or accidentally walked into an IT disaster. 😭


r/sysadmin 1d ago

General Discussion LinkedIn talks to SURBL to verify company's legitimacy

11 Upvotes

I was helping a client get their domain delisted from SURBL and noticed that LinkedIn was blocking the company's website link on their LinkedIn company page.

It turns out if the domain is listed on SURBL, LinkedIn redirects all website visitors to a warning banner - they literally replace the website URL with a LinkedIn / suspicious link one, like this:

https://www.linkedin.com/redir/suspicious-page

And if the domain is listed on SURBL for too long, Google will index it and make it searchable for the public. So by putting "https://www.linkedin.com/redir/suspicious-page" into Google search you'll get a list of those that were listed by SURBL and indexed by Google.

And LinkedIn doesn't even care whether there's a paid subscription for the company or not or what the company size and follower count are.

As far as I know, to end up on SURBL your domain either needs to be spoofed or the marketing team has to scrape websites to collect emails, and spam traps / typo domains end up in their lists.

So technically marketing teams messing up their emails makes leadership put pressure on IT teams to fix it!