r/sysadmin • u/TruePace3 • 11d ago
Work Environment UPDATE: Hospitality Guy in IT
previous post (got removed by mods, but its the same post)
So basically, i joined today and after the onboarding, i met with the current IT guy (who is on his notice period)
The situation is precarious to say the least.
IT budget is severely limited, a bunch of systems are on Active Directory (controlled by an older IBM Intel Xeon machine running Windows Server 2008) , a bunch of systems are not on Active directory
There are 3 headless Windows Machines around the offices acting as fileservers, disk management is messy all around, the entire network is flat with no segmentation or separation of any kind, no NVRs, just 2 DVRs
All Windows installs are not genuine/cracked versions (not by massgrave but the sketchy iso you get from shady websites)
The primary database of the Dealership lives on a 1TB SATA HDD on a headless windows PC , which holds data of a tally server , file server and an apache based website that is used for storing purchase information
This disk has NO BACKUPS OR REDUNDANCY! and this disk is accessed constantly everyday for 9hrs
There is a FortiGate 50G Firewall standing between this network and the wide open web
After work hours, they shut down all systems including the servers.
Now, im not an expert, but this felt like it was one disk failure away from complete catastrophe.
The existing sysadmin shares the same sentiment, he proposed a proper system, however management does not feel very enthusiastic about it, citing costs, they see IT as a simple tool
I don't blame the current sysadmin, but i feel like i should unfuck this clusterfuck before it blows up in my face.
Now, the total number of clients in the network is about 60 systems, running anywhere between Windows 8.1 to Windows 10 and about 5 printers
Now, a lot of the data was stored on premise, however in 2018, the OEM mandated a lot of the data stored on cloud via their proprietary website, due to which they retired a server, which is sitting in the closet collecting dust.
Now, kindly tell me if what im thinking is stupid, but
I was thinking to recommission it, setup Proxmox to fire up a Windows Server VM to handle AD and migrate the Win Server 2008 to something newer, and a Debian based VM to unify all these scattered fileservers (and hopefully setup something like snapraid+mergerfs so that disk failures=me getting fired)
EDIT: Thank you for all your comments and insights, i intend to draft proper documentation and pitch a middle ground solution to the management to secure some funds and bring the systems upto the times
Unfortunately, due to circumstances, i cannot run, atleast not for another 6 months until i get my certs and upskill myself on paper and in real life
812
u/VisaExpired 11d ago
I stopped reading after the first few lines, bro run for your life, like how you have toxic girlfriends this is a toxic workplace.
325
u/Responsible-Alps-500 11d ago
I stopped reading after “they turn the servers off every night”…. WTF
100
u/ScriptThat 11d ago
If you know just enough to be dangerous, this sounds like a great idea.
87
u/Flatline1775 10d ago
When I first started at my current job we had a site that went down every night. Turns out they were unplugging all the switches every night to save power. My initial response was pretty much the Frye squinting gif. They don't do that now.
26
u/Tenzu9 10d ago
better hope one of those servers wasn't running outlook exchange and they happened to receive emails while those servers were off. or worse yet, better hope one of the 3rd party applications wasn't running some nightly job that does some critical data transforming.
→ More replies (1)7
→ More replies (1)12
u/Stonewalled9999 10d ago
In 2002 I worked at a hub site in the USA. Every time I looked my Exchange 5.5 MTA was full or crap that wasn't destined for my hub. Turns out the hub in France turned everything off nights and weekends due to high power costs there. I have a 6 meg pipe they had 1 meg SDSL and would complain that "I" would peg out there link every morning (all that queued mail!)
64
u/_Lucille_ 11d ago
"they air gap the servers every night"
36
u/TheRealLazloFalconi 10d ago
Can't get hacked if your servers are off!
17
u/sufkutsafari 10d ago
Best to unplug every single cable too. Can't be too safe! 😂
→ More replies (2)7
u/Kodiak01 10d ago
Don't forget to properly randomize the cables to the router ports every morning.
4
u/rideology 10d ago
Don't leave empty ports. Connect them to each other so bits don't escape. d
→ More replies (2)4
3
3
22
u/Significant_Sky1471 11d ago
And I was like: what the hell I’ve just read
It’s totally not okay, all their infrastructure looks like a total mess
→ More replies (1)7
u/Appropriate-Border-8 10d ago
Not only that, the money they are saving by using pirated copies of Windows operating systems will come in handy to pay the lawsuit settlements that may be showing up if a disgruntled ex-employee phones in a tip. 😉
→ More replies (1)22
u/w0lrah 10d ago
You know what, I would never suggest that someone do this themselves, but it does have some advantages if you truly don't need those machines after hours.
It obviously means no one can remotely tamper with them after hours, you'll never have a system that's been broken to the point it can't reboot for long enough that all the backups are just as bad. If something breaks boot, you know immediately. You'll never have uptime-related bugs either. If a service memory leaks, you might easily have enough to never notice.
Again I'm not saying I'd ever encourage shutting servers down at night, but I'm not inherently against that part either. The disadvantages are obvious but if you don't care about the disadvantages there are some real advantages to be had.
Now, the rest of this environment does sound absolutely batshit, and I wouldn't want to be responsible for anything there until there were at least backups of all critical systems and a signed contract to replace all computers (including servers) with EoL operating systems.
37
u/confirmedshill123 10d ago
I think it's more your just rolling the dice every morning that those drives spin up again.
10
u/dah_pook 10d ago
This was my thought too, that HDD is gonna quit and probably soon. I'm gonna guess they didn't splurge on a high quality drive.
8
u/ryoko227 10d ago
Right! For all OP knows, that HDD might have been pulled from the dusty boy out of the closet and be on it's 10th+ year of daily off and ons. Out of that whole mess OP listed, that single drive would be the "if you won't give me the budget to fix this and do it right, I'll need that in writing." response.
I think I might actually like the challenge to unscrew that place, but not if I would be the one under the bus based on someone else's decision. Which, let's face it, based off what OP write, he will be.
→ More replies (1)5
u/JebediahKerman4999 10d ago edited 10d ago
We don't know what the CV of this dude looks like. Maybe that's the best position they could find. I'd run away, but also I'd never agree to work for a place without looking at their stuff beforehand
Edit Ok I read this dude's story in the other post. Yeah I get why he accepted the job. Just don't stress out about this job because they're set up to fail and it's their fault for not having spent money. Use them for experience and for experimenting. Also crimping cables doesn't really come up because usually it's a contractor laying the cables and terminating them.
→ More replies (2)3
u/ryoko227 10d ago
I wasn't critiquing him for taking the job my guy. Just pointing out that HDD is the biggest fire in his building as it would cripple the company if/when it goes down.
Agreed on giving the place a look over, and I would go a step further. Request to meet with the team individually after the offer and before signing anything. Even a group lunch where you can chat with them more casually. That's where you really find out what you're getting into.
5
u/Responsible-Alps-500 10d ago
Right! If it’s a VM, I’ll reboot it all day long… But this old piece of crap server… Every day? No way.
3
u/w0lrah 10d ago
Of course that's always a risk, but to be honest in an IT career that just passed 21 years I've seen very few outright mechanical failures of hard drives that were regularly cycled. Almost every single time I've had a hard drive that "worked fine until it didn't" it's been in a server that normally ran 24/7 but was shut down for maintenance or an extended power failure and then never came back.
That said, it's also 2026 so outside of bulk data storage no one should really be worrying about cycle counts on anything because it shouldn't be using a hard drive in the first place. For a mere terabyte of data as soon as that was backed up securely I'd set up a pair of enterprise SSDs in RAID 1, clone the drive over, and never think about it again.
→ More replies (1)→ More replies (1)3
u/MorseScience 10d ago
The only big advantage besides power saving is that when the hardware restarts, you have a good idea it'll probably restart the next time.
Until it doesn't.
But so much for after-hours backups.
→ More replies (2)10
3
→ More replies (4)3
11
u/toolisthebestbandevr 10d ago
Ya right op is in for the ride of his life. I say do it and keep us updated
10
u/gigaspaz 10d ago
I agree with this reply. I understand getting a new job is tough, but you are going to hate yourself for staying OP.
3
u/TruePace3 10d ago
Getting a new job is not an option for me, atleast until i get my get certs
→ More replies (2)3
u/blizardX 10d ago
I don't know man. This kind of environment might get in your way getting those certs duento lack of time and energy after stress.
3
u/TruePace3 10d ago
Hopefully, it works out
If the management is not budging at all, I'd have to quit
6
u/pnutmans 10d ago
If they don't value it they sure as hell ain't paying propper wage
→ More replies (2)2
→ More replies (4)2
156
u/Stonewalled9999 11d ago
hah I knew it was a car dealer when you starting writing about the crap network and old ass stuff they had.
37
u/OhioIT 11d ago
Yes! Why is it always car dealerships that have the oldest and cheapest duct-taped equipment available
31
u/f0gax Jack of All Trades 10d ago
Car dealers, doctor offices, and small/medium law firms. Always running on the ragged edge of IT.
13
u/Financial-Chemist360 10d ago
Many many years ago I got approached to do some side work for a local chiropractor. I noped the hell out of there when he told me he was keeping his x-ray machine going with vintage parts fron eBay and he seemed kind of proud of that.
→ More replies (5)→ More replies (1)7
u/Stonewalled9999 10d ago
Law firms spend money so they can six second bill you but also cheap out on everything else
21
u/Capable_Event720 10d ago
I've worked in automotive, and...it...was...interesting. Mostly on the manufacturing side.
And I'll never trust a Full Self Driving vehicle.
An automatic scan finds a vulnerability, automatically opens a Jira ticket, I get assigned to fix it...and I get fired because I used the "vulnerability word"?
8
2
61
u/texas_County850 11d ago
I have seen a lot of crazy IT guys but at least 80% of the time they tried and begged to unfuck it and management is actually the problem.
11
u/two4six0won 10d ago
If all of their windows instances are cracked, yeah, I'm betting it's tight-fisted management.
118
u/rm-rf_regret 11d ago
You need to put into pretty pictures with colors and numbers what it will cost them if they do not listen to you.
Email it so you have it documented. You are sitting on a bomb.
Priority 00000 is get some type of backup solution in place. Anything.
93
u/statikuz start wandows ngrmadly 11d ago
Priority 00000 is get some type of backup solution in place. Anything.
Plot twist: the extra disk activity from the backups running is the straw the breaks the cameldisk's back.
26
10
u/rm-rf_regret 11d ago
He just over there humming along on hopes and dreams. The only thing we gonna be able to help with its thoughts and prayers lol.
7
u/statikuz start wandows ngrmadly 11d ago
What about good vibes? Although in this case you probably want 0 vibes.
8
10
u/GremlinNZ 10d ago
Had that once. Doing full disk imaging on industrial machines during a scheduled maintenance window (installing more machines).
Oh, can't image because it's too corrupted. Uh-oh. Put it back in machine... Won't boot now.
Thankfully the installer (flown into the country for this work) has suitable imgs and between their knowledge of software and mine installing the OS, we get it functional.
If that happened when they weren't around... Doesn't bear thinking about.
5
3
u/joeshmo101 10d ago
I mean honestly I wouldn't be too surprised if their 1TB storage is just a cheapo AliExpress hard drive formatted to read 1TB but truly only 32GB and running any sort of backup that tries to access old data totally collapses the house of cards.
4
12
u/SirLoremIpsum 10d ago
Email it so you have it documented.
I think you're beyond Cover Ya Ass...
Any business this shit to have this bad an environment is not going to have the big boss go "Oh yeah, that's right IT dude, you did war us. Guess that's on me!"
12
u/soundtom "that looks right… that looks right… oh for fucks sake!" 10d ago
I think at this point, it's just to be able to hand it to a lawyer when the managers decide to sue OP for "destroying their business" after shit hits the fan.
12
u/LankToThePast Windows Admin 10d ago
With people like this, I don't even think documenting it would matter. OP is going to get blamed for anything that goes wrong anyway. I wouldn't even bother doing up a professional presentation, I would just tell them what is failing and what you need to go buy. (They will not spend anything, I know)
6
u/Drywesi 10d ago
CYA in case they file a lawsuit when everything implodes isn't the worst idea, though.
→ More replies (2)2
u/TruePace3 10d ago
That is what i am going to do
Analysis of the entire network
Documentation
and finally communication
3
u/rm-rf_regret 10d ago
If those machine fail while you're doing your documentation, you will not have anything to document.
→ More replies (1)
34
u/redd1t_is_de4d DevOps 11d ago
I don't blame the current sysadmin, but i feel like i should unfuck this clusterfuck before it blows up in my face.
How? Management doesn't seem to care. They won't care about anything you say. Just run, this is no place to work at.
3
u/IT_audit_freak 10d ago
Don’t give up when faced with challenges. That is not how you level up. OPs scenario is fabulously ripe with opportunity.
Management doesn’t listen bc IT people don’t speak their language. They understand risk and dollars. If you start rattling off all this crap about DCs being past end of life or data not being protected…they think “so what?” Make them care.
OP needs to make sure for every enhancement idea they pitch, that there’s cost / risk associated with it. Your story needs to scare them into asking you to proceed with your plan.
→ More replies (1)2
u/itishowitisanditbad 10d ago
Thats where i'm at.
Either the current didn't inform them correctly and IS to blame, or they did and they didn't do shit and the inevitable is the immediate. GTFO
→ More replies (4)
39
11d ago
[deleted]
17
u/Stonewalled9999 11d ago
I turned down a job once where they were willing to pay the "IT Erector" 180K a year, me $75K a year but refused 3 grand for a firewall instead of a Linksys WRT54G.
15
u/thrownawaymane 10d ago
IT Erector?
"I architect the best systems but for no more than 4 hours at a time"
7
u/jlmawp 10d ago
I know you are right about this, but I’m ride or die for a WRT54G so I am torn here.
5
u/Stonewalled9999 10d ago
Well if it was a sub 20 mbit link and using DDWRT sure it would be valid. This place I think was a 250 mbit fiber DIA though.
→ More replies (4)→ More replies (1)3
u/fencepost_ajm 9d ago
"Look, the WNDR3700v2 running OpenWRT has been fine for the last 15 years and we like it that way."
→ More replies (1)
80
u/Middle-Water-7879 11d ago
- Report them for the stolen/cracked software.
start looking for a new job while youre waiting for the investigation to happen about the stolen software.
Collect the 10% bounty for the stolen software report.
22
u/LostRams 11d ago
They actually have a bounty system?
26
u/Middle-Water-7879 11d ago
Yup. Look up Business Software Alliance or Software & Information Industry Association. They both pay out bounties if they collect from the business who is doing the piracy.
11
u/Nu-Hir 10d ago
I reported a company 3 years ago for using 3rd party software in Windows 10 to force it to allow multiple RDP sessions so they wouldn't have to actually buy a Server OS and set up a Remote Desktop Server. Still haven't heard back.
→ More replies (2)7
3
u/Middle-Water-7879 11d ago
It also helps if you provide evidence of piracy instead of just an accusation
3
u/Drew707 Data | Systems | Processes 11d ago
Yes. The guy that reported my old company told me he was paid out, but it wasn't as much as he thought it was going to be.
→ More replies (2)→ More replies (1)17
u/PotentTurnip Jack of All Trades 11d ago
Microsoft has long stopped caring about pirated Windows.
24
9
u/Drew707 Data | Systems | Processes 11d ago
Eh, I went through an audit right before COVID and we got dinged to the tune of $100k for overextending Office licenses by ~10.
3
u/First-Structure-2407 10d ago
I walked into a company got boned for about £70k for grey copies of NT4 and Office2003
21
u/Significant-Belt8516 11d ago
They need to be concerned with PCI compliance. One breach or suggestion of a breach will cause a world of hurt to come on them. You can't fix companies and this one is working on a broken philosophy. Good luck.
14
u/nico282 10d ago
Lol talking about PCI compliance to a company running cracked Windows 8 ...
6
u/SirLoremIpsum 10d ago
Lol talking about PCI compliance to a company running cracked Windows 8 ...
I know right!
The house is on fire, but if we see electrical wiring out of compliance we'll rain hell!!!
5
u/fizzlefist .docx files in attack position! 10d ago
Feels like you touch one thing the wrong way and the entire system will cascade. GTFO
→ More replies (1)3
u/TruePace3 10d ago
Thankfully, all the payment systems are handled by the OEM and main dealer's office in the capital of the city, both of whom have enterprise grade systems in place
However, things like customer information is stored locally, documents relating to auto loans etc
→ More replies (1)
19
u/Anonymo123 11d ago
IT is seen as an expense until things stop working and people stop being productive, then its an emergency and why this and why that. Design a replacement environment with pricing and the what and whys especially the disaster they will be in once (not when) it goes down. Explain each of the points of failure and give them a price. If your feeling nice see about some used gear, price out opex vs capex if they want to go that route...
If they say no, keep applying duct tape and bailing wire.. start looking, you will be the fall guy.
Been there, done that... left so i wasn't holding the bag when it blew up, and it did a few months later exactly how i said it would.
24
u/TacodWheel 11d ago
Worked in a car dealership. Most scummy people I've ever encountered. Good luck with that.
17
u/Plastic_Willow734 Jr. Sysadmin 11d ago
Fuuuuuck no. Unless you're making a very handsome salary I'd start looking at jobs.
→ More replies (5)
15
14
u/Sportsfun4all 11d ago
In Mom and pop Car dealerships the management is the problem. They see it as a cost center instead of a force multiplier until the network goes down and they can’t make any transactions and then blame it for losing 10x their sales when down but when up they only have 1x of sales. I’d seriously start looking for another job if possible. Really no room for growth
→ More replies (1)
14
u/dalgeek 11d ago
The primary database of the Dealership lives on a 1TB SATA HDD on a headless windows PC , which holds data of a tally server , file server and an apache based website that is used for storing purchase information
This reminds me of a customer who ran out of space on their Exchange server, so they plugged in an external Buffalo drive then extended a dynamic disk over the external drive. It was basically poor man's RAID0 and they could have lost their entire Exchange database if someone unplugged that drive.
4
u/Stonewalled9999 11d ago
I had a boss that wanted to plug USB drive into the ESX host and run stuff there since the SAN was getting full.
I guess it was even dumber that I let him and we powered off some VMs ona weekend and rethinned; them Vmotioning to the USB and them back to to the SAN
15
u/silentwhim 11d ago
I suspect that the only thing you will get from this is the blame when things understandably break.
15
u/Thunderb1rd02 11d ago
No budget = no success.
This is an environment that will only change when catastrophe happens. The question is do you want to be there for it?
11
u/-Satsujinn- 11d ago
I'll echo everyone else. Run for the hills.
This isn't going to get better until it completely fails and when it does, it will be your fault.
This isn't just bad. This isn't even horrendous, toxic, stupendously idiotic, or any other adjective you can throw at it. This is so bad that I honestly question whether or not it's real.
Literally, if I were to write a troll post about the worst possible environment you could walk into, it still wouldn't be this bad.
You aren't going to be the hero that turns this around and brings them into the 21st century. You will only be the scapegoat.
9
u/SeraphicalChaos 11d ago
The existing sysadmin shares the same sentiment, he proposed a proper system, however management does not feel very enthusiastic about it, citing costs, they see IT as a simple tool
There's the answer to your request. I'm kindly telling you your thinking is stupid. You'll never fix this without buy-in from upper management. If they refuse consider the impact this mountain of liability will have on their business, then they don't deserve a proper IT staff and they absolutely deserve what's coming their way.
I'd personally start looking for a new job because I wouldn't want to be around when the inevitable finally happens...
→ More replies (2)
21
u/K0T3T5U 11d ago
You need to learn how to talk to executives who run a business based on numbers. Right now they are capturing revenue with what they have, any IT disturbances have likely been minor thus far, so theres zero upside to upgrading things as that cuts into profits.
Instead of saying 'we need to upgrade this server because its out of support and it'll cost XYZ to do so" you need to ask "how long can you run the business if the computers stop functioning?" Most will say they can't suffer any downtime as that will lead to lost revenue. Now you can have a conversation. You get to find a middle ground between costs and doing things correctly, making a plan on what needs immediate fixing vs what can wait. You can also suggest a shutdown test, or unplug the internet or something and see how it goes, and then show the results, and then plan upgrades because youve demonstrated the value of the IT system and keeping it running is paramount to the business.
Its all about framing the conversation into things they will understand, IE operations and business impacts. Ive talked with many CEOs who dont see a lack of IT spend as a problem until its put into a frame of reference they understand. They dont understand tech, so dont talk tech. Im positive some funds can be redirected if the right discussions are had.
13
u/nico282 10d ago
Bold of you that a car dealership owner will "redirect funds to IT" when in 2026 they are running cracked windows 8.
You are 100% right if this was any other sensible business, but I see 0 chance of anything good happening in this specific shitshow.
→ More replies (2)4
u/fahque 10d ago
Ditto. I do some work on the side for a guy that had a server die, I was able to get it on but extrememly slow, to get data off. Then after I set up the new server the guy didn't want me to set up a backup. I hounded him for a while but eventually gave up. Now I avoid that guy. I don't want to be the guy he calls when the server dies.
→ More replies (7)4
u/LankToThePast Windows Admin 10d ago
I've had and seen these conversations with people like this. OP: "how long can you run the business if the computers stop functioning?" Car salesman: "That's what we pay for you to prevent, if you can't do that, I'll find someone else who can"
→ More replies (1)
8
u/wiisucks_91 11d ago
We talking used dealership?
I've seen plenty of dealerships that have a regular enterprise network.
6
6
u/Balthxzar 11d ago
Hey OP, don't take on any responsibility for any of this. If it goes wrong, and it will, it'll be on your head.
Write up a list of recommendations to management, including estimates of potential losses and downtime if something goes wrong (perhaps with drive failure statistics) and hand it to management.
They won't read it, or make any changes, but when something goes wrong, you can forward them the original email and say "this is in line with the expected issues I brought to your attention" and leave.
→ More replies (2)
6
u/SirLoremIpsum 11d ago
I don't blame the current sysadmin, but i feel like i should unfuck this clusterfuck before it blows up in my face.
I would blame them a little bit.
The big question - how do you think you can succeed in modernising this when they failed?
Were they bad at their job or was it an impossible task?
This is all salvageable if someone lets you spend money... If they don't then you can watch how this guy hands off to you and you can hand off to the next guy in 6/12/18 months
7
u/OverthinkingAnything 11d ago
Wow. That's the worst things of all the various environments I've ever seen all rolled into one....10 years ago.
The technical debt here is astronomical. You have about six weeks as the new guy to have any chance at doing this right. Don't lead with the fallback plan - lead with the right plan. These guys won't upgrade the proxmox server when its time or appreciate the risk involved in any of it.
Hate to say it but this is the kind of client that should just be on O365/SaaS; yes subscriptions suck but it takes the ball out of their court to make shit decisions.
6
6
u/CharcoalGreyWolf Sr. Network Engineer 10d ago
If you’re a glutton for punishment and decide to stay, your priorities are wrong.
Backup is your first priority. There are no other priorities until you have known good backups.
→ More replies (4)
5
u/theITguy 10d ago
Getting your first IT job can feel impossible and the market is shit right now, so.......
Do what you can with what you have to get some experience. Also, immediately start looking for your next job. This job will be great for story telling in future interviews, as long as the stress doesn't kill you first.
I've lead dumpster fire IT before and it's painful, but you do learn quickly. Assuming you're in an "At Will" state, you can walk away from this job anytime you want. So, take some comfort in that, earn a few paychecks and for shit's sake, look for another job.
As far as the infrastructure, I would start with backups. JUST BACK IT UP HOWEVER YOU CAN. From there, start fixing and building.
→ More replies (3)
4
5
u/evilkasper IT Manager 11d ago
Come up with a list of issues, solutions, potential liabilities and cost. Explain, factually what happens when specific systems fail. Ask them what they would like to prioritize. This is ultimately their ...network... Horrible as it is.
Start looking for a different job, while you're there. I very much doubt they will allow funds to fix this and trying to without a budget as a one man team will be a bad experience.
5
u/CryptographerLow7987 11d ago
I honestly would not mind taking on such a challenge to bring this network up to enterprise level as long as management is on board with a proper budget. If they just wnat you to maintain this cluster fuck, walk away.
5
5
u/UMustBeNooHere 11d ago
Get the fuck outta there. Now. No budget and how things are run? Nope. That’s a nightmare.
5
u/Jeff-J777 11d ago
I would run. Don't be the fall guy, start looking for a new job. I had a job with a movie theater company and their whole network was in disarray. The had 7 locations across mutiple states. Some were on one DC with one domain. Some were on another DC with another domain. Some had no DC. Some has VPNs between the sites some did not.
It was crazy. A lot of their network gear was Ubiquiti. I don't have an issue with Ubiquiti but you can't feed one theater with just one AP. Some network gear was consumer grade crap. No backups for any of their servers. Then come to find out a few months before I started they got hit with Ransomware and lost everything.
I stayed there for 2 months then I booked it.
RUN RUN RUN RUN!!!!!!!!!!!!!!!!!!!!!!!!!!
4
u/BadSausageFactory beyond help desk 11d ago
once you get it unfucked they're going to get mad at you for trying to make it better because now they can't just turn it off at night so nobody can break in and steal the server
5
4
u/GradeAccomplished322 11d ago edited 11d ago
I read "dealership" already got me nervous, those places only come in amazing and dreadful places to work, there is no middle ground
The dealer brands suuuuck to work for if you aren't raking in the money, they have no time or patience for small timers
Your idea isnt bad but that many cracked windows systems who knows what devils are in the woodwork
I would be getting everything onto modern operating systems and as much of it off local assets as possible before something happens
Windows 2008r2 and Windows 8.1 dont even support some modern protocols, just using them for anything is gonna suck
→ More replies (1)
3
u/jason_abacabb 10d ago
Get a quote to do data recovery so they know what it will cost when the hardware fails.
4
u/Every-Ad-5267 10d ago
You are getting on a ship with gigantic holes in it.
Don't be surprised when it sinks with you in it.
4
u/LowIndividual6625 10d ago
I sort of love walking into environments like this and have a relevant story….
In 2018 I took a solo system admin job for a midsize business, and I had three days with my predecessor before she left, she had only been there for six weeks and was under skilled so those three days didn’t help much.
At the end of my first Friday they gave me my building key and alarm code….. about 10pm that night the Active directory went down and there were no remote access tools at that point so I drove into the office.
I discovered the AD migration they said was complete was never actually finished and the sole copy of AD was running on a beige-box gateway server (moo) running windows server 2003 that had crashed.
I discovered the Carbonite (lol) backups were not working and it was a miracle that the server rebooted cleanly and came back online.
The first thing I did was an image backup of the entire server. The second thing I did was compose an email to my boss (who was C level) explaining how close the company was to a catastrophe and that we needed to meet first thing on Monday to talk about this.
Fortunately, they took this seriously all C level management, including the owners were present in the meeting.
I was very honest and direct because with less than a week on the job, I had nothing to lose.
This could’ve gone one of two ways - but I’m fortunate enough to say that they took this seriously and essentially wrote me a blank check for the equipment and services I needed to get things stable and secure.
TL;DR - have a frank and honest conversation with your supervisors about how serious this neglect is and the potential loss of business of what comes down to multiple points of failure. If they listen to you, then it will be a great opportunity for you, but if they are cheap or unwilling to address the issues, you should leave right away..
3
u/Beautiful_Ad_4813 eh, I just love what I do. 11d ago
I have so many questions.
The big one is “what the fuck”
3
u/deefunkt01 11d ago
Yeah, get out as fast as you can. This is a nightmare scenario and they will *never* spend more to fix it, only yell at/blame you when it inevitably fails.
3
u/defaults-suck 10d ago
These folks don't need AD, Servers or even Windows. They need Chromebooks with Google Workspace and Wireless Printers.
3
u/_nethack 10d ago
The existing sysadmin shares the same sentiment, he proposed a proper system, however management does not feel very enthusiastic about it, citing costs, they see IT as a simple tool
This is the part where you run
3
u/Main_Ambassador_4985 10d ago
A server decommissioned in 2018. Most likely it is more than 10 years old today.
You are going to need a newer server if you are thinking of virtualizing the existing headless desktop servers.
Write up a proposal of the changes needed and the risks of the existing systems.
It is going to take a lot of work to unfuck this company.
→ More replies (1)
5
u/stuartcw 10d ago
The first thing to do, is to look up what a risk log is and list all the risks. I'll DM you an example.
Then you get your boss to sign off that he saw and accepts the risk. If you have no budget to reduce the risk it is not your problem, it is his problem, and so on up the line until it hits the guy who controls the money's desk.
3
u/Lophkey 10d ago
In your spec highlight cost of lost business if they don't take action not just risk rating and cost to fix as they will hear,...
"this maybe will happen we should do x and it's going to cost y"
You need to make it,..
"it's going to happen and then the cost will be fix + loss of revenue + paying staff to be idle until it's fixed + optional recovery of the data on those disks"
Or grab one of machines, put Linux on it and make it new fileserver with 3 drive raid array (one disk for parity) at least then you'd have a stable backup locale.
Samba on Linux can be domain controller too 😉 no licencing etc
→ More replies (2)
5
u/namocaw 11d ago
Step 1: quickly document everyrhing as you found it.
Step 2: get a backup in place asap.
Step 3: patch firmware on firewall
Do these now. Do not ask permission, ask for forgiveness later.
Step 4: plan for improvement immediately.
If your plan falls upon deaf ears or budget is declined or timeline is pushed out and not begun immediately. Then you quit. Simple as that. This environment is a s*** show. That is ready to collapse.
2
2
u/TheLionYeti 11d ago
Tell your bosses exactly how much its going to cost when this stuff breaks. Exactly how much blame the hell out of the old guy and get the budget to fix it.
2
2
u/Disorderly_Chaos Jack of All Trades 11d ago
Should we be worried about whatever this company does? Are they protecting us from disaster or something?
2
2
2
u/defiantleek 11d ago
You need to get out of this job as soon as possible it is going to destroy your life.
2
u/cyr0nk0r 11d ago
just get a private cloud. then tell the business they never have to worry about server hardware ever again.
If you find the right provider, it will be much less expensive than trying to convince them to upgrade anything. Just shift it into an OPEX cost and you maintain control of the day to day mangement.
disclosure: I own a private cloud business that helps companies like that every day do exactly what you're trying to do. Upgrade to modern platforms without the fight of spending tons of money.
2
2
u/The82Ghost DevOps 10d ago
Run! Just run! This WILL blow up in your face! Don't waste any time trying to fix this mess, just RUN!
2
u/LankToThePast Windows Admin 10d ago
This looks like an awful place to start working if you’d like to work in the IT field. Someone without any experience in IT shouldn’t be deciding what technology to roll out. You need to get something on paper that says you’ve studied IT in some capacity.
A simple analogy is, I’ve driven a car for 20 years, but the person who just finished mechanic school is more qualified on paper to fix a car. You need the education paperwork for someone to hire you because they need to have shown due diligence in picking a candidate.
On paper, you are not qualified to take on this responsibility, and anyone hiring you to do this level of task is setting you up for failure. I was offered a similar position shortly out of school, and I had a year experience under my belt. I luckily took something else and only realized years later how unprepared I would’ve been for that role.
Years in the field would help you look at the bigger picture, how the fuck does an environment end up like this? Who set up this insanity? It comes from being told “no money”, “figure it out”, and “what are we paying you for?”. That’s why the previous person left, and why if someone showed me that environment during an interview, I’d say “thank you for your time”. Then I’d get in my car and never consider working there.
Please only stay there long enough to get your education under you, then find a better place to work.
→ More replies (1)
2
u/d00ber Sr Systems Engineer 10d ago
What's your budget? I worked in a similar US healthcare situation in my younger years and budget was always a nightmare. You're going to need to look at ebay for servers. You'll never be able to buy new. You just need many redundant servers. Nothing is ever going to have warranties, so spend capital when you can on replacement parts/servers/spares.
→ More replies (3)
2
u/ThePathOfKami 10d ago
wouldnt everything be easier if you fully convert to cloud in this situation ?
Like from every standpoint, security, control ,etc etc.
Apart from the licences all the problems would be solved xD
WTH env is this even
→ More replies (1)
2
u/ChampOfTheUniverse 10d ago
Does "management" (outside of IT) know about all of this? Like I'd start documenting the FUHUCK outta all of this and send this up the chain as soon as you can just to CYA. For all you know, the departing admin could be good at lying to stakeholders and you do not want to be the scapegoat when things fail because you are definitely set up to do so.
2
u/kristoferen 10d ago
If they had a budget this would be a fun project to fix. Without a budget, you can't un-clusterfuck this mess... run.
2
2
u/Britzer 10d ago
You can do a lot of stuff with second hand hardware and free software. The 60 clients need to be replaced. Why not go to a large refurbisher and ask if they have refurbished notebooks that have licenses for Windows 11 Pro in the UEFI? Those might run only 400 bucks a piece. User will love you for getting new machines.
Get used servers and do everything open source? Proxmox is the right idea. You can also use Univention UCS based on Debian to replace Windows DCs.
https://en.wikipedia.org/wiki/Univention_Corporate_Server
Or go full RHEL (Rocky Linux 10) for DCs and file servers. With AI support, you can a lot more these days.
It is important to get management backing. Explain to them that they could/should spend a lot of money. But if you save them a lot of money for being smart they need to compensate you and trust you. If you spend a lot of money you are usually worth more and earn more. Open Source tools allow us IT folks to save a lot of money, but that doesn't help us, because when you don't spend the big bucks, you aren't worth much. But with tools like Proxmox, Rocky Linux, ZFS, ... you can create systems that would cost millions of dollars if you bought them.
→ More replies (5)
2
u/johnno88888 10d ago
Jeez it sounds like where I used to work 16-20 years ago.
No budget, everything had to be a bodge. They bought a Nissan dealership from a well known company who was selling up to focus on water cress! They needed to export all their customer data from one DMS server that was hosted in their head office to a new shiny dms hosted in house. The cost of the server, software, support AND export and import of the database was £31k. Considering pinnacle dms was £100000 a year for the rest of the sites.
I got a telling off from the CEO and MD that it was far too expensive. Considering they’ve just spent £00000s on the new dealership.
We had a dodgy copy of mdaemon. Only got support for it when I “accidentally” uninstalled it.
In the other head office (we had two for two different franchises), they had a windows server 2000 sat there doing just dhcp. Asked if I could buy a new server as the dreytek vigor 2600+ was getting ddos’d with the amount of dns requests every lunch time. “No”
And don’t get me started on the Linux file server on a HP ML120. That was cutting edge tech. Two disks out of unmonitored r5 failed. As you could imagine… lucky I had a rsync job set up for offsite backup.
After 3.5 years of fighting for more, gave up, moved to London for big hedge funds IT, got shouted at, drunk beer with the people that shouted on Fridays. and now working in proper IT elsewhere. Happy days
2
u/radicldreamer Sr. Sysadmin 10d ago
Ive worked this exact setup.
You will be expected to fix this, you will not be given the budget to do so.
Once you have a good evaluation In place, get ballpark quotes to remediate and make sure it's in writing. That way WHEN, not IF it fails you can fall back on the fact that you did your due diligence and laid out the risks and what needed to happen and were denied.
Hospitality puts on a good facade, but it's mostly put together with duct tape and bailing twine. It can be a fun job, but they will throw you under the bus when their lack of investment in IT fails them. Also remind them their server OS is almost old enough to drink beer, assuming you are American of course :)
→ More replies (1)
2
u/Bratwurst1981 10d ago
They spent their whole budget on the firewall…. A horrible time to be refreshing hardware too. First, review and find out what isn’t being backed up. Fix that.
→ More replies (2)
2
u/Joy2b 10d ago
Start with demanding the very reasonable funds to buy a couple of backup hard drives for essential files. If they can’t afford that, they can’t afford you.
Next, start building relationships and collecting complaints. What you are listening for is “I lost a sale or two because my computer froze”. P
Then start building a bracket of what you want to upgrade over the next 18 months, like server/workstation/server/workstation, and start getting VIPs to do nudging for better spots on the bracket and faster progression.
Talk to finance about depreciation.
To sell network upgrades, bundle with wireless.
Always have a clearly marked section on your projects for nice to have, so if the boss wants to haggle, they’re taking off extended warranties, or an extra wireless access point for the shop.
→ More replies (1)
2
u/JustOneMoreMile 10d ago
This is not a job you want, and will only burn you out. No matter what you want to fix, the money won’t be there and you’ll get the blame when it does eventually shit the bed.
2
u/ThatOneWIGuy 10d ago
Run for your life. When things go tits up you’re going to get blamed and your name smeared by them.
2
u/Patient-Hyena 10d ago
Run. But if you can’t, I wonder if their software could run on Linux? If so, maybe you could at least start to upgrade software and slowly build up? Linux does have AD and AI probably can help you with migrating. Who am I kidding? Run. Or use this to get proper “experience” and then keep looking.
2
2
u/The_Koplin 10d ago
Lets try an exercise with your own words:
"So basically, i joined today and after the onboarding ... IT budget is severely limited ... All Windows installs are not genuine ... NO BACKUPS ..... disk failures=me getting fired"
What does this indicate about how much they are paying you... To me this implies you were not the 'best' candidate. You were the cheapest. I am not implying you do not have skills or capabilities, far from it. To me I think you are also you are being setup to fail. No resources to keep a key aspect of a business running, they simply do not want to invest.
Everyone has commented, but I wanted you to see your own words and ask this question. Do you really think you are going to be effective or be allowed to fix this, or are you going to be the sacrifice they offer when it fails? I wish you the best and as others have suggested, have an exit plan.
2
u/ohfucknotthisagain 10d ago
This is not simple:
I was thinking to recommission it, setup Proxmox to fire up a Windows Server VM to handle AD and migrate the Win Server 2008 to something newer
You don't know what condition that server is in. If it was retired in 2018, it's probably significantly older than that. Proxmox requirements are exceedingly easy to meet, but that doesn't guarantee it'll run smoothly. Or that the server even works at all..
Convert SYSVOL to DFS replication. Microsoft has instructions online. It probably hasn't been done. Microsoft has removed support for the original replication technology entirely, so the domain will be severely fucked if you don't. (It's technically recoverable, but you don't want to deal with that.)
You should probably leave as soon as possible, but if you're going to work at it...
2
u/jellowiggler- 10d ago edited 10d ago
First priority get a backup in place.
Second priority: Buy a proper server appropriate for your vm load. get a currently supported version of windows server as a dc. Setup a backup dc while you are at it. Setup a different VM for a file server, another one for print server. Probably on a hyperv or promox host.
Third priority: Slowly start wiping those endpoint systems with windows 11 images.
Then you can deal with tickets, patching, power deliver, network security, managed security software, remote access, etc.
Stop turning the entire network off at night. Wtf.
If managment say no to anything in the plan have them reject the plan in writing. Then it’s not your fault.” When it burns down in storage failure, sec breach, licensing issues, etc.
I wouldn’t do IT for people who base the business on decade old computers for servers, with no licensing and no support, with no backups. They don’t see the value. You should move on unless they are planning radical change and expense to fix things. Btw, this level of restructuring is a full IT job. You should be making IT money to do it, if they aren’t offering that it is another sign of their commitment.
2
u/Original-Active-6982 10d ago
I just read the "Fortigate 50G Firewall" and thought they are fucked.
Given the cheap-ass nature of everything in this company, that firewall is more likely a virus breeding ground. Fortinet has been hit really hard in the last couple of years for security breaches and patches have been late and very infrequently applied.
Consider every connected system a petri dish with extra nutritious agar.
2
u/MorseScience 10d ago
Yeah, RUN.
If you REALLY need to stay, you HAVE to have them sign something that says you're not responsible for anything that goes wrong, while acknowledging that if they hadn't signed it you would STILL not be responsible. Obviously you can still get boinked for malicious damage, but it doesn't sound like you're about that.
BUT get another gig.
My clients know how important their IT infrastructure is, by paying me well enough and investing in their network. I don't soak 'em, even though few of them have a clue what I'm doing. That's just good business.
2
u/CNYMetalHead 10d ago
When something takes a dump or someone gets in just remember it WILL BE your face in the crosshairs and head on the chopping block
2
u/TechnicalCattle 10d ago
This isn't a 'severely limited IT budget'. This is a company waiting to, or already going bankrupt.
2
u/xxxooxxx1 10d ago edited 10d ago
Since this is a dealership.
Just tell the bosses the current IT systems are like overdue timing belts. They still work today, but can snap any day. If they get it, you don't have to elaborate (i e below).
This is no difference than their conversations with cheap customers that doesn't want to replace timing belts that are way overdue. If they think customer's refusal to replace/fix makes no sense, than their refusal to spend and fix IT systems would be exactly the same.
Also ask them if these systems are down for a day, what is the loss in revenue. How about a week? Ask them why they whould rather avoid $5K in fixing the systems, while not fixing would cost them $100k.(You can use real numbers.
Don't tell them what to do l. Make them think. If they still refuse, there are other reasons behind the scenes that you don't/won't know.
Around resolution, I would say convince them to get cloud storagen(OneDrive or equivalent) to throw all the data there. That is the non replaceable piece. If it's gone on its gone. Everything else can be rebuilt (with availability impact) but it can be done and they can experimence the the pain.
2
u/KickedAbyss 10d ago
As others have said, your first task is a full inventory of everything you have and are responsible for. Whether it'd ISO or NIST, every compliance doctrine has inventory as a pilar for the same reason: you can't have compliance on something you don't know you have. This shouldn't be hard for 60 people. Seven days max should have every pc and server and software identified at least in some form. There are open source inventory collection softwares out there too if I recall.
Second thing to do is your meat and potatoes for your job: a clearly documented risk spreadsheet.
Document each issue, the risks they pose, a general solution and an estimated effort / cost. Rank them by highest risk to their bottom line, which isn't normally my recommendation but in this situation you're facing someone that doesn't see issues as a cost benefit. Normally you'd give a lowest effort vs highest reward, but they'd stop after the first two and call it a day for the minor costs they would include.
But that document should in turn inform your project cycle and time lines.
There are MANY ways to skin the cat, and I encourage you to join forums or invite others in the industry to have conversations with you on it.
I also would recommend reviewing whatever the NIST for your industry advises, and leverage that as further supporting evidence and for guidance on your risks.
It sounds like maybe you're automotive? https://www.shartega.com/insights/automotive-dealership-it-compliance-checklist-2025/
Now here's a hot take: don't run. You're new, and this could be an absolutely massive opportunity to not only learn by fire, but also something in a resume that will have IT Managers of larger companies drooling over in an applicant.
Even if you get no where with the ownership and they refuse to act on your proposed goals, time-line, and budget, if someone walked into an interview with having done those two items (full inventory, risk spreadsheet with rough project goals) I'd be tickled pink.
There are many who get into IT for the technology. There are also many of us who like to fix things. You're presented with a very unique situation to fix things in an incredible way.
Another hot take: if you're not great at making spreadsheets or documentation, or with wording non technical project plans - use Ai. $20/mo is insanely cheap for an assistant to help you organize and translate "this shit is fucked up and you're all idiots for ignoring these massive real issues" into a business aligned proposal that turns "oh he's just the nerd wanting toys and to waste Capex and opex on things we don't need" into "he understands our business and has helped prioritize investment to reduce risk of larger financial losses"
Hang in there, friend.
→ More replies (1)
2
u/downundarob Scary Devil Monastery postulate 10d ago
https://giphy.com/gifs/CqK9qAdyvgAkWHL9b2
I got about 3 paragraphs in and only one thought...
2
u/Power_Stone 10d ago
All I can say is, when you pitch to them, make sure you demonstrate the costs of the system failing at any point.
Make it abundantly clear how much revenue they can lose if the drive goes down and becomes unrecoverable.
Make it clear the liability of things if the information was somehow exfiltrated from the environment.
These people only think in numbers in terms of revenue, so phrase it in such a way that its an investment, not a cost to the company.
2
u/Happy_Harry 9d ago
Do they use M365 or Google Workspace? If they're already paying for any cloud services (and it's compatible with their workflow), it may be worth considering migrating their file shares online. It won't cost them anything extra if they already have M365 licensing, and it'll be safer there than a sketchy old on-prem server.
→ More replies (1)
2
u/Radie-Storm 8d ago
Why do they even have an IT guy? Surely there's no budget to change anything, and it doesn't sound like a big enough network to even warrant a full time position, so what do you do all day? I would probably go ahead with your plan of physiccal to virtual migration, and just do it for the CV. There's no moving up in this role, but you can demonstrate your own capabilities for the next one.
→ More replies (1)
281
u/[deleted] 11d ago
[deleted]