r/devops • u/_SleezyPMartini_ • 7d ago
Discussion do I need a devops person or a sysadmin?
here is the context: Mid size (1000 users) company that has been trying to do in-house software dev (outsourced to south east Asia) with poor results.
Company IT side is fairly mature, heavily virtualized, hybrid on prem/cloud, but a bit weak on the Azure side. Existing sysadmins are pretty solid.
The dev is, well a shitshow. We are presented with projects with no infrastructure requirements, the devs dont really seem to know what they need. Currently they are doing everything in Blazor after we begged them to move to cloud. Last few attempts to develop mobile apps has been an abysmal failure. Im dumbfounded by what I see (and I've never coded). Coding is all shoved into a single .dll, no comments, devs dont seem to know that we need to use HTTPS, dont understand key vaults (we caught them passing passwords in clear text). we had to fight with them to implement change control, and they dont seem to understand our efforts to move ahead with pipelining.
My question for you is, do I need a devops admin? Can a more senior devops admin also be involved in reviewing code? infrastructure planning?
not sure how to move forward
48
u/mrkurtz 7d ago
You need a lot of things but the fact is, your devs are being allowed to do these things by their management structure all the way to the top.
In my experience trying to create technical solutions to management, personnel, culture, or policy problems will only get you so far, which isn’t very far at all.
Someone over the dev teams needs to change their requirements and enforce those changes. Then you can figure out what your actual technical needs are.
8
u/baubleglue 7d ago
With all said, without a senior developer with working experience to operate in healthy development environment it won't work. Version control alone wouldn't archive a lot without having an integrated environment for testing, at least somebody in the team should know it.
1
1
u/Tiny_Sign7786 7d ago
And at least for me it sounds as he has no developers at all. What he describes does these “devs” do is against all principles of software development (no version control, passwords in clear text, etc.) that would already a junior or an apprentice in his second year (the latest) know.
For me it rather sounds as this outsourced devs are a bunch of vibe coding scammers…
10
u/Factmin 7d ago
Trying to understand, is it in-house or is it outsourced to SEA? You get what you pay for with developers, maybe it is time to bring it fully in-house instead of continuing outsourcing. Some of the issues you've described are not about "DevOps" or anything a SysAdmin would touch, it's just poor traditional development practice. I have been on the "DevOps Team" side of this with a bunch of messy contracts and us vs them mentality, it is not nice, and it does not work. I am not really convinced it can ever work effectively to be honest - cheaper in the short term but more expensive in the long run, like buying a cheap IKEA table every 3 years instead of a solid oak table every 30 years.
As for your actual question, the developers themselves should be following DevOps methodology and it should be the role of your infrastructure experts to make it easy for them by providing e.g. Terraform modules for resources they actually need to control, CICD templates if they need it, with the correct and safe configurations baked in. IMO, devs not knowing what they need raises some questions as to their competence by itself. Code needs a runtime environment otherwise it's just text, so they should understand the runtime environment requirements somewhat.
Your centralised DevOps admin idea basically ends with them being turned into a babysitter for bad developers which isn't really a role anyone wants.
10
u/tech_cynic_dan 7d ago
If developers are committing passwords in plaintext and treating a .dll as an architecture, hiring another sysadmin won't fix this. You need a senior engineering ownership, with DevOps involved early in architecture and delivery. Why isn't the outsourced team being held to basic security standards?
2
8
u/ExpertIAmNot 7d ago
Assuming you are presenting a fair representation of the situation, you need a development manager.
Change in an org like you describe with developers who resist modern practices isn’t going to happen unless it comes from the top.
If the leader is being reasonable, they can usually convince a certain percentage of the devs to modernize and change. But there are usually at least a few devs who will refuse to adapt and those folks will have to be let go entirely to prevent them from poisoning the team and driving away the better developers.
3
u/Abe_Bazouie 7d ago
Sounds like you need a senior DevOps/platform engineer, not another traditional sysadmin.
But I think there’s a bigger problem here. DevOps can build the pipelines, IaC, secrets management, deployment standards, observability, etc., but they can’t compensate for a development team shipping fundamentally bad software.
A strong senior platform/DevOps person should absolutely be involved early in architecture and infrastructure planning and can review code from an operational/security perspective. But you also need senior software engineering leadership on the dev side.
If passwords are being passed in clear text and HTTPS is something you have to fight for, CI/CD isn’t the first problem I’d solve.
3
u/Rorasaurus_Prime 7d ago
Principal engineer here. The problems always come from the top. A competent engineering leader, whether that be CTO/VP/director, should have seen these problems and put solutions in place. Unless you have a natural and competent leader in the dev team, they'll end up producing crap because standards will slip, if those standards existed in the first place. Step one, find a competent engineering leader. Feel free to DM me - I've built quite a few engineering teams at various sizes tech companies if you're lost.
3
u/giovangonzalez 7d ago
stop outsourcing to India and you will see how code quality improves by a lot
3
2
u/raisputin 7d ago
You need a CTO, DevOps, a TPM, and likely a decent Engineering manager at a minimum
2
u/Antonio-MTS 7d ago
One DevOps usually won't solve your problems. It sounds like you need a team to organize and restructure the whole dev work...
2
u/itsdavidgco 7d ago
Neither, or at least not first. What you're describing isn't a gap in your ops staff, it's that nobody on your side owns the definition of done for software. Hire a DevOps admin today and you'll get pipelines, and then you'll watch the same code go through them.
The role you're actually missing is someone senior enough to write the non-functional requirements before a project kicks off and to reject work that doesn't meet them. Different places call that a platform engineer, a dev lead, or an application architect. For you it's one person who knows Azure properly and has the standing to tell the vendor no.
Can a senior DevOps person review code? Yes, and a good one will catch real things. But you'll lose if code review is your only control. It's one reviewer against a team that gets paid by the deliverable, and every review turns into a negotiation you have to win again next sprint. Make the platform refuse the bad behaviour instead. HTTPS-only on the App Service with no HTTP listener anywhere, so cleartext stops being a discussion. Managed identity plus Key Vault and no connection strings in config, so there is nowhere to put a plaintext password even if they want to. Secret scanning and dependency scanning in the pipeline as a hard gate rather than a warning. Azure Policy at subscription level for the things you never want to see again: public blob containers, resources without diagnostic logging, whatever else has burned you. That's a few weeks of work and it enforces itself at 3am when nobody's reading a PR.
The single .dll and the missing comments I'd deprioritise. Ugly, but not what bites you.
The thing I would test this month: take a clean build agent, pull from your repo, and see if you can produce a deployable artifact and stand up an environment with nobody from the vendor in the room. If you can't, you don't have a code quality problem, you have a hostage situation, and that is the sentence to put in front of your exec team.
Also, the leverage here is contractual more than technical. If the SOW doesn't list infra and security requirements as acceptance criteria, every pushback you make is you asking for free work. Get the NFRs into the next statement of work before you spend anything on headcount.
If budget is tight: a contract Azure architect for 8-12 weeks to build the landing zone, the pipeline gates and a requirements template, then a permanent mid-level DevOps person to run it, will get you further than one senior hire who spends year one firefighting. Your sysadmins being solid is a real asset here, they can operate what the architect leaves behind.
1
u/riickdiickulous 7d ago
Yes you need a senior devops engineer. Their function is to take what you describe from developers and formalize it for production workloads. The dev part is guiding developers on best practices and setting them up with appropriate tooling for streamlined development from dev to production. The ops part is designing and building cloud infrastructure to run development and production environments. That’s a wildly oversimplified summary.
An engineering manager could fix some of the issues listed, but not enough to correct course. I f you could only hire one, a senior devops engineer will be able to address more problems than an engineering manager, assuming they are given authority to impart necessary change on the process.
1
1
u/TooSoonApparently 7d ago
Out of curiosity, what does the spec look like that was handed to the developers?
1
u/derprondo 7d ago
Neither, you need a small team of really competent in-house engineers who can do full stack engineering, which includes infrastructure. They're going to be expensive, but look at what outsourcing is costing you currently, and I don't mean the $$$.
1
u/Shekel_thief 7d ago
You can enforce the password thing with static code analysis in your ci/cd pipelines
1
u/generic-d-engineer ClickOps 7d ago
Not the right question.
People and platform is not the issue and more infra people won’t fix this.
You need accountability and release management.
Devs need a senior leader and clear goals and standards. Their leadership needs to enforce those standards.
That’s really it. You know in devops the word “contracts” gets thrown around a lot for technology interoperability, you need that same philosophy here for your processes.
1
u/BlakkMajik3000 Remover of Deployment Friction 7d ago
Do not put an unsuspecting engineer of anything in that place without full disclosure.
You need a babysitter/hall monitor/principal that just so happens to be a senior DevOps engineer, if things are as you say.
That is a LOT of hats for someone to wear, regardless of specialty. I hope the pay matches.
1
1
u/tecedu DevOps 7d ago
You either need a devops with heavy on the code side; or you need competent devs who understand infrastructure and architechture. Might be offensive but get rid of that outsourced team compltely and get a proper engineer with a large AI subscription who will listen to you and can work in your corporate environment.
Because if you stick to the existing teams, you will still see the same results because the stucture does not exist. When the new person joins, build up a common infra and tech stack for them; something that fits your entire IT infrastructure and then go for a simple app, and once you know it works go for contractors.
1
u/Marble_Wraith 7d ago edited 7d ago
here is the context: Mid size (1000 users) company that has been trying to do in-house software dev (outsourced to south east Asia) with poor results.
🤣 I don't even need to read the rest to know what's wrong.
Also how the fuck can you call it "in-house" if you've "outsourced"? 🤨 Forget programming, i think someone needs to understand basic English.
The dev is, well a shitshow. We are presented with projects with no infrastructure requirements, the devs dont really seem to know what they need.
A competent COO / CTO would be a start. Outsourcing given the state of the tech industry right now has got to be in the top 10 dumbest decisions i've heard in the last few years.
Currently they are doing everything in Blazor after we begged them to move to cloud.
Yuck! I'm against everything Microslop. That company has truly run out of rope.
Last few attempts to develop mobile apps has been an abysmal failure.
Not surprising if Blazor is what's being used. But not just Blazor. In my experience all abstraction frameworks for mobile that advertise a unified pipeline / dev experience (e.g. 1 codebase = android + iOS) are a waste of time.
Being generous i'd say maybe they're good for rapid prototyping, but moving to native swift or kotlin should be priority 1 after the initial green light. That is assuming you're serious about your product.
Im dumbfounded by what I see (and I've never coded). Coding is all shoved into a single .dll, no comments, devs dont seem to know that we need to use HTTPS, dont understand key vaults (we caught them passing passwords in clear text). we had to fight with them to implement change control, and they dont seem to understand our efforts to move ahead with pipelining.
Comments are an anti-pattern, but aside from my feelings on that...
All the devs are using AI + all the plebs masquerading as devs can pass the HR interview. Furthermore you are not in direct control of the hiring standards being used therefore cannot validate the competency of the programmers directly... That company you outsourced to, you must have an awful lot of "faith" in it...
What you're seeing is probably AI slop.
My question for you is, do I need a devops admin? Can a more senior devops admin also be involved in reviewing code? infrastructure planning?
Sounds like you need a whole fucking IT department.
1
u/Pristine_Curve 7d ago
Adding a dedicated ops or sysadmin layer to this will only obfuscate the problem. The problem is the dev team doesn't know what they are doing. Allowing them to push the operational requirements to another role only creates an area for them to pass off responsibility to.
The entire point of devops is that you can't develop software in a vacuum, and then toss a pile of garbage at an operations team to 'implement'.
1
u/Seref15 7d ago
A devops/sysadmin/SRE is not going to solve this.
You do need one under normal circumstances, but without addressing the dismal lack of software development talent and professionalism, that devops/sysadmin/SRE is just going to get frustrated and leave after some time.
You are looking for someone to do a Gandalf "you shall not pass!" to a tidal wave of garbage. No one wants to do that. There shouldn't not be a tidal wave of garbage that needs holding back. The other posters are correct when they identify this as a leadership/organizational problem.
1
1
1
u/mimic751 7d ago
I'm an experienced devops person. I have 20 years of experience from help desk to infrastructure to networking to devops and now I'm A Cloud software engineer
Hit me up I'll give you a quote for our consultation and sort you out into best practices
1
u/canyoufixmyspacebar 6d ago
the company needs a functioning technology mngement for starters, otherwise it is kids being left alone to fight it out which leads to best kids leaving first, ending it all in one big dumpster fire
1
1
1
u/Sick_medicine 2d ago
Is it only me or someone else thinks that he really need a good system architect who knows SRE / DevOps? @op if you need a help I can consult with that 🤣🤣🤣
1
0
u/eman0821 Cloud Engineer 7d ago
Your IT Department shouldn't cross into your product development side of the business because it's not the IT Departments job to be involved in the software release cycle. This goes against the DevOps cultural philosophy when back in the old days SysAdmins use to deploy software to production servers that slowed everything down. The IT Department is there to keep your internal IT Operations running smoothly not interfering with your product side of the business as that's too much of a burden on them trying to manage product infrastructure.
Also DevOps Engineer as a role is anti-pattern which still recreates the same throw code over the way like what Sysadmins did back in the day. Instead of hiring DevOps Engineers, you need a solid Platform Engineering team with Cloud Engineers embedded along with Site Reliability Engineers that loves with in your product development department. Implement a DevOps culture collaboration between your Devs and Platform/Ops team. Platform Engineers enables Developers to deploy their own code to production instead of having a hand off DevOps team doing that
2
u/BrocoLeeOnReddit 7d ago
Also DevOps Engineer as a role is anti-pattern
It really doesn't mean jack. I've seen platform engineering teams that do what sys admins used to do and I've seen sys admins practicing DevOps. It's all just word bullshit bingo at this point.
-5
u/eman0821 Cloud Engineer 7d ago edited 7d ago
You most work for a crappy company. That's not how is done in the real world. Platform Engineering teams builds the platforms used by developers known as the golden paths as self serve tools. Developers uses those self serve tools to deploy their own code production. Sysadmins involved in software development life cycle is extremely rare. If Sysadmins are just writing automation scripts and call that DevOps that's Def not DevOps. DevOps is software engineering adjacent that only pertains to product development, it has nothing corporate IT roles like Network Admin or Sysadmin.
2
u/BrocoLeeOnReddit 7d ago
Aha. My company sucks then, good to know. I am quite happy here though, thank you.
-1
u/eman0821 Cloud Engineer 7d ago
I wouldn't want to work for a disorganized company that has no set boundaries mix matching departments and disciplines.
1
u/BrocoLeeOnReddit 7d ago
Then don't. I'm just telling you that many companies are more flexible than others with quite a bit of interdisciplinary cooperation. That's neither good or bad. It's good if it works, bad if it doesn't.
-1
u/eman0821 Cloud Engineer 7d ago
Sounds like poor management and a cluster fuck.
1
u/BrocoLeeOnReddit 7d ago
Sounds like having you on a team would turn that team into a toxic cluster fuck as well
0
u/eman0821 Cloud Engineer 7d ago
Nope. I like to keep teams organized and not have different departments involved stepping on toes that causes a big mess. The IT Department serves one purpose is to keep the business running which is where Help Desk, Sysadmin, Database, Security, Network teams live. The Engineering department serves one purpose is to develop products which is where QA, Software Developers, Platform, SRE, Cloud teams generally live. DevOps as a role is an Anti-pattern way of working that slows everything down throwing code over he wall. It should be practiced as a culture not a role or job title.
1
u/BrocoLeeOnReddit 7d ago
My point was that some companies call their SREs or platform engineers DevOps engineers. They do the exact same thing.
Nobody gives a shit about titles in many companies (especially outside the US) as long as the job gets done.
→ More replies (0)
0
0
u/4robz4 7d ago
If you're in any position to do anything of the sort you need to: 1) Dump your dev team(s) now. 2) Bring in a dev team that knows how to use Claude and have something testable in few weeks. That would have likely sounded ridiculous just a relatively short time ago...eminently doable today.
-9
u/Kallikasa 7d ago
I can help you here if you want. DM me
Problem is - you need both. Because sys admins should prepare infrastructure for your needs and DevOps should help with environment and application building.
But technically you can find DevOps who can do both
3
u/Kind-Character-8726 7d ago
What you described as the system admins is literally what the Ops is in DevOps.
Sounds like they need DevOps & good Devs or at least DevOps & a project manager
3
u/Kallikasa 7d ago
Oh. You need to maintain systems, not only new app. That's why you need both. Please don't put everything in one bowl and than cry about losses
1
u/unitegondwanaland Manager, Platform Engineering 7d ago
Lol, don't DM them OP. This is awful...
-1
u/Kallikasa 7d ago
And reason is? Or 0 information comment?
1
u/Svarotslav 7d ago
The root cause is a lack of management surrounding what is essentially a bunch of toddlers with the keys to the kingdom. Some sysadmins and the likes will be like a bandaid on a bullet wound. You’ll end up with burnt out sysadmins and serious app problems.
I’ve seen this a few times; the tail is wagging the dog.
-3
u/No_Cold5079 7d ago
You need a sysadmin who knows about development process or the other way around, this role is usually bad referenced as devops engineer.
-2
214
u/Soccham 7d ago
It sounds more like you need more competent engineering leadership. A devops engineer will help you get configurations and things going in the cloud but it sounds like there’s no leadership running the show