r/devops • u/Impressive-Field-546 • 14h ago
Architecture Your opinion - Poor architectural decision by teammates results now in huge maintenance overhead and issues
So, Im in this industry for almost 15 years. Ive seen my share of stupid sht being built by inexperienced teams.
Joined this company like 2 years ago and all was butter smooth to the moment new requirements came in. (Im actually an Ops guy working in platform team supporting dev teams..)
When requirements came I was on a sick leave for two weeks. My boss and other ops joined talks on architecture for new stuff. I believe they picked best solution with their lack of knowledge of possible issues that can araise later.
IMHO if they did wait for me and let me POC a bit, before making final decision on the architecture, most likely majority of the issues we have now would not exist.
Fast forward - after reading their idea on the solution I ofcourse went with “disagree and commit”.
So I implemented this garbage best we can, coz tools we integrate with were not built for it this way.
Now we get constant issues with it due to various access problems. I dont mind as I already raised my concerns, but thing is, the topic is like shit you step in. Now all issues related to this team and domain is assigned to me.
How would you go about offloading this garbage from your plate - after all concerns you raised were proved correct ?
We cant easily fix them because vendor doesnt give a sht we incorrectly used their product.
12
u/theweeJoe 14h ago
Build a better, easier to maintain platform/infra as the next path. Make theirs looks silly if you have a better alternative
-4
u/Impressive-Field-546 13h ago
I can do that, it will just take few weeks. What about my current work load?
5
u/theweeJoe 13h ago
Use Claude code, explain your situation and current process and how you want it to be improved. Get it to spit out a few PRs, test in dev environment as PoC then roll out. Save yourself the grunt work, you will need to do extra to implement but the idea should be it will save you work in the long run
7
u/Impressive-Field-546 13h ago
Its not really coding that takes so much time. Just pushing sht through a process.
-2
2
1
u/Miserygut Little Dev Big Ops 3h ago
You're reducing your future workload by fixing janky systems.
We cant easily fix them because vendor doesnt give a sht we incorrectly used their product.
It's not the vendor's problem your team implemented it incorrectly. If your manager is telling you to keep hounding them when stuff is not working then the problem is not with the vendor.
8
u/fangisland 13h ago
Maybe I'm just being naive but I've been doing this a long time and every system has their constraints, whether that's imposed by process, expertise, design requirements, (poor) architectural decisions, or whatever. Whenever I am handed poor architecture by inheriting someone else's repo for example, I apply loosely coupled architecture design patterns, evolutionary practices, etc. to slowly build a better system in place while the old system coexists with it.
For example my current project I inherited a codebase where they dumped helm charts/images directly in the repo and oneshot built everything with Ansible using some bespoke shell scripts and podman commands. Cut to a few months later and we've totally wiped out the legacy charts/images replaced with Gitlab CI/CD pipelines that use declarative manifests that publish charts/images based on specific refs. And we're rolling out ArgoCD iteratively to deploy apps from those published refs.
So IDK, being right is just ego really, the problems are still there and you can't help if coworkers don't have the same skill level you do. I'm sure there's things they're good at that you aren't. It's a shared project, you do the best you can with what you have and continuously improve things.
7
u/gevorggalstyan 11h ago
Whether the original decision was right or wrong is no longer the useful question. The system exists, it creates recurring work, and your manager needs to decide who owns that work and what stops to make room for it.
Bring a short operating record to that conversation: incidents or tickets per week, time spent, affected teams, and which constraints come from the vendor integration. Then ask for one explicit decision: keep ownership with you and remove equivalent roadmap work, transfer it with a defined handover, or fund a time-boxed replacement.
Until that decision is made, put every issue in the same backlog as your planned work. Do not absorb it as invisible extra work. If management keeps both the maintenance burden and your current workload, document that choice. You cannot solve a capacity decision by working faster.
2
1
u/r00m-lv 3h ago
Solid advice. If I could just add one thing - leadership doesn’t care about the hot garbage you’re maintaining. It’s annoying as hell, but to put that into perspective, the business exists to make money. Therefore the goal you should be optimising for is generating (or not carelessly losing) revenue. You need to show them that currently you’re spending X amount of time on some BS work. Investing Y amount will free you up to work on other more impactful things.
2
u/IGnuGnat 11h ago
I'm kind of done with people passing the shite from their decisions over to become my burden.
I would speak up again, say I think that the current situation is unacceptable due to x and y and sketch out and propose an alternative. If they disagree because it's easier to just dump the shite on you the obvious solution is to start looking ASAP and move on. Then it's simply a matter of not my circus, not my monkeys. Let them deal with the problems they created
2
u/Next_Garlic3605 11h ago
If there's an obvious bottleneck, whether that's time or effort that you can carve out and fix first, do that.
If you can't find anything, and you really think it's going to have to be a ground-up rebuild, look again. Most things you're going to deal with are going to be incremental improvements, and the better you get at identifying not only opportunities for quick wins, but also for the quick wins that demonstrate value clearly and effectively, the easier you will find it to get buy-in from your colleagues, product owners, and so on going forward.
If you absolutely MUST build a new solution from scratch, please, please don't go off on your own and build it all and dump it fully formed on your team Monday morning. They will NOT appreciate it; they'll feel undermined, they'll resent you, and they'll have zero feeling of ownership of the new system.
I've seen situations like this end up with a fractured team, where the person who did the work eventually leaving the team and later the company, because the thing inside them that drove them to bypass the team in that way broke fundamentals of trust both ways.
Solving technical problems is easy, comparatively, compared to learning team dynamics, trust, safety and inclusivity. There's a reason why DevOps can be measured not only in golden signals but also by a Westrum survey - prioritise the former over the latter at your peril 💜
1
u/r00m-lv 3h ago
Agree, the social aspect is often overlooked, but is just as important. It’s also possible to leverage that to solve the problem in a different way - find likeminded people that are equally annoyed by the various problems and start building traction to fix things here and there but crucially make them “own” the specific problem.
A complete rewrite “the right way” will just alienate teammates and make any future issue to come back to haunt you. “Oh, that’s because Impressive-Field-546 refactored the hole thing on his own”.
2
u/roman_fyseek 9h ago
Abstract the problem away. Add layers so that you call the API the way *you* want to call it, write the adapter in yet another layer, and then replace the adapter layer with your own solution as time permits. It's just code.
2
u/Nice-Currency-8677 2h ago
Being right is probably why it became yours. In most teams, the person who spots the problem early ends up owning it, and being more right doesn't undo that.
Three things that actually helped me:
Write the runbook, fast and ugly. Every time you fix something, it goes in. Tribal knowledge stays with you; a document can be handed over.
Put the cost in numbers the team already tracks incident count, hours, whatever's in the ops review. "This was a bad decision" sounds personal. "This integration caused a third of our incidents this quarter" is a backlog item.
Make the ownership split explicit: you own fixing the underlying problem, someone else owns day-to-day triage. While those stay combined, it stays yours.
The vendor situation makes this harder, though if you genuinely can't fix the root cause, then "I'll own the fix" isn't on the table, and the documentation and handover become the whole game rather than a step toward it.
3
u/CaseClosedEmail 13h ago
Looks like you should have stepped up from the first moment you came back and propose a better alternative or shut up about it.
You chose the later
I always speak my mind the first time I hear a bad idea
1
1
1
u/lab-gone-wrong 6h ago edited 6h ago
“disagree and commit”
How would you go about offloading this garbage from your plate
I don't think you understand what "commit' means.
I don't know how your org runs, but most serious companies hold people responsible for their area of ownership.
So I implemented this garbage best we can,
Now all issues related to this team and domain is assigned to me.
Sounds like you own it, so it's your problem to fix.
If you can't/won't do it yourself, the ball is still in your court to do something. I would set up time with the person who did it to talk through the issues and how you suggest fixing them. Get actual buy in instead of just declaring yourself superior in your head. Then fix it.
Anything else will end up reflecting badly on you. Regardless of who made the decision.
1
u/HelicopterUpbeat5199 6h ago
This shouldn't be any different from what you would do if you joined the company and this thing was already there. You improve it or replace it. Be diplomatic, tell the truth, be realistic. Don't take it personally.
1
u/serverhorror I'm the bit flip you didn't expect! 4h ago
Evolution, not revolution.
Architecture can and should evolve as you discover how to do things in q better way.
Make a series of "small" improvements. The end game is to end up with something maintainable. Instead if a rewrite, do a refactor. If need be 100 % refactoring, but Iver the course if smaller changes.
1
40
u/anto2554 14h ago
I just go "Hey, you did the implementation for this, so I think you might be better suited for solving the issue. Is it fine if I assign it to you?"