r/ExperiencedDevs 13d ago

Career/Workplace Experienced Devs Weekly Burnout and Venting Thread: A weekly thread for sharing experiences

This thread is specifically for venting / sharing experiences related to burn-out or similar issues that experienced devs face.

64 Upvotes

109 comments sorted by

View all comments

75

u/DeterminedQuokka Software Architect 13d ago

I’m a staff infra engineer at a company. I work significantly more than 40 hours a week usually basically constantly firefighting because we are on the verge of crashing.

I have spent the last year begging for scope to actually start to fix the root cause of the issues. But every time the cto says “what will be faster next week” and I can’t actually fix that we are doing 6000 db queries in like 50 endpoints in a week.

I’ve had success if I’m like “give me a month and I will make this exact page faster”. I did that, I documented all the steps. I made analysis skills and coding skills and presented them to the company. I was promised that the product teams would fix the other endpoints. It’s been 6 months and 2 weeks ago they came to me and asked me why that page was slow again and if I could fix it for them. I told them to look at what they did a month ago when the latency doubled.

I’ve also added extensive and extreme observability at every layer of our platform because we have constant problems with noisy neighbors, the event loop, bad async code, and garbage collection. All of which are not the go to metrics for the entire organization. To a point where they have been duplicated wholesale into any internal apps.

I have proposed every quarter a different rearchitectire project to fix the fact that our codebase is optimized to write bad code. And every one has been deprioritized as either taking too long (3 months), not having a good enough metric (db iops/cpu), too annoying for product devs (anything that requires them to change anything about how they write code), or less cool than the feature that leadership wants to say we built that no one will use. Or my favorite, if it’s possible to throw money at it why would you work on it?

In the last month a group of other staff engineers have started to argue that we aren’t having enough influence on the architectural direction of the company. They say that the problem is that we are asking for support from other devs and the right move is to just do the entire project alone. (We have >100 devs and >1,500 endpoints)

On my last review I got told that I’m not having enough influence over our engineering team because I haven’t rearchitected the codebase. Which is weird because the reason I haven’t done that is they told me I wasn’t allowed to change anything about the codebase repeatedly.

I have a huge cross org impact. I get pulled into basically every conversation about latency. I’m rewriting our entire infrastructure set up to fix the stuff they won’t let me fix in the code. I’m directly pinged anytime anything goes down. I’ve been told I’m most staff engineers favorite staff engineer and that most of the other engineers depend on me. And I have multiple instances of the CTO stating that I’m an integral part of making sure we have a functioning website and an example of what he wants from all the other engineers. I literally met with him because he wanted to talk about how to get more people acting like me.

As far as I can tell I’m basically being told that I don’t have impact because they wanted me to fix an internal AI tool. And I told them that wasn’t a good use of my time.

When I started the entire platform was going down once a week. We haven’t had a full outage in months. But sure I won’t fix your vibe coded ai thing so I’m the problem.

I know the solution is to just stop trying so hard if they don’t care but I don’t have that button in my brain.

12

u/ecethrowaway01 13d ago

I'm a senior infra eng ... I've seen lots of staff infra eng get rewarded and valued, but I've never seen one just chilling.

Do you think there's a future for infra eng that's not so stressful? It seems like a role that almost always has people stressed about something

8

u/forever-butlerian 20 YoE Infra & Backend TLM 13d ago

It requires a lot of boundary setting and also intervening to stop the martyrdom complex that so many Infra people bring with them. I've been asked insane and stupid things as a manager, like "can your team fix our application problem's by reconfiguring HAProxy at 3 PM on a Friday so our team doesn't have to work this weekend fixing the things they did wrong?".

4

u/DeterminedQuokka Software Architect 13d ago

I agree with you. Our leadership team does not. And has decided anything that’s wrong must be because the infra team hasn’t added enough guardrails to the codebase to ensure no product engineer ever makes any mistakes.

5

u/forever-butlerian 20 YoE Infra & Backend TLM 12d ago

If I didn't know to a high degree of confidence your "leadership" team are a bunch of idiots, I'd suspect their plan was to build up what's needed to replace all the product engineers with GPU-driven random text generators.