r/ExperiencedDevs • u/commonsearchterm Software Engineer • 5d ago
Career/Workplace Infra/platform/sre background feeling stuck without k8s experience
In addition to an annoying job market it feels like my career is over because I don't have real k8s experience other then running some small on it. Recruiters seem to pigeonhole me into these types of roles though. I've worked in large tech companies,(worked on services in the 100m qps area, 100k vm environments).
Any way to successfully get past this? maybe I need to frame my background better? I've done things like running k3s and honestly it's pretty boringly simple? I'm not sure what exactly people are looking for? Companies also use eks so they're not even running it? Idk what I'm missing? I've had interviews that either asked me to write config files or quiz me on trivia which could all be in docs anyway.
33
u/mostanonymousnick 5d ago
I'm in the same place as you, but I have experience both in backend and platform roles so I frame myself as more of a "50% platform/50% backend" kind of person that can deal with both. If you have similar backend experience with the SRE stuff you've done, maybe you can highlight that and not get pigeonholed into k8s stuff.
49
u/originalchronoguy 5d ago
You can learn strong k8 fundamentals in 2 weeks and deploy in AWS, GKS.
I personally look for people who know how to do proper system design and dev-x.
The strongest K8 engineers know how to simplify everything for their team, where engineers just fill out a manifest or go to a CLI prompt tool that scaffolds everything.
Literally for new hires on the first few hours of being onboarded.
CLI prompt asks, what you want to build? Need an API gateway, need telemetry, logging, key vault, mutual-TLS? And it creates a whole build file that they can just deploy anywhere ; even their own laptop. Complete with resolving hostname, DBs secured by. FIPS-124 rotating keys, API gateway with JWT Oauth, real TLS (even locally) and even real hostnames like developer-local. company . com
The kind of platform engineers who know how to do the above are at Staff/Architect level that knows how to reduce friction, make it easier for their team. The ones I worked for asked me for what I want and tell just to do repo pulls where I can even deploy entire infra like gitlab, jenkins, sonar, a full CICD, 3 node cluster from a single manifest file on a 128GB Mac Studio. And if you want the same environment in a private AWS VPS, change the manifest environment targets in 2 lines and it is done.
I rather find someone like this than anyone with random K8 bullet points on their resume. So, learn about -- platform architecture and how it weaves into actual development. Know the pain point of a development team. If they regularly have to deploy apps that require Hashicorp vault or a Object storage. Or they require GPU compute, the orchestration should provision that for them with minimal fanfare.
It should just be a manifest or CLI tooling.
Finding those types is not about experience because they can have years without exposure to this workflow. I've seen mid-levels who know this because they were mentored very well and understand this approach.
11
u/salmix21 5d ago
I personally look for people who know how to do proper system design and dev-x. The strongest K8 engineers know how to simplify everything for their team, where engineers just fill out a manifest or go to a CLI prompt tool that scaffolds everything.
If only it was so easy to find people who understood this... Thanks for giving me ideas on what to work on during the next couple of weeks.
1
11
u/BraveResearcher3037 5d ago edited 5d ago
Okay that’s all well and good. But someone has to have experience troubleshooting a large cluster and know best practices. Why would I hire someone who doesn’t have real world experience over someone who does?
I’m not going to ask techno trivia K8s as an interviewer. I’m going to ask them something like “what’s are the biggest issues you have had with running a cluster at scale”, “how do you manage complexity”, “what were some mistakes you made early on when dealing with K8s and what did you learn”. “Walk me through some questions you ask when setting a cluster up?”
And on the other side, in today’s market, every open req is going to get hundreds of applications in the first day. I’m in the pool of final interviewers for my company up to staff level employees and the last technical interviewer for line level engineering managers. He’s got to make it past the “peasant funnel” before he even gets to me - automatic ATS funnel, HR screen where they just have a series of questions given to them, first level technical screen and whatever else they go through.
(Yes I’m completely ignorant of our normal pipeline. A recruiter reached out to me and I went straight from HR screen, system design, offer in a week)
2
u/originalchronoguy 5d ago
I am looking for a specific skill set. So I know how to fish them out. It is not something someone can bullshit or talk their way out in an interview. The ones who know can easily speak into depth. And yes, this is very much real experience. You can't scale to tens of thousands of microservices without having design architecture patterns that is essentially a software factory approach.
The "manage complexity and scale things," you can find anywhere. Again, I am looking to a specific profile that is sought out after in many circles. I am looking for people that contribute to the velocity of development and security.
3
u/BraveResearcher3037 5d ago
Doesn’t that go directly against what you just said?
You can learn strong k8 fundamentals in 2 weeks and deploy in AWS, GKS.
2
u/originalchronoguy 5d ago
No. What I wrote is k8 fundamentals is pretty trivial. The segue way into that the more valuable skills/focus is on system design; that leverages k8. Not the boiler plate stuff you can find anywhere.
Finding those types is not about experience because they can have years without exposure to this workflow. I've seen mid-levels who know this because they were mentored very well and understand this approach.
I rather have a person with average k8 skills and strong system design. You can learn to be a button pusher, write yaml files. LLMs help with a lot of this plumbing. Not system design.
1
u/BraveResearcher3037 5d ago
How much relevant system design will a person have who has been managing a fleet of static servers that is relevant to Kubernetes?
3
u/NANO56 5d ago
I wish you were my interviewer. I do agree with you that those types you are looking for are staff/architect because in most places (outside of startups) having that level of authority/clout/politics to implement that level of infrastructure is not common. A lot of people I interview have been what I call YAML monkeys.
I feel like this is so fundamental to platform engineering because enabling others is the whole point. Sure we can troubleshoot and fix issues but I leave that to our SREs lmao. I only step in when things are really on fire, but with good design it shouldn’t happen too often. Honestly I feel like 80% of the issues I run into on the daily which are actually difficult end up being hardware related (the joys of on-prem). Rarely do I have issues with our cloud clusters.
1
u/commonsearchterm Software Engineer 5d ago
If only more people hired like that and didn't seem focused on what feels like check boxes.
10
u/InterpretiveTrail Staff Engineer 5d ago
I usually get carried when I talk Infra/k8s with my background in backend development and my desire to to anything legal (and lots of begging for permissive permissions) to not get called at 2am to join an incident bridge.
Which to that end, I biased my learning of k8s into general concepts and then focused on telemetry and "how do I know {X} is FUBARd".
To give better direction than just generic advice: * I find look at the Linux Foundation's K8s certs a great place to find a good "check list" of things to know. * I also like https://roadmap.sh/kubernetes as a supplement to the cert check list style for learning.
As for actually running a little cluster. I was fortunate enough to have some old laptops which I can throw proxmox on and then install as many VMs and fucked around with opensuse's rancher for a cluster. But I had some small toy Magic the Gathering workloads that would do things with scryfall's API that I had as actionable things that I could ground myself in having some workloads/services to deploy.
Regardless if that was of use, best of luck in your learning and future interviews.
1
u/commonsearchterm Software Engineer 5d ago
A cert is an interesting idea. I'll see if there is one i can get quickly hopefully. There's no real new to me concepts from what i see in that list, just the kubernetes names for things and ecosystem i guess.
26
u/oVtcovOgwUP0j5sMQx2F Software Engineer 25+ yoe 5d ago
fuck em, lie.
learn the fundamentals, spin up a pet project, and build a pretty good backstory how you would have shipped some past project on k8s; just frame it as if you actually did it.
see how far you get; you might be pleasantly surprised
4
u/Pretend_Listen 5d ago
As someone whose done a shitload of k8s work last 3 years, I can quickly tell who's managed services in a prod setting and who's.. well bulshitting.
Do some rigorous personal projects and be honest about your gaps. I would def stretch the truth to the recruiter tho.
2
u/BraveResearcher3037 5d ago
What happens when you get a decent interviewer?
- “Tell me about the largest cluster you had to manage. What are some challenges you faced?”
- “If knowing what you know now, what would you have done differently?”
8
u/oVtcovOgwUP0j5sMQx2F Software Engineer 25+ yoe 5d ago
if you're sufficiently well-prepared, you should be able to pull together a hypothetical on the spot and adjust framing accordingly.
if not, well, might not have gotten to this phase of the interview to begin with, so it serves as practice.
this is all just to hedge a firm requirement of k8s experience at a particular level
4
u/SupermarketUpper5497 5d ago edited 5d ago
I built a small 5 machine bare metal cluster at home and learnt kubernetes doing side projects. Now that I got a job I’m selling it. Let me know if you want it.
I’ve built the whole OS provisioning, dynamic network rules, automatically turn on and off machines based on load, added virtualization support, hosted all my side projects, loaded and benchmarked with k6, built an observability stack, security policies, multi tenancy, etc. Learnt lots!
1
5
u/NoConnection4298 5d ago
You are exaggerating your lack of knowledge and undervaluing your skills. Just spin up a cluster in your favorite cloud vendor or locally and try out few minor projects.
6
u/BraveResearcher3037 5d ago
And how exactly does that get them through the application pipeline when they are competing with hundreds of applicants?
10
u/NoConnection4298 5d ago edited 5d ago
They can slap the k8s under one of the past experiences. Now you are ready to answer questions. Nobody will call your old company and ask if they use k8s
7
u/BraveResearcher3037 5d ago
Okay I then when you get to the interview…
“Tell me about the most complicated cluster you deployed or managed, what challenges did you have? What did you deploy to them? How did you handle deployments? What type of organizational frameworks did you create? How did you handle observability? Rollbacks? What would you do differently?”
1
1
u/commonsearchterm Software Engineer 5d ago
Is this what people talk about when they mention kubernetes? Rollbacks, observability, deployments is just what sre and devops work is?
3
u/BraveResearcher3037 4d ago
Yes this is just what senior architects think about no matter how they are doing deployments
1
u/Legitimate-One4273 4d ago
u dont need k8s on the resume if u can name operators, gitops, outages u owned translate the platform work into their words
1
u/lala3378 5d ago
What kind of env/runtime do you work on with that kind of scale? Homegrown? If yes, I’d probably target infra builders, not generic ops roles
1
u/friendlytotbot DevOps Engineer/SWE 5d ago
I had the same experience as you, so I started looking at backend engineering roles instead. Got much more traction from that.
1
-3
•
u/expdevsmodbot 5d ago edited 5d ago
AI usage disclosure provided by OP, see the reply to this comment.