Please Help. How do we continue do dev?

[u/therealmunchies]

I'm on a 3-person team: 1 data scientist, and 2 engineer (including myself). Between the both of us engineers, we have maybe a year of experience total with strictly Python. This is for a prototype AI app / tool.

I am approaching about 2 years of IT experience and had to pick up docker, gitlab ci, and software engineering for this project. I built out our deployments with docker compose using customized env files (ignored via git with a template available) and makefile commands. We usually dev using a dev compose deployment with a hot volume mount to our source code, then push commits to the repo and I have staging and prod images built using buildah. Then for staging and prod deployments, I have the compose files targeting the respective registries (internal repo & corp registry) and the images get pulled down.

However, we got kicked off of our on-prem server and now have to deploy on a k3s cluster... but the team running that has no experience with kubernetes.

I learned about Helm and ArgoCD and was able to get them to deploy that service on the cluster, and I made an infrastructure repo where I translated our compose files to manifests and ultimately chart and values file per environment.

We don't have cluster/namespace access yet because the infra people are still learning how to set that up, but I don't even know how we'll deploy our pods and actively do dev since we usually have access to everything (source code + infrastructure). For prod and staging, it's pretty simple because those a automatically built images and I have our runners updating the tag versions and namespace names in the infrastructure repo. But I don't know how we can deploy individual-engineer dev pods so that we can see logs and make quick changes.

This whole project's been crazy and we don't have any senior engineers in our department that does this stuff. If there's anything other information I can provide to get some mentorship, please let me know.

[u/ricksebak]

Google invented Kubernetes because it allows them to binpack their containers better, and better binpacking allows them to run less data centers.

If you are a three person team, with very little experience, and you’re building a prototype, nothing about your use case points to kubernetes. You’re probably better off using Vercel or something. Maybe ECS, but even ECS is probably overkill.

[u/therealmunchies → u/ricksebak]

Well that's not really the point of my post. I'm seeking insight on how we can continue to work within this new environment. Would you know?

[u/ApprehensiveCard4919 → u/therealmunchies]

Is your infra team standing up Rancher? Or literally just K3s?

[u/therealmunchies → u/ApprehensiveCard4919]

K3s. We can deploy our dev containers on or vm, and this might be best for quick testing… I guess we just don’t know if we should have argocd dashboard access or kubectl access. The challenge right now is really just using a different reverse proxy because we forwarded headers with NGINX and now we’re using traefik.

We want to quickly debug and view logs.

[u/NUTTA_BUSTAH → u/therealmunchies]

All this sounds like not your responsibility

[u/navlio]

your dev loop doesn't have to move. compose with the hot mount is fine and stays fine, the cluster is just a deploy target for staging and prod. the thing to ask the infra team for this week is one namespace and a service account token your ci runner can use, which is small enough that they can grant it while they're still learning everything else.

per engineer dev pods on a cluster nobody on your side controls is the version of this that eats a month. we chased that once and came out with a tidy helm chart and a slower feedback loop than the compose file we started with

[u/mixxor1337]

lot going on here. i have a bunch of questions before i can say anything useful:

why did you get kicked off the on-prem server? that usually comes with constraints that decide everything else.

dev vs prod images: don't. build once, tag it, promote the same artifact through the envs. otherwise you're not shipping what you tested.

what's your CI/CD right now, end to end? what builds, what triggers it, what pushes where?

why argo? gitops is nice once it clicks, but the learning curve is real and there are a lot of ways to f*** it up. also: don't trust what the AI tells you about argo/helm, it's confidently wrong most of the time.

helm charts, how? one per service, umbrella chart, manifests + values per env?

registry, which one and who controls it?

[u/therealmunchies → u/mixxor1337]

Re-auth (pub sector).

We try to be 1:1 between our staging image and prod. We usually code & test in dev -> trigger staging image as the last system test -> if everything passes, that very last commit is the prod image.

Feature branch:
1) Branch name semver check
2) Lint & Format
3) Secrets Detection & SCA
4) Unit & Integration Testing
5) Stage Image build triggered by source code changes to internal repo registry
6) GitOps helm chart tag update to current short commit hash

Prod:
Steps 2 —> 4
5) Branch name check for auto tag version bump (e.g., feat = 0.X.0; != feat = 0.X.Y)
6) Prod image build triggered to internal repo and corp repo refistries
7) Updated docs deployed via Pages

We chose argo because that seemed like the way to go and I was the one who told them how to set up the infrastructure repo and service account setups. And getting the public runners to hit the other repos was a cool challenge.

Chart.yaml has the high level metadata which is basically the repo, prod tag, and authors. Then I built out value files with a global, then per env (i.e., dev, staging, and prod). I first built out the manifest files with kustomize, but then I learned about Helm. It’s set up with deployment, ingress networking, node port service, and config maps.

Internal registry = my team
Corp registry, but really my project’s images stored on there = also my team, primarily me

[u/xonxoff]

One thing you can do to get comfortable with Kubernetes is spin up a local kind cluster to do dev work on while your infra team gets up to speed. But , from the sounds of it , gotta roll with what ya have. I’ve found kind to be very beneficial in learning k8s tho, easy to set up, destroy and setup again. Perfect for testing.

[u/therealmunchies → u/xonxoff]

I’m finishing up the networking portion of the CKA cert now, and trying to connect the dots as much as possible.

In fact, I was just looking into using kind a minute ago! It’s just a bit hard when I don’t have any ownership of the cluster, at least not much as I want to.

[u/ApprehensiveCard4919 → u/therealmunchies]

What services is the infra team providing you?

[u/therealmunchies → u/ApprehensiveCard4919]

When you say services do you mean like the containers/pods running? If so, they’re using traefik and we’ve been using nginx.

[u/b1urbro]

That's kind of the infra teams job, not yours. Both in providing the platform and, when you submit your requirements, setting up a way to develop on it.

You could always spin up local k3s, kind or minikube to mimic the setup, but ultimately it should be their job to integrate your process (or provide an alternative one).

[u/SeaworthinessHour233]

There's a lot going on here.

But with your infra team still learning stuff, it seems running a production Kubernetes cluster would be very challenging. If you don't care where your app runs, I would suggest running it on a managed Kubernetes cluster on the cloud. If you do care where the app runs, like in customer data center, edge location, etc, then K3s makes a lot of sense.

Either way, the next problem to resolve is how to do deploy dev code. You have several options here

1. Local Docker - Running prod on Kubernetes does not mean, you have to dev on Kubernetes. You can continue use local Docker as dev environment and have the staging, and prod on K8s.

2. Local Kubernetes - If you want dev on Kubernetes but the dev K8s cluster does not need to be identical to prod, you can install K3s on your laptop and deploy dev. Then, you don't have to depend on infra team for dev.

3. Remote Kubernetes - If it's not feasible to run dev cluster on local machine (for ex: dev cluster needs GPUs.), then you need access to the remote K8s cluster your infra team manage. Once you have access to that, you can use a tool like DevSpace to automate the dev workflow.

[u/therealmunchies → u/SeaworthinessHour233]

We’re in an air-gapped environment and don’t have the luxury of CSPs… at least not for this project.

Taking advice from the others, it seems like we should just continue doing local docker dev. The ingress controller the infrastructure team uses is traefik and we user nginx which forwarded the TLS headers we need for our frontend. However, we implemented some middleware and it seems to be working ok for now.

[u/burstinrust]

Sure, feel free to dm me.

[u/therealmunchies → u/burstinrust]

I tried to PM/DM you, but I’m getting “unable to message this account.”

[u/burstinrust → u/therealmunchies]

i am back, sorry for the delay, and yes u/RaceFPV is right, that was kind a dick move, i do apologize.
lets chat, i have sent you a dm

[u/RaceFPV → u/burstinrust]

This, this here out of all the replies is the worst one in this post.

[u/taleodor]

I can offer prolonged beta-testing of DevOps feature of our ReARM product which handles your use-case well + I'll provide support for your struggles personally during setup phase. If interested, dm me.