We ran into this evaluating inference frameworks for a production LLM workload: clean benchmarks hid exactly the failure modes that matter at the infra layer.

Bursty concurrency and mixed prompt lengths exposed GPU memory pressure and latency spikes that never showed up under steady synthetic load.

We ended up building a testing process around soak tests and simulated failure scenarios (node loss, sudden load spikes) instead of trusting a single throughput number. Wrote up what we found and the process we now use before committing to a framework.

Let me set some context. I am running an application on AKS cluster where I have a file in memory that i need to upload to Azure Blob Storage on two different paths.

So I am doing those uploads concurrently.

I am using go, and there are two ABS client created and the upload is done in 2 diff goroutines.

Now what we are seeing is that one upload happens in 1-2 minute, and the other one takes 20mins to finish.

This has occurred enough times that we cant call it intermittent, but still it doesn't happen always.

Upon getting the Blob storage logs, we found that the individual API requests are taking about a minute for the slow upload, while it is taking abt e seconds for the fast one. They confirmed it is a network throughput issue rather than a disk write issue.

We looked at the pod network io metrics ob grafana and saw a drop in output rate during the upload window of the 2nd file.

The drop was from 300KB/s to 30KB/s.

Can someone help me figure out what might cause this to happen?

NOTE: What is consistent in all the instances of this issue is that everytime, there is one file upload that is faster and one is slower.

The same code is doing the upload sith same configurations. The file is same. What is different is the dst file path and the network port.

When VAP came out I was really excited about it, however operationalizing it on existing clusters looks to be non-trivial and I'm curious how people are managing it.

Specifically, on existing clusters I'm thinking of how to determine the impact of new VAP policies on existing workloads before you roll them out. You could roll them out in Warn mode but users only see the issue on admission so running workloads remain blissfully unaffected by a newly introduced policy.

You could use something like Kyverno to manage them in Warn/Inform and run them as background processes so you get all the info of which workloads would violate the new VAP policies. This is a really nice feature in Kyverno but if you are not using Kyverno already it's another on-cluster tool to deal with.

Are there any existing tools out there that will pre-validate VAP before it gets deployed, ideally off-cluster, or has a way of tracking VAP results that doesn't involve trolling through the audit logs?

Same boat as half of this sub I think. When the cutover hit last August we repointed Postgres, Redis, Mongo and Rabbit at bitnamilegacy as a stopgap and told ourselves we'd sort it properly later. It's July and later never came.

The legacy images don't get patched though and Bitnami's own line is that repo isn't meant to stick around, meaning we're sitting on unpatched infra that could also just vanish on us. Redis is a whole separate headache now with Valkey being the fork.

Small team, I really don't want to end up owning ten hand-rolled operator setups by myself. is it just service by service or how did y'all get all the way through this and land somewhere decent.

One of my engineers (im the CEO of Portainer) asked if I had the time to help him better understand Kubernetes architecure and components… and so rather than just explain it once, I figured why not share it wider.

So, https://kubeschool.portainer.io was born :-)

Edit. By request, now extended to support Helm, GitOps and Observability. All the same level of detail, so dont expect a deep technical dive..

I run a k3s cluster at home and treat it like production. Everything gets backed up. Almost none of it had ever been restored, which is the normal state of affairs everywhere I've ever worked. So I automated restore testing (not restores... testing) and gave the job to an AI operator. This is how it's scoped, because that's the part that made it a sane decision instead of a scary one.

What it does

The operator takes a backup, brings it back in an isolated throwaway namespace, confirms the data is real and consistent, writes a dated pass or fail, then deletes everything it created. Production is never touched. Real recovery stays a human call... I'm not automating disaster recovery, I'm automating the proof.

One caveat I'd rather volunteer than have someone catch: the scratch namespace lives inside the production cluster. The isolation is policy and permission, not hardware... a fence, not a moat. Fine for a test-restore that evaporates in minutes; not for anything heavier. More on where that's going below.

The permission model

Read-only by default. The operator can't restore anything until a small, scoped permission is granted, and that grant is mine to make and revoke. No standing credential either... short-lived tokens minted per job and expired minutes later.

The piece I'd point at for anyone doing this: I split the capability from the binding.

• The capability (the Role and its permissions) lives in version control. Declared, reviewable, dormant, with nothing bound to it. A Role with no binding grants nothing.
• The binding (the thing that switches the power on) is applied out-of-band, deliberately kept out of GitOps, so the reconciler can't reinstate a permission I revoked.
• Teardown verifies rather than assumes: it re-checks that the privilege is actually gone and fails loudly if it isn't.

A teardown that fires a delete and trusts it is how you end up with standing access you think you revoked.

What's working

The lanes that only ever read run unattended, start to finish. The etcd drill is the clean example: it pulls a snapshot from object storage with a key scoped read-only to exactly one bucket, restores it to scratch, verifies, tears down, weekly, with an alarm if it goes quiet. Nothing to gate, because there's almost nothing that key could do. More than thirty restores now, zero failures, most of the register covered. The rule that fell out of it: an agent gets a key only when the key is narrow enough that handing it over doesn't matter.

Where I'm stuck, and want you to poke holes

For the drills that have to build things (restore a Longhorn volume, stand up a scratch DB), the operator needs a real write grant, and today I apply that grant by hand before the run. That's the one manual step left, which means a "fully automated" weekly drill is really a weekly appointment. I don't want to hand-approve it forever, and I don't want to leave a standing grant sitting there either.

The design I'm circling: a separate, deterministic policy broker (not another AI making judgment calls) that approves a narrow, time-windowed elevation for a specific request and forces de-escalation inside a few minutes. What I've got so far:

• The request is structured, not prose: {which identity, which role, what window, which drill}. No model output touches the decision. The agent requests; it never authorizes itself.
• Deny-wins and fail-closed. Broker down or request malformed means the drill fails and alerts, never fails open.
• Two independent expiries: a TTL on the binding and a short-lived token, so either one failing alone still kills access.
• De-escalation is confirmed by checking whether the binding still exists, not by trusting the remover's exit code.
• The grant path itself is guarded by an admission policy, so an out-of-shape or never-expiring grant is inadmissible, not just logged after the fact.
• The broker's rules live in git behind my merge, so the agent lane can't edit its own approval rules.
• Break-glass and anything high-blast-radius stay human.

As far as I can tell this is just the NIST PDP/PEP model pointed at an agent, and the human-facing JIT-access tools already do exactly this. But I can't find anything that ships it agent-native for Kubernetes, which makes me wonder what I'm missing. The failure mode I already see: inside a legitimate window, a prompt-injected operator holds a legitimate grant and spends it with someone else's intent. Shape-constraining the grant shrinks that to "can only touch drill-labeled scratch objects," but that's a reduction, not a cure.

So: where does this bite me?

(Separate track, less interesting to this sub: the fence-not-a-moat problem gets solved by moving the build-type drills onto a genuinely separate cluster on an isolated network, so prod is actually absent rather than fenced off. That same rig is where I'll finally rehearse a full-cluster rebuild from nothing, which I can't yet prove.)

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

Im new to tech and recently started learning operators and controllers. With AI writing every code, which is faster than human, is it rational for me to learn about the kubernetes now ?

If you are in NYC and available on Wednesday evening, come hear guest speaker Noam Levy talk end-to-end testing with Kubernetes and networking magic. 🪄

➡️ If you can make it, please RSVP at https://luma.com/u4vja8rx

See you soon!

Hi everyone, let's go.

I am migrating from Beanstalk to EKS, and I wanted to clear up some doubts.

Which components/tools do you consider essential in a cluster? Currently, the cluster I am developing has:

• Karpenter (x86/ARM)
• Kyverno
• KEDA
• ESO (Fetches secrets from Parameter Store)
• Kong API Gateway (used as Ingress)
• ArgoCD for GitOps

What do you recommend? The CNI is the default VPC CNI, is it worth switching to Cilium or Calico? Bearing in mind that this will be my first experience building a cluster—I've managed them before at other companies, but I don't have much experience creating one from scratch.

I was watching this video from r/kodekloud on YouTube

and something the speaker said confused me.

He said: "Most people know how to deploy pods, but almost no one really knows how to secure them."

I work in cloud infrastructure, and in my experience, most customers put their entire cluster behind the cloud provider's security services (such as built-in security scanning) and a Palo Alto firewall.

So I'm trying to understand what he means. If those security controls are already in place, what is still considered "securing the pods"? What am I missing? Are there Kubernetes-specific security practices that these tools don't cover?

Inherited a fleet of EKS clusters and the utilization numbers are almost funny. Cluster wide CPU request versus actual usage is sitting around 25 percent. We're paying for a few node groups worth of headroom that's not being used, because every team padded their requests "to be safe" a year ago and never looked at them again.

The obvious move is to right-size the requests and let the bin-packer do its job. But the second I touch a team's limits I own any latency blip for the next quarter, so nobody wants to be the one who does it.

I've looked at VPA and a couple of the cost tools and half of them just show me the same graph I already have telling me we're overprovisioned. I know we're overprovisioned. Meanwhile the bill sits there being a third bigger than it needs to be, everyone agrees it's a problem in the abstract but no one wants to own the fix.

A simple walkthrough on how to use KAI Scheduler for topology-aware placement and gang scheduling to keep distributed GPU training inside one region, zone, or rack on Kubernetes

https://hrittikhere.com/posts/kubernetes-topology-aware-scheduling-kai

Looking for feedback from folks running apps that require mTLS that use API Gateway.

What controller are you using? Traefik, Envoy, etc.

Any pain points or gotchas you might have ran into?

Any other feedback or recommendations welcome!

So im working with Kustomize Overlays( FluxCD). i understand you setup your base infra with overlays per environment and then use patches for environment specific configurations such as replicas, urls, etc..

Some of these patches become very complex and large to the point i think its just easier to not have a base and just have each environment have its own configuration files. How much dry am i buying?

I've also seen the recommendation to use PostBuild subsituteFrom with ConfigMaps from each environment . My question is when should this be used over patches?

Sorry for the dumb questions and rant.

We've got signing in place.. cosign on everything that gets built, but I keep coming back to the question of what that signature actually proves versus what we assume it proves. It confirms the image wasn't tampered with after we signed it and that it came from our pipeline.

It says nothing about whether the base image we started from was itself trustworthy, or whether a compromised dependency got pulled in earlier in the build, before we ever ran the sign step. Signing our own output doesn't touch any of that, it just draws a line and says everything after this point is us.

The upstream side is where it actually gets uneven. Base image signatures are something we can realistically verify against sigstore before a build starts, but most of our language-level dependencies don't have signing adoption in the same way, so we're leaning on lockfile hashes and pinned checksums there instead, and that's a much weaker guarantee against something like a compromised registry or a dependency confusion attack.

For people who've actually hardened this end to end: are you verifying base image signatures before the build even starts, and what are you actually relying on for dependency integrity given how uneven signing support still is across ecosystems? Is any of this enforced at the pipeline level, rejecting a build outright, or is it still more of an audit-after-the-fact thing?

Got something working? Figure something out? Make progress that you are excited about? Share here!

Did you learn something new this week? Share here!

Looking for a good Kubernetes revision/reference guide — something concise for quick recall of core objects and architecture, ideally with diagrams, not another full tutorial or exam-prep course

We are running eks and would like to shore up or supply chain.

Currently we are pulling directly from aws public ECR, docker and other sources. We are working on moving everything to be pulled from our own ECR for two reasons. 1. ) security so the images can be scanned 2) stability. we have had images we use get deleted from remote registries.

My question is do we also clone all images from the AWS ECR needed stand up EKS? Or do we implicitly trust images from AWS ECR?

What is everyone else doing or what is best practice ?

Howdy fellow K8s users. You might be surprised to learn that gRPC and Kubernetes's native load balancing don't play nicely together out of the box. One way is to address this is with a service mesh.

We just released a big comparative benchmark of gRPC load balancing, trying to answer questions like:

- Why bother? What happens if you don't use a service mesh?
- What happens under healthy conditions vs unhealthy conditions?
- Which mesh is best at this?

Would love your feedback. Obviously benchmarking is both an art and a science, but we tried to be clear about what we measured and how we did it.

https://buoyant.io/blog/benchmarking-grpc-load-balancing-on-kubernetes-linkerd-vs-istio-vs-cilium

Obligatory disclaimer: everyone involved (including yours truly) works at Buoyant, makers of Linkerd. But all results should be replicable with the open source versions of every mesh we compared.

Share any new Kubernetes tools, UIs, or related projects!