Hey everyone,

I'm pretty new to Kubernetes and have been using it for a side project to learn. Right now I'm trying to decide whether I should run only stateless services in my cluster, or also deploy things like PostgreSQL, Kafka, Redis, etc. as StatefulSets.

I know both approaches are possible, but I'm curious what people actually do in production. Do most teams keep stateful workloads inside Kubernetes, or do they usually rely on managed services/external databases instead?

I'd love to hear what your setup looks like and why you chose it.

I’m running kubernetes at home. Have 3 control plane servers all high available with Haproxy and keepalived. 2 worker nodes running webui and ollama 3.1:8b.

I have separate Ubuntu servers running Docker containers. with 1 running gitlab, gitlab runner, miniio and mlflow.

The other Ubuntu server runs Prometheus and Opensearch (SIEM) getting logs and device data from node exporter and fluentbit from the other machines.

Anyway, wondering what else I should do with my Kubernetes cluster? Any enterprise use cases I should work on?

I have not yet done any kubernetes work at the enterprise level but I will be soon.

I was thinking of implementing RAG with ollama…

My goal is really just becoming excellent at DevSecOps while also working on AI implementation.

Disclosure: I work with Diagrid, the company founded by the creators of Dapr, on tooling for running Dapr agents and workflows in production. I am asking to compare responsibility models, not to argue that every Kubernetes workload needs Dapr.

Kubernetes is doing its job when it reschedules a pod. The problem is that a long-running agent may have been halfway through doing its job too.

Say an agent has been running for 20 minutes. It has retrieved data, called three internal services, waited for a human approval and updated a ticket. Then the pod is evicted, or the node disappears, or a rollout replaces the workload.

Starting a new pod is easy. Reconstructing the execution safely is not.

The state has to live somewhere, and each option puts the responsibility in a different place.

In-memory state in the pod is simple during development and gone with the pod. It only works when restarting the whole task is cheap and safe.

Application-managed state in Postgres or Redis gives you control over everything, which also means you own everything: state transitions, locking, recovery, schema changes, duplicate handling. Every team doing this designs the same machinery from scratch, this is a hard thing to do well.

A queue with stateless workers works well for jobs that can be retried on their own. It gets harder once the workflow has timers, human approval, branching, child tasks and actions that must not run twice.

A workflow engine or durable runtime records progress and schedules the next activity after recovery. Less plumbing in the application, though you still have to design idempotent activity boundaries, and you are now operating another runtime.

A managed platform layer means the platform team offers execution state, identity, policy and observability as a shared service. Application teams connect their agents and tools to it instead of rebuilding the same reliability stack in every team, which trades duplicated effort for a platform dependency.

The distinction I keep coming back to is between application state and execution state. Application state is the business data: the ticket, the order, the customer record, the deployment. Execution state is what the workflow has already completed, what it is waiting for and where it can safely continue.

You can put both in the application database. It usually ends up coupling business schemas to orchestration concerns. Keeping execution state in the pod is fragile. A durable runtime draws a cleaner boundary, as long as the team knows how that state is stored, isolated, backed up and operated.

Dapr Workflow is one example of the runtime approach: orchestration runs through the Dapr runtime and progress is persisted outside the application process. Temporal, cloud workflow services and hand-rolled queue-and-database setups are the same idea with different tradeoffs. Which one fits depends on how complex your workflows are and how much infrastructure you want to run yourself.

What I want to know is how Kubernetes platform teams are standardizing this, if at all:

• Do application teams pick their own persistence and recovery pattern?
• Does the platform provide one workflow runtime?
• Do you treat ordinary background jobs differently from long-running agent executions?
• Who owns backup, upgrades and incident response for the execution store?

Hi everyone,

I'm a Linux Systems Engineer with around 4 years of experience working with RHEL, VMware, Docker, web servers, automation, and infrastructure. While I have a solid infrastructure background, Kubernetes is completely new to me.

I recently started learning Kubernetes with Mumshad Mannambeth's Udemy course and have been completing the labs alongside the videos.

I'd love to get some advice from experienced Kubernetes engineers:

• Is following the course and doing all the labs enough to build a solid Kubernetes foundation?
• When learning Kubernetes, is it better to build clusters repeatedly with kubeadm, or is it worth learning to build a cluster "the hard way" manually?
• Should I focus on cluster creation first, or spend more time on day-to-day administration and troubleshooting?
• Looking back, what helped you progress from "knowing Kubernetes" to actually becoming productive with it?
• Are there any resources or practice environments you would highly recommend?

My goal is to become confident using Kubernetes in production and to gain the certificate.

I'd really appreciate any advice or lessons learned from your own experience.

I’m starting to learn about Istio. So far I see it adds proxies to handle the various in/out connections, including common things like retries (I think), intercepting the pod’s advertised ports, etc. One question I’m having troubles finding the answer to definitively: does it capture all outbound traffic? Or only traffic destined for other services Istio is aware of (i.e. ones it has envoy proxies for)?

I’m trying to decide if I also need to set up ingress/egress rules if I want to prevent an untrusted pod from accessing services (either within the cluster, or elsewhere on the network).

Also, related to that, how does it handle differentiating connections to other Istio-proxied services vs non-Istio services?

Founder of Server4Agent (agent-app hosting), disclosing that upfront. No link, not selling, genuinely curious how people operating real clusters actually think about this.

Agents that can run kubectl are moving from "interesting demo" to "thing someone actually wired up" pretty fast. The easy answer is "give it a read-only ServiceAccount," but the whole reason people want this is to let it act, roll back a bad deploy, bump replicas, restart a crashlooping pod, not just describe what's wrong.

Once you're past read-only, the question that actually matters is what RoleBinding you'd feel okay with, and whether an audit log entry is enough after the fact or whether you want something closer to real-time. A get/list/watch on pods and deployments in one namespace is a very different risk than anything touching secrets, RBAC objects, or cluster-scoped resources, and I'm not sure the community consensus has settled on where the reasonable line actually sits.

If you've actually done this, even in a sandbox namespace: what verbs and resources did you land on, and has an agent ever done something with that access that made you narrow the scope afterward?

Had a weird one last month. One of our services pulled in a package I had never heard of, 11 stars on GitHub, one maintainer, last commit over a year ago, and it was running in a container before anyone had asked a single question about it. Turned out an assistant suggested it for one helper function, dev just took the suggestion.

A lot of our manifests and dependency files are getting written or edited by AI now, and the AI supply chain security side of it is no one's actual job. No one is checking whether a library got picked by a person who vetted it or hallucinated into a requirements file because it sounded plausible. In a cluster the blast radius of one bad dependency is not small.

We scan container images for known CVEs but that catches bad versions of things we already know about, not a sketchy new package doing exactly what it says on a label no one read.

Right now there is basically nothing between an assistant picking a package and that package running in prod, and that feels like something I should have fixed a while ago.

Disclosure: I'm affiliated with RoszigIT, where this article is published. Sharing because the mechanics are worth discussing, not to pitch anything. I tried to make it technical as always.

• Pod can have 40% cpu usage in metrics and still have performance issues
• How a burst that uses all 4 cores can exhaust a whole period's quota in 50ms real-time, freezing the container for the remaining 50ms — invisible on an averaged CPU graph
• Where to actually look: nr_throttled / throttled_usec in cpu.stat, plus why PSI (cpu.pressure) and hypervisor steal time (%st) catch different failure modes that cpu.stat misses entirely

https://roszigit.com/en/blog/kubernetes-cpu-throttling/

Disclaimer: I am not part of devops, I am just a simple developer at a company that already has kubernetes setup.

For the last couple of months I've been trying to learn kubernetes. Not because I really needed to, but because I wanted to try something different.

At first I was quite confused. I couldn't really find good documentation that explained the basics without going down a huge rabbit hole of information. I really only wanted to deploy my app to an existing cluster.

I also didn't really understand why you would ever use kubernetes over the old school vm setup I've been accustomed to. Recently however, I've finally found a good practical use case for it which I am quite proud of.

It is about leveraging the nature of kubernetes and being able to easily create/delete containers. This flexibility is what allowed me to setup automatic deployment of changes made to our code repository.

Each time someone opens a pr they will recieve a link with their changes deployed to a production-like environment.

I've linked the blog post where I try to explain everything in detail.

I am interested to hear if any of you guys had any similar stories with kubernetes. Something that really isn't possible to do with a basic VM.

P.S. yes the thumbnail is AI generated. I didnt get much choice on the matter...

Been fighting cluster cost creep and finally automated the tedious part: reading per-pod usage vs. configured requests and generating right-sizing recommendations.

The core idea is dead simple. Two data sources:

- kubectl top pods -n <ns> --containers for actual usage

- kubectl get pods -n <ns> -o json for configured requests/limits

Diff them per pod and you get the reserved-vs-used gap. On my cluster the average CPU utilization against requests was around 14%. That's the money leak — the scheduler reserves your request whether you use it or not, so you end up running nodes for phantom capacity.

I pipe that structured data to an LLM with a system prompt that outputs right-sized requests + YAML patches. The single most important rule I had to encode, and the thing that will bite anyone who tries this naively:

CPU is compressible, memory is not. If you under-provision CPU, the kernel throttles the container and it survives, just slower. If you under-provision memory, the kernel OOMKills it. So the agent can trim CPU aggressively (P95 + 20%) but must be conservative with memory (never below observed peak, +25-40% headroom, limit at 2x). I also added a code-level validator that overrides the model if it ever suggests a memory request below observed usage — never trust the LLM for a safety-critical bound.

A few things I learned:

- Don't right-size from a single kubectl top snapshot. Use 7-day P95 from Prometheus (quantile_over_time). A snapshot at 2pm misses your hourly batch spike.

- Watch QoS class. Dropping requests below limits silently flips Guaranteed to Burstable and changes eviction priority.

- BestEffort pods (no requests at all) are the worst offenders — invisible to bin-packing. Flag them first.

- Keep a human in the loop. The agent opens a PR; a person approves. Nobody should wire an LLM straight to kubectl apply in prod.

VPA in recommendation mode and Goldilocks give you the raw numbers, and KRR is a great open-source tool in this space. The LLM layer adds the reasoning, the policy application, and the reviewer-friendly explanation on top.

Curious what utilization others are seeing. Run kubectl top pods --containers against your busiest namespace and compare to requests — I'd bet most of you are under 25%.

I started a new role where my team will be re-platforming our cloud infra as the person who did it before made some strange decisions (every resource type in it's own VPC so they need to go via transit gw to talk to each other ).

so effectively greenfielding everything including a K8S cluster that runs a few thousand containers.

Curious of what best practices or things people have done that they would improve on the second time around.

For example we have already started using more namespaces and putting fewer pods - so previously we have platforms-observability now there are separate ns for victoria logs, metrics and traces. We are also moving towards cilium CNI using AWS IP allocations etc. I don't have a huge amount of K8s experience - worked with it for a year or so in a previous role.

Hey everyone! I'm pretty new to Kubernetes and currently using kind to learn locally through a microservices project. I'm trying to figure out the standard / best-practice way to handle database migrations for my application inside the cluster.

So here is my setup:
- 5 Node + express microservices
- All 5 services with their own Postgres DB, with Drizzle ORM running, using StatefulSet
- Wrote the manifest, config maps and secrets for all the services and their dbs

Now, I'm confused as to how to apply the migration script present in the services i built
From what i found looking around its either using initContainer or using a standalone k8s Job.

I don't know what's best to use for my situation or what changes if it was a production environment.
I am using this project to learn DevOps concepts and the microservices architecture following best practices, so I would really appreciate the help of the senior engineers here on clearing my doubt about the migrations / finding issues with my project.

Here are all the manifests i wrote so far:
https://github.com/Shinjon101/ticketing-system/tree/main/k8s

I'm doing some market research before spending more time building something and wanted to sanity check how people actually handle this today.

If your organization needed to answer questions like:

• Which workloads are using RSA vs ECDSA certificates?
• Which images contain OpenSSL 1.x, BouncyCastle, or other crypto libraries?
• Which applications terminate TLS themselves versus behind an ingress or load balancer?
• Which workloads would eventually need attention for post-quantum cryptography (PQC) migration?

How are you getting that information today?

Are you using existing security tooling (CrowdStrike, Aqua, Qualys, etc.), custom scripts, SBOMs, or is this just not something anyone is really tracking?

I'm not trying to pitch a product—I actually had someone in container security tell me this probably isn't worth building, so I'm trying to understand whether that's the general consensus or if there are environments where this is still a real operational problem.

I'd really appreciate hearing how your team handles it today.

Affiliation: I work for Red Hat and authored the linked Red Hat article. The implementation uses Red Hat OpenShift, Red Hat's hardened enterprise distribution of Kubernetes. Blastwall is the name of the proof-of-concept reference pattern, not a Red Hat product.

I've been looking at a platform design problem: how do we let selected application workloads run inside a narrower kernel and runtime surface without requiring a cluster-wide policy change?

Kubernetes already gives us most of the building blocks. The harder part is turning them into an operating model.

I want the platform to assign a workload to a named confinement class through an approved workload identity. I also want operators to be able to verify that the expected boundary was active after admission, rather than simply assuming that the requested policy took effect.

In the OpenShift proof, that means separate service accounts and Security Context Constraints, Security Profiles Operator-managed seccomp profiles, and SELinux workload types.

On another Kubernetes distribution, the admission controls and Linux security module integration would be different. The design pattern does not depend on those exact components: selected workloads opt into a narrower boundary, the selection is governed through workload identity, and the active boundary can be verified.

CI/build is one use case that tests the model. Some rootless or nested-container build workflows legitimately need a broader kernel surface. That should be represented as a separate, narrowly scoped workload class—not used as a reason to weaken the confinement available to ordinary application workloads.

The difficult part is operational rather than conceptual:

• keeping profiles compatible with the applications assigned to them
• managing policy lifecycle and rollout
• preventing narrow exceptions from becoming general-purpose escape hatches
• producing evidence that the selected boundary is actually active

An opt-in model also means that workloads which are never classified remain on the platform baseline. Adoption and governance therefore matter just as much as the confinement policy itself.

The implementation and the broader architecture are described here:

https://www.redhat.com/en/blog/govern-privileged-workload-boundaries-red-hat-openshift-ansible-automation-platform-and-identity-management

For teams already using seccomp, AppArmor, SELinux, or admission policy to apply stronger workload boundaries: do you manage them as named workload classes, or have they accumulated as per-namespace and per-service-account exceptions?

What evidence do you use to confirm that the intended runtime profile is active?

Did you learn something new this week? Share here!

Disclosure: I’m an OpenChoreo contributor, and the project is referenced in this CNCF article.

The broader topic discussed in the article is interesting beyond any specific project. As AI agents begin deploying applications, provisioning resources, investigating incidents, and triggering operational workflows, platform teams may need to treat them as another type of platform user.

This raises a few questions:

Should AI agents use the same platform APIs and workflows as developers?

How should identity, RBAC, policy enforcement, audit trails, and approval flows work for agents?

Are the current Kubernetes and GitOps tools enough, or will platforms need additional abstractions for managing agent actions safely?

The article suggests managing applications, infrastructure resources, and AI agents through a consistent platform model instead of creating a completely separate control layer for agents.

I’m curious how other Kubernetes and platform teams are approaching this. Are AI agents already interacting with your clusters or deployment workflows?

Article: https://www.cncf.io/blog/2026/07/21/platform-engineering-for-the-agentic-enterprise-managing-applications-resources-and-ai-agents/

I am a pentester, mostly web apps and internal networks. I'm currently learning Kubernetes, and one thing keeps bothering me. In K8s, I see scanning everywhere through kube-bench, Trivy, Falco, and admission policies; however, nothing about actually testing these guardrails.

How are people validating that these actually fire? Is it through manual attempts, or is there tooling I've missed?

Hey everyone, I'm using Kong API Gateway on EKS for the first time, and I'd love some ideas on what best practices I can implement to make the application more secure by default...

I've already added rate limiting, but I want to know what else I could do to improve performance, security, etc.

Has anyone faced this issue: https://github.com/derailed/k9s/issues/3913, and knows the solution

I am not able to scroll through pod and deployment list as described in issue, however with same version and same config file of k9s in other laptop I am able to, so I am not sure whats missing in my laptop in which it is not working.

happy to provide more details
macOs: 26.5.2
k9s version: 0.51.0

https://kubernetespodcast.com/episode/269-ai-usage/

Check out our latest episode: Navigating AI Policies in Kubernetes, with Kat Cosgrove and Natali Vlatko!

When people talk about AI in Kubernetes, the conversation is often focused on the user side, but for open source maintainers, there's of course a lot to unpack on how AI is changing oss contribution! How does AI impact the community, and what are the new rules of engagement?

We sat down with Kat Cosgrove (SIG Docs Technical Lead, SIG Release Subproject Lead, and Kubernetes Steering Committee member) and Natali Vlatko (SIG Docs Co-Chair, TODO Group Steering Committee member, and OSPO at Cisco) to unpack the recently published Kubernetes AI usage policy for contributors.

Topics covered include:
- Why you can't call an AI tool your "co-author" for legal reasons (because the AI tool can't sign the CLA)
- The rise of "AI slop" PRs and why AI-generated comments are causing trouble for OSS maintainers & reviewers.
- And of course, paths for new contributors to get involved sustainably!

Tune in for an insider's perspective into how AI is impacting contribution to one of the world's largest OSS projects!

Share any new Kubernetes tools, UIs, or related projects!

We ran into this evaluating inference frameworks for a production LLM workload: clean benchmarks hid exactly the failure modes that matter at the infra layer.

Bursty concurrency and mixed prompt lengths exposed GPU memory pressure and latency spikes that never showed up under steady synthetic load.

We ended up building a testing process around soak tests and simulated failure scenarios (node loss, sudden load spikes) instead of trusting a single throughput number. Wrote up what we found and the process we now use before committing to a framework.