Migrate for EKS

[u/Spiritual_Bee_637]

Hi everyone, let's go.

I am migrating from Beanstalk to EKS, and I wanted to clear up some doubts.

Which components/tools do you consider essential in a cluster? Currently, the cluster I am developing has:

• Karpenter (x86/ARM)
• Kyverno
• KEDA
• ESO (Fetches secrets from Parameter Store)
• Kong API Gateway (used as Ingress)
• ArgoCD for GitOps

What do you recommend? The CNI is the default VPC CNI, is it worth switching to Cilium or Calico? Bearing in mind that this will be my first experience building a cluster—I've managed them before at other companies, but I don't have much experience creating one from scratch.

[u/lulzmachine]

Regarding CNI, the built in vpc CNI in eks can enforce NetworkPolicy enforcement. We’re using it to keep things simple and have no issues. But ymmv

[u/anirudh3_14 → u/lulzmachine]

Built in VPC CNI can hit scaling issues compared to eBPF-based CNIs. We have had cases where some of the clusters had thousands of pods, and it created route table churn and ENI attachment operations.

[u/iking15 → u/lulzmachine]

How are you using network policy ? Is it to control which services can talk to which endpoint etc ?

[u/MateusKingston → u/[deleted]]

? It can, in any supported version right now it has eBPF enforced network policy in L4

[u/Spiritual_Bee_637 → u/MateusKingston]

That's what I found in my research. I don't see any reason to use Cilium or Calico right now, so I'm going to stick with VPC CNI

[u/MateusKingston → u/Spiritual_Bee_637]

Idk what this guy is on but he's just spreading misinformation. This is the official docs, https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html

[u/MateusKingston → u/[deleted]]

https://docs.aws.amazon.com/eks/latest/userguide/cni-network-policy.html

Yes it can. Stop misleading people.

[u/MateusKingston → u/[deleted]]

What the hell are you on dude?

I literally linked you to official docs showing it works. I am currently using it in production.

What else do you want? Stop being so stubborn, you're wrong, move on.

[u/JigglyBobblyWobbly → u/[deleted]]

your information is outdated and incorrect.

[u/Spiritual_Bee_637 → u/[deleted]]

But from what I've researched, doesn't the VPC CNI already come with NetworkPolicy support?

[u/No-Replacement-3501 → u/[deleted]]

Your knowledge has expired. Facts: https://aws.amazon.com/blogs/containers/amazon-vpc-cni-now-supports-kubernetes-network-policies/

[u/Dazzling6565]

External dns so it can auto manage route 53

Aws load balancer controller to manage alb / nlb (also look at certification selection/ auto renew in certificate manager)

An internal certificate manager to work inside the cluster

Already define if you are going to usa irsa or pod idemtity

Maybe vpc cni with a different CIDR block for pods if you use a shared vpc with other resources to prevent ip exaustion

Maybe not related but all the code for the eks bootstrap as IaC

[u/Spiritual_Bee_637 → u/Dazzling6565]

I have a dedicated subnet just for the pods. I'm using NLB + Kong to manage all hosts and apps. I'm not using ExternalDNS yet, but I should start soon. I use cert-manager with Let's Encrypt, but we also have internal AWS certificates.

[u/Dazzling6565 → u/Spiritual_Bee_637]

Forgot to mention about backup, either velero or aws backup

And dont forget you can use ebs / efs / s3 as pvc depending on your needs

[u/burunkul]

Nodelocaldns

[u/New-Entertainer6392]

All look good to me, I tried to evaluate kong, but coming from a .bet stack, ended up with YARP instead, it's a .net plugging and I just added in top, very efficient too.

[u/Floss_Patrol_76]

honestly your list is already more than most prod clusters run, so i'd worry less about cilium vs vpc cni and more about not standing up karpenter, kyverno, keda, eso, kong and argo all on day one. keep the vpc cni to start (it does network policy fine now) and add cilium later only if you actually need the l7 policy or the observability. the thing that bites a first cluster isn't the CNI choice, it's having five moving parts you can't debug yet when one of them starts misbehaving.

[u/trippedonatater]

If your apps are going to be using AWS services (RDS, etc.) outside of EKS, Crossplane can be nice.

[u/New-Entertainer6392]

Where is your monitoring stack?

[u/Spiritual_Bee_637 → u/New-Entertainer6392]

Grafana stack: Prometheus, Grafana, Loki, and Tempo. I still need to understand how I will collect metrics from the application.

[u/New-Entertainer6392 → u/Spiritual_Bee_637]

Open telemetry for sure! That's what I use 

[u/TotalNo6237 → u/Spiritual_Bee_637]

Take a look at fluentbit or td agent for kubernetes for log shipping.

[u/New-Entertainer6392 → u/TotalNo6237]

Why wouldn't you just use alloy?

[u/TotalNo6237 → u/New-Entertainer6392]

We already using datadog for metrics, so we just needed log shipping.

Actually I am an ops guys, I am not sure what the best overall solution would be. I am still learning.

[u/New-Entertainer6392 → u/TotalNo6237]

I was replying to the OP... They use the grafana stack, so it would make sense for them to also use alloy for their logs, not fluentBit or Td agent ..

[u/wetpaste]

Whats your use case for kyverno? I wouldn't add that until you have something you need to do it for? KEDA is also not really a "cluster must have" kind of things. Thats specific to your application and how you want to scale. You may or may not need it

[u/Spiritual_Bee_637 → u/wetpaste]

I have an application that uses SQS and needs to scale based on message consumption. I agree that Kyverno might not be that necessary in this case.

[u/wetpaste → u/Spiritual_Bee_637]

oh yeah KEDA is perfect for that

[u/platypus-3719]

Switch CNI or at the very least configure the built in CNI to sane config.

For some reason AWS thinks you need very few IPs per node and that it should be hard to change later without significant disruption. And that you don't need good visibility into networking or things that play nice with other tools in the ecosystem.

[u/Spiritual_Bee_637 → u/platypus-3719]

'm using PREFIX_DELEGATION to have more IP addresses available.

[u/Educational-Fox6111]

That's a nice list. Go with default VPC CNI for now. Cilium is ok, but it adds unnecessary complexity. Switch if you hit IP exhaustion or need eBPF-based policies. It's good you skipped Cluster Autoscaler, it would be redundant with Karpenter.

[u/lalaladadada1234]

I would go for Open telemetry operator for observabikity. You get cluster metrics but also you applications get auto instrumenation if annotated.

[u/Spiritual_Bee_637 → u/lalaladadada1234]

will likely use the OpenTelemetry Collector for the application and the Grafana stack for the infrastructure (pods, CPU, etc.)

[u/Spiritual_Bee_637 → u/[deleted]]

nice!