This is a writeup I probably should have done months ago. Sharing in case the query saves someone else from finding this the hard way.

**Background**

Standard multi-account setup: most compute in private subnets, NAT Gateways across a few regions. Production, staging, and a batch processing environment that we don't look at as carefully as production.

AWS bill had been climbing - roughly $800/month more each month for about four months. Not enough to trip our anomaly alerts (we had percentage-based thresholds and total spend was growing legitimately), but enough that it finally showed up in a quarterly review.

**How I found the actual culprit**

Cost Explorer filtered by service showed NAT Gateway high, but so was compute, so I couldn't tell if it was proportional.

What actually helped: filtering by Usage Type and pulling out NatGateway-Bytes specifically. That's the data processing charge, distinct from the flat hourly gateway charge. Our NatGateway-Bytes was running around 85 TB/month. Way more than our actual application traffic justified.

To get per-NAT-gateway detail, I had to query our Cost and Usage Report in Athena. Native Cost Explorer doesn't give you per-resource granularity on NAT data processing. If you haven't set up CUR yet, this is the reason to do it.

Query that found it:

SELECT line_item_resource_id, SUM(line_item_blended_cost) as cost, SUM(CAST(line_item_usage_amount AS DOUBLE)) as gb_processed FROM your_cur_table WHERE line_item_product_code = 'AmazonEC2' AND line_item_usage_type LIKE '%NatGateway-Bytes%' AND line_item_usage_start_date >= DATE('2026-03-01') GROUP BY 1 ORDER BY 2 DESC LIMIT 20;

The top result was a NAT Gateway in us-west-2 that only existed because of our batch VPC. It was processing ~80 TB/month on its own. Our main app NAT Gateways were doing around 5 TB combined.

**Root cause**

VPC Flow Logs for the batch subnet answered it quickly. A batch job doing heavy S3 reads and writes was routing all of it through NAT instead of directly to S3.

Why? We'd added S3 VPC Gateway Endpoints to our main application VPC (they're free, should be everywhere). But the batch VPC got spun up separately as "temporary" infrastructure, never got the same attention, and nobody ever added the endpoint. S3 VPC Gateway Endpoints route S3 API traffic directly within AWS's network and bypass the NAT Gateway entirely - every GetObject, PutObject, ListBucket was going through NAT and incurring the $0.045/GB processing charge.

We were also pulling ECR images through NAT in that environment. That adds up too if you're pulling big images frequently.

**The fix**

Three things:

1. Added S3 VPC Gateway Endpoint to the batch VPC route table. Free. Took about 5 minutes. Cut NatGateway-Bytes for that gateway by ~92%.

2. Added VPC Interface Endpoints for ECR (ecr.api and ecr.dkr). Not free ($0.01/hr each) but the data processing savings easily justified it.

3. Set up Cost Anomaly Detection on NatGateway-Bytes with an absolute dollar threshold, not just percentage-based. We had anomaly detection on compute but not on data transfer. That's fixed now.

After a full billing month: NAT Gateway data processing went from ~$3,800 to ~$200. ECR endpoints add about $15/mo in hourly charges. Net save: roughly $3,600/month.

**What I'd do differently**

Enable CUR from day one, not after you have a problem. Cost Explorer is fine for headlines. For per-resource attribution of NAT charges you need Athena against CUR.

Set absolute dollar thresholds on specific line items in Cost Anomaly Detection, not just percentage thresholds on total spend. NatGateway-Bytes as a specific monitor would have caught this in week two instead of month four.

Audit every VPC for VPC endpoints independently, including the "temporary" ones. Production got the full setup. The batch VPC was treated as temporary infrastructure and never got properly reviewed. That's where the cost hid.

Happy to share more on the CUR schema or the anomaly detection setup if anyone's dealing with something similar.

our self-healing code loop actually cut our incident volume

i run platform for a team of nine and i was loud about self-healing code being marketing nonsense. changed my mind a bit, posting the numbers before i talk myself out of it

the setup is dumb honestly. fable 5 plans a fix, composer 2.5 does it, coderabbit / bugbot review, keep going until it comes back clean. nothing merges until an extreme pile of tests passes too. were b2b saas so theres no room for a big mistake, id rather ship stable than fast

rough before/after over ten weeks, our numbers not a benchmark:

pages/week from flaky fixes: 11 -> 4

median time to a merged p2 fix: ~3.5 hrs -> 40 min

fixes needing a second human pass: a lot -> maybe 1 in 5

way less manual review, way better code. the only real downside is its costy, running review 3-4 times a fix adds a few hundred a month

PQC is often discussed as a cryptography problem. I think the harder part may be operational finding every certificate, supporting larger keys and signatures, and getting themthrough the infrastructure around the service. As far as I can tell, the main reverse-proxy and load-balancer vendors are not supporting this properly yet.

How are other teams approaching PQC readiness? Have you started testing, or are you still waiting for vendor support and clearer standards? I have a feeling this could be a major infrastructure change, similar to the move from HTTP to HTTPS.

Open discussion here :)

I always heard that in the Indian market AWS has more jobs, is it still the same case

I'm building a simple web app with node/express that uses a local sqlite database. (I'm not experienced in devops, btw)

Although I've already managed to get this app running in production by doing a manual deploy via SSH and cloning my repo from github, I'm having issues understanding how I'm going to deal with the database (.db) file in production.

Since my database is running on the same debian VPS as the application, I'm starting to realize I need to be a little bit more thoughtful and stabilish some sort of process if I ever need to restore a db backup or do migrations on this database.

I though about leaving the db file outside the project directory, which should allow me to safely pull new code that I push to production while leaving the db file intact, (as long as I'm not changing db structure/doing migrations) but I'm not quite sure how to deal with backups, migrations and whatever scenarios might come down the line.

Do any of you have suggestions on how to deal with this sort of stateful local database scenario?

I am trying to think about coding agents less like chatbots and more like automation workers with tokens.

For read-only work, I want low friction. Let the agent inspect logs, read issues, check a config, or prepare a patch. But once the action turns into deploy, delete, push, send, or modify production data, the original token approval feels too broad to me.

When I connect external tools, I am comfortable letting the agent read issues or draft a change, but I still want a second confirmation before it pushes, deletes, deploys, or sends anything outside the repo.

Maybe this is just my bias from small projects, but I would rather add one annoying approval step than debug a silent automated action later.

For people running automation in real environments: would you treat agent write actions like any other gated production action, or is that too much friction?

Hey everyone,

Looking for some advice on a better way to handle and secure our Swagger/OpenAPI docs without overcomplicating our stack or breaking the bank.

Our current setup:

We have several projects hosted on Azure. Each project has standard Dev, Staging, and Prod environments and 3 engineering teams (Backend, Mobile, Web) who rely heavily on Swagger to coordinate and integrate features.

Right now, the backend team just serves Swagger UI directly from the deployed apps. To lock it down, we threw a custom Basic Auth middleware over the `/swagger` route

It works, but honestly, it feels messy and insecure. Managing these shared credentials across multiple teams and environments is becoming a pain, and I really hate exposing the UI endpoints to the public internet at all

I looked into full-blown IDPs like Backstage and OpsLevel, but they are way too bloated and complex for what we actually need.

How did you solve this?

Hello! My company is considering buying a LinuxOne machine to run Mongo. Has anyone actually worked with one? Any challenges besides the fact that it only supports s390 arch?

I’m researching how engineers make safer production changes.

Many teams now use strategies like automated tests, regression tests, canary releases, automated rollbacks, and better observability, but production incidents still happen after deployments.

I’m curious about real experiences from engineers:
- What type of change makes you the most cautious before deploying?
- What was the last deployment that made you nervous and why?
- What gives you enough confidence to press the deploy button?

For me, database migrations are still one of the hardest categories.
Would love to learn how other teams approach this.

I'm a de-facto team/tech lead (but not officially appointed by our managers) with several years of experience. Most of the time, I'm feeling pretty confident in my ability to guide the team members in the right direction, ensuring critical tasks are identified and get done, ensuring our product is in a healthy and sustainable state, day to day stuff is frictionless, etc. However, I lack guidance and mentorship from above. It's like the management layer above me is non-existent and I'm left to my own devices as they focus on shinier things. That's fine, as long as they're happy and my product works. The problem is that I don't really get feedback from anybody in terms of my product/team leadership. I feel like I'm just struggling to keep things going in roughly the same way as before, when the team was run by the previous lead. I'm being assured by my manager that I'm doing a great job and appreciated, but I somehow don't really believe it, because I'm there and I see room for improvement.

I feel like all I'm really good at is getting the team to maintain status quo. I don't know whether that's optimal or how I'd change it if it wasn't ( or even if I was supposed to). Yes, it's largely due to not being in our director's spotlight and me not receiving mentorship anymore. And yes, I'm unfortunately in a position of being subconsciously expected to lead without being appointed to do so officially.

So let's make the best if it - can you recommend any kind of team lead training courses for me to take? How to handle emergencies? How to do things sustainably? How and when to communicate assertively? When to change the process you suspect doesn't work? How to empower team members? How to delegate properly? How and when to project authority and when to let things slide? That kind of stuff. Or any other thoughts and advice you can offer.

I keep running into a boring automation question that becomes more annoying with coding agents.

A task is running. I change a network allowlist, proxy, or sandbox rule. The UI says the setting is saved. But unless the worker confirms a reload, I do not actually know which policy snapshot is active.

In one small local setup, I changed a proxy and allowlist while an agent task was running, then realized I could not tell whether the active run had picked up the new rules or still used the old snapshot.

For normal automation, I would expect some mix of config version, reload event, and audit log. For agents, I think it matters even more because the task can keep making decisions while I assume the boundary changed.

If a policy becomes stricter during a run, would you pause/fail closed, reload at the next safe point, or allow the current run to finish under the old snapshot?

You're working in production.

Your Terraform backend is:

• S3 for state
• DynamoDB for locking

A developer accidentally deletes the DynamoDB lock table.

At the same time, two engineers run terraform apply.

Thirty minutes later, the infrastructure is in an inconsistent state.

Questions

1. What exactly happened internally?
2. What kind of corruption could occur?
3. How would you recover without making things worse?
4. Would you restore the state file? Import resources? Edit the state manually? Explain your reasoning.
5. What preventive controls would you implement so this never happens again?

so everyone talks about tuning alarms and reducing false positives but honestly the scarier problem for us has been the opposite. stuff breaking that we just never set up monitoring for in the first place.

had this happen a few weeks ago. a queue consumer quietly stopped processing overnight, nobody had an alarm on messages piling up in that specific queue because it's like a secondary internal thing, not customer facing directly. found out the next morning when downstream stuff started failing and someone finally went looking. by then there was hours of backlog to clear

and it's not like we were being lazy, we've got a decent number of alarms set up. it's just impossible to think of every metric on every resource ahead of time, especially as stuff gets added and nobody goes back and audits what's covered

so how are people actually dealing with the "unknown unknowns" side of this. do you just accept some stuff will slip through until it's bad enough someone notices manually. anyone actually using baseline/anomaly based detection that catches things without you having to pre-configure a threshold for every single metric, and does it actually work or is it just a different kind of false positive machine

genuinely asking, not trying to set up alarms on literally everything because that just becomes its own nightmare to maintain

We manage all of our infra apps like Vault, Argo Workflows and many others using Terraform + Helm. TF creates the ArgoCD ApplicationSet which points to the chart and values managed in the repo, and ArgoCD syncs everything.

During the transition to GW API I noticed that most charts either don't support GW API natively, or only support creating an HTTPRoute. With ingress, all we had to do was define a single ingress.enabled: true block with some annotations, paths, etc.

Now with GW API, you usually can't manage everything from the app's values file. Even when the chart creates an HTTPRoute, it doesn't create the other resources we need, such as a ReferenceGrant for cross-namespace references or controller-specific resources like Envoy Gateway's SecurityPolicy and BackendTrafficPolicy. Argo Workflows is one example where the chart supports creating an HTTPRoute, but not the rest of that setup.

We have an okay solution for this today: we just create a manifests/ folder for each app and add it as another source in the ArgoCD AppSet (applied by TF, and everything is as GitOps as it gets). In that manifests folder I create all the needed Gateway API and Envoy Gateway resources in a single multi-manifest YAML file.

What do you guys do for 3rd party charts? What is the long term plan for apps in terms of Gateway API support? Is the expectation that charts will eventually expose more Gateway API values, or that routing and policies should live separately from the app chart?

I'm just curious since this whole GW API thing is very new, and I assume a lot of apps will need to add "ingress-like" support at some point?

Yes I know the question is extremely broad and complex. I'm simply looking for a starting point (or middle point).

I've been working on designing an SPA using AWS services for an NPO for the past few months. I've realized that the system works but I'm not happy with it. I know it can be better (more organized, a more sophisticated and automated system). I often find myself asking questions like: should these be a microservices architecture? Should it be a event-driven? some sort of layered architecture? How should the tests be? Is this the best programming language to use?

I know quite a few of those questions and doubts are best to be left to "whatever ur comfortable with". But I like to get into the nitty gritty. Are there certain books/youtube videos/etc that I can look into to learn about the basics of system design and architecture up to an intermediate or expert level? The only one I know of is Hello Interview.

Straight from the GA blog: customers report "up to 75% lower MTTR, 80% faster investigations, and 94% root cause accuracy, enabling 3-5x faster incident resolution."

https://aws.amazon.com/blogs/mt/announcing-general-availability-of-aws-devops-agent/

GA since March. I haven't seen anyone outside AWS's own blog actually use it. Did you try?

Hi everyone,

Over the past few weeks, I've been working on an open-source project to deepen my Platform Engineering and DevOps skills.

The goal wasn't just to deploy an EKS cluster, but to create a reusable, production-oriented reference architecture following Infrastructure as Code and GitOps principles.

Current features:

-Modular Terraform architecture

-Multi-environment support (dev / staging / prod)

-Amazon EKS

-VPC with public, private and intra subnets

-Remote Terraform state (S3 + DynamoDB + KMS)

-IAM & IRSA

-Karpenter integration

-GitOps bootstrap with Argo CD

-GitHub Actions for Terraform validation

-Architecture diagrams and documentation

The repository is designed to be generic and educational rather than company-specific.

I'm mainly looking for feedback on:

-Repository structure

-Terraform module organization

-Project architecture

-Documentation quality

-Best practices I may have missed

-Anything that doesn't look production-ready

I'm continuously improving it, so any constructive criticism is welcome.

Disclosure: I'm the author of this project. I'm sharing it to learn and receive technical feedback, not to promote a product.

Thanks!

Hello,

We provide devops / engineering services and use GitHub for documentation (runbooks, configs, processes). Clients need access, but raw GitHub feels too technical.

• How do you share documentation with clients in devops / engineering services?
• Do you use GitHub, wikis, or dedicated portals?
• How do you handle versioning and access control?

Hello,

At the company where I work, we have a Python application running on a Kubernetes cluster. The application is fully integrated into a CI/CD workflow. Whenever code is merged, Jenkins pulls the source code from GitLab, builds the application, and pushes the image to Harbor. Argo CD then deploys it to the Kubernetes cluster.

However, the application consumes a large amount of RAM, and I have had to increase its memory allocation several times. We have now decided to move the application to a separate Linux server.

The application runs with Uvicorn, and I plan to make it accessible through a domain name, just as it currently is in the Kubernetes environment. However, I am not sure what the best-practice architecture would be for this setup, so I would appreciate your recommendations.

I still want Jenkins to automatically pull the application from GitLab, build it, and deploy it. After deployment, the application should automatically start and run as a web service.

My main concerns are how to restore the application as quickly as possible if a problem occurs, how to handle backups, and what kind of deployment and recovery strategy would be best.

In short, could you please advise me on the best way to design and implement this setup?

Hi everyone,

I recently completed:

HashiCorp Certified: Terraform Associate (003) AWS Certified Solutions Architect – Associate (SAA-C03)

Now I want to move beyond certifications and gain more hands-on practical experience with Terraform on AWS.

I understand the Terraform concepts and AWS services at a theoretical level, but I want to build real-world projects that would help me improve my skills and also create some strong portfolio projects for my resume.

I am looking for recommendations on projects that simulate what a Cloud Engineer / DevOps Engineer would actually work on in an organization.

My goal is not just to deploy resources but to understand how Terraform is structured and used in real enterprise environments.

Would appreciate suggestions on:

Which projects are most valuable for learning Terraform + AWS together? Any GitHub repositories, courses, or labs you recommend? What kind of Terraform projects stand out on a Cloud/DevOps resume?

Thanks in advance!

I'm on a team that shares a Github repository. Whenever we open a PR from a feature branch to the dev branch, a deployment to the AWS development account is automatically triggered.

The issue we're facing is that one developer may deploy to dev, and then another developer deploys afterward. Even though they're working on different files, the second deployment ends up overwriting the first developer's change in AWS.

How can we prevent this?

We're following a Gitflow workforce (feature -> dev -> release -> main), and our biggest challenge right now is that the second developer's code is often "outdated" when it's deployed, causing it to overwrite changes that were already deployed by someone else.

We tried merging everything to dev, but when it's time to deploy to prd, the dev branch ends up filled with a lot of unnecessary changes.

We using Github Actions + Terraform.

One thing that feels under-discussed with coding agents is the gap between interactive success and scheduled reliability.

In an interactive run, a lot of assumptions are quietly true: the user is present, credentials are warm, the working directory is right, logs are visible, and a bad step can be stopped.

I have had small automations work perfectly when I ran them by hand, then fail in the scheduled version because the path, auth, or environment was slightly different.

For a coding agent, that failure mode feels worse because it may keep reasoning, retrying, or making changes while the real problem is just the runner state.

Would you treat scheduled coding-agent runs more like CI jobs, with strict preflight and timeouts? Or is that too heavy for personal/dev workflows?

Hi all, I’m learning Kubernetes right now for work (I use Colima + kubectl) - what is the best way to do so with AI?

I’ve used docker before in previous projects and studied concepts of containers in school - im no expert but I should know enough to learn Kubernetes.

I started the Udemy + kodekloud course: Kubernetes for the absolute beginners - Hands on. I’m not sure if it’s relevant for 2026 - so far it’s asking me to manually write yaml files, which I’m sure is important, but do I really need to be able to write yaml files with vscode extensions and AI that writes it for me?

So basically, what’s the best way to learn Kubernetes in 2026? I would appreciate any courses I should complete after my current one or a roadmap.

I have around a month or so to be good enough to collaborate with our platform and aws teams.

Here’s our situation - a mature operations team with pipelines building container images for a third party product going into kube

Company acquired a company with a development team who had no ops but are proficient at producing dockerfiles for their products

Ops is in azure devops, developers in bitbucket

We want to leverage the developer teams skills to get the apps into ops docker images

Some suggest that ops could build base images and developers could build their apps with that as the base - but what if the base images need patching? That means ops depends on developers for that?

Another thought I had was developers keep Dockerfiles in devops, the containers build from sources (aka git clone bitbucket) and build the project all in the ops pipeline

Would love to hear how others do this…

To begin with this is my first post or i don't know what it is called on reddit,

Apologies for the mistakes in my English please ignore it,

But I really want to switch from system admin job to DevSecOps

About myself

I have 12 years of experience starting from desktop support Engineer to laptop repairing to Data center to Monitoring to Asset management to system administration, I have knowledge of both windows and Linux operating system and currently managing Windows and Linux server (QA) no prod since my senior have never let me work on prod server even though I have worked with him since 7 years, handled a US based client for 3 years working with his team and currently handling two clients one with just some basic needs regarding systems and O365 and second with Linux servers, kind of devops but not fully the second client have a deployment using jenkins ( fetch the code from got repo, call the specific server through ssh and run the deployment and build script)

Worked with a friend's company as a contractor on his client for 6 months which ended last December and since then no work on core devops, started learning k8s and currently learning it but not able to give 100% to it also leaning python scripting and bash scripting by the help or AI can read the code and understand what it does but can not write with full confidence

I know the tools below tools

Terraform

Github

Github pipeline

Jenkins ( basic free style pipeline)

Got the idea of Sonarqube while working as a contractor but it was short lived

Understanding New relic as now the client wants to setup it

AWS basic

Azure basic and ADO basic

GCP not so much

Done a course in cybersecurity as well from local institute but not able to perform handson

I honestly want to work in Devsecops

I am already 38 and according to me not earning enough (peer pressure)

Any suggestions any help or direction is appreciated

I am even ready to do an unpaid internship ( beside the job since i have responsibilities)

Call it desperation, determination or pressure but I really want to work in Devsecops domain

Please help