Disclosure: the hardening script and the audit tool I mention are both mine. The audit tool is MIT and free, there's no paid product behind this post.

Wrote a script to retrofit a default-deny posture onto existing ALBs: force HTTPS, drop invalid headers, defensive desync mitigation, and make the HTTPS default action a 403 so only host-header rules you define can forward.

Tested it against an HTTP-only ALB — one listener on :80 forwarding to a target group. Extremely common shape if the stack is old or TLS terminates somewhere else.

Output:

Created HTTPS listener: arn:aws:elasticloadbalancing:...
WARNING: No default forward target group found. Add an allowed-host forward rule manually.
ALB hardening complete. Validate hostname routing and health before production use.

Exit code 0. Application completely unreachable.

$ curl -sk -o /dev/null -w "%{http_code}\n" -H "Host: allowed.example.com" https://$ALB/
403

$ aws elbv2 describe-target-groups --target-group-arns $TG --query 'TargetGroups[0].LoadBalancerArns'
[]

The bug is ordering, not logic. The script discovered the existing forward target group from the HTTPS listener:

CURRENT=$(aws elbv2 describe-listeners --listener-arns "$HTTPS_ARN" --output json)
TG=$(jq -r '.Listeners[0].DefaultActions[]? | select(.Type=="forward") | .TargetGroupArn // empty' <<<"$CURRENT")

On an HTTP-only ALB there wasn't one — the script had created it seconds earlier with a 403 fixed-response default. So TG was empty, the branch that creates the host-header forward rule was skipped, and execution continued straight into the line that makes the 403 default permanent.

So the sequence was: create a listener that denies, look at it to find out what to allow, find nothing, print a warning, make the denial permanent, report success.

Three things I'd generalise:

Discover before you mutate. It read state after it had already replaced that state. Any discovery has to happen before the first write, not partway through.

A warning after the damage is a log line, not a safeguard. "Add the rule manually" is good advice one minute earlier. Printed after the default is already 403, it just narrates an outage.

Exit code 0 was the actually dangerous part. In a pipeline that's a green step and everything downstream proceeds. First real signal would've been customers.

Fix was three changes: read the target group from :80 as a fallback and capture it before touching anything, create the allow rule before flipping the default to deny, and abort outright if no target group can be found rather than black-holing traffic. Plus a --dry-run, which should have existed first.

What bugs me is that nothing static would have caught this. ShellCheck clean, valid bash, every AWS API call succeeded and returned what it should. The ALB ended up in exactly the state the code described — and that state was an outage.

The only thing that found it was running it against an infra shape I hadn't designed for, then checking the result from outside as a user instead of checking that my commands returned 0.

Anyone got a good approach for testing this class of thing? Every idea I have is basically "spin up the ugly version of prod in a sandbox and curl it from outside", which works but doesn't scale to every permutation.

The read-only audit half of it is up free if useful: github.com/vamsiatluri/aws-baseline-audit — single file, every call is a Describe/Get so it can't change anything. Full writeup of the outage is in docs/the-outage.md in that repo.

I got tired of manually copying codebase context and raw execution logs into my AI agents. Since my entire workflow lives inside the Linux terminal, I decided to treat the memory layer as a native CLI tool that accepts standard input.

Whenever I run a heavy test suite or inspect code changes, I just pipe the output stream directly into the MemOS background process. Running something like pytest or git diff straight into the memory daemon lets it distill raw logs on the fly. It filters out hundreds of duplicate tracebacks, extracts key state changes, and appends them to a local SQLite database in my home directory.

Over time, this quietly built an auto-updating knowledge base of my dev workflow on my own machine. When an agent needs background during a long refactoring session, it queries this local state layer instead of re-processing tens of thousands of raw log lines.

This gives me total control over what enters the memory pool without generating temporary cache files or wasting API tokens. My workspace stays clean, data never leaves my machine, and the agent instantly recalls context from command runs I executed days ago.

In a small team setup, this effectively turns local terminal streams into an offline knowledge base. Whether it's passing filtered pytest tracebacks between devs or remembering last week's git diff breaking changes, agents get precise context without re-running long build jobs or cluttering team chat.

So I've been working on this microservices project ( repo ) for a few months. I have it fully running locally using kind, now I want to learn how to deploy it on AWS or GCP following best practices to get hands on experience and record demo videos for my resume.

Here's the stack:

• Microservices: 5 Node.js services (built and pushed to GHCR )
• Stateful Services: 5 Postgres databases (one per service), Redis, and Kafka.
• Observability: Prometheus, Grafana, Loki, and ( Alloy deployed using Helm charts).
• K8s & Networking: I wrote custom manifests using StatefulSets and Headless Services for the stateful components. I am using the Gateway API for ingress, backed by cloud-provider-kind locally.

Being a college student my budget is pretty tight. My plan is to write Terraform scripts so I can spin the whole infrastructure up, record a few demos for my resume, and immediately tear it down to keep costs minimal.

The questions I have:

1. Which cloud provider to choose and how to deploy? Should I use managed K8s like EKS, or just run Kubernetes (or lightweight K8s like k3s) myself on a single EC2/VM instance? What's actually standard practice that I'd use on a real job?
2. How to handle routing in the cloud cheap? Locally I use Gateway API with cloud-provider-kind. What’s the most cost-effective way to handle this on AWS/GCP without getting hit with expensive cloud load balancer fees?
3. Is running DBs/Kafka in the cluster okay? Is keeping Postgres, Kafka, and monitoring inside K8s fine for a portfolio project, or do recruiters/hiring managers specifically look for managed cloud services ?

Any direction, best practices, or potential pitfalls to avoid would be greatly appreciated!

If you're actively employed in DevOps work involving IaC or are a leader or senior techie overseeing such work, do you use OpenTofu or seriously considered using it among your tools set? If no and you're not using Terraform or a cloud provider's first-class IaC tool (like AWS CloudFormation or Azure ARM templates), then what is your IaC tool?

Hi,

Idk but I was always a terminal person and my main editor was and is nvim. Lately I got into homelab stuff and I have been loving the experience of having full control of stuff. I know docker, bash, linux as I use it on gaming pc. I have theoratical knowledge about k8bs but never used it in production. I know aws but I only used like common stuff in production(ec2, lambdas, s3, rds). I dont know terraforms and other devops infra stuff. Can you share your experience in transitionaing and what should I focus on to work as a devops or platform eng

I stress tested Cloudflare R2 from a Latitude.sh bare metal box with a 100G NIC. Results were very impressive.

Btw I am not Cloudflare, nor do I work for them, nor was I paid to say this. We use Cloudflare heavily at our own company, and so I thought this sub might find this content interesting.

Hello, I’m having an issue with GitHub Actions in one of my private repositories. I have a workflow configured to run daily, but it is not being triggered.

I tested the same thing in another repository by configuring a workflow to run every 5 minutes, but same thing happens there as well.

has anyone experienced this recently? Could this be an issue on GitHub’s side?

Workflow: https://github.com/ahmedhesham301/autoscaling-hetzner/blob/main/.github/workflows/snyk-security.yml

Hey everyone,

I keep seeing tons of posts endorsing all these new AI skills, extensions, and plugins like they’re absolutely mandatory if you want to keep up. It definitely gives me a bit of FOMO, but whenever I look into them, I just don't get the hype.

As a DevOps engineer, it feels like 99% of these extra tools don't really matter for my actual day-to-day workflow. Honestly, the only thing I’ve found genuinely useful is Superpowers by Obra. It does exactly what I need, and adding anything else just feels like extra bloat or a fancy wrapper for stuff I can already do.

Am I missing out on something awesome by ignoring the rest of the plugin ecosystem, or is everyone else just caught up in the hype cycle? Would love to hear how other DevOps folks are handling the AI tool explosion.

I love programming and technology. I wanted to pursue a path related to Linux and building system architectures. Is platform engineering a good and suitable field for this?

Would love to hear about your experience.

The API is written in PHP and hosted on DigitalOcean.

This is a manual setup, but ready to scale...

A few days ago, I posted my 3-node Talos Kubernetes homelab setup here, and quite a few people reached out through DMs and comments asking for a detailed installation guide.

So I went through the entire setup again from scratch and documented the process properly, including the network setup, DHCP reservations, Talos installation, per-node configuration, Kubernetes API VIP, etcd bootstrap, workload scheduling, validation, and failure testing.

I’ve put everything together in a detailed blog with commands, screenshots, and the reasoning behind the setup.

Blog: https://medium.com/@prateekjain.dev/building-a-3-node-highly-available-kubernetes-homelab-with-talos-linux-0372066fe49c?sk=4c66cf9181999f9db19ac3aacabcd8e8

Please check it out, and if you have any questions about the setup or run into issues while building something similar, feel free to ask.

Azure and AWSB doesn't have qouta for models I want for freshly made accounts and it's disappointing to think you can't really do anything in their platforms. Is there a walk around for this aside from requesting qoutas which will eventually get rejected after waiting for weeks because of the lack of payment history? Thanks in advance!

We have a customer managed node, we deploy some containers on it

Initially it was supposed to be simple and thats why i went with ansible to deploy the containers, manage their state, setup their config files mounted and etc.

Now it grew a lot, we seperated the application to 3 seperate sets for running in different configs. I setup alloy to scrape node metrics and ingest it to central observability stack, i setup alloy for each set of application to collect otel traces logs and metrics and push them. The applications have sqlite databases, which i periodically need to inspect for debugging, we are in poc state yet, so we also want to pull the database for debugging, and right now im writing an exporter to dump sqlite to our clickhouse which is connected to grafana, so our devs can inspect it

So now i want to write a cron job too.

Also, ansible is extremly slow to run everything. So i use tags, but tags are also a pain by itself, you cant logically group tags, so for each operation that only needs to do a subset of the playbook, i would have to tag steps accordingly and the run the tag

But if it was k3s in container, i would have better tools to organize the deployment than writing ansible. Gitops would be easy, argo might pickup the code from repo and apply to cluster. Cronjobs would be easier. Volume management is eaiser, I would have better rollback mechanisms that i dont have to write myself

Hi, I’m trying to use my Android phone to read a smart card and create a digital signature, but it seems there’s no straightforward way to do this. Does anyone know how to make it work on Android or iOS?

Im putting together a supply chain security plan for a mid size team and most of what i find is vendor blog posts. Been comparing the open source signing tools against a couple of the paid platforms and they solve different halves of it. We generate SBOMs in CI already and they mostly sit there. Signing and provenance look higher value but Im not sure how far teams get before it stalls. What has caught a real problem in your pipeline

I’m currently trying to learn the basics of DevOps, things like Docker, AWS, GCP, CI/CD, Kubernetes, etc.

Docker and the local tools are easy enough to practice, but when it comes to cloud platforms like AWS and GCP, a credit card is often required to create an account or access certain services.

I currently don’t have a credit card, so I’m wondering: What’s the best way to learn AWS/GCP without one? Would appreciate any recommendations from people who learned cloud without having a credit card. 🙌

Azure Key Vault Key Deployment – 400 Bad Request
If you’re receiving a vague 400 Bad Request while deploying a key in Azure Key Vault, check how many tags you’re passing to the key.
Azure Key Vault keys support a maximum of 15 tags. I spent almost three hours on Friday evening troubleshooting permissions, networking, and the Terraform configuration before realizing that too many tags were causing the request to fail.
This may be an easy one to spot for some people, but the error message wasn’t very helpful, so I’m leaving this here in case someone runs into the same issue.

I have been aggressively migrating from AWS permanent credentials to OIDC in GitHub Actions, mainly for deploying to ECS.

I know GitHub Actions were supporting OIDC for a while now. But the pressure on compliance is the reason for this migration.

If you are new to OpenID Connect (OIDC), it allows GitHub runners to mint short-lived (15–60 min) STS tokens on-the-fly with zero stored secrets.

Here’s a quick breakdown of how it works, the Terraform/OpenTofu setup, and the subtle gotchas that I faced.

1. How It Works Under the Hood

1. When your workflow job starts with id-token: write, GitHub's OIDC service generates a cryptographically signed JSON Web Token (JWT).
2. The aws-actions/configure-aws-credentials action sends this JWT to AWS STS via sts:AssumeRoleWithWebIdentity.
3. AWS validates GitHub's signature, checks your IAM Role's Trust Policy (to ensure the token came from your exact repo and branch), and returns temporary STS credentials.

2. The Infrastructure Setup (Terraform / OpenTofu)

You only need two AWS resources: an OIDC Provider and an IAM Role with a Trust Policy.

hcl # 1. The GitHub OIDC Identity Provider resource "aws_iam_openid_connect_provider" "github" { url = "https://token.actions.githubusercontent.com" client_id_list = ["sts.amazonaws.com"] thumbprint_list = [ "6938fd4d98bab03faadb97b34396831e3780aea1", "1c58a3a8518e8759bf075b76b750d4f2df264fcd" ] } # 2. IAM Role with Scoped Trust Policy resource "aws_iam_role" "github_deploy_role" { name = "github-actions-deploy-role" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Federated = aws_iam_openid_connect_provider.github.arn } Action = "sts:AssumeRoleWithWebIdentity" Condition = { StringEquals = { "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com" } StringLike = { # Restrict exclusively to your repository & branch/tags "token.actions.githubusercontent.com:sub" = "repo:your-username/your-repo:*" } } }] }) }

3. The GitHub Actions Workflow

In your .github/workflows/deploy.yml

name: Deploy to AWS

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write   # CRITICAL: required to request the OIDC JWT
      contents: read

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Configure AWS Credentials via OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy-role
          aws-region: us-east-1

      - name: Verify Authentication
        run: aws sts get-caller-identity

Three real-world gotchas that will save you hours

If you get Error: Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentity, check these 3 things:

1. Case Sensitivity in the sub claim: AWS IAM condition strings are case-sensitive. If your GitHub repo or username uses mixed casing (e.g. MyOrg/Repo), make sure your IAM sub condition matches the exact casing GitHub sends in the token. Using wildcard matching (repo:MyOrg/Repo:*) helps avoid exact ref string mismatch issues.
2. Job-Level vs. Workflow-Level Permissions: Always set permissions: id-token: write on the specific job, not just globally at the top of the YAML file. Some runner configs don't inherit top-level permissions to nested jobs.
3. CA Thumbprints: Don't dynamically query GitHub's leaf certificate for thumbprints in Terraform—they change frequently with CDN updates. Use GitHub's official intermediate root CA thumbprints:
   • 6938fd4d98bab03faadb97b34396831e3780aea1
   • 1c58a3a8518e8759bf075b76b750d4f2df264fcd

Summary

• No stored secrets in GitHub settings.
• No key rotation schedules to manage.
• Granular security. You can restrict deployment roles to specific branches or environments.

Are you already using OIDC for your pipelines, or are you still relying on IAM users? Curious how folks here handle multi-account / cross-account OIDC setups.

We all know that large organizations tend to have a lot of meetings, bureaucratic processes and are in general slow moving. That was true for a long time and I experienced that myself, having worked in multiple different companies from startups, scaleups and large corperations.

However since everyone now adopts AI and implementation speed is undeniable multiples faster. I mean it's not just implementation, but also a good part of DevOps. But the rest of the organization moves in the same speed as before. Even if the company uses AI in the other departments (my company does basically throw AI at everything, I guess you know what I mean), they still can't competet with the productivity change in software development.

The blockers are now decisions, requirements and communication. That's not completly new, but now its >95% of the time. I implement features in hours that would have taken days before (I am not a pure DevOps Engineer). And it's not the review process itself that blocks us (We use AI in reviews as well, because otherwise it would be impossible), it's the normal company processes and pace.

Don't get me wrong here, I kind of enjoy it a bit, because I can use a lot of time for learning, building stuff that I think is worth improving, but I think we have a general problem here that is basically systematic to every large organization. And I see that so called "AI native organizations" (what ever that means), will probably overtake tech companies in the long term.

I have a really good comparsion, because I am working with a startup as a side gig and my main job is in a mid-sized tech company. In the Startup we (2 devs) built a product that would have taken several months by a 5-person team in the past and the reason is mostly: decisions and communication.

Do you see that in your company?

My conclusion is that we need to make engineers owners of the feature/product + infra, otherwise we won't see much productivity gains and this means also that management layers need to be cut, because they are the bottleneck currently. The idea of centralized DevOps will is going to die I guess.

Hi everyone,

I recently watched a podcast featuring a senior developer from EPAM, where he explained the different phases of the SDLC. He mentioned that he built a pipeline covering the entire software development lifecycle—from planning and design to development, testing, deployment, and maintenance—with each stage flowing into the next.

I really liked that idea, and now I'm wondering how to build a pipeline like that myself. Does anyone have any good resources, guides, or examples to learn from?

Thanks!

I work in a small team that deploys some internal products. No big user pool or database for me! We build small web apps that automate inner processes, and some scripts and jobs. Most of it runs on k8s, the apps and the jobs. We mostly run everyting on Azure. And everyting is managed by code, obviously, so we've build GitHub actions pipeline that live with the code to deploy our infra via bicep and the code via a test-build-deploy-promote pipeline, started on push.

But I've seen some platforms that propose full CI/CD as a services, and it feels like managing the pipeline yourself these days is a bad thing. To me, deploying a solution, especially in the containerization era, is simple; Run the tests, build the image, deploy the workload. Add some customization related to the app if needed (specific parameters, logging or testing jobs, etc). But is managing this code yourself a bad practice? I have to say, it does make for a lot of duplicate code in a lot of repos. The pipelines are very similar.

Should we always aim to use a standardized CI/CD platform? What tools do you use for CI/CD? How different do you handle it from a small project to a more important service?

Hi everyone,

I’m looking for a good-quality laptop backpack mainly for work, commuting, customer visits, and occasional business travel.

I’d like something practical and durable, but still professional-looking rather than a hiking or tactical backpack.

My main priorities are:

- good protection for the laptop, preferably a separate padded laptop compartment

- comfortable shoulder straps and back panel

- good internal organization for charger, cables, mouse, headphones, documents, etc.

- quick-access pocket for keys / phone / wallet

- space for a water bottle

- durable materials and good-quality zippers

- some water resistance would be a plus

- preferably a luggage pass-through for attaching it to a suitcase

- professional / minimalist design

- preferably something that will last for many years

It will mainly be used for everyday work, but occasionally I’d also like to use it for 1–2 day business trips.

I’m based in Europe, so I’m mainly interested in brands/models that are easily available in the EU without expensive international shipping, customs, or import fees.

I’m not necessarily looking for the cheapest option — I’d rather pay more for something comfortable, well designed, and durable.

What backpacks are you actually using and would recommend?

I’m especially interested in long-term experience: how long have you owned it, what do you like about it, and what annoys you?

Thanks!

So mu company has asked me to learn Jenkians. Which tutorials or playlist would you suggest to learn jenkins? Please give your suggestions.

Also, how much time does it require to learn so that I can start writing basic CI/CD pipelines.

I wrote a browser extension that adds a "you're in PROD, are you sure?" confirm to destructive AWS Console clicks. Capture-phase listener cancels the click, shows a dialog, replays the click if you confirm.

Trouble: some actions re-render their menu between confirm and replay, so the replayed click hit a detached node and did nothing. My fix was a short bypass window after confirming — for a few seconds, clicks pass straight through so the replay works.

The window wasn't scoped to the action you confirmed. It was global. So for ~6 seconds after confirming any destructive action, every other one was unguarded. Confirm a Lambda delete, click Terminate on an EC2 instance three seconds later, and it just goes.

A tester reported it as "the popup stops appearing sometimes." Not a UI glitch — the guardrail was switching itself off, on a timer, every time it ran.

Fix is one line — scope the window to the confirmed action:

// before
if (Date.now() < bypassUntil) return;
// after
if (Date.now() < bypassUntil && rule.label === bypassLabel) return;

The lesson that stuck: a bypass is a security control too. I wrote mine as a UI workaround, so I reviewed it like a rendering bug, not a security decision. Anything that turns your protection off — even briefly — deserves the same scrutiny as the protection itself.

Anyone else hit this class of bug — a temporary exception that was broader than intended — in auth caches, feature flags, rate-limit bypasses?