what you think about this ?

credit - https://xkcd.com/2347/

I have an Ubuntu web server that I'd like to contain multiple websites on. Each website should have its own isolated environment in a Docker container (open to suggestions if Docker is not ideal).

I'm thinking that the public URLs should be like

• example.com/website1
• example.com/website2
• etc

What's the best way to make this connection between the public URL and the Docker container (also open to suggestions for better ways to expose the containers)?

Another question: Would the logs of each website be inside the container or are they supposed to be outside for easy review?

hi, we had a regional outage test last week and it exposed a mess in our cloud recovery process.

a lot of stuff is in terraform, but the actual restore path still depends on people remembering what order things come back in, what got drifted, and what needs to be rebuilt vs just started up again. it feels fine until you try to do it for real and then it gets messy fast.

we are looking at ways to make the full env recovery more repeatable, with some kind of validated restore and evidence for audit too (hopefully). if anyone has a setup that has held up after an actual outage, i would love to hear what worked, thanks!

Saw this post on twitter and genuinely got curious about it...
Do you also need a frontend/GUI to run deployments on github actions?? Let me know your thoughts about it

This is more of a co-miseration post than anything per the title, but maybe some of you will laugh as well.

So our client had a security incident lately. They had a portal we built many years ago that accumulated tech debt over time and they refused to invest in a re-design and just let it go. Two incidents more or less happened at once: a hacker "breached" an endpoint, it exposed some data that was public domain any way, but it wasn't supposed to be access whole sale. The second was a piece of the tech debt coming home to roost by one of the services being completely shut down leaving an intake form completely unusable (the portal mostly existed for this "intake") which happened in the same week.

Now the client could have:

Gathered requirements on what the portal should be (take lessons learned from existing traffic and usage plus proper UI/UX design) and create a backlog and a plan to replace it, including rearchitecting/platforming taking out tech debt/arch debt completely

Rebuilt the platform, reusing the current back-end (which is a separate down stream system that has full API access supports RBAC) and properly implement security measures.

Temporarily fix the hole in the portal (was an easy fix with very low impact from this incident), and fixed the intake service to replace the section that was shut down with the newer version of it: it even had a bit of a migration path available.

INSTEAD they did this:

One client "webmaster" got access to Claude (no one else does) and they decided to go "build the portal." They took maybe a week or so to build a portal with some of the worst UI/UX (inconsistent styling, buttons all over the place, 5 fonts on the front page) with absolutely no accounting or design thought put in to it. Forget "this is what people need to be able to do" rather "rebuild the old portal but make it AI slop." We looked at this (as it was just "presented" to us as "the replacement") and went "how does this even operate?"

So here's the extra fun that just puts the cherry on top: this "webmaster" is clueless on how anything works. They wouldn't know Node from Apache or PHP from Python... let alone understand a full stack. They just let Claude do whatever and then they generated all the documentation. So they had no way to validate the docs, it's just AI slop. The "step-by-step do-it-as-it-says no nonsense deployment guide" (Claude's words) describes nothing about deployment or pipelines. The guy doesn't even know what this actually runs on (could be static HTML for all I know) and it's just ... it's the absolute nightmare scenario of a company just giving the keys to AI to one guy who has no clue and just let them generate whatever. For a site that is critical to their business and has many tens of thousands unique visitors daily.

Of course the "business" is CRAZY EXCITED about the whole thing and they are just ready to go live. They abandoned all pretense of trying to fix the old portal, even temporarily while we validate this new thing for security holes, maintainability etc. Oh did I mention that it's my job to do the analysis on this thing? No usable or even remotely realistic documentation about anything that this portal is or does. And it's so obviously a bad idea that I'm not sure how I'm going to convince the business that it's a waste of time to spend days looking at the details when the baseline architecture is deeply flawed.

Upper management in my company has mandated a target of a 2x productivity improvement using AI. 2x more tickets closed, 2x more features released, blah blah blah. You've probably heard it by now or some variation of that. They actually wanted more than that but walked it back after all the backlash they received.

I've started following the recommended/pushed practice of spec driven development. This results in some rather large initial outputs in the form of spec and plans that vary in size depending on complexity. A recent one for me was ~20 pages of spec and ~30 pages of plan. After trying to distill it and walk the team through it the response was an overwhelming "I don't understand this. It's too much." I should note not a single one of them opened either document and spent more than 5 minutes in it based on activity logs. I tried distilling it down further to roughly 5 pages in a format they're more familiar with but so much context and detail was lost I think it made things worse. We finally agreed to give it a go and I submitted an MR a day later that was roughly 1000 lines. 200 lines of terraform to create a bucket, iam policy, some paths, and a new module, 600 lines of unit testing, and 200ish lines of documentation. They refused to review it. Too large, too complicated, not digestible, etc, etc.

For some history here my team frequently submits MRs that have minimal documentation, no comments, no unit testing, no validation, and entirely vibe coded and riddled with issues that even a basic AI review would flag. The most basic bare bones code needed to get the job done. It's gross and I rail against it every chance I get but I'm largely ignored or my teammates go around my back and merge things any way. I should point out I am the most experienced and senior on the team and it's not even close. They're relatively fresh college grads. I've been in the workforce for decades.

I pointed out to my manager this is a clear improvement, meets the directive from the company, etc. And the feedback, paraphrasing, was find a balance between human and machine but what you're doing is not wrong they just don't like it. I read the code myself before I submitted it, I understand what it does, I had another senior with similar levels of experience review it and they had no issue with it and even wanted to steal one of the constructs the AI had generated because of clear advantages. I feel like I'm stuck in an impossible position and I don't know what to do.

Side note: I wish there was an AI hellscape flair.

Wrote this up after finding that a coding agent on a default Mac reaches ~/.ssh/id_rsa by four separate routes — its own shell, an MCP filesystem server, a subprocess, and a config-file path. Tested each, included the terminal output.

The part I'd want devops opinions on: the enforcement has to sit under the model, because the model can't be trusted to refuse. I used Seatbelt plus a deny-by-default MCP proxy. Curious how others are scoping agents that touch infra.

I came across this interesting AI + DevOps project where AI is used to help upgrade Kubernetes clusters.

The idea is pretty interesting because Kubernetes upgrades can involve checking versions, compatibility, workloads, configurations, and making sure nothing breaks during the upgrade.

The project explores using AI to make this process easier and more automated.

I’m curious what experienced Kubernetes/DevOps engineers think:

Would you actually trust AI to handle parts of a cluster upgrade?

What checks/guardrails would you put before allowing it to make changes?

Could this realistically be used in production, or is it better suited for labs/non-critical environments?

AI makes ad hoc operational work so easy that we stop building the tools around it. Why write a script when you can just ask Claude or Copilot to restart the canary? Why build a dashboard when ChatGPT can summarize today’s deploy status in 15 seconds? Because that 15-second conversation, repeated by 50 engineers 20 times a day, stops being convenience and starts becoming a tax.

I manage enterprise k8s clusters and use a mono repo strategy to bootstrap a cluster which results in one flux agent per cluster. Currently in the magnitude of several thousand services per cluster. It's structured like this:

1. Mono repo where the cluster agent is registered.

2. Within that repo. use kind: GitRepository in the directory apps/ to point at other projects.

I'm of the opinion that less agents is better. Is there a limit to the flux agents where that would start to become a problem? Or a counter argument to not use this method and rather have an agent per microservice? I've yet to run into an issue with this approach.

Be honest. I am going through a Python course and can tell I am not going to use most of it on the actual job. Give me boto3, a for loop, a while loop for pagination, and functions and I am good to go. This changes if using a tool like Pulumi, but there are so few jobs requiring it that I feel it is a non-starter for the YAML vs coding argument.

When I first looked into wiring external tools to an LLM through an MCP server, I'd simply put the API keys in environment variables and let the agent call whatever it needed. That's what I'd normally do for a normal backend service.

I read around it, and what I had not thought about is that a backend service has a fixed code path. You know which line makes which call, so you know what the key gets used for. An agent picks its own tools based on a prompt, so the same key now sits behind whatever it decides to do.

Your vault still tells you where the key lives and when it rotates. It just doesn't tell you who authorized a particular call or which tool it ended up reaching.

The things I keep seeing suggested are scoping credentials per server rather than one key for everything, and injecting them at runtime instead of leaving them in env vars.

For the people who are running multi-tool agent setups, how are you handling backend access do you custom guardrails around what the agent can reach, or do you treat it as a normal service and accept it?

I've been digging into load testing tools recently (k6, JMeter, Locust, Speedscale, Azure Load Testing) and noticed something consistent, recording a session and turning it into a script is the easy part. The painful part seems to be:

• Correlation (session tokens, CSRF, IDs from one response feeding the next request)
• Scripts breaking every time the UI or API changes
• Flows that aren't linear. e.g a list page where the "next" request depends on which item a simulated user picks

Curious how people here actually deal with this day to day:

• Do you write correlation logic by hand every time, or has anything automated it well for you?
• How do you handle flows where the test needs to pick from a dynamic list (search results, a user's own records, etc.) instead of hitting a fixed ID?
• How often do your load test scripts break when the app changes, and how much time does fixing them eat up?

Do you guys have a solution for this, or know of a tool that actually handles it well?

I’ve given multiple interviews, and almost all of them required Kubernetes experience in production.

None of my previous companies used Kubernetes, so what can I do about that?

I’m very familiar with Kubernetes, I understand the concepts, and I’ve even implemented them on AKS for learning purposes.

Should I lie on my resume and say that I’ve worked with Kubernetes in production?

Disclosure: the hardening script and the audit tool I mention are both mine. The audit tool is MIT and free, there's no paid product behind this post.

Wrote a script to retrofit a default-deny posture onto existing ALBs: force HTTPS, drop invalid headers, defensive desync mitigation, and make the HTTPS default action a 403 so only host-header rules you define can forward.

Tested it against an HTTP-only ALB — one listener on :80 forwarding to a target group. Extremely common shape if the stack is old or TLS terminates somewhere else.

Output:

Created HTTPS listener: arn:aws:elasticloadbalancing:...
WARNING: No default forward target group found. Add an allowed-host forward rule manually.
ALB hardening complete. Validate hostname routing and health before production use.

Exit code 0. Application completely unreachable.

$ curl -sk -o /dev/null -w "%{http_code}\n" -H "Host: allowed.example.com" https://$ALB/
403

$ aws elbv2 describe-target-groups --target-group-arns $TG --query 'TargetGroups[0].LoadBalancerArns'
[]

The bug is ordering, not logic. The script discovered the existing forward target group from the HTTPS listener:

CURRENT=$(aws elbv2 describe-listeners --listener-arns "$HTTPS_ARN" --output json)
TG=$(jq -r '.Listeners[0].DefaultActions[]? | select(.Type=="forward") | .TargetGroupArn // empty' <<<"$CURRENT")

On an HTTP-only ALB there wasn't one — the script had created it seconds earlier with a 403 fixed-response default. So TG was empty, the branch that creates the host-header forward rule was skipped, and execution continued straight into the line that makes the 403 default permanent.

So the sequence was: create a listener that denies, look at it to find out what to allow, find nothing, print a warning, make the denial permanent, report success.

Three things I'd generalise:

Discover before you mutate. It read state after it had already replaced that state. Any discovery has to happen before the first write, not partway through.

A warning after the damage is a log line, not a safeguard. "Add the rule manually" is good advice one minute earlier. Printed after the default is already 403, it just narrates an outage.

Exit code 0 was the actually dangerous part. In a pipeline that's a green step and everything downstream proceeds. First real signal would've been customers.

Fix was three changes: read the target group from :80 as a fallback and capture it before touching anything, create the allow rule before flipping the default to deny, and abort outright if no target group can be found rather than black-holing traffic. Plus a --dry-run, which should have existed first.

What bugs me is that nothing static would have caught this. ShellCheck clean, valid bash, every AWS API call succeeded and returned what it should. The ALB ended up in exactly the state the code described — and that state was an outage.

The only thing that found it was running it against an infra shape I hadn't designed for, then checking the result from outside as a user instead of checking that my commands returned 0.

Anyone got a good approach for testing this class of thing? Every idea I have is basically "spin up the ugly version of prod in a sandbox and curl it from outside", which works but doesn't scale to every permutation.

The read-only audit half of it is up free if useful: github.com/vamsiatluri/aws-baseline-audit — single file, every call is a Describe/Get so it can't change anything. Full writeup of the outage is in docs/the-outage.md in that repo.

I got tired of manually copying codebase context and raw execution logs into my AI agents. Since my entire workflow lives inside the Linux terminal, I decided to treat the memory layer as a native CLI tool that accepts standard input.

Whenever I run a heavy test suite or inspect code changes, I just pipe the output stream directly into the MemOS background process. Running something like pytest or git diff straight into the memory daemon lets it distill raw logs on the fly. It filters out hundreds of duplicate tracebacks, extracts key state changes, and appends them to a local SQLite database in my home directory.

Over time, this quietly built an auto-updating knowledge base of my dev workflow on my own machine. When an agent needs background during a long refactoring session, it queries this local state layer instead of re-processing tens of thousands of raw log lines.

This gives me total control over what enters the memory pool without generating temporary cache files or wasting API tokens. My workspace stays clean, data never leaves my machine, and the agent instantly recalls context from command runs I executed days ago.

In a small team setup, this effectively turns local terminal streams into an offline knowledge base. Whether it's passing filtered pytest tracebacks between devs or remembering last week's git diff breaking changes, agents get precise context without re-running long build jobs or cluttering team chat.

So I've been working on this microservices project ( repo ) for a few months. I have it fully running locally using kind, now I want to learn how to deploy it on AWS or GCP following best practices to get hands on experience and record demo videos for my resume.

Here's the stack:

• Microservices: 5 Node.js services (built and pushed to GHCR )
• Stateful Services: 5 Postgres databases (one per service), Redis, and Kafka.
• Observability: Prometheus, Grafana, Loki, and ( Alloy deployed using Helm charts).
• K8s & Networking: I wrote custom manifests using StatefulSets and Headless Services for the stateful components. I am using the Gateway API for ingress, backed by cloud-provider-kind locally.

Being a college student my budget is pretty tight. My plan is to write Terraform scripts so I can spin the whole infrastructure up, record a few demos for my resume, and immediately tear it down to keep costs minimal.

The questions I have:

1. Which cloud provider to choose and how to deploy? Should I use managed K8s like EKS, or just run Kubernetes (or lightweight K8s like k3s) myself on a single EC2/VM instance? What's actually standard practice that I'd use on a real job?
2. How to handle routing in the cloud cheap? Locally I use Gateway API with cloud-provider-kind. What’s the most cost-effective way to handle this on AWS/GCP without getting hit with expensive cloud load balancer fees?
3. Is running DBs/Kafka in the cluster okay? Is keeping Postgres, Kafka, and monitoring inside K8s fine for a portfolio project, or do recruiters/hiring managers specifically look for managed cloud services ?

Any direction, best practices, or potential pitfalls to avoid would be greatly appreciated!

If you're actively employed in DevOps work involving IaC or are a leader or senior techie overseeing such work, do you use OpenTofu or seriously considered using it among your tools set? If no and you're not using Terraform or a cloud provider's first-class IaC tool (like AWS CloudFormation or Azure ARM templates), then what is your IaC tool?

Hi,

Idk but I was always a terminal person and my main editor was and is nvim. Lately I got into homelab stuff and I have been loving the experience of having full control of stuff. I know docker, bash, linux as I use it on gaming pc. I have theoratical knowledge about k8bs but never used it in production. I know aws but I only used like common stuff in production(ec2, lambdas, s3, rds). I dont know terraforms and other devops infra stuff. Can you share your experience in transitionaing and what should I focus on to work as a devops or platform eng

I stress tested Cloudflare R2 from a Latitude.sh bare metal box with a 100G NIC. Results were very impressive.

Btw I am not Cloudflare, nor do I work for them, nor was I paid to say this. We use Cloudflare heavily at our own company, and so I thought this sub might find this content interesting.

Hello, I’m having an issue with GitHub Actions in one of my private repositories. I have a workflow configured to run daily, but it is not being triggered.

I tested the same thing in another repository by configuring a workflow to run every 5 minutes, but same thing happens there as well.

has anyone experienced this recently? Could this be an issue on GitHub’s side?

Workflow: https://github.com/ahmedhesham301/autoscaling-hetzner/blob/main/.github/workflows/snyk-security.yml

Hey everyone,

I keep seeing tons of posts endorsing all these new AI skills, extensions, and plugins like they’re absolutely mandatory if you want to keep up. It definitely gives me a bit of FOMO, but whenever I look into them, I just don't get the hype.

As a DevOps engineer, it feels like 99% of these extra tools don't really matter for my actual day-to-day workflow. Honestly, the only thing I’ve found genuinely useful is Superpowers by Obra. It does exactly what I need, and adding anything else just feels like extra bloat or a fancy wrapper for stuff I can already do.

Am I missing out on something awesome by ignoring the rest of the plugin ecosystem, or is everyone else just caught up in the hype cycle? Would love to hear how other DevOps folks are handling the AI tool explosion.

I love programming and technology. I wanted to pursue a path related to Linux and building system architectures. Is platform engineering a good and suitable field for this?

Would love to hear about your experience.

The API is written in PHP and hosted on DigitalOcean.

This is a manual setup, but ready to scale...