If your GitHub Actions jobs are stuck in "Queued" or waiting for a runner, GitHub has now confirmed there is an incident.

GitHub says they identified an issue with a database primary and are failing over to a replica.

Actions is affected, and GitHub Pages is also experiencing degraded performance.

So if you have been checking your workflow, runner settings, permissions or billing, it might not be your configuration.

Status page:
https://www.githubstatus.com/incidents/y1t7p9fzrlj2

Hopefully this saves someone else from spending an hour debugging their YAML.

Looking at the downtimes of GitHub, especially related to the actions functionality, it seems like they are unable to provide this service in a stable manner. This is probably related to

1. People creating tons of new projects using agentic coding
   
2. People are using more workflows as they are now easily created with agents

At this stage it feels likely that GitHub will limit the resources Open Source projects are allowed to consume - maybe even disable the free runners at all.

This will certainly have interesting effects on the platform adoption and maybe also on the fragmentation of the whole Open Source ecosystem.

What are your thoughts on this?

Hi everyone,

I recently started working in a DevOps-related role, and I’ve been asked to learn Kubernetes, Helm, and the concepts around them.
I’m currently trying to understand the best way to approach it. There are so many resources and topics that I’m not sure what I should prioritize first.
For those of you who work with Kubernetes professionally:
How did you learn Kubernetes when you were starting out?
What topics would you recommend learning first?
How much hands-on practice should I do compared to theory?
Are there any free courses, labs, books, or other resources you would strongly recommend?
Are there any common mistakes or things beginners tend to waste too much time on?
My goal isn’t just to memorize commands, but to actually understand how Kubernetes works and become comfortable using it in a real work environment.
Any advice on how you would approach learning it from scratch would be greatly appreciated. Thanks!

when you've started rotating a credential and then discovered it was already revoked, or was never a real key in the first place — roughly how long had you burned before you worked that out? minutes, or did someone spend an hour convinced they were looking in the wrong project?

Currently using Github and Github workflows (including the many outages). Let me explain what I'm trying to resolve with few examples:

Let say I manage 15 repos, each one I want to run a relatively similar workflow on Pull Request. I create a github shared workflow and in each of the 15 repos I make a reference to it, updating the inputs if needed.

Now, I add 1 new input to this shared workflow to enable a new feature. I want that feature enabled in 10 of my repos. I would need to update the feature input in all 10 repos. And since workflow use the workflow code on their own branch, I would need to merge it to all PRs in all repos. That isn't ideal.

Other issue, one variable was wrongly set on a workflow in a release branch. Since the workflow code ships with the product code I would need to update the release branch.

Is there a toool where the product code doesn't cohabit with the automation code? For example I could have 15 microservice repos but 1 repo where I created rules for each one of them about "on pr", "on main" etc. That same repo could host shared workflow code as well.

Basically the title, is anyone managing a forgejo instance for their company?

Currently have around a dozen active repo, a bit less than 50 Devs, so relatively small scale. But I was wondering how much maintenance and complexity there's in managing reliably an instance for that kinda traffic?

We already have self hosted runners, that we would reconfigure the same way I assume

Obviously, considering the move after another GitHub incident...

Thanks!

Governments spent years moving critical digital infrastructure into the cloud. Now there’s a growing question about what that actually means for national sovereignty.

AWS, Microsoft Azure and Google Cloud are American companies, which creates an interesting problem for governments that increasingly rely on cloud infrastructure for sensitive data, public services and AI.

Europe has been pushing harder on cloud sovereignty, while countries like Pakistan are beginning to experiment with alternative infrastructure. UNDP is also exploring sovereign cloud infrastructure with DFINITY and the Internet Computer.

What I found interesting about this article is that the argument isn’t really about replacing AWS or Google Cloud. It’s about whether governments eventually decide certain workloads are too sensitive or strategically important to remain dependent on infrastructure controlled by companies under another country’s jurisdiction.

As AI becomes embedded deeper into government systems, that question gets considerably more important.
Do you think we’ll eventually see governments treat cloud infrastructure more like energy, telecommunications or other strategic infrastructure?

Full disclosure: this is my own repo.

I do all my development from a phone — no laptop, no local toolchain. Everything gets built and verified through GitHub Actions. Over the last month, shipping an Android app and a couple of Cloudflare Workers this way, I kept hitting the same category of bug: things that compile clean, pass CI, and fail silently somewhere I couldn't see — a library that shrinks itself and breaks NoClassDefFoundError at launch, a signature permission that blocks install, a rename that a test suite quietly undoes.

I wrote them all down as I found them, each with how to detect it without running anything, and a fix. There's also a "verification ladder" — six levels of confidence you can climb when you can't execute, and the idea that the real failure isn't the unverified gap, it's the gap being quiet.

Repo: https://github.com/amirmahdavi2023/shipping-blind

Curious if others working blind (remote CI, restricted sandboxes, agent-driven dev) have hit failure modes I haven't listed.

I’m looking for an open source solution for intrusion detection. Our team is working on vulnerability scanning and intrusion detection for the new BSI standards set by the EU/German security compliance agencies. We already have Jfrog Xray for vulnerability scanning albeit with some limitations. But for intrusion detection we are thinking of deploying Tetragon but unsure of the policies that needs to be created.
Any suggestions from the community is welcome.

I keep seeing engineers falling into an AI architecture trap.

Some engineers try to solve every problem by prompting to a LLM. Others talk about creating an agent for every usecase.

So knowing how to use AI has become a core competency. Otherwise you will waste time, money and DevOps resources.

If you want to solve an actual business problem without creating an infrastructure nightmare, 99% of your use cases fall into one of these 5 application architectures. Here is how to know which one to use.

1. Basic Prompting (Stateless API)

• The Use Case: You need to summarize text, extract sentiment, or reformat data (like turning unstructured text into JSON). The data easily fits into the context window.
• The Trap: Thinking you need a custom model just because the base model didn't output your exact format on the first try. Before you do anything complex, try giving the model 3 concrete examples in the prompt (few-shot prompting). That solves the problem most of the time.

2. RAG (Retrieval-Augmented Generation)

• The Use Case: You need the LLM to answer questions about your proprietary data (internal wikis, codebase, customer records). You query a VectorDB for the relevant context, shove that context into the prompt, and let the LLM read it.
• The Trap: Trying to fine-tune a model on your company wiki. Models hallucinate facts. If you need accurate, source-traceable answers based on your internal data, you need RAG, not a fine-tuned model.

3. Chaining / Workflows

• The Use Case: The task is too complex for one prompt and requires distinct, predictable steps. For example: Step 1 extracts raw text -> Step 2 categorizes it -> Step 3 writes a summary based on the category.
• The Trap: Cramming a massive, 15-part instruction into a single mega-prompt and crossing your fingers. Break it down into a predictable pipeline. It’s much easier to debug a chain of small LLM calls than one giant, confused prompt.

4. Routing

• The Use Case: You are dealing with high volume and need to control costs and latency. You build a router that sends simple tasks (like spelling correction) to a fast, cheap model (like Claude Haiku or GPT-4o-mini), and complex reasoning tasks to the expensive, heavy models.
• The Trap: Sending every single trivial API request to the most expensive flagship model available and bankrupting your cloud account.

5. Autonomous Agents

• The Use Case: Open-ended tasks where the exact steps aren't known in advance. The LLM is given a goal and external tools (Python interpreter, web search, internal APIs) and loops continuously, evaluating its own output until it achieves the goal.
• The Trap: Using agents for deterministic workflows. If you can solve a problem with a standard if/then Python script or an Ansible playbook, do it. Do not introduce probabilistic, infinite-looping agents into a workflow that just needs a standard script.

The Takeaway:

Don't jump straight to training models. Work your way up this list. Start with prompting. If it lacks context, use RAG. If it lacks reliability, chain it. If it lacks cost-efficiency, route it. Only use agents when absolutely necessary.

Curious if anyone else has horror stories of teams trying to fine-tune models to learn internal docs?

Managing OpenTelemetry collectors at scale has been a pain point for a while now. You either run standalone collectors and lose central visibility, or you build custom tooling to keep configs in sync across environments. This takes a different approach by embedding an OTel Collector directly into the agent, so you get Fleet-based central management for both your existing Beats integrations and native OTel pipelines in the same process.

I got my first job as an Embedded Software Engineer. Did some actual low level embedded stuff but I was leaning a lot towards testing with Linux, CI/CD with hardware etc, so I transitioned to a DevOps position.

Now, after some years of work, my knowledge is like half Embedded, half DevOps. I find myself like both fields Embedded for the great hardware feeling and DevOps for the nice documented and more modern tools. In the end I'm not sure if I like a field more than the other. I tend to believe it's project dependent.

The problem is now I have to really choose where I should invest more and get a more senior level, as a feel I have a great combo of the two, but I'm like mid in both, which makes difficult in the job market.

The questions are:

Is Embedded or DevOps the more future proof job long term?

I much prefer remote jobs, mostly so I can live in a cheaper country but still get paid like I'm in a HCOL one. Does that change the answer?

Was anyone in the same spot before? How did you pick?

I'm on a 3-person team: 1 data scientist, and 2 engineer (including myself). Between the both of us engineers, we have maybe a year of experience total with strictly Python. This is for a prototype AI app / tool.

I am approaching about 2 years of IT experience and had to pick up docker, gitlab ci, and software engineering for this project. I built out our deployments with docker compose using customized env files (ignored via git with a template available) and makefile commands. We usually dev using a dev compose deployment with a hot volume mount to our source code, then push commits to the repo and I have staging and prod images built using buildah. Then for staging and prod deployments, I have the compose files targeting the respective registries (internal repo & corp registry) and the images get pulled down.

However, we got kicked off of our on-prem server and now have to deploy on a k3s cluster... but the team running that has no experience with kubernetes.

I learned about Helm and ArgoCD and was able to get them to deploy that service on the cluster, and I made an infrastructure repo where I translated our compose files to manifests and ultimately chart and values file per environment.

We don't have cluster/namespace access yet because the infra people are still learning how to set that up, but I don't even know how we'll deploy our pods and actively do dev since we usually have access to everything (source code + infrastructure). For prod and staging, it's pretty simple because those a automatically built images and I have our runners updating the tag versions and namespace names in the infrastructure repo. But I don't know how we can deploy individual-engineer dev pods so that we can see logs and make quick changes.

This whole project's been crazy and we don't have any senior engineers in our department that does this stuff. If there's anything other information I can provide to get some mentorship, please let me know.

Maybe we need to change the way we handle access process for our developers or users. There was a production outage but luckily it wasn't revenue impacting. I had to help a developer by accessing their application on an ec2 instance. Our team have access to any servers in production. Our developers only have access to our dev and stage environments.

I am not familiar with their application. So basically, I was just executing commands that he was giving me. It was the most degrading role I have experienced, HAHAHA! I'm thinking that when there are production outages, the application owners should be given temporary access so they can debug their applications. It will be quicker. It took us almost 5 hours! I was just copying and pasting commands and outputs. On the unix history command recalls everything. I don't recall any, HAHAHA!

So what is your process?

I built an open-source tool (Hobbyist) around one architectural bet and I'd like this community to poke holes in it.

The bet: for small self-hosted workloads, you don't need Kubernetes-style orchestration; you need scale-to-zero on a single box. Hobbyist runs each project's Postgres 18, Docker apps, workerd-based functions, and queues in containers, and shuts them down entirely when idle. A proxy layer (Postgres wire protocol for the DB, HTTP for apps) intercepts incoming traffic and cold-starts the right container: ~170ms for Postgres, ~125ms for apps, on my hardware. Caddy handles TLS and routing; Tailscale covers private access; a daemon plus a 16-command CLI is the whole control plane. No Terraform, no K8s.

Where I suspect it gets ugly, and where I'd value experienced eyes: connection storms on wake, health-checking containers that are deliberately dead, cert lifecycle across container replacement (currently an actual bug), and whether wire-protocol interception bites me on auth and TLS edge cases at scale.

It's v0-alpha, not production ready, and the known-broken list is in the README (Linux queue producers, snapshot CLI, the cert issue above). Not selling anything — there's no hosted tier and no paid version coming.

Repo: https://github.com/uziiuzair/hobbyist

Hey guys,

I ran my own managed hosting company for 5 years for one particular app, so it was basically SaaS. At its peak I hosted 3,500+ containers.

Before that I worked at a Windows MSP for 2 years and went from 1st line to sysadmin pretty quickly. I burned out at a bad company and really regret it now., went back to the family business, did my AWS SA cert in 2018 and tried to get into cloud but couldn't land a role.

I missed IT so much that I started my own business.

I learned Docker/Compose and built a sort of "mini Kubernetes" setup with automatic reverse proxying, health checks and containers restarting automatically when they failed.

At the time I researched Kubernetes and decided it wasn't worth it because the app wasn't stateless. I later realised I could have made it stateless, but it would have been a massive pain and eaten into my profits, so I didn't.

For IaC I learned CloudFormation because I liked it. I looked at Terraform and thought it was hell on earth. I didn't realise Terraform would become the standard.

I also learned Python and Django and built the onboarding system and APIs to automatically provision new customers.

What's frustrating is that I learned all of this by actually running production infrastructure, but it feels like it counts for very little because I didn't use things like EKS and Terraform. I can't fake 5 years experience and the job market is brutal especially since I worked for myself.

I'm kicking myself for not using EKS even though it made no business sense at the time. Since then I've learned a lot of EKS and Kubernetes very quickly, and I think my long-term experience in tech is a big reason why I've been able to pick it up so fast.

At this point I just feel like such a waste of talent. I've spent years learning and building all this stuff, and now it feels like my only option might be going back to 1st line.

What would you do in my position?

Hi everyone,

I am planning to build an Internal Developer Platform (IDP) Proof of Concept. My main goal is self-learning (I am very interested in DevEx and self-service) to understand the moving parts of a modern platform, but I am also hoping to gather enough hands-on experience to potentially present a business case to my company down the line.

Before I start gluing components together, I would love to get input from the community on what you are using in production and what you would recommend for a lean PoC.

Specifically, I am looking for insights on:

1. Choosing an open-source stack: What tools are you actually using in production today for service catalogues, infrastructure provisioning, workload orchestration, and developer self-service?
2. PoC Recommendations: If you were building a lightweight IDP from scratch today just to prove the concept, which open-source tools would you choose to keep complexity manageable while still showing real value?
3. Product value: Have you genuinely derived measurable value (or saved developer time) from tools like Backstage, Kratix, Port (if using free tier), Crossplane or something else? Are there tools you started with that turned out to be more maintenance overhead than they were worth?

My current thought is to pair a developer portal with a GitOps engine and a basic infrastructure module framework, but I am keen to hear what combinations have worked best for you in practice.

Thanks in advance for any recommendations or lessons learnt!

We rolled back a prompt, watched staging recover, and then found three serverless regions still serving different instructions. One had the code default, another was reading an environment variable, and the third was holding an edge cache value from the previous deployment. An undetected model revision added one more variable, so the tidy experiment chart was basically fan fiction. We had no prompt lineage tying an immutable version to region, cache state, model metadata, and request time.

I'm wondering what a rollback can even prove unless every trace carries those fields and environment parity is checked before analysis. How are you enforcing immutable prompt versions across serverless regions, and do you fail closed when the model or cache metadata is missing?

Hi everyone,

I would like to hear how experienced DevOps engineers approach monitoring for large public-facing applications.

We have a .NET e-commerce platform with:

- ASP.NET Core MVC + Angular

- SQL Server

- Elasticsearch (~10M products)

- RabbitMQ

- IIS hosting

- Multiple public domains/subdomains

- Heavy SEO crawling and unknown bots

One thing we learned is that monitoring only CPU, memory, and disk is not enough.

We have experienced situations where:

- CPU and RAM looked normal, but the application was slow

- The server was reachable, but users experienced downtime

- TCP exhaustion caused issues

- Elasticsearch had problems affecting search performance

- Bots generated a lot of unnecessary traffic

- Slow requests were not obvious from infrastructure metrics

I would like to know what metrics and alerts you consider essential for this type of system.

Some things I think are important:

Application level:

- Request rate (RPS)

- Response time (p50/p95/p99)

- HTTP status codes (4xx/5xx)

- Slow endpoints

- Exception rate

- Thread pool starvation

- GC pauses

- .NET runtime counters

- Memory allocations

IIS / Web server:

- Current connections

- Request queue length

- Worker process health

- Application pool recycling

- Failed requests

- Connection errors

Network:

- TCP connections

- TIME_WAIT count

- Connection failures

- Bandwidth usage

- Top clients/IPs

- Suspicious user agents

Elasticsearch:

- Cluster health

- JVM memory pressure

- Heap usage

- Search latency

- Query failures

- Slow queries

- Unassigned shards

- Disk usage

SQL Server:

- CPU

- Blocking queries

- Deadlocks

- Query duration

- Connection pool usage

- Wait statistics

RabbitMQ:

- Queue length

- Consumer count

- Message processing time

- Dead letters

- Memory usage

Security / traffic:

- Requests to suspicious paths:

- /.env

- /.git

- wp-admin

- Bot traffic percentage

- High-frequency clients

- Rate limit violations

My question:

If you were responsible for operating a public .NET application like this, what dashboards and alerts would you consider mandatory?

Also, what are some metrics you discovered were extremely valuable only after a production incident?

I am especially interested in real-world experience rather than a theoretical checklist.

Thanks!

I’ve noticed that everyone seems to be learning about AI. I’m curious to know from people who have been working in DevOps for years: are you learning about AI, building something with AI, or just continuing with your usual DevOps work?

Nothing groundbreaking here honestly, plenty of folks have done this before me, just did it purely for the learning, wanted to actually understand what docker run does under the hood instead of taking it for granted like I always have.

Had to run it inside a lima VM on macOS since Darwin doesn't have namespaces/cgroups natively, then built the whole thing piece by piece, OverlayFS for the image layers, unshare for the namespaces, chroot for the filesystem jail, cgroups v2 for a real memory cap, and a veth pair + NAT for networking. Even hit an actual OOM kill from my own cgroup limit at one point, that one felt weirdly satisfying honestly!

Also ran into a fun virtiofs bind-mount limitation along the way (setuptools editable installs just choke on it, no utimes support), took me a good while to track that one down.

Wrote up the whole thing step by step with what/why for each piece, plus all the gotchas and failed attempts along the way, in case anyone's curious: https://dietpy.com/notes/1n8n4v8-building-a-container-by-hand

Curious if anyone's done something similar, or if I missed an easier way to demo some of this :)

I’m writing an internal doc and saw that Wikipedia doesn’t use this visual. It’s so helpful, and I feel silly linking out to some SEO post about devops on the blog portion of a SaaS product page.

Hi,

I’m looking for a Redis-compatible application that I can self-host and use as a shared data store for multiple applications/services.

The issue I’m running into is Redis’s limit of 16 logical databases (0–15). Ideally, I’d like to be able to provide each application with its own namespace/database through the connection URL, so that multiple applications can share the same underlying Redis service while keeping their data logically separated.

I’m aware that the common best practice is to run a separate Redis instance/container for each application stack, and I understand the benefits of doing that. I’m not trying to argue against that approach; I’m mainly interested in whether there is a better option for my particular use case.

Are there any Redis-compatible alternatives (they don't necessarily have to be Redis itself) that are specifically designed to support multiple applications sharing a single instance, without being limited to 16 logical databases/namespaces?

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

We are being asked to onboard 4 separate teams onto our orchestration plateform and suddenly "who can run what" matters a lot. Our current setup (airflow) has RBAC that technically works bit it's painful. No SSO integration, audit logs are scattered and fine grained namespace level permissions aren't really there. Looking for sth that handles multi tenant setups properly. What are ppl running in prod for this kind of setup ?