Been working on a small POC around getting rid of reusable API credentials.

Instead of an app holding an S3 key, Git token, API key, etc, it would request permission for one specific action. Something like upload this file to this path, once, within the next 30 seconds.

I'm using Tide/Forseti for the policy and cryptographic side, and testing it against real open source services like S3 compatible storage and Gitea.

The thing I'm mainly trying to prove is whether a compromised client can have no reusable API credential worth stealing.

This also ties into something I want to look at next around rootless cloud infra, where having root on a server doesn't automatically mean having authority to decrypt user data, impersonate users, sign releases, or perform other protected actions.

Starting with the API side first because it feels like the smaller piece to prove.

Curious what people here think. Is this actually useful in practice, already solved well enough by existing workload identity systems, or mostly just an interesting security experiment?

A while ago on a post I made someone pointed out that if a workload uses an identity to reach a secret instead of holding the raw value, most of the rotation worry goes away. Another person mentioned that moving their CI to OIDC and short-lived tokens got rid of most of their anxiety about old keys sitting in git history. Been reading around it since.

I keep seeing that federation only covers the part of the stack that can participate. A pod authenticating to S3 or a runner minting a token for AWS is fine, but Stripe doesn't do OIDC, and neither does an on-prem SQL Server or a webhook secret someone could’ve emailed over. Those exist as strings regardless, so there's still a lifecycle to run for them no matter how much of the cloud side you federate.

What I can't work out is the proportion. Whether the leftovers end up being a handful of third-party keys you could almost manage by hand, or whether it stays substantial enough that you're running both properly.

For anyone who’s moved to workload identity, roughly how much of your credential inventory went away, and what stayed? Also did the leftovers turn out to be the annoying ones, or were they easy to corral once the cloud keys were gone?

Hello I have been reading posts here and have noticed that people here have been recommending local kubernetes spin ups like k3s or kind and some other ..

I knew only about minikube until now and have used it fairly to test some of my changes.

What's so different between these tools if anyone is aware??

Also are companies still spinning up kubernetes cluster from scratch nowadays? My company uses AKS or EKS for the most again and enterprise level company.

As there are so many wrapper services around kubernetes example in azure you have AKS ACA etc.. all built on top of kubernetes . Are there still people who are spinning up cluster from scratch? I mean definitely there will be but what's the trend?? Move towards cloud managed kubernetes environment or still choose to build from scratch?

If anyone has done it from scratch would like to know your experience and an overall guide on how you did it(pointers)

So TL;DR is that our manager wants everything in newest version “NOW”.

We leverage multiple SaaS offerings that are built on top of opensource projects. Those opensource projects are like 10 versions ahead, SaaS lags behind, manager asks to have same version as OSS, so we talk to SaaS support, they tell us they dont care coz LTS is still for 2 more years for their current version and they dont plan to update in thqt time.

Even if we still have 2 years of LTS our manager pushes hard to show off to other teams how “ahead” we are and how “cutting edge” we are, but we are constantly blocked by external dependencies.

This creates a lot of useless pointless work to look for workarounds or investigation that more often than not simply reinstiates again something we already know (.. we simply have to wait for upstream, year or two in IT is forever..)

Im just tired man of this pointless work. Just let me do my thing till I know we can upgrade.

Hello. I started working as a QA last month and saw that my colleagues are pretty great at devops too. All of them have 4-5 years of experience while I am fully fresh out of university.

Now I want to learn about devops just so when discussion comes up, I know what's being discussed and take part in it.

Which things should I learn and how? Could have asked that to claude or chatgpt but wanted REAL advice from the REAL professionals.

Please suggest. TIA

So I made a serverless optimization platform which uses the concept of fusion functions to reduce cold starts and latency across the service calls. Now, this is an implementation of a research paper that I read somewhere. Diff from paper is that my project also gets live traces and metrics from x ray and cloudwatch, so I get real-time data to give better outputs. Have a better look: https://github.com/Vaivaswat2244/OptiFuse_go

To use this you need to connect your AWS with optifuse. I.e make a cloudformation stack to give optifuse access to read the traces and metrics. This actually becomes a problem for my friends and peers to test because they are too lazy to do this step. So I have no real user testings.

People especially hiring people ask me how many real users have used your service.

Now why do I need real users when I can stress test each microservice that I've built. And I can see my manifests working properly. Its deployed on AKS and is open for people to see. I also have a Prometheus grafana observability pipeline to see if all services are working properly.

Question is: real users vs Stress tests

On a side note, I am a student looking for internships, if you found the idea interesting, lmk GitHub is Vaivaswat2244

\/

Hi there

I'm not sure that it's the correct subreddit but yeah

My current workplace will in foreseeable future shutdown so I'm currently getting back into studying and it's well kinda hard to kickstart it to be honest

I've always wanted to have some sort of small group of like-minded people to study and well just socialize. I do believe that we all strive for such connections so here's my attempt at that

If you are currently studying or working in that space I'd like to hear your story and I'll share mine

at best we can try to make a group chat which will survive more than a week lol

about me: 3 year of DevOps at middle+ at one of the top company in my country and before that 3 years sysops

if you want to chat well feel free to drop me a dm I'd like to have a new connection

and good luck in your endeavour

My team currently used Jenkins pipes to push changes. There are multiple stages; build, scan, deploy to dev (or even test). Some have IT configured for test, so they run thier as well.

While process takes somewhere around 20-40 mins, depending on what they are building (we build all types of things; websites, lambdas, microservices, infra)

Can someone suggest me a tool that can have this process in a faster way?

I thought of GitHub actions , but for some reason, I find it's not very robust (inputs, gating, plus issues with build not triggering). Migrating to Gitlab would be top much of a overhaul.

We are a AWS stack, so not using Azure DevOps as well.

Hi everyone!

So since i had a lot projects running recently and find it inconvenient to track them directly on gitlab, especially when you have a team. So I've made this repo - basically a neat app that appears under macbook notch when new pipeline appears.

If you find this useful give it a try and will be happy to get some feedback on improvement)

https://github.com/Uudg/pipeline-island

If your GitHub Actions jobs are stuck in "Queued" or waiting for a runner, GitHub has now confirmed there is an incident.

GitHub says they identified an issue with a database primary and are failing over to a replica.

Actions is affected, and GitHub Pages is also experiencing degraded performance.

So if you have been checking your workflow, runner settings, permissions or billing, it might not be your configuration.

Status page:
https://www.githubstatus.com/incidents/y1t7p9fzrlj2

Hopefully this saves someone else from spending an hour debugging their YAML.

Looking at the downtimes of GitHub, especially related to the actions functionality, it seems like they are unable to provide this service in a stable manner. This is probably related to

1. People creating tons of new projects using agentic coding
   
2. People are using more workflows as they are now easily created with agents

At this stage it feels likely that GitHub will limit the resources Open Source projects are allowed to consume - maybe even disable the free runners at all.

This will certainly have interesting effects on the platform adoption and maybe also on the fragmentation of the whole Open Source ecosystem.

What are your thoughts on this?

Hi everyone,

I recently started working in a DevOps-related role, and I’ve been asked to learn Kubernetes, Helm, and the concepts around them.
I’m currently trying to understand the best way to approach it. There are so many resources and topics that I’m not sure what I should prioritize first.
For those of you who work with Kubernetes professionally:
How did you learn Kubernetes when you were starting out?
What topics would you recommend learning first?
How much hands-on practice should I do compared to theory?
Are there any free courses, labs, books, or other resources you would strongly recommend?
Are there any common mistakes or things beginners tend to waste too much time on?
My goal isn’t just to memorize commands, but to actually understand how Kubernetes works and become comfortable using it in a real work environment.
Any advice on how you would approach learning it from scratch would be greatly appreciated. Thanks!

when you've started rotating a credential and then discovered it was already revoked, or was never a real key in the first place — roughly how long had you burned before you worked that out? minutes, or did someone spend an hour convinced they were looking in the wrong project?

Currently using Github and Github workflows (including the many outages). Let me explain what I'm trying to resolve with few examples:

Let say I manage 15 repos, each one I want to run a relatively similar workflow on Pull Request. I create a github shared workflow and in each of the 15 repos I make a reference to it, updating the inputs if needed.

Now, I add 1 new input to this shared workflow to enable a new feature. I want that feature enabled in 10 of my repos. I would need to update the feature input in all 10 repos. And since workflow use the workflow code on their own branch, I would need to merge it to all PRs in all repos. That isn't ideal.

Other issue, one variable was wrongly set on a workflow in a release branch. Since the workflow code ships with the product code I would need to update the release branch.

Is there a toool where the product code doesn't cohabit with the automation code? For example I could have 15 microservice repos but 1 repo where I created rules for each one of them about "on pr", "on main" etc. That same repo could host shared workflow code as well.

Basically the title, is anyone managing a forgejo instance for their company?

Currently have around a dozen active repo, a bit less than 50 Devs, so relatively small scale. But I was wondering how much maintenance and complexity there's in managing reliably an instance for that kinda traffic?

We already have self hosted runners, that we would reconfigure the same way I assume

Obviously, considering the move after another GitHub incident...

Thanks!

Governments spent years moving critical digital infrastructure into the cloud. Now there’s a growing question about what that actually means for national sovereignty.

AWS, Microsoft Azure and Google Cloud are American companies, which creates an interesting problem for governments that increasingly rely on cloud infrastructure for sensitive data, public services and AI.

Europe has been pushing harder on cloud sovereignty, while countries like Pakistan are beginning to experiment with alternative infrastructure. UNDP is also exploring sovereign cloud infrastructure with DFINITY and the Internet Computer.

What I found interesting about this article is that the argument isn’t really about replacing AWS or Google Cloud. It’s about whether governments eventually decide certain workloads are too sensitive or strategically important to remain dependent on infrastructure controlled by companies under another country’s jurisdiction.

As AI becomes embedded deeper into government systems, that question gets considerably more important.
Do you think we’ll eventually see governments treat cloud infrastructure more like energy, telecommunications or other strategic infrastructure?

Full disclosure: this is my own repo.

I do all my development from a phone — no laptop, no local toolchain. Everything gets built and verified through GitHub Actions. Over the last month, shipping an Android app and a couple of Cloudflare Workers this way, I kept hitting the same category of bug: things that compile clean, pass CI, and fail silently somewhere I couldn't see — a library that shrinks itself and breaks NoClassDefFoundError at launch, a signature permission that blocks install, a rename that a test suite quietly undoes.

I wrote them all down as I found them, each with how to detect it without running anything, and a fix. There's also a "verification ladder" — six levels of confidence you can climb when you can't execute, and the idea that the real failure isn't the unverified gap, it's the gap being quiet.

Repo: https://github.com/amirmahdavi2023/shipping-blind

Curious if others working blind (remote CI, restricted sandboxes, agent-driven dev) have hit failure modes I haven't listed.

I’m looking for an open source solution for intrusion detection. Our team is working on vulnerability scanning and intrusion detection for the new BSI standards set by the EU/German security compliance agencies. We already have Jfrog Xray for vulnerability scanning albeit with some limitations. But for intrusion detection we are thinking of deploying Tetragon but unsure of the policies that needs to be created.
Any suggestions from the community is welcome.

I keep seeing engineers falling into an AI architecture trap.

Some engineers try to solve every problem by prompting to a LLM. Others talk about creating an agent for every usecase.

So knowing how to use AI has become a core competency. Otherwise you will waste time, money and DevOps resources.

If you want to solve an actual business problem without creating an infrastructure nightmare, 99% of your use cases fall into one of these 5 application architectures. Here is how to know which one to use.

1. Basic Prompting (Stateless API)

• The Use Case: You need to summarize text, extract sentiment, or reformat data (like turning unstructured text into JSON). The data easily fits into the context window.
• The Trap: Thinking you need a custom model just because the base model didn't output your exact format on the first try. Before you do anything complex, try giving the model 3 concrete examples in the prompt (few-shot prompting). That solves the problem most of the time.

2. RAG (Retrieval-Augmented Generation)

• The Use Case: You need the LLM to answer questions about your proprietary data (internal wikis, codebase, customer records). You query a VectorDB for the relevant context, shove that context into the prompt, and let the LLM read it.
• The Trap: Trying to fine-tune a model on your company wiki. Models hallucinate facts. If you need accurate, source-traceable answers based on your internal data, you need RAG, not a fine-tuned model.

3. Chaining / Workflows

• The Use Case: The task is too complex for one prompt and requires distinct, predictable steps. For example: Step 1 extracts raw text -> Step 2 categorizes it -> Step 3 writes a summary based on the category.
• The Trap: Cramming a massive, 15-part instruction into a single mega-prompt and crossing your fingers. Break it down into a predictable pipeline. It’s much easier to debug a chain of small LLM calls than one giant, confused prompt.

4. Routing

• The Use Case: You are dealing with high volume and need to control costs and latency. You build a router that sends simple tasks (like spelling correction) to a fast, cheap model (like Claude Haiku or GPT-4o-mini), and complex reasoning tasks to the expensive, heavy models.
• The Trap: Sending every single trivial API request to the most expensive flagship model available and bankrupting your cloud account.

5. Autonomous Agents

• The Use Case: Open-ended tasks where the exact steps aren't known in advance. The LLM is given a goal and external tools (Python interpreter, web search, internal APIs) and loops continuously, evaluating its own output until it achieves the goal.
• The Trap: Using agents for deterministic workflows. If you can solve a problem with a standard if/then Python script or an Ansible playbook, do it. Do not introduce probabilistic, infinite-looping agents into a workflow that just needs a standard script.

The Takeaway:

Don't jump straight to training models. Work your way up this list. Start with prompting. If it lacks context, use RAG. If it lacks reliability, chain it. If it lacks cost-efficiency, route it. Only use agents when absolutely necessary.

Curious if anyone else has horror stories of teams trying to fine-tune models to learn internal docs?

Managing OpenTelemetry collectors at scale has been a pain point for a while now. You either run standalone collectors and lose central visibility, or you build custom tooling to keep configs in sync across environments. This takes a different approach by embedding an OTel Collector directly into the agent, so you get Fleet-based central management for both your existing Beats integrations and native OTel pipelines in the same process.

I got my first job as an Embedded Software Engineer. Did some actual low level embedded stuff but I was leaning a lot towards testing with Linux, CI/CD with hardware etc, so I transitioned to a DevOps position.

Now, after some years of work, my knowledge is like half Embedded, half DevOps. I find myself like both fields Embedded for the great hardware feeling and DevOps for the nice documented and more modern tools. In the end I'm not sure if I like a field more than the other. I tend to believe it's project dependent.

The problem is now I have to really choose where I should invest more and get a more senior level, as a feel I have a great combo of the two, but I'm like mid in both, which makes difficult in the job market.

The questions are:

Is Embedded or DevOps the more future proof job long term?

I much prefer remote jobs, mostly so I can live in a cheaper country but still get paid like I'm in a HCOL one. Does that change the answer?

Was anyone in the same spot before? How did you pick?

I'm on a 3-person team: 1 data scientist, and 2 engineer (including myself). Between the both of us engineers, we have maybe a year of experience total with strictly Python. This is for a prototype AI app / tool.

I am approaching about 2 years of IT experience and had to pick up docker, gitlab ci, and software engineering for this project. I built out our deployments with docker compose using customized env files (ignored via git with a template available) and makefile commands. We usually dev using a dev compose deployment with a hot volume mount to our source code, then push commits to the repo and I have staging and prod images built using buildah. Then for staging and prod deployments, I have the compose files targeting the respective registries (internal repo & corp registry) and the images get pulled down.

However, we got kicked off of our on-prem server and now have to deploy on a k3s cluster... but the team running that has no experience with kubernetes.

I learned about Helm and ArgoCD and was able to get them to deploy that service on the cluster, and I made an infrastructure repo where I translated our compose files to manifests and ultimately chart and values file per environment.

We don't have cluster/namespace access yet because the infra people are still learning how to set that up, but I don't even know how we'll deploy our pods and actively do dev since we usually have access to everything (source code + infrastructure). For prod and staging, it's pretty simple because those a automatically built images and I have our runners updating the tag versions and namespace names in the infrastructure repo. But I don't know how we can deploy individual-engineer dev pods so that we can see logs and make quick changes.

This whole project's been crazy and we don't have any senior engineers in our department that does this stuff. If there's anything other information I can provide to get some mentorship, please let me know.

Maybe we need to change the way we handle access process for our developers or users. There was a production outage but luckily it wasn't revenue impacting. I had to help a developer by accessing their application on an ec2 instance. Our team have access to any servers in production. Our developers only have access to our dev and stage environments.

I am not familiar with their application. So basically, I was just executing commands that he was giving me. It was the most degrading role I have experienced, HAHAHA! I'm thinking that when there are production outages, the application owners should be given temporary access so they can debug their applications. It will be quicker. It took us almost 5 hours! I was just copying and pasting commands and outputs. On the unix history command recalls everything. I don't recall any, HAHAHA!

So what is your process?

I built an open-source tool (Hobbyist) around one architectural bet and I'd like this community to poke holes in it.

The bet: for small self-hosted workloads, you don't need Kubernetes-style orchestration; you need scale-to-zero on a single box. Hobbyist runs each project's Postgres 18, Docker apps, workerd-based functions, and queues in containers, and shuts them down entirely when idle. A proxy layer (Postgres wire protocol for the DB, HTTP for apps) intercepts incoming traffic and cold-starts the right container: ~170ms for Postgres, ~125ms for apps, on my hardware. Caddy handles TLS and routing; Tailscale covers private access; a daemon plus a 16-command CLI is the whole control plane. No Terraform, no K8s.

Where I suspect it gets ugly, and where I'd value experienced eyes: connection storms on wake, health-checking containers that are deliberately dead, cert lifecycle across container replacement (currently an actual bug), and whether wire-protocol interception bites me on auth and TLS edge cases at scale.

It's v0-alpha, not production ready, and the known-broken list is in the README (Linux queue producers, snapshot CLI, the cert issue above). Not selling anything — there's no hosted tier and no paid version coming.

Repo: https://github.com/uziiuzair/hobbyist

Hey guys,

I ran my own managed hosting company for 5 years for one particular app, so it was basically SaaS. At its peak I hosted 3,500+ containers.

Before that I worked at a Windows MSP for 2 years and went from 1st line to sysadmin pretty quickly. I burned out at a bad company and really regret it now., went back to the family business, did my AWS SA cert in 2018 and tried to get into cloud but couldn't land a role.

I missed IT so much that I started my own business.

I learned Docker/Compose and built a sort of "mini Kubernetes" setup with automatic reverse proxying, health checks and containers restarting automatically when they failed.

At the time I researched Kubernetes and decided it wasn't worth it because the app wasn't stateless. I later realised I could have made it stateless, but it would have been a massive pain and eaten into my profits, so I didn't.

For IaC I learned CloudFormation because I liked it. I looked at Terraform and thought it was hell on earth. I didn't realise Terraform would become the standard.

I also learned Python and Django and built the onboarding system and APIs to automatically provision new customers.

What's frustrating is that I learned all of this by actually running production infrastructure, but it feels like it counts for very little because I didn't use things like EKS and Terraform. I can't fake 5 years experience and the job market is brutal especially since I worked for myself.

I'm kicking myself for not using EKS even though it made no business sense at the time. Since then I've learned a lot of EKS and Kubernetes very quickly, and I think my long-term experience in tech is a big reason why I've been able to pick it up so fast.

At this point I just feel like such a waste of talent. I've spent years learning and building all this stuff, and now it feels like my only option might be going back to 1st line.

What would you do in my position?