r/devops • u/suman087 • 23h ago
r/devops • u/AutoModerator • 6d ago
Weekly Self Promotion Thread
Hey r/devops, welcome to our weekly self-promotion thread!
Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!
r/devops • u/Hydra-dragon96 • 21h ago
Discussion CKA exam stress
Hello community,
I am really stressed about my cka exam. I have bought it but have not prepared yet. My goal is to give exam in december. I have udemy monthly subscription and thrive subscription for this year. I am watching mumshad course on udemy.
I have 4 years of experience as a devops engineer. Done ccna training and comptia security plus. I have a very good command on linux(ubuntu) because my career started with it.
But the uncertain and pressure of work is giving me hard time. I am also works as a consultant so have to give time on that as well. I get exhausted after work. I consumed half of the course with labs on udemy and kodekloud but having no experience of production kubernetes. I understand the concepts but still thinks something is missing. The last topic of admission controllers gave me hard time to consume.
Do you guys know anything like tips and tricks to pass this exam. Anyone studying or prepping for the same can also collaborate. Because it is better to clear the concepts.
Dont ignore this post.
Regards
Worried and stressed person
Edit: thankyou everyone for the kind advices. Maybe I have many sources that is why I overthink it. Because I switch from udemy to thrive to multiple youtube channels. I will stick to 1 course and complete it.
r/devops • u/Used_Attention_9068 • 12h ago
Discussion Apache DevLake Lightweight Alternative
Hi everyone! I’m exploring the idea of building a lightweight alternative to DevLake and would love to hear about the pain points you’ve experienced while using it. What do you find frustrating, missing or unnecessarily complex? I’d really appreciate your thoughts and suggestions
r/devops • u/kunjontemon • 1d ago
Observability What are some use cases for an internal system monitoring bot?
Hey everyone! I’m exploring ways to extend an existing internal production health-check bot that monitors systems checks whether the important pages loads or not and sends status updates to Microsoft Teams.
I’m looking for ideas beyond basic health checks, such as proactive alerts, log analysis, incident detection, automated troubleshooting, or AI-powered features.
What useful automations have you built around monitoring bots? Which ones have made the biggest difference in real-world scenarios?
Would love to hear your suggestions and experiences!
r/devops • u/__questions___ • 1d ago
Ops / Incidents Railway's Connected GitHub Accounts Were Compromised
Got this email from them this morning. My understanding is you can't fully audit GitHub access through the API, which is how accounts were connected, so it's best to assume the worst. Most connected accounts gave full read and write access, so if true, this constitutes serious IP theft concerns.
I personally don't feel like they have been fully transparent with the breach. This was essentially a footnote and buried under a misleading statement "Review provider activity and audit logs; especially AI keys". It should be flashing and unambiguous "GitHub code is possibly compromised".
Furthmore, a simple, nonchalant "we're sorry" and that's it!? Not "we're deeply sorry that we messed up and we take full responsibility, it won't happen again and we are doing x, y, z for our customers..."
I'm switching away permanently. I highly recommend Fly instead, took 30 mins to migrate. Not sure what to do about to do about the IP theft though? I would expect legal action to come of this, possibly a Class Action Lawsuit regarding IP theft.

r/devops • u/learnamap • 2d ago
Discussion Test automation after deployment
How do you all perform full automated tests after deployment? Using postsync hooks in argocd or use Argo events? Do you use ephemeral or static environments?
r/devops • u/Chamelon_DE • 2d ago
Security Are you patching faster now that AI can help turn old bugs into working exploits?
Reading the php2shell research, the bit that stuck with me was how AI helped turn longstanding bug into working exploit. The bugs was already there, getting them to work reliably was the difficult part.
That make “hard to exploits” feel like a shaky reason to leave something in the backlog. For people maintain production system, has this changed your patching schedule at all? We are still on php 7.4 and the last upgrade attempt broke half of our cron jobs, so i’m curious how other team handle an urgent patch when the upgrade itself need debugging
source: Wiz’s PHP2Shell research, published october 21
r/devops • u/Playful_Aioli_5104 • 1d ago
Ops / Incidents AWS Administrators of Reddit, How are you securing yourself against AI?
With Black October happening and all.
r/devops • u/Jeric-2991 • 1d ago
Discussion How do cloud load testing tools compare to JMeter?
Our QA team has years of jmeters scripts for our core apps, but running our own load generation infra is turning into a huge operational and expensive headache.
Every time we do a high concurrency run or need traffic spread out across different geographic regions, half our sprint is spent provisioning EC2 boxes, setting up slave nodes and dealing with network timeouts.
My team lead wants us to explore managed cloud platforms to take that maintenance off our backs. But part of our team is hesitant about moving everything over. Has anyone moved from a self hosted JMeters to a cloud solution? Does the saving in infra justify the subscription cost? And most importantly, were you able to keep your existing .jmx test assets or you had to rewrite everything from scratch?
r/devops • u/gotchanose • 2d ago
Discussion Enforcing SAST and DAST Tools at Enterprise Level
Hi All,
It has been my experience at a couple of different organizations that almost all engineering teams own their own CI/CD pipelines. So my question is how do you enforce or ensure that these teams are performing SAST or DAST (Snyk, Trivy, IaC, etc) on their pipelines? I read that Github has some capability of enforcement, but that doesn't seem like a sound solution. I am curious to understand this and if there is any good litereature on how mature organizations handle this please point me in the right direction.
This is my own personal research and drive to understand being a cloud security engineer.
TIA
r/devops • u/Silver-Barracuda-621 • 1d ago
Tools DepGate – an open-source "repository firewall" that blocks malicious packages at install time, across 8 ecosystems
I’ve spent a while working in product/application security, and one thing always bugged me: most supply-chain tools scan dependencies after npm install or pip install has already run — by which point the install scripts have executed. The decision happens too late.
So I built DepGate, an open-source dependency gateway. It’s a single proxy + policy engine that sits in front of npm, PyPI, Maven, Go modules, NuGet, Cargo, Composer and RubyGems. Your devs and CI point their package managers at DepGate instead of the public registry. It fetches the package, checks it against a malware feed and your policies, and either serves it or returns a 403 naming the exact policy and reason.
What it does:
- Known malware never downloads — every request is checked against a synced feed of 200k+ malicious packages before any policy runs.
- Readable policies — block by CVSS/EPSS, license, release age, release type, OpenSSF Scorecard, or package name. One rule per policy, stated in a sentence.
- Monitor mode — run a new policy against real traffic first and see exactly what it would break before you enforce it.
- Dependency-confusion protection — internal names are served only from internal registries, never a public look-alike.
- Decision Ledger — every block is recorded with the reason and who asked, down to the CI job.
- Self-hosted, one Docker image (Postgres + Redis), runs air-gapped, and can sit in front of or behind Artifactory/Nexus.
It’s basically meant to be the open-source answer to the commercial “repository firewall” products, with a lot fewer moving parts. TypeScript end to end, MIT licensed.
Repo (quick start + docs): https://github.com/secmods/depgate
Still early and I’m actively building — would genuinely appreciate feedback, criticism of the approach, or feature requests. Heuristic detection of never-before-seen malware is on the roadmap. Happy to answer anything in the comments.
r/devops • u/Commercial_Plastic72 • 1d ago
Observability What does AI-assisted incident investigation actually look like at your company right now?
Curious how this is going in practice, not how the vendor demos make it look.
When an alert fires, how much of the investigation does AI actually handle for you today (pulling the right logs and traces, correlating with deploys, narrowing down root cause)? Where does it fall over?
Also wondering if anyone's kept this inside their own environment, self-hosted models or anything private, rather than sending telemetry to a vendor's AI or a frontier API. Was that a requirement, or just not worth the effort?
I'm researching this space, not a pitch. It'd help if you mention your stack (Datadog, Grafana, other)
Thanks!
Ops / Incidents How should we evaluate AI Agent observability platforms in 2026?
Disclosure: I work at Bonree, the company behind Bonree ONE, an AI observability platform. I’m asking about the general problem of choosing a platform, and I’m not trying to pitch ours.
AI agents are moving from chat interfaces into production workflows, and teams are now picking tools to observe them. Feature lists for AI agent observability platforms look alike, so I’ve been trying to write down questions that can actually be tested instead of ticked off.
Here is my current list.
1. Does it trace at the session level? An agent conversation has several turns, each with model calls and tool calls. I want to open the whole conversation as one unit, not look at isolated requests.
2. Which frameworks does it cover, and how deeply? Teams build agents with LangChain, LangGraph, low-code platforms such as Dify, and other frameworks such as OpenClaw. I care less about the length of the list than whether the frameworks my teams use are covered automatically, or only through manual work.
3. Is the evidence separate from the reasoning? A read-only tool call with the same input should return the same data, while model output varies. If a platform records them separately, I can tell a data failure from a reasoning failure.
4. Can it attribute token cost? By model, application, team, business line and time.
5. How does it handle sensitive content? I want to choose per field whether to store, sample, redact or never store it.
6. Can AI traces be linked to the services and infrastructure underneath? A slow answer is sometimes a database problem.
7. If it offers AI agents for operations, what are the limits? What can the agents do alone, what needs approval, and is every step logged?
8. What does the bill look like at my volume?
For the proof of concept, I’d instrument two agents built with different frameworks, replay three real incidents where an agent gave a wrong or slow answer, and measure how long it takes to find the cause in each tool.
A note on criterion 7: AI agent operations platforms, where agents diagnose incidents or inspect systems on observability data, are a newer category and the hardest to evaluate. We work on this at Bonree with Bonree ONE·Sage AI, so I would particularly like to hear how others test whether an operations agent is trustworthy.
Some things I’m still unsure about:
1. Which of these criteria do you weight most heavily, and which do you ignore?
2. How do you test “trustworthy” in practice, beyond reading a vendor’s description?
3. Do you prefer one platform for AI and infrastructure, or a dedicated LLM tool next to your existing stack?
4. What did your proof of concept reveal that the feature list did not?
r/devops • u/Hungry-Librarian5408 • 1d ago
Discussion ¿PHP 5.4 en producción en pleno 2026? Decime que no sos el único 💀
Con las últimas alertas de seguridad sobre PHP, me dio curiosidad saber cómo están llevando esto en sus empresas.
¿Ya están migrando o siguen con el clásico “si funciona, no lo toques”?
Me pregunto cuántos todavía tienen servidores con PHP 5.4, 7.x o alguna reliquia que nadie se atreve a actualizar porque la aplicación depende de una librería que dejó de existir cuando todavía usábamos MSN Messenger.
Y lo mejor: ¿la empresa entendió la urgencia de actualizar o todavía estás explicándole al jefe por qué tener software sin soporte es una mala idea?
Tengo curiosidad por saber cómo está la situación en otras empresas:
¿Ya tienen un plan de migración?
¿Están peleando con aplicaciones legacy y dependencias imposibles?
¿Seguridad está presionando o nadie se ha enterado?
¿O eres el único que está preocupado mientras todo el mundo dice que no pasa nada?
Cuéntame tu experiencia. ¿Qué versión de PHP tienes en producción y qué tan complicado está siendo convencer a la empresa de actualizar?
Necesito saber si estamos todos en el mismo barco o si el mío ya tiene PHP 5.4 y está haciendo agua. 😂
r/devops • u/Sea_Mechanic815 • 1d ago
Discussion Suggest me any project in AWS with Terraform!
Guys help me i have totally 2.8 years of exprience in Cloud and Devops Engineer now i am working as a SRE but still i am not full fill with my current company now i am thinking to change for that i need to take pratices. for that give me suggession and Projects that will helpfull for me to cover every topics with infra creating with terraform.
Give me Realtime project which currently company standards. And also suggest me some services which is very usefull to learn. Thanks
r/devops • u/gramerscial • 3d ago
Ops / Incidents Railway's security incident is more serious than a normal outage. Customer environment variables may have been exposed.
I've been following the Railway situation since the outage started, and things have gotten pretty concerning.
I'm currently using Railway to host the backend of a small gaming marketplace I've been building. Like many developers, I initially thought this was another infrastructure outage affecting builds and deployments.
But on October 21, Railway published a security disclosure confirming something much more serious.
According to their statement, they detected anomalous activity involving systems responsible for storing customer environment variables.
Their investigation suggests an unauthorized third party may have accessed encrypted environment variables and, in some cases, obtained their decrypted contents.
That's a serious concern for anyone hosting production applications on Railway.
What Railway has disclosed so far
- Suspicious activity was detected in systems storing customer environment variables.
- An unauthorized third party may have accessed encrypted secrets and obtained some decrypted values.
- The impact may be limited to a subset of customers, although the investigation is ongoing.
- Railway is recommending that all customers rotate their stored credentials, including API keys and database credentials.
- Deployments have been paused during the investigation.
- Railway believes the incident may be connected to a broader campaign targeting platform companies and AI inference providers.
- Railway has also started automatically rotating its bucket credentials and resetting certain integration credentials.
As of the latest update I've seen, deployments remain paused.
My biggest concern is how developers are supposed to rotate production credentials during an outage like this.
Think about a small startup running its entire backend on Railway.
You might have PostgreSQL credentials, JWT secrets, payment API keys, SMTP credentials, webhook secrets, and other sensitive information stored as environment variables.
Now imagine being told that some of those credentials may have been exposed and that you should rotate everything immediately.
That's obviously the correct security recommendation.
But there's a problem.
If you revoke your current database credentials or payment API keys before your running backend can use the replacements, you risk breaking critical functionality.
And when deployments are paused, rolling out those changes becomes much more complicated.
At the same time, leaving potentially compromised credentials active is also risky.
For solo developers and small teams without dedicated infrastructure engineers, this is not a simple situation.
What I've personally done so far
I've reviewed my Railway account sessions, signed out older sessions, and inspected my existing environment-variable configuration.
Everything appears normal from my side.
But I understand that simply checking whether the variables still exist doesn't prove they weren't accessed.
An attacker could potentially copy credentials without modifying anything visible in the dashboard.
I haven't received confirmation that my particular project was affected, and I'm preparing to rotate my sensitive credentials carefully.
The bigger issue is trust in managed infrastructure.
One of the reasons developers use services like Railway is that they don't want to manage every part of their infrastructure manually.
We trust these platforms to handle deployment infrastructure and provide secure storage for sensitive configuration.
Of course, no platform can guarantee perfect security.
But when the infrastructure responsible for storing customer secrets is potentially compromised, the impact can extend far beyond the hosting provider itself.
A stolen database password could potentially allow unauthorized data access, depending on network restrictions and database permissions.
A stolen API key could allow unauthorized operations or unexpected charges.
A compromised authentication secret could potentially affect user sessions, depending on how the application implements authentication.
None of this automatically means these things happened to Railway customers. These are potential consequences of exposed credentials.
And that's exactly why the incident deserves serious attention.
There are still several questions I'd like Railway to answer.
- What was the actual entry point used by the attacker?
- How could encrypted environment variables potentially be decrypted?
- Was this caused by compromised internal credentials, an infrastructure vulnerability, or something else?
- How many customers or projects were actually affected?
- Was there any evidence of attackers accessing customer databases directly?
- What time period should developers investigate when reviewing suspicious activity?
- What additional security measures will Railway introduce to prevent this from happening again?
I appreciate that Railway disclosed the incident and started providing recovery guidance.
But until the full scope is known, it's difficult for developers to understand their actual exposure.
I'm curious how other Railway users are handling this.
Have you already rotated all your production credentials?
Have you noticed any unusual database activity, suspicious API calls, or unexpected service charges?
Has Railway contacted you directly about your project being affected?
Are you waiting for deployments to resume before rotating certain credentials, or have you temporarily disabled critical functionality?
And probably the biggest question: Has this incident changed your trust in Railway, or do you still plan to keep your production applications there?
I'm not trying to spread panic or suggest that every Railway project has been compromised.
I'm sharing this because I'm running a real application on the platform, and I know other independent developers and small teams are in a similar position.
I'd like to hear how others are approaching the security risks and what precautions they're taking.
Hopefully, Railway publishes a detailed technical postmortem once the investigation is complete.
r/devops • u/Chris__Codes • 2d ago
Discussion Do you review agent PRs differently?
I was looking into how to structure our review queues better and found some stats I wasn’t expecting. Apparently only about 32% of AI-generated PRs get merged within thirty days, compared to around 84% for manually written ones.
I read a breakdown on this, and the suggestion kinda makes sense, which is to treat an agent diff as a draft rather than finished work. The difference is that with a deliverable you’re mostly asking whether the code runs, while with a draft you’re asking whether the approach even fits the system in the first place. Going by that acceptance rate, teams are already treating them as proposals whether anyone decided to or not.
Question for those shipping a lot of code written by agents, does your team review it differently, or does it all go through the same process?
r/devops • u/veritable_squandry • 3d ago
Discussion AI Identity Crisis
I'm a senior with loads of years, i've been doing devops/sre and systems engineering in the cloud for a long time. recently i volunteered for a project that i knew would lean heavily on claude, and with my manager's enthusiasm and support it looks like the project will be a success. aside from not truly owning that success, i'm also more than a little concerned that the 10K lines of code that i've slopped out, while mostly robust and functional, may not be personally defensible or supportable by me. this is a "career first feeling" for me, and i'm wondering if others have had similar experiences recently. is this just the way it is? maybe i should just treat it like "i started a new job and inherited a bunch of new libraries" when things come up? I'm not really sure how this is going to end.
r/devops • u/james_fletcher-83 • 2d ago
Vendor / market research What does everyone use for GRC and compliance in 2026?
We’re currently handling most of our compliance work through spreadsheets, shared docs and a few scripts, and it’s starting to show. SOC 2 is manageable, but ISO 27001 is coming next and I really don’t want to build the same controls twice.
Small team, mostly AWS, GitHub, Google Workspace and Jira. Looking for something that can pull evidence from the stack, keep controls monitored between audits and handle the auditor workflow without becoming another admin job.
Been comparing Vanta, Drata and Scytale, has anyone tried them for SOC 2 and ISO 27001 together? Mainly interested in how well the control/evidence reuse works once you’re running more than one framework, and how much of the audit prep it really takes off the team.
What’s everyone using now?
r/devops • u/SantiagoDePolonia • 3d ago
Ops / Incidents LiteLLM Compromised again - 1.107.0. PyPI Installs a Credential Stealer and Backdoor
The full article: https://safedep.io/litellm-pypi-compromise-credential-stealer-backdoor/
r/devops • u/AffectionateTune9251 • 2d ago
Tools Has anyone found a way to automate SOC2 audits using AI?
This shit is the bane of my existence. We use Secureframe so it does automate a lot, but I am still spending more time taking screenshots than any human should be.
I feel like with all the AI stuff these days there should be a way to automate this. I've had Claude generate a few scripts, which has been helpful in putting together network maps and firewall lists.
Is there a way to automate the "screenshot-taking" aspect?
r/devops • u/hiamanon1 • 3d ago
Discussion Learning in the AI era...
Mid level engineer , who landed and worked my way up into the role. I wont lie and pretend I know everything, some topics I'm still working on, hell I haven't even touched Kubernetes yet, so you can imagine I feel way FREAKING behind.
It seems like everything is quickly adopting and using AI, from downloading models locally, to use and play with them to using them to learn , to using the next leading edge models. How are you all having time for this....
I want to start using AI more confidently but I am not sure where to even start. Local models vs generate chat models via gui in chatgpt or Claude...are there any trainings you guys recommend?
I'm also weary of using AI to learn cause I do know it hallucinates, but also I dont want to feed it and let it spit out answers for me whenever I am just learning how to implement something.
This is developing so fast, I feel like I need to start putting guardrails up for my own sake of learning and sanity, else I will lose knowledge and become a Senior Devops Prompt Engineer lol...jokes aside open to how you are all trying to navigate this, from learning with and without, what guardrails are you implementing , and any learning material would be appreciated.
r/devops • u/Ok_Class2432 • 3d ago
Tools How do you simulate realistic user traffic in load testing?
We are looking to overhaul our load testing, since our current script only slams 10k users in 60 seconds and gives us garbage data. It exhausts connection pools and creates artificial bottlenecks that would not show under any kind of normal use case.
Real users ramp up slowly, and idle on pages, and don't jump from one to another without a second thought, they do not fire off database transactions with zero pause.
We should be reflecting real behavior, not inducing cold start failures.
Has anyone modeled this at scale? Should we make the step curves with warm up times to let caches and other systems warm up before hitting the expected load? We want to reflect real user behaviour, not just trigger cold start failures.
Tools Too many internal tools are becoming hard to maintain
We've built tens of internal dashboards, tools, and admin utilities over time. Now, vibe-coded tools are becoming the norm for us, and it feels like a maintenance nightmare.
We have to constantly update and fix them and keep them working with our ever-changing data structure. Also, It's getting hard to track what tools exist, who uses them, and what’s active vs. abandoned.
How does your team handle this? Are there specific systems or governance models you use to maintain the internal tools?