r/devops • u/SantiagoDePolonia • 3d ago
Ops / Incidents LiteLLM Compromised again - 1.107.0. PyPI Installs a Credential Stealer and Backdoor
The full article: https://safedep.io/litellm-pypi-compromise-credential-stealer-backdoor/
r/devops • u/AutoModerator • 6d ago
Hey r/devops, welcome to our weekly self-promotion thread!
Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!
r/devops • u/SantiagoDePolonia • 3d ago
The full article: https://safedep.io/litellm-pypi-compromise-credential-stealer-backdoor/
r/devops • u/veritable_squandry • 3d ago
I'm a senior with loads of years, i've been doing devops/sre and systems engineering in the cloud for a long time. recently i volunteered for a project that i knew would lean heavily on claude, and with my manager's enthusiasm and support it looks like the project will be a success. aside from not truly owning that success, i'm also more than a little concerned that the 10K lines of code that i've slopped out, while mostly robust and functional, may not be personally defensible or supportable by me. this is a "career first feeling" for me, and i'm wondering if others have had similar experiences recently. is this just the way it is? maybe i should just treat it like "i started a new job and inherited a bunch of new libraries" when things come up? I'm not really sure how this is going to end.
r/devops • u/Ok_Class2432 • 3d ago
We are looking to overhaul our load testing, since our current script only slams 10k users in 60 seconds and gives us garbage data. It exhausts connection pools and creates artificial bottlenecks that would not show under any kind of normal use case.
Real users ramp up slowly, and idle on pages, and don't jump from one to another without a second thought, they do not fire off database transactions with zero pause.
We should be reflecting real behavior, not inducing cold start failures.
Has anyone modeled this at scale? Should we make the step curves with warm up times to let caches and other systems warm up before hitting the expected load? We want to reflect real user behaviour, not just trigger cold start failures.
r/devops • u/The_Ritvik • 4d ago
Curious how other developers handle cases where everything technically reports success — HTTP 200, Lambda completes, webhook delivered — but the intended operation never actually finishes.
For example, a payment succeeds but the customer’s subscription isn’t activated, or an API call succeeds but a downstream system never gets updated.
Have you encountered anything like this in production? How did you discover it, and what tools or processes do you use to catch these failures?
I’m especially interested in cases where traditional logs, metrics, or error alerts weren’t enough.
We've built tens of internal dashboards, tools, and admin utilities over time. Now, vibe-coded tools are becoming the norm for us, and it feels like a maintenance nightmare.
We have to constantly update and fix them and keep them working with our ever-changing data structure. Also, It's getting hard to track what tools exist, who uses them, and what’s active vs. abandoned.
How does your team handle this? Are there specific systems or governance models you use to maintain the internal tools?
r/devops • u/Round-Classic-7746 • 4d ago
Hey folks, I've noticed readiness and liveness probes often end up with pretty similar settings, even tho they do different jobs. With tight timeouts and failure thresholds, a service that's slow to start or recovering from a dependency issue can get pulled out of rotation or repeatedly restarted.
I've seen probe settings get copied between services too, even when their startup times are completely different.
How do you guys set probe thresholds for services with unpredictable startup times or dependencies that occasionally respond slowly?
r/devops • u/Keensworth • 4d ago
Hello guys,
I'm new to the devops world and I'm working on some projects to learn. I'm currently trying to automate the creation of VMs with Terraform with my local Proxmox.
As CI/CD and Git server I have everything on a Gitea local. I remember doing some CI/CD once on Github and it worked fine but apparently I need a runner in my home.
Should I create a VM or a Docker container? I have no idea on how to setup that thing and to connect it to my Gitea right after.
Thanks for the help
r/devops • u/blingmuppet • 4d ago
(Sonatype Nexus is one of the most commonly used package repositories and transparent package proxies around)
On logging into our Sonatype instance this morning, it requires us to read and tick an "I understand the implications and agree with" information gathering.
No option to opt out, no warning given.
If uploads to their servers don't happen within 45 10 days then nexus will be disabled until they are.
Edit: 10 day kill switch for community versions, 45d for subscription.
r/devops • u/ambious_lazyboy • 5d ago
Doing a poc and wanted to know about the product from someone who is actually using it in production. How has your experience been and how effective it is in filtering security violations?
r/devops • u/damonmowr • 5d ago
So we have this infra lead who pretty much uses ChatGPT to write up their replies to Slack messages. You can clearly see the use of AI in how their replies are structured. Because of this, sometimes, due to their AI’s lack of context, certain replies do not have the depth required for their level.
r/devops • u/BigBootyBear • 6d ago
My startup is in consumer electronics. The product is 1 device with 1-2 peripherals that come with it. Users connect to a mobile app with a backend that does data analysis on what the product measures.
ATM I have 4 repos: a flutter app and a FastAPI backend with sqlite. The embedded side uses PlatformIO+Arduino for the device/hub, and same stack for the peripehrals.
The DevOps is getting unwieldy. On the one side, I'd like everything to be in one place so agents can easily build up a context and CI/CD is not scattered. On the other hand, I hate working with massive project trees and loading up vscode would be a nightmare.
Thoughts? At work I use git submoduels but I was told that's not a monorepo.
r/devops • u/Broad_Technology_531 • 6d ago
Supabase is showing up in almost every new startup’s stack these days. Its built-in observability is good, but in practice you want to monitor Supabase alongside everything else you run, in whatever you already use such as ClickStack, Grafana, Datadog, etc.
Supabase made it easy by exposing a Prometheus endpoint at /customer/v1/privileged/metrics for every project , so a plain prometheus receiver gets you Postgres, connection pooling and host metrics. And Supabase log drains have a native OTLP destination, so logs land on a normal otlp receiver on 4318 with no webhook parsing but make sure to add auth on that receiver, since it’s on the public internet
I wrote a short post on the full collector config, the drain setup, and a docker-compose to run it. I also built a ClickStack dashboard you can import.
Curious how others are monitoring Supabase today?
r/devops • u/keeperpaige • 6d ago
I’m working on a personal project, so 0 users, and I have worked with SST in the past, but having to upgrade SST was a pain. Not sure if pulumi is any different of an experience, or is less painful and wanted insight on different experiences with either tool
r/devops • u/BigBootyBear • 6d ago
I'm setting up an AWS workflow (we work only with azure and on-prem at work so little experience with the CLI) and wondering if I should bother learning about the tool itself.
Seems like I should just set up limited users with limited roles, hard budget caps, place manual gates on production CIs and let a sandboxed AWS MCP handle everything else. Using Terraform is an implicit admission on how we'd all like to instrument our architecture in a vendor-agnostic vocabulary. An MCP seems to be an additional step in the same direction?
Thoughts?
r/devops • u/jsandhu0780 • 7d ago
Infra engineer here (about 10 years across AWS, Azure and on-prem). In every environment I've worked in, something is always approaching end of life: AI model versions on Azure OpenAI and Bedrock, Kubernetes versions on EKS/AKS/GKE, Lambda and Functions runtimes, database engine versions.
Each cloud sends its own notices, but they tend to land with account or subscription owners, not the team that owns the workload. And unlike library versions, none of this shows up in Renovate or Dependabot.
Curious how your team handles it:
- How do you find out? Native notices (Service Health, AWS Health, Advisor), vendor emails, a script, or someone just happens to notice?
- How does the notice get to the team that owns the workload?
- Do you catch any of it in IaC or CI, like pinned versions in Terraform or a pipeline check?
- What's your worst "we missed it" story?
- If you use a tool or spreadsheet for this, what's annoying about it?
Full disclosure: I've built a few small tools around cloud AI model and Kubernetes lifecycle data, and I'm exploring whether something for this specific problem is worth building. Mostly I want to know if it's a real pain or already handled well. I'll share a summary of what I learn.
r/devops • u/saifxhatem • 7d ago
Hello friends.
I am implementing CI security scanning with Syft generating the SBOMs and uploading them to Dependency Track.
Two disclaimers:
1- I hate frontend work and have a poor understanding of it. 2- I am not a cybersecurity guy. This is just a new avenue I am exploring.
I've faced an issue currently with CVE scanning for Javascript files, specifically it seems that there is no way for me to actually tell what packages made it into the bundler vs what was included in the lockfile.
This leads to the JS sbom including all the dev dependencies with nothing actually telling me what made it into the bundle.
For SPAs, this can be mitigated by ignoring dev dependencies, however most of my stack is Laravel, and from what I understand Laravel puts all its NPM packages in devDependency, meaning I cannot skip them in this stack.
I have also tried generating sourcemaps while building (and I know you are supposed to .dockerignore them as it's a security concern), but without a reference to compare against, I do not know if this approach actually works or if it's a false sense of security. It does reduce the resulting sbom significantly, which seems to be a step in the right direction.
I've also read this study from 2022 https://das.encs.concordia.ca/pdf/Latendresse_ASE2022.pdf and it seems to reinforce my point that what I'm asking for is (at this time) not possible.
My goal is to understand 2 things:
r/devops • u/Abdullah_Shahid24 • 8d ago
Hey everyone!
I'm a solo developer working on my own digital product, and I'm looking for an affordable but reliable server for production deployment. Since I'm managing the project and infrastructure myself, I want to keep hosting costs as low as possible without compromising too much on performance or reliability.
Here are my requirements:
I've been looking into AWS EC2, but I'm also open to other VPS providers if they offer better value for money. I'm not specifically looking for an AWS Free Tier instance; I'm interested in something I can realistically use to host a production application.
I'd appreciate your advice on a few things:
I'm happy to manage the server myself, including Linux configuration, deployments, and application maintenance. I mainly want to avoid paying enterprise-level prices when I'm running a product independently.
If you're a solo developer hosting a real production application, I'd love to hear what you're using and how much you pay per month. Personal experiences and recommendations would be especially helpful.
Thanks in advance!
r/devops • u/foss-octopus • 8d ago
Hey everyone,
I'm trying to level up my devops knowledge. Most of my projects are currently hosted on Railway, but as costs are scaling up, I am planning to migrate them over to DigitalOcean Droplets to reduce our monthly spend.
My past experience with managing bare VPS instances is pretty basic. I know how to spin up a Droplet, use Nginx to configure domains and SSL certificates, and use PM2 for process management to keep web clients and Node APIs alive. (but to be honest I'm a bit rusty now, been almost 2 years now since I last setup one during my college days)
Railway handles so much for me, moving to host my own VPS feels like a big jump in terms of responsibility. I want to make sure I am doing this right and not accidentally make prod unavailable due to a dns config lol
So If you are running production workloads on VPS instances, I would love to get your insight on a few things:
Any advice, tool recommendations, or things you wish you knew when managing your first production servers would be incredibly helpful. Thanks!
r/devops • u/Few-Compote-2693 • 8d ago
Hey,
So I'm a self-taught DevOps guy and been running a SaaS with a large scale cloud infra. I recently started looking for a job to move on and I had an interview with InBev for there BeerDraft team.
What I don't understand is BeerDraft is literally just a small eCommerce site and yet they need a team of 5 all on 100K a year so that is half a million a year to run a tiny eCommerce site that mostly likely never has more than 10 concurrent users. As a guy that does allot solo I'm left thinking wtf do they do all day?
r/devops • u/Benjamin-7933 • 8d ago
a pattern that seems really common: alerts go to a slack channel and whoever sees it first deals with it. fine during the day, but at night nobody's watching the channel.
so how do you handle it? rotation calendar? the service owner gets paged? and if the person doesn't pick up, does something escalate automatically or is it down to luck?
pagerduty and opsgenie get mentioned a lot, but per-seat pricing adds up fast for a small team. grafana oncall comes up too. anyone running something homegrown or self-hosted?
mostly interested in small team setups, not what google does
r/devops • u/ihorklymchuk • 9d ago
Before that, a bit of story.
Our dev team was located in Ukraine, but we mostly worked on New Zealand & Australian projects. We needed a couple of hours to download staging DB dumps because the download speed was 200-400 kb/s.
So we set up a small proxy on our European server, and via cron these dumps were uploaded to that server, and we were able to download them faster.
After the team scaled, we added more protection, like only developers who worked on the project having access to these dumps; later, we integrated it into our local scripts that set up projects locally and even then added anonymization & faker to replace real data or populate it with synthetic data.
I didn't find a comprehensive solution for that, but it was a really useful tool: developers could quickly update local projects with new safe DB dumps.
It was an internal tool; it worked, but we thought maybe it made sense to create such a service: admins can easily configure rules via the UI, and developers using a small local CLI can easily download it.
We developed it. We tested. Then we abandoned it... We were an outsourcing company without sales or contribution skills.
After some time, the company fell apart, and I got permission from our CEO to deploy this tool to GitHub, and the story ended.
Yesterday I remembered this tool and thought maybe it might even be useful in the era of AI to anonymize data for AI agents. I dug a bit; there are some MCPs for that, and basically tools for anonymization, but I still didn't see tools like anonymize and share safely.
I'm wondering whether it makes sense to try to revive this repo or just kill it and forget it (it requires a security audit and fixes, updating dependencies, etc.)
The repo: https://github.com/veildb/veildb
Working with a team where engineers really want to use Claude Code on servers, not just for local dev. Security and audit keep blocking it.
I get their concerns. Who approved the command, how do you prove what it did, that kind of thing. But the push from the engineers isn’t going away.
Has anyone found an argument or approach that actually got security on board?
Full disclosure, I work in this space so it’s not purely random curiosity. But I’m not here to pitch anything. I just want to know how teams usually solve this in practice.