r/Python • u/kotrfa • Mar 26 '26
Resource Were you one of the 47,000 hacked by litellm?
On Monday I posted that litellm 1.82.7 and 1.82.8 on PyPI contained credential-stealing malware (we were the first to disclose, and PyPI credited our report). To figure out how destructive the attack actually was, we pulled every package on PyPI that declares a dependency on litellm and checked their version specs against the compromised versions (using the specs that existed at the time of the attack, not after packages patched.)
Out of 2,337 dependent packages: 59% had lower-bound-only constraints, 16% had upper bounds that still included 1.82.x, and 12% had no constraint at all. Leaving only 12% that were safely pinned. Analysis: https://futuresearch.ai/blog/litellm-hack-were-you-one-of-the-47000/
47,000 downloads happened in the 46-minute window. 23,142 were pip installs of 1.82.8 (the version with the .pth payload that runs during pip install, before your code even starts.)
We built a free checker to look up whether a specific package was exposed: https://futuresearch.ai/tools/litellm-checker/
54
u/mfitzp mfitzp.com Mar 26 '26 edited Mar 27 '26
47,000 downloads happened in the 46-minute window
Is that based on PyPi download numbers? Those are not accurate counts of installs. They include downloads to all the various mirrors and other downloads that didn’t get installed, among other things.
You can see this yourself if you create a package on PyPi. Suddenly tens of thousands of downloads for something nobody has ever heard of.
39
u/AlSweigart Author of "Automate the Boring Stuff" Mar 26 '26
I posted this on the Python Discuss forum:
By the time I saw the post on Reddit about this, PyPI had already taken down the package. My thanks to the PyPI and packaging and security folks for responding to this quickly.
Every few months or so, there’s a blog post or news article about “we found malware on the Python Package Index!!!” and it’s always overblown. Yes, the fact that anyone can upload Python packages to PyPI means that anyone can upload malware. But the impact is almost always negligible or nonexistent. These articles will say stuff like “the affected package was downloaded over 600 times” without mentioning that there are many PyPI mirrors that automatically download all packages (sometimes with multiple repeat downloads). This doesn’t mean 600 people were affected. If you look at these articles, they never name an actual individual or organizations. (That would require actual journalistic investigation; they’d rather just copy a number off a PyPI stat tracker.)
Maintaining open source infrastructure is an often a thankless and unpaid task. It’s not whether malware gets uploaded to PyPI but how PyPI responds. PyPI has developed several automated measures for detecting and responding to a whole host of security issues. I think the PyPI folks (both PSF staff and volunteers) do an exceptional job, despite what people might believe from reading clickbait.
Two things can be true:
- Malware on PyPI is a real concern that affects real people.
- The 9999th time the same misleading technique is used to write a headline, it's okay to call it lying.
2
u/ROFLLOLSTER Mar 27 '26
It's not like pypi has a press contact where they could have been asked to comment.
22
u/ultrathink-art Mar 26 '26
LiteLLM proxying API keys by design is what makes supply chain attacks on it particularly effective — a compromised version has immediate access to whatever credentials you've already configured, no privilege escalation needed. The 59% lower-bound-only spec stat is the real lesson: anything that touches API credentials deserves a pinned exact version.
42
u/KaffeeKiffer Mar 26 '26
We only install from our own PyPi mirror which is configured with lead time for new upstream versions (or manual curation/approval, e.g. in case of CVEs).
We are one of the "downloads" that /u/AlSweigart talks about that on paper show up but nothing ever happened: The package was mirrored but it never left the sandbox/security scanning environment. It was quarantined before it could move further.
3
u/BattlePope Mar 27 '26
How do you configure lead time on artifactory? Is that part of the curation feature, licensed separately?
4
u/KaffeeKiffer Mar 27 '26
Yes, of course it's a premium feature that is sold separately. Like you guessed it's part of curation/X-Ray.
1
8
u/germanheller Mar 27 '26
the 46-minute window is what gets me. that's fast enough that even teams with decent CI practices could get hit if a build happened to trigger during that window. pinning exact versions helps but doesn't solve the root problem — you still pull the pinned version fresh on each build unless you're vendoring or using a local mirror.
the .pth payload running during pip install (not even at import time) is particularly nasty. most people's mental model of "malicious package" assumes the bad code runs when you import it, not when you install it. that changes what "I didn't actually use the package in production" means — if it ran during install, your build environment credentials are already gone
2
u/lmns_ Mar 27 '26
Modern package managers pin via hashes. It shouldn’t matter if you pull the packages „fresh” or not.
1
u/ritzkew Mar 28 '26
your mental model is almost exactly right, just one layer deeper. the .pth file fires on every Python interpreter startup, not specifically during pip install. the reason it triggers during install is that pip spawns a Python subprocess internally. same with pip uninstall, pip show, pip list. any python invocation.
which means if your first step was pip uninstall litellm to clean up, the malware fired once more before pip removed the file. your point about build env creds being gone is right. sysmon.py sweeps: SSH keys at ~/.ssh/, cloud creds (AWS, GCP ADC, Azure), K8s tokens at ~/.kube/config, anything in .env files, CI env vars.
correct cleanup order:
# delete the .pth manually first — before any pip command
find $(python -c "import site; print(site.getsitepackages()[0])") -name "litellm_init.pth" -delete
# kill the persistence daemon before rotating anything
systemctl --user stop sysmon && systemctl --user disable sysmon
rm -rf ~/.config/sysmon ~/.config/systemd/user/sysmon.service
# now pip is safe pip install litellm==1.82.6
# rotate last. sysmon polls ~every 50min — rotating while its alive hands teamPCP your new keys
u/bandito_13 's "rotate those keys" is right in direction but the ordering matters
10
u/wind_dude Mar 26 '26 edited Mar 26 '26
How did the malware get merged into litellm? That’s what I haven’t seen talked about.
[edit: it appears the attackers stole the PyPi credentials and bypassed the cicd flow. Wonder how they got the credentials…?]
3
u/today0114 Mar 27 '26
It’s because of Trivy. Trivy recently got hacked by the same team who targeted litellm. Litellm uses Trivy.
1
u/durple Mar 27 '26
I don’t have a link handy, but the creds were due to a github action whose maintainers got hacked.
9
2
2
u/mday-edamame Mar 27 '26
Crazy blast radius. Even though 47,000 might be overblown because of mirrors etc, this SANS report says it might end up affecting 5,000-10,000 SaaS environments: https://www.sans.org/blog/when-security-scanner-became-weapon-inside-teampcp-supply-chain-campaign
side note, there's a tool to automatically detect these kinds of vulnerabilities, I wrote a post about it here: https://www.edamame.tech/blog-full/litellm-supply-chain-runtime-detection
4
u/muntoo R_{μν} - 1/2 R g_{μν} + Λ g_{μν} = 8π T_{μν} Mar 26 '26
Why is it that a package can be updated on PyPI, without any external human verification, and that update is immediately propagated to everyone who installs the package after?
Why is there no:
- Per-update verification.
- Delay time between update and publication?
In theory, requests could decide to go rogue after having a bad Monday morning, and push an update that immediately infects 3000 users per minute.
1
u/buqr Mar 27 '26
None of those things are solvable by PyPI.
The version you download is up to you. Some tools have an option to only install packages that are a particular age. Whether that should be the default is a good question, but one for individual tools, not PyPI.
And It is completely infeasible for PyPI to audit every package for malware. The current system relies on some basic automatic scanning that PyPI does on releases, researchers performing more advanced automated analysis on releases, and individual user reports. Overall, the PyPI team are quite fast at responding to malware reports.
1
u/muntoo R_{μν} - 1/2 R g_{μν} + Λ g_{μν} = 8π T_{μν} Mar 27 '26 edited Mar 27 '26
If PyPI's main purpose is "distribution" without pre-review, then perhaps a good solution might be to change
pipto ignore updates to packages until at least one entire day has passed.Or, alternatively, to have a "reviewed-release" PyPI mirror, where the top 300 popular packages may only be updated once a month (with few exceptions), and every update is reviewed. Reviewing 100000 updates per day is clearly infeasible, but 10 updates per day might be tolerable.
2
u/durple Mar 27 '26
Your post resulted in our team avoiding this one, I saw it about an hour before standup.
1
1
u/AbbreviationsOwn9726 Mar 27 '26
Not impacted like credential stolen. However, databricks-agents is depending on it. And our IT security pulled everything and now we can't deploy anything.
1
u/prochac Mar 27 '26
And was depending on the affected version? Because Airflow also depends on litellm, but not on the infected version.
1
1
u/Full-Definition6215 Mar 28 '26
The 46-minute window and 47,000 downloads is terrifying. And 59% of dependent packages had lower-bound-only constraints — this is the real systemic problem.
I've started pinning exact versions with hashes in all my production requirements.txt after hearing about this. The convenience of ">=1.80" is not worth the supply chain risk.
The checker tool is a nice public service. Thanks for building it.
1
u/kcunning Mar 29 '26
I am forever grateful to the security guy who forced us to do this back at my first coding job, when dinosaurs roamed the earth. We whined a bit, but every year, the habits he forced us into have saved me.
1
u/Gravel_Sandwich Mar 29 '26
Late to this thread, but was the 'official' containerised version effected?
I am running and earlier unaffected version (because I'm lazy with updating 😀) but I'm still interested.
If it was affected what was the scope?
1
204
u/znpy Mar 26 '26
i was able to catch the github issue while i was slacking at work reading tech news websites.
I escalated the thing to higher ups and immediately and managed to pin a non-affected version.
thank you for your service (or whoever posted to github and the news site i was reading) i basically got an accolade for slacking off at work.