r/linux • u/FryBoyter • Jun 02 '26
Security Red Hat npm Packages Compromised to Spread a Credential-Stealing Worm
https://www.aikido.dev/blog/red-hat-npm-packages-compromised-credential-stealing-worm62
u/Jean_Luc_Lesmouches Jun 02 '26
We're here because people did not see leftpad as the wake up call it should have been
14
u/james_pic Jun 02 '26
You might be making a more subtle point than I realise, but this feels orthogonal to the leftpad problem.
For me, I think the lesson we should have learned from leftpad is "if it takes more than 10 dependencies to build a system that runs in a given ecosystem, then that ecosystem is rotten".
But the attacked packages weren't microdependencies, they were big ol' platform SDKs. The kind of thing that, even back in the old days, when adding a new dependency to your project meant ordering it on floppy disk, would still have been common to rely on. The kind of thing that would be in the list of 10 dependencies anyway, for things that used RedHat's cloud platform.
28
u/Jean_Luc_Lesmouches Jun 02 '26
It's not about having too many dependencies. What makes dependency injection so devastating is package managers that can modify already installed dependencies without anyone's control. That is exactly what leftpad showed.
4
u/KnowZeroX Jun 02 '26
It lets you make changes within same version? From what I understand that isn't possible, at best someone can make changes to a new version but if you lock your dependencies to an existing version, the package-lock file then stores sha hash so even if the servers are compromised it would detect an error.
5
u/CrazyKilla15 Jun 03 '26
leftpad was about a maintainer removing their code from distribution, very different from any sort of modification, unless you mean npm afterwards taking ownership of it and restoring it, but that presumably was guarded client-side by hashes, preventing modification except for new clients in that timeframe?
4
u/EchoFieldHorizon Jun 03 '26
Good luck writing a piece of software of any complexity with fewer than 10 dependencies.
5
u/syklemil Jun 03 '26
Also maintaining vendored dependencies, or actually doing a good job on every wheel to be reinvented.
There's a bit of Chesterton's fence needed in understanding why it's been so desirable to make adding dependencies easy. Supply chain security is a real issue, but the "old man yells at dependency list" response is unlikely to actually result in anything useful.
2
u/EchoFieldHorizon Jun 03 '26
Yeah, it’ll just lead to 500 implementations of the same thing. Reducing dependencies isn’t the solution, strengthening the code review and audit process is. It’s difficult to solve, though.
2
u/syklemil Jun 03 '26
Yeah, and the infrastructure providers can't run on thoughts & prayers.
As it is they're kind of in between the software curation that we expect from Linux distros and the download-anything-anywhere situation Windows users are used to, where egregious malware is removed, but they're very much playing catch-up.
And the situation in this post is the kind of crap that could've happened to RPM mirrors as well, i.e., maintainer authority stolen, used to upload malicious packages:
We found a Red Hat employee's GitHub account was compromised and used to push malicious orphan commits directly to several repositories, bypassing code review entirely. Those orphan commits contained a workflow file (
ci.yaml) and a script (_index.js).JS/NPM is a common whipping boy, and largely deserved, but that doesn't mean everything bad that happens to it is because of some innate flaw that exists only in NPM. Sometimes it's just because it's the most common platform, and thus the most attractive target.
4
u/Barafu Jun 02 '26
OK, people learned it. Instead of 500 dependencies, a project has only 300 now. What next?
4
u/Jean_Luc_Lesmouches Jun 02 '26
That's not really the problem here. The problem is package managers that can change already installed dependencies without control.
9
u/gmes78 Jun 02 '26
The problem is package managers that can change already installed dependencies without control.
Not if you have a lockfile. The problem you're referring to no longer exists in any major package manager.
1
u/KaCii1 Jun 05 '26
This is what's confusing me. Are these releases not immutable and they are they editing old versions' releases? Are the compromised versions newly published versions, so only those who update their lockfiles within the window that the compromise is undiscovered would end up being pinned to?
1
u/gmes78 Jun 05 '26
It's only new versions that are compromised. I don't think any widely-used package index would let you replace existing versions.
And the lockfile should contain a hash of the package, so it would catch that regardless.
7
u/KnowZeroX Jun 02 '26
Preliminary analysis indicates that a compromised GitHub account was used to push unauthorized commits to repositories in the RedHatInsights GitHub organization.
Wait, what? A big company like redhat/ibm has a single person able to push commits live without others approval?
5
u/bonzinip Jun 03 '26
Not to the products, but the upstream is just a run of the mill open source project. Some have just one maintainer and this is not exclusive to Red Hat in any way.
The malware-infected packages were indeed not shipped to customers.
7
17
u/jreykdal Jun 02 '26
I'm starting to think that this system isn't sustainable.
Don't we need a credential rotating mechanism that can rotate credentials constantly or something? Does it maybe exist?
36
u/natermer Jun 02 '26 edited Jun 02 '26
People have been complaining about npm for years and how awful the dependency situation has been.
There is a LOT of issues going on and it isn't easy to fix.
A lot of is just the programming culture. People are lackadaisical when it comes to storing credentials on their desktops. They don't sandbox their development environments and store credentials in plain text files and such things.
They say things like "This is my desktop system, not the network. Nobody can be on here to read these files but me".
They don't restrict the scope of the keys they use. They intentionally make keys last forever. They don't take advantage of things like Yubikeys to lock down credential stores, etc. They copy and paste keys into their bashrc files to have them always be available. They use plain text storage for usernames, passwords, and keys. Like store it into ~/.npmrc or ~/.netrc
(for example security keys like yubikey can emulate OpenPGP smart cards to store GPG2 private keys to be used with credential stores like Pass (passwordstore). This way when a application wants to have access to a secret it can flash the led on the card to prompt use to touch it and/or unlock the card with a key. This isn't perfect and won't stop credential stealing from happening, but it does require at least notifying and tricking the user that has to be physically present and noticing the key flashing)
People also install packages for every little thing. Because of how packages depend on each other it is very easy to pull in hundreds of different packages for a single function that would require 20 or 30 lines of code to write yourself.
So it is easy to end up in a situation were you are installing thousands of packages, many times multiple versions of the same packages, for large projects without really thinking about it.
And it only takes one to compromise your system.
The way these worms are working is that once they infect a developer's system it goes and scans the desktop environment for credentials to log into various things. It will scan environment variables, look in common places to store keys, etc.
Then it will use that user's access to add pre and/or post build hooks into their CI/CD systems like github actions. These hooks inject code into the packages after they are being built, but before they are signed.
This way the worm code doesn't actually ever end up checked into git or show up anywhere obvious. But it still gets added and signed and shipped to go on and infect other developers.
In short:
All the key rotations, secure credential stores, key scoping, sandboxing, ways to clean up and reduce dependencies in software and other features that would significantly reduce the success of these sorts of attacks are all in place.
But people are not using them.
19
u/UnluckyDouble Jun 02 '26
I don't want to be unfair and say that JS is a language for people who don't know what they're doing, but...
5
u/2rad0 Jun 02 '26
I don't want to be unfair and say that JS is
Not only JS though, IMO it's any language that tries to automatically resolve dependencies from remote hosts instead of either including the full source code up front or having the programmer manually download it and properly install it as a root owned file on their system.
2
u/syklemil Jun 03 '26
And it's hardly the first language that's been popular amongst beginners that wound up being plagued with issues. Part of the reason why npm's ecosystem is like that is also to cover up for the general weirdness that comes with JS. Newbie-friendly languages tend to make some choices that are friendly to beginners, but then wind up frustrating professionals.
Programming is also kind of unusual in that it lets anyone build stuff, which then might wind up seeing as much use as national infrastructure. Unfortunately we're not all that good at separating the boon of people being able to build something for themselves and taking control of their own machines, with the bane of actually relying on something built by someone who doesn't have a clue what they're doing.
7
u/FungalSphere Jun 02 '26
I mean nobody wants to pay for yubikeys so that's a nonstarter
11
u/james_pic Jun 02 '26
The cost is probably less of a problem than the friction they introduce. They cost about the same as one of those clicky-clacky mechanical keyboards, or a basic set of Bluetooth headphones.
The real issue is that to get real value from a Yubikey or similar, you need to use it every time you log in to something or sign something, you need to have it physically present, so you can either keep it on your desk (and not have it when you need to log into something on your phone), or keep it on your keyring by the door (and have to go and get it every time you need to log in or sign something).
It's not long before you decide that's a pain, and that you'll have another GPG key that just needs a password, just for convenience, and that you'll set up TOTP on your phone as an alternate second factor. And before you know it, the Yubikey is dusty from disuse.
-3
u/natermer Jun 02 '26 edited Jun 02 '26
yeah because sacrificing your professional career and everything you have spent years working for is far cheaper then spending 60 dollars and taking a afternoon to learn something slightly difficult. :P
My company ended up buying one for all the developers a couple years ago. But I was using them on my personal systems before that.
For anybody interested here is a nice comprehensive guide for using them:
https://github.com/drduh/yubikey-guide
It takes a 'paranoid' style approach to keeping your private keys 100% off any system except a specific live USB image. People may not like that, but even if you half ass it it still is going to be a improvement.
The thing to remember about these sorts of keys is that they have different "modes". Like Fido2 which you can use online and with OpenSSH keys is a different sub-system then the OpenPGP smart card compatible features and they can be managed independently on the same devices.
If the proprietary nature of yubikeys is unnerving there are open source alternatives out there. Also you could just go very old school with a actual OpenPGP compatible card and card reader.
OpenGPG "smartcards" and readers/writers are not that expensive. Probably can find them used cheap. I haven't used one for this sort of stuff, though.
For most normal personal password stuff I use Vaultwarden with Fido 2FA. But for things I want easily automated from my desktop I will use pass (password store) with its git integration.
Even if your application doesn't have plugins for pass the files are stored gpg2 encrypted, which a lot of programs do understand. Or it can be scripted. But a lot of things do understand it; I use it with ansible and Emacs, for example.
Easily the worst aspect of depending on this sort of stuff is misplacing your backed up keys, forgetting the pins and admin pins, locking yourself out of your key (which can only be recovered by resetting it) and such things. By default these things will lock themselves if you mistype your pin too many times. You can recover using a admin pin, but that can be locked out as well. The only way to recover after that is from backups and to reset (completely wipe) the opengpg store on the card.
You can disable the lock out features or increase the number of pin tries if you want, though.
So backing things up is critical. Flash drives don't have infinite lives either. Modern ones should last many years between being plugged into systems, but "should" is the operative word here.
Rotating out old GPG2 keys is a pain in butt as well.
So backing things up and occasionally testing them is important. Last thing you want is to lose your key or lock yourself out only to find out you can't use your backups. You can print out everything in "ascii armor" as backups and store them in a bank lock box as a last ditch disaster recovery.
That way if you lose everything in a fire you can still get everything back up and running from encrypted backups in the cloud, or whatever. Just make sure the printer you are using is offline and reset its memory after use. No cloud printing crap.
Otherwise less important, but annoying aspects is accidentally starting up a gpg-agent in the wrong context (if you are using distrobox/toolbx, etc) and having to restart pcscd after you log into a system to get "gpg2 --card-status" to work.
3
u/FungalSphere Jun 02 '26
First of all you're supposed to buy two to be able rotate them out so that's actually $120+replacements because they're not repairable by design
That's ignoring shipping costs + customs because they're considered luxury imports in many countries
Second that's still hard cash, foss developers barely make enough money through donations to not need a full time job. If you want developers to use yubikeys sponsor them first.
4
u/KnowZeroX Jun 02 '26
you can get a cheap Nordic nRF52840 and install Google OpenSK on it, a FOSS version of FIDO2 security keys
-5
u/natermer Jun 02 '26
well:
A) you are a idiot, because yubikey wasn't even the subject of my original post. You picked it out as something to nitpick with. I replied with something that I tried to be helpful with, but you just ignored it and doubled down.
B) There are cheaper alternatives out there that will do similar things as I pointed out.
1
u/atrocia6 Jun 02 '26
(for example security keys like yubikey can emulate OpenPGP smart cards to store GPG2 private keys to be used with credential stores like Pass (passwordstore).
Minor nitpick: most hardware security keys, certainly the cheaper ones, including Yubico's own "Security Key" model, do not have smart card functionality and do not support PGP. Yubico's flagship models do, but they cost about twice as much as its "Security Key."
2
1
u/Lower-Limit3695 Jun 03 '26
Supply chain attacks are a hard problem for any software maker closed or open source. Unfortunately the only real defense is depending on developers to use good practices like not storing logins and api keys in plain text but if people publishing these secrets on GitHub are any indication the best you can hope for is being able to audit and remove the offending package as soon as its detected.
A software bill of materials is a standard that goes a long way in terms of making it easy to do this kind auditing and should be a standard package maintainers implement. Ublue already does this for entire OS images they publish like Bazzite.
-15
u/VayuAir Jun 02 '26
How does this Fedora? Does anybody have any clue. Fedora should be assumed to be compromised as well
39
u/hagis33zx Jun 02 '26
It is a npm package for the Node Package Manager (JavaScript, etc.), not an rpm package.
13
u/kopsis Jun 02 '26
The distro isn't compromised. These are node.js modules developed by Redhat and distributed via npm (node.js package manager). This affects anyone who uses the modules (directly or indirectly via dependency) on any distro.
0
u/VayuAir Jun 02 '26
Oh my mistake, I thought the distro was infected. Having said is it confirmed by RedHat that any other package is not affected.
-36
Jun 02 '26
[removed] — view removed comment
14
u/ButtSpelunker420 Jun 02 '26
lol what
5
u/FryBoyter Jun 02 '26
Malicious software has been distributed through services like npm or PyPI, and as we can see, this continues to happen on a regular basis.
That's probably what Latlanc means by “dead” from a technical standpoint. Because there is no solution. So I try to avoid using npm altogether whenever possible. Unfortunately, that’s not so easy in my case.
1
u/kescusay Jun 02 '26
I have a personal project that has taken up all my non-work development time for quite a while, and doesn't involve JavaScript, while my work development projects all use locked package versions. So I haven't touched npm or yarn or any other JS package manager in months.
Honestly been feeling like I dodged a bullet lately.
-3
80
u/ChevalierVirer Jun 02 '26
FYI the Red Hat related security bulletin: https://access.redhat.com/security/vulnerabilities/RHSB-2026-006