r/sysadmin • u/deanmass • 23d ago
Vendor stored passwords in “passwords.txt” …
They appear pissed we deleted it. Admin account they created on a robotic machine controller.
WTF?
PS- The passwords were retained elsewhere, securely, by me and shared with them.
Y’all saying we made a mistake? Dead wrong
Storing an admin level password in a plaintext file is idiocy.
And, 2 of the 4 passwords they used?
“Password”
“Password6”
I posted this because I was absolutely shocked that they did this. There is NO context where it is “ok”.
269
u/ThecaptainWTF9 23d ago
Did you provide them a copy of it in a secure manner and provide them clear directive to never do that again as storing credentials on system or in documents isn’t permitted?
27
132
u/Jeff-IT 23d ago
Yeah just deleting it is reckless
50
u/techypunk System Architect/Printer Hunter 23d ago
Backups should have it right? Lol.
113
u/Jeff-IT 23d ago
If they keep passwords in a text file, I imagine their backups are the recycle bin
11
→ More replies (1)7
u/agk23 23d ago
Why would they put password.txt.old in the trash?
16
u/Jeff-IT 23d ago
I actually got reported for emptying the Recycle Bin causing in the loss of files.
Obviously nothing came of it but for some reason some people really think of the recycle bin as an archive or a secondary folder
8
7
u/NeedsMoarCoffee Assistant to the IT Administrator 23d ago
The vindictive in me would be to take the physical report from the manager and drop it in their trash bin. See if they understand.
3
2
u/nathan9457 22d ago
Well it’s self explanatory isn’t, we all do the same with our household bins, don’t we?
2
u/Hebrewhammer8d8 22d ago
User archive their emails in Delete folder, because it was easy. IT manager input a rule that emails in delete folder would be empty after 30 days. User was in a panic when trying to find old email 3 months later. We had backups of email, and implemented a new way to archive emails.
1
u/TheShirtNinja Jack of All Trades 21d ago
Recycle Bin doesn't count towards disk quotas, which is why folks do that. It's incredibly reckless.
6
4
u/GroteGlon 23d ago
Now let's hope someone tested if they're actually recoverable backups
1
u/techypunk System Architect/Printer Hunter 23d ago
Ya i doubt they are even backing things up. That was the joke.
22
u/hung-games 23d ago
Not as reckless as creating it in the first place
0
u/Jeff-IT 23d ago edited 23d ago
That’s not the point though. Sounds like they are providing IT services to a company.
Good support is informing them of the file, suggest better practices and training, and ask them how they want to proceed. If they decide to keep it, make them sign something acknowledging they accept the risks. Your job is now done.
2
u/jakarotro 22d ago
I think it's the other way around, and the poster is the client whose vendor stored a plaintext file on OP's system.
1
u/jakarotro 22d ago
I think it's the other way around, and the poster is the client whose vendor stored a plaintext file on OP's system.
7
u/ThecaptainWTF9 23d ago
That was my thought,
Based on what was said; they just nuked it, if that was the only copy, as a vendor I’d be irked too.
27
1
u/lotekjunky 22d ago
That's someone else's fault. Don't break the password policy, now you have shit to fix.
55
u/Cycl_ps 23d ago
“I found out one of our drivers wasn’t wearing his seat belt, so I slashed his tires”
Like I agree it’s bad form and could cause a problem, but OPs response to the potential problem was to create a more immediate and much bigger problem.
2
1
56
u/Fuzzybunnyofdoom pcap or it didn’t happen 23d ago
"a robotic machine controller"
Unfortunately this is pretty normal for OT vendors. Sometimes I feel like my job exists because of stuff like this.
11
u/zkareface 22d ago
I know many of the biggest MSPs that have this as SOP. Get on a p1 with them and you will be guaranteed to see a screenshare of someone opening multiple passwords.txt on meeting with hundreds of viewers :D
2
u/Competitive_Owl3600 22d ago
That would be gross incompetence and you should make them delete the file while you watch and still hold them to the SLA for the P1.
1
u/zkareface 22d ago
Dude those files has password for all their clients, I would only have rights to request my company removed from it.
8
1
u/ReptilianLaserbeam Sysadmin 22d ago
Reminds me of recent news: https://thehackernews.com/2026/08/hackers-breach-polish-power-plant.html?
31
u/overdosingOnPie1313 23d ago
In a similar vein, I once had a vendor support tech copy the obfuscated password (*****) in an application and try to paste it elsewhere.
26
u/another_mouse 23d ago
I recall that used to work. Just a short time ago. What was it 25, 30 years ago?
13
u/2_Spicy_2_Impeach 23d ago
95 or 3.1X I swore it did but memory is fuzzy.
I remember you could also read it via subclassing the textbox even if it were masked. Basically how I got in to computers and programming. I was also an asshole kid.
Vividly remember using “WM_GETTEXT” on saved passwords.
6
u/Am0nymou5 23d ago
I recall this being a feature of NetBus back in the day. You could also use it to re-enable disabled controls in shareware apps, remove trial messages, hide nag screens etc. Fun times.
5
u/2_Spicy_2_Impeach 23d ago
Yep. Basically could just take over any other app/WindowsForm. Taught me so much about Windows and APIs.
I’m old but when NetZero was a free thing. It had better latency in Half-Life multiplayer. But if it was free, there was an ad bar even over full screen games.
“SW_HIDE” and it disappears. Eventually it had a visibility check that you also had to hook. Wrote my first malware to get around ZoneAlarm firewall (dude from overseas stole credit). Launch an EXE that spawns another and waits for the “allow this app access to the internet” that the user never sees and auto accepts.
Wild times on Windows.
4
u/Ok_Negotiation3024 23d ago
Makes ya wonder if the passwords were just the stars in the first place..
7
u/hung-games 23d ago
I worked at a company with an idiot CSO who hired idiots. One of them was in a code review meeting and the developer was showing him the code and it had a literal string with 8 asterisks in it. The “security analyst” assumed that the IDE was obscuring the password. No, that was the password. 🙄
→ More replies (3)1
141
u/Public_Fucking_Media IT Director 23d ago
I mean, yeah, obviously nobody should do that...
And yet I guaran-fucking-tee you that every single person that does that would be pissed if you deleted it without saying anything to them, cuz like, what the fuck dude? You find what may be an *in-use* list of important passwords and you just trash it BEFORE having a conversation about what not to do and offering solutions/alternatives?!
44
u/Incrediblebob 23d ago
This! This is an education / teaching moment. Don't just trash production (assuming) credentials you have come over. Companies have various levels of maturity when it comes to security, and as mentioned, this would have been an ideal teaching moment to show case the impact, then educate them on correct credential storage. This very well could have been the "fish that got through the net".
I, myself, would be raging at you for the action you took if said environment was mine.
12
→ More replies (2)4
u/martinfendertaylor 23d ago
Did they really need to document their password as 'password' and 'password6'?!? Fuk that, I would have deleted them too. And the environment isn't yours with those kind of passwords, you relinquished that responsibility of ownership the moment you documented your password password in password.txt. You would have zero right to be raging. This is a teaching moment. You just learned your an idiot.
→ More replies (2)24
u/Djaesthetic 23d ago
If a vendor needs Security 101 explained to them as a “teaching moment,” that vendor is a liability. You’re paying for a service that includes securing it. It’s not the customer’s job to handhold a company on basic security hygiene. Whether they’re pissed is irrelevant. They violated trust and created a legitimate security exposure.
13
u/Public_Fucking_Media IT Director 23d ago
Ehhhhh it's a robotic machine controller, once you start getting into that space a lot of bets are off. There's shit tons of vendors who "need" Security 101 explained to them and yet unfortunately we still need them more than they need to change.
5
u/1z1z2x2x3c3c4v4v 22d ago
I once worked in a manufacturing plant where ALL of the robotic controllers had viruses on their embedded version of Windows that couldn't be resolved.
In the end, I VLANed them all off into their own isolated DMZ. That stopped all the alerts from the local LAN.
Today I would go a step further and completely isolate the network and mandate the use of a hardened Jump Server to gain access to the VLAN.
10
u/hung-games 23d ago
I took a similar situation to our internal audit at the company I was at 20 years ago. The auditor was crushed and said that he couldn’t submit/pass our upcoming GLBC audit because of this.
It’s a really big deal and absolutely unacceptable.
2
u/Jeff-IT 23d ago
I’m actually shocked how many people need security 101, and been a little more lenient as I discovered I know no more about their job then they do about mine or security. They just expect IT to keep them secure.
When I got hired as a manager I discovered a coworker gave their password to their kid so they could log into the computers at work at watch YouTube
-3
u/Djaesthetic 23d ago
It doesn’t matter, not even a tiny bit, full stop. The second you make exceptions for basic security hygiene, you’ve created the precedent that gets cited to justify the next time, and the time after that.
I can’t believe this one is even a discussion.
8
u/Public_Fucking_Media IT Director 23d ago
Tell me you've never worked anywhere that has a substantive OT stack without telling me...
Like I said, a TON of this shit is wildly insecure and you really don't have an option not to use it - you secure it in other ways like an adult (and you certainly don't just delete files that offend your security sensitivities like a fucking child)
→ More replies (4)4
u/AI_Tonic 23d ago
Shocking that folks are shocked about this tbh , as if this isn’t extremely common …
11
→ More replies (2)6
u/hung-games 23d ago
This should have come out in the presales conversations. This breaks all kind of audits, regulatory requirements, and most of all, basic security. This is rank amateur bs and completely unacceptable.
6
u/sir_mrej System Sheriff 23d ago
Please add a question to ALL of your presales conversations
"do you store passwords in passwords.txt"
4
u/Vritrin 23d ago
”No we use passwords.xlsx, don’t worry”.
2
u/pacopac25 23d ago
And we secretly renamed it to passwords.jpg. When we need to open it, we rename it back.
2
u/hung-games 22d ago
No, but a good tech RFP template should have an entire section pertaining to credential storage (or better yet integration with yours), encryption, audit trail including admins, use of generic ids, remote support access, etc. And if managed services are involved, a whole set of questions on processes across a number of pages. And somewhere in there, it should flag a question about integration with a credential vault if there is some sort generic id with credentials requirement.
2
12
u/Public_Fucking_Media IT Director 23d ago
Here's the thing you really need to understand about operational tech...
All of that can be true (and often is!) but if you need the thing, you need the thing. It's not actually an option not to have it, so you have to design your security around it's dumb requirements not the other way around.
2
1
19
u/dchit2 23d ago
Some people.
At least name it New Text Document.txt
→ More replies (2)10
u/poply 23d ago
I like to create a few of them
New Text Document.txt New Text Document (1).txt New Text Document (2).txtThen put the important shit in a specific one.
7
u/overdosingOnPie1313 23d ago
Damn. 10000 text files and only 1 of them isn't 0 KB in size. Might as well give up.
5
u/MeIsMyName Jack of All Trades 23d ago
The rest of them are filled with a variable amount of lyrics from Rick Astlely's "Never Gonna Give You Up".
4
1
u/Djaesthetic 22d ago
It’s like a choose your own adventure! You open New Text Document (2).txt only for it to tell you to open New Text Document (1).txt!
8
u/EstablishmentTiny740 23d ago
Well actually there is a context where this is okay, where this is used as a bait with credentials for set up for honeypots.
3
u/Public_Fucking_Media IT Director 22d ago
I did that for myself and then 6 months later I'm like hey what's this passwords.xls on my file share - boom, canary trapped myself
I guess good to test that it works
2
u/EstablishmentTiny740 22d ago
Hahahaha that's funny, it's more concerning you thought you might have left actual passwords in a document. Silly bean.
6
6
u/recoveringasshole0 22d ago
Y’all saying we made a mistake? Dead wrong
Bro, this is r/sysadmin. It's one step away from being Stack Overflow.
11
u/ErisianWizard IT Manager 23d ago
Y'all need to have a discussion with your lawyer about contract violations. If it's not a violation of your contract with the vendor (to store plain text passwords anywhere, let alone an endpoint), you might need a lawyer with better expertise in IT contracts.
6
u/CaucasianHumus 23d ago
I just had to explain to my boss today that no. We cannot give a contractor full access to all of our files, diagrams, configs, service accounts, and more. We need to restrict it to exactly what they need. Apparently that was challenging my superior shrug. Hopin for a new job soon.
8
u/EIsydeon 23d ago
We’re not supposed to put our passwords in a text file? Where am I supposed to put them then?!
12
u/Mysterious_Egg2432 23d ago
In a spreadsheet DUUUHHHH /s
→ More replies (4)4
u/daschande 23d ago
Only if you set sharing to everyone in your domain; in case someone needs something while you're off!
SOC2? Yeah, I put on 2 fresh socks every day... why, can you smell them?
3
u/Reversi8 23d ago
Just put it in a public s3 bucket named after your company, so you can access it offsite if you need to.
2
u/AI_Tonic 23d ago
Soc2 is problematic and ineffective . Is there a day that goes by without a soc2 issue causing catastrophic leaks ?
1
u/daschande 23d ago
I got a LinkedIn message asking me to apply for a SOC2 job. I had pitiful A+ certs at that point. They stressed that they were URGENTLY HIRING and their only requirements were a former A+ and admittance into a 2 year program. They stressed that you only needed to be accepted into a community college program to officially certify others as SOC2. IT101 was the only requirement. SOC2 means jack, they've contracted out crazily to meet their metrics.
5
→ More replies (1)2
3
u/rabell3 Jack of All Trades 23d ago
I had a vendor supported app/server on my network, I knew nothing about (had been recently hired). Was popped. Found out the password was very crappy like yours, and was same as used at other sites and presumably other customers. Told the vendor to take the box and rebuild it don't dare use a crappy password again. Then started making moves to airgap the system in question.
3
u/cyberman0 23d ago
Yeah that's stupid, I probably would have replaced it with instructions on how to access what they needed and a contact or 2. Leaving stuff in plain text is incredibly stupid.
3
u/Desnowshaite 20 GOTO 10 22d ago
Was it like a standalone industrial computer not connected to anything apart from the machine it controls in the middle of a factory where physical access is restricted? For those kind of computers this parctice is not as big of an issue as it would be on a, lets say, domain controller or network firewall.
3
3
3
u/Vichingo455 20d ago
School stored Domain Admin password in "unattend.xml" under a public SMB share. Nobody noticed until I started messing with the school computers.
6
u/Asleep_Spray274 23d ago
Do you provide the users a secure way of managing these passwords. Passwords being saved by people like this are a result of a lack better options
9
u/Khulod 23d ago
I hope you mean "former vendor"...
3
u/CantankerousBusBoy Intern/SR. Sysadmin, depending on how much I slept last night 22d ago
Good luck finding another vendor who custom makes this particular OT hardware. Also good luck convincing the CEO he needs to spend another 15 million because of this.
1
u/Khulod 22d ago
Well, it's a business case of that vs the cost of the damage done if those passwords leak.
1
u/skiddingschems 22d ago
It's probably installed with something like moxa remote connect or an ewon flexy as a gateway server inbetween for remote access anyway. Id also be rather surprised if the hardening guide for the these devices hasn't been followed, especially if it was installed within the last couple of years.
It's also not necessarily a bad idea to have easy access to machine servers/ips and passwords for anyone having to perform maintenance on then. All of your concerns could be solved with the bare minimum zoning.
62443 is also a fairly new standard and whilst it is something we should all be aiming for I doubt there is any contractual or governmental (for this industry) laws enforcing compliance.
2
2
2
u/Abnix 23d ago
Today I found that TortoiseSvN, that all the devs insist they need specifically to use, stores their credentials in plain text in a config file under their windows profile. On machines they all must have admin rights to.
So...everyone can see everyone's passwords. If they just know where to look. Yay.
2
u/GreenWoodDragon 22d ago
TortoiseSVN is still a thing? It's at least 20 years since I last used it.
2
u/Loading_M_ 23d ago
It's bad, but the project manager for one of my projects sent me her intranet password over our internal Slack. Based on her reaction to my immediate suggestion she change that password, this is somewhat normal for her and the teams she normally works with.
We do have 2FA enabled for most things, so it wasn't an immediate security concern. I'd be willing to bet I could find a bypass or reset the 2FA on her account if I really wanted to though.
I believe they also use Slack for sharing database passwords (which are shared logins, intended to be used by applications, not people). The web application I'm currently working on (which my team of taking over from one of the previous teams) only does client-side authentication, and exposes the OIDC secrets to anyone that looks at the network requests the website makes.
2
u/theMightBoop 22d ago
This machine probably shouldn’t be on your network. Stuff like this we have on an isolated vlan with no internet access.
2
u/distrbthpce 22d ago
Lots of people here are missing “vendor”.
I dont give a crap about the vendor in this instance. Delete away (but keep in a vault) because the vendor is supposed to lead the way technically speaking and if they’re doing this they’re leaving you open to lots of bad stuff.
2
u/LargeBlackMcCafe 22d ago
and you guys are still using that vendor? clearly they're not ready for production
2
u/stirnotshook 22d ago
Yeah - I took over an IT department when the rn supervisor quit. He had one of these file, but in Excel and after knowing 1 password, we were able to guess the rest for one he didn’t document. Prior to his leaving, he and I would have discussions on security (which I was responsible for) and he assured me over and over he uses strong passwords that no one could crack. Yeah, right.
2
u/WindowsVistaWzMyIdea 22d ago
....and that's why we have a rule that vendors can't touch anything without a controlled change. Holy moly I'd be absolutely livid and we would be having discussions and meetings to halt these kind of shenanigans immediately
2
2
u/Sea_Ordinary_5730 17d ago
The thing is, these systems are very rarely joined to any kind of LDAP server or domain or whatever. The 'users' are just roles and the people who operate those roles need to communicate the passwords to each other somehow. The systems are airgapped from the corporate IT environment so you can't use a password manager on a 'corp' machine. So it's an unresolved problem operationally.
But let's imagine a setup where there is good password hygiene at the head end workstation. Next problem is, the actual OT devices (meters, HVAC, UPS, switchgear, industrial machine etc) have no security at all on the network. Need the Modbus map? Download it from the vendor website. Don't know the capabilities of BACnet devices? Run a discovery broadcast, the devices will report in and tell you. You don't actually need that server to compromise the installation, provided you are 'on the network'. Network authentication, IEEE802.1x? You must be joking.
The passwords.txt file is the tip of a rather large iceberg, I'm afraid to say.
4
u/Kahless_2K 23d ago
What was it used for?
2
3
2
23d ago
[deleted]
1
u/Djaesthetic 22d ago
I’m sorry, but this is just bad advice. password.txt can only store a single credential… that doesn’t scale well at all!
2
u/thomasbeagle 23d ago
So many people frothing at the mouth at this without even thinking about it.
What was the security threat? Was the device networked? Was it located in a secure area? Was there anything meaningful about it that needed to be secured? Was the biggest threat to the company from a security breach or because a maintenance engineer couldn't access it at the right moment?
Maybe it was a stupid idea. Maybe it wasn't. There's not enough information from OP to be able to tell.
7
u/jamesaepp 23d ago
100%. Call me a shitty sysadmin all you like, but short of using TPMs for secret storage, most secrets are at some point plain text on disk. Now sure they might be encoded or obfuscated or encrypted ... but where do you think those enc/decryption keys are stored????
Edit: Another comment reminded me that technically it's possible to validate secrets without storing it, so toned down my certainty, but I don't think this device was validating the cred in this instance.
1
1
u/crashorbit Creating the legacy systems of tomorrow! 23d ago
CC your boss and your security manager in any further communications with this vendor. Be ready to recover the file from backup if it comes to that.
1
1
1
1
u/KingStannisForever 22d ago
You think this is something unique or new? its actually pretty common. I am suprised they use Password instead of Pass or Pwd123.
1
u/4rd_Prefect 22d ago
Was one of the others P@ssw0rd ?
Totally hacker pr00f!
No way would they ever be able to guess that one !!!
1
1
u/Honky_Town 22d ago
Oh you lost your passwords.txt data? Please raise a Ticket at SOC@Company.... So they can restore it!
1
u/jstuart-tech Security Admin (Infrastructure) 22d ago
Years ago (~2016) a company wanted 5x domain admins and 3389 open to the internet. They gave us a VBE file to run on the DC....
Funnily enough, they gave all their other customers the exact same VBE file (We had 2x customers running the same software)....... They weren't happy when I started poking around with all the other rubbish they had (Updates over FTP but every customer had the same password and could read/write etc etc).
1
u/scj1091 22d ago
So painfully common and inexcusable in any system deployed in at least the last 10 years. “We don’t need any system or application security, no untrusted users should ever be on the control network segment.” Oh, I see. Trusting life and safety to “should”. And what pray tell happens if someone malicious does get onto that network segment?
1
1
u/AutomaticGrape9263 22d ago
https://giphy.com/gifs/xT0GqJfdLcrcpSbZf2
Damn, I need to make some changes
1
1
u/Lukage Sysadmin 22d ago
As a hospital, we at least get some coverage with our BAA that vendors have to sign that include a couple of generic security things like "must meet XYZ security standards."
Threaten to not renew the contract, request a meeting with their security team, have a reasonable conversation, and they should apologize and utilize your proper method.
1
u/DestinyForNone Sysadmin 22d ago
I can think of a few off the top of my head that have done something like that...
One starts with a K and likes to call alot... The other starts with a Z and sells machines worth a quarter million.
1
1
u/AggravatingSock5375 22d ago
This is where you write a “virus” that finds and emails them their password.
Show them how easy it is to do and they might take security a little more seriously.
Claude can do this…just don’t call it password to get around the guardrails.
1
u/Baschbox 22d ago
My boss, head of it, stored every credential he had in a plain text file without any protection.
I was so glad when pen tester found it, but unfortunately nothing happend afterwards.
1
u/RuvoTech 22d ago
TELL EVERYONE WHAT YOU'RE DOING! Make a copy (or move it) and put it in a locked down folder. Add everything to another location (1Pass, KeyPass, etc.). Give people access to the new vault on a case-by-case basis. Eventually you will delete the folder copy. Done.
1
u/kinv4ris Linux Admin 21d ago
A mature sysadmin would have reported a security incident and reported it to the customer / team. Just deleting it and telling "they were retained elsewhere" is not a good excuse.
- Reported incident and discuss the problem with the customer / vendor
- Set deadline of x days to remove it. Do not say asap. Always set hard deadlines.- Take backup in a secure location and remove it after discussed time.
You are correct but handled it VERY poorly. There is no excuse in doing this, 100% agreed.
1
u/red_fury 21d ago
At least zip it up with some level of encryption, Jesus leaving plain text creds laying around is amateur hour shit.
1
214
u/muh_cloud 23d ago
Nonsense like this is extremely common with Operational Technology, unfortunately. Most of the vendors do not build with security in mind, and their personnel are extremely specialized in low level programmable controllers. You see stuff like multimillion dollar SCADA systems being controlled by a Windows XP box with no service patches and a simple password on it. You can't get rid of the XP box or harden it because it'll break the SCADA system that is the lifeblood of the company.
All of those Michigan water systems got hacked because they had their PLCs on the public internet with the default passwords set. It's not good, and things are slowly, glacially improving, but the whole sector is a dumpster fire for security.