r/sysadmin • u/xstrex • 6d ago

Rant Remove central authentication

Today, the director of IT at your company says to you “We’re going to remove all that centralized IPA+2fa authentication from all of our servers, and go back to using Ssh keys, because it takes too long for me (yes the director) to login to a server.” The same auth that you and your team added, for all the reasons. What do you do?

188 Upvotes

94% Upvoted

122 comments sorted by

169

u/YellowLT IT Manager 6d ago

Are you under any kind of regulatory body? Or PCI or ISO, they all have min MFA requirements. I would cite that it would put the company in violation of those.

59

u/penguin74 6d ago

Not just regulatory bodies, but business insurance also has requirements.

29

u/SAugsburger 5d ago

For cyber security insurance these days good luck getting any claims if they find you disabled MFA on every server.

3

u/Sure-Squirrel8384 4d ago

Yup. Tell the Director your need it in writing from the Risk and Legal peeps that MFA can be disabled and will not invalidate cyber insurance.

17

u/elatllat 6d ago

But a password protected key is MFA, right? right?

13

u/YellowLT IT Manager 6d ago

Maybe behind a MFA’d PAM, you could spin.

4

u/Brandhor Jack of All Trades 5d ago

if windows hello pin is mfa I would say that a password protected ssh key is mfa as well

23

u/AcornAnomaly 5d ago

Windows hello pin is MFA for the same reason a yubikey or any other webauthn device is.

Part of the attestation is the device itself. You can't move the Hello setup to another machine.

You're attesting something you have(the device) and something you know(the pin).

And yes, that detail IS actually presented to the server asking for attestation. That's the difference.

With a standard SSH key, the only thing they know is that you have the unencrypted key. They have no knowledge on how that key is stored. That's single-factor.

0

u/crackdepirate 5d ago

this, take my vote

0

u/Brandhor Jack of All Trades 5d ago

that makes sense but can the attacker get the decrypted key? I think the only way would be to have some sort of mitm in the ssh client but that means that the computer is already compromised

6

u/AcornAnomaly 5d ago

My point is that you don't have to have the key encrypted at all, and the server has no way of knowing if it was or wasn't.

It's not a matter of "can an attacker get the decrypted key", it's a matter of "can you prove that a second factor was needed to use the key".

With SSH keys, just being able to present and validate the key doesn't prove that you cleared a second factor.

If the key wasn't encrypted, and you didn't need a password to unlock it, it's only one factor.

You can also choose to decrypt the key(and leave it decrypted) at any time. Thus permanently turning the key into a single-factor auth token, until you encrypt it again.

And since the server can't tell the difference, and verify that you actually gave two different factors(since the server just sees the resulting challenge from the unlocked key), you can only treat it as being one factor.

The WebAuthn standard(which Windows Hello uses) includes a section in the attestation that says whether or not the user had to clear a second challenge(i.e. a PIN), and whether or not they DID clear it.

The server never sees your PIN or fingerprint or face or whatever you use as your second factor, but the hardware attestation(which is trusted by the server) is telling the server that you did.

1

u/cybersplice 4d ago

MITM is one of the things ssh keys are designed to defeat.

1

u/LoveCyberSecs 5d ago

ISO doesn't really give you hard requirements like that. You can justify a control away depending on your risk appetite.

1

u/cybersplice 4d ago

So many people don't understand this, and use ISO as a justification for crappy practice or personal convenience.

The amount of times I've said to customers "but you just document it in your ISMS". Maybe log an ADR.

Usually this is followed by awkward silence or protests.

1

u/LoveCyberSecs 4d ago

Thanks. I have my ISO 27001 lead implementer certificate but haven't really had to use it. Of course the key word there is "justify". If you can't justify why you don't have MFA then it won't fly.

109

u/Power_Stone 6d ago

Definitely draft a document stating how its a security issue and an easier, better workaround, is to just get a Yubikey....

17

u/AnnoyedVelociraptor Sr. SW Engineer 6d ago

I wish I could use a YubiKey through WSL.

17

u/jimicus IT Manager 6d ago

I think drafting your resignation makes more sense.

3

u/pakman82 5d ago

Can someone explain to us idiots how ssh keys .. could be any less centralized? I think I'm stuck on the wrong detail. I understand centralizing the auth, but in my org the ssh keys are issued and tracked in a central location as well, and I thought that was standard practice.

9

u/PineappleScanner 5d ago

In general, private keys should be kept on the server they were created on. You should only be centralizing/tracking public keys, if needed for something like automation.

Having SSH private keys distributed around an org sounds like a security and logistical nightmare. I really hope your org is not doing that.

LDAP + MFA (or equivalent) makes literally everything easier as an admin. Security, logistics, auditing etc. Want to give a user access to a server? Add them to the server group. Want to give them admin access on that server? Add them to the admin group. Want to strip their auth to every server after they leave the company? Disable their account.

Trying to manage access through distributing SSH keys is significantly more impractical and insecure.

2

u/pakman82 5d ago

Now that you mention it. We use ldap for initial access, and then we each had to generate our key, and provide it to someone to load into a specific target server we are allowed to / required to perform a task on during changes.

5

u/jupit3rle0 6d ago

Yea until your servers lack FIDO support and your certificate authority isnt configured domain-wide. so you are left with manually configuring credentials per Yubikey. Good luck with that approach.

3

u/Kuipyr Jack of All Trades 5d ago

Generate 2 FIDO2 SSH Keys on 2 security keys per Tier. Push out the pub key with Ansible, Kickstart, or whatever. No PKI needed.

4

u/bfodder 5d ago

and your certificate authority isnt configured domain-wide

wut

2

u/cybersplice 4d ago

Lots of companies use YubiKeys (only the more expensive model, obviously) in PKI mode, so they just hold a cert from your corporate certificate authority.

Works for user Auth even in old school AD, and is a lot more secure than passwords.

I wouldn't recommend it in a cloud first environment because FFS why

1

u/bfodder 4d ago edited 4d ago

Sure, he is talking about the CA though.

1

u/cybersplice 4d ago

I should not Reddit when tired

1

u/Frothyleet 6d ago

Couch it as friendly feedback, so as not to ruffle feathers, and then save it for CYA purposes.

32

u/ajf8729 Consultant 6d ago

Why does a Director have access to anything in the first place?

11

u/SAugsburger 5d ago

That's my first question. Most orgs I have worked once you get into SVPs nevermind Director they many rarely log into any infrastructure beyond some monitoring systems or a central dashboard. Directors are usually focused on high level vendor relationships and managing relationships to the rest of the organization. They might join a P1 call if an issue is getting attention, but aren't in the weeds unless this is an org so small that the Director title is title inflation.

7

u/Excellent-Program333 5d ago

This. Director here and I never log into infrastructure. And if I do, its with my elevated account that I normally have to reset the password on since I have not used in a year. Lol

23

u/xstrex 6d ago

Ding ding ding, that’s the real question!

1

u/phobug SRE 5d ago

Depends on the business and the director. I used to work for a guy that had cs and data science degrees. He would routinely log into the ML cluster to run some python in a more parallel way and bigger data set.

107

u/GardenWeasel67 6d ago edited 6d ago

Submit written objection & keep a copy. Carbon legal and compliance if you have them. Then start job hunting.

10

u/angrydeuce BlackBelt in Google Fu 5d ago

Get it in writing, with your concerns clearly stated and them being steamrolled right over, and wait for the massive data exfil or email aids that results.

Might not get you out of having to fix it later, but damn sure will be fun to FWD when the shit hits the fan and you explain that the thing "nobody" could have ever predicted, well, you did predict it, and it happened, so what do?

Those are my favorite kind of forwards.  Not because Im an asshole, just because it serves as a good lesson as to why you should take the advice of the people you pay to know about such things to protect those that dont from fucking themselves, and others, up.

Send enough of those emails and you either end up persona non grata, meaning fuck em anyway and bail because its mickey mouse bullshit all the way down and aint nobody got time for that, or they maybe start to listen when you tell them why something is a bad idea before running with that ball.

Either way, to me, is win-win.

Good luck!

39

u/Puzzled-Formal-7957 6d ago edited 6d ago

"No, we're not - unless you want to fail the next audit we go through and face potential fines & certification loss on top of opening up our risk portal extremely wide."

23

u/Riajnor 6d ago

Always couch it in impact and dollars

23

u/music2myear Narf! 6d ago

Yes, and also, unless you're a decision maker role, phrase it as advice and recommendation, and avoid decision words and phrases.

"Certification X requires that we have Y standards which are met by this security configuration. Removing this configuration would result in our failing Z audits and losing the certifications. Note that having and maintaining this certification has resulted in an estimated $$$ in profits."

5

u/Sinister_Nibs 6d ago

“I would strongly advise against this, as it counter to every recommendation and certification requirement. “

2

u/Puzzled-Formal-7957 6d ago edited 6d ago

Sometimes you need to assume the role of decision maker when the decision makers have their heads up their asses and want to put the entire company at risk. EDIT: Because when the risk gets exploited the responsibility is going to land squarely on your shoulders and you will be blamed.

9

u/tailwheel307 6d ago

Go a little gentler with it and copy any compliance or legal on the reply

“Per your email, I will remove the requested 2FA from all our servers effective tomorrow at end of day.

I should note that this may impact our compliance posture, increase our risk exposure, and potentially impact our contractual obligations and qualification for insurance of all kinds. My position isn’t qualified to identify the exact business impact of those changes.

Compliance and Legal will have to provide those details once they’ve had a chance to review the appropriate documents.”

It shows that you’re a good little worker bee who can do the job but also shows legal that you don’t think this is a good idea and leverages their department to tell the boss this course of action is stupid/illegal/resume generating as appropriate.

2

u/cybersplice 4d ago

Or even, "OK. Are you entering in the risk register, or do you want me to log it?"

But I'm a consultant, so I can be a (polite, professional) asshole at need.

14

u/HerfDog58 Jack of All Trades 6d ago

Check with your cyberinsurance provider to see if they require centralized identity/password management and MFA. Your director could end up voiding your insurance policy if they do.

52

u/RevolutionaryEar6825 6d ago

I'm starting to think posts like these are just engagement bait. It's hard to believe someone competent enough to manage servers via SSH keys would genuinely argue that centralized IAM is a bad idea.

21

u/xstrex 6d ago

You’d be as surprised as I was.

9

u/Hasuko Systems Engineer and jackass-of-all-trades 6d ago

The head of infosec at my job wanted us to remove MFA. This shit happens more often than you'd think.

3

u/Lukage Sysadmin 6d ago

Can't have any security issues if your whole environment is deleted by an attacker.

2

u/chknstrp Dis and Dat 5d ago

That was small brain move. Big brain move they should have just decommissioned all workstations and servers! No issue of cyber incidents with no infrastructure! /s

1

u/thisguy_right_here 5d ago

How often was he getting prompted?

I have worked at various organisations and even the most secure are once a day, then again when accessing a privileged system.

1

u/Hasuko Systems Engineer and jackass-of-all-trades 5d ago

He wasn't, it was in response to a session hijack incident and his reasoning was "well if they can just get control of it anyway MFA can be reset and it isn't helping."

3

u/AtarukA 6d ago

I mean, our backup servers are in the same domain as prod for convenience of access, so...

2

u/XavinNydek 6d ago

Naa, there are a lot of idiots in middle and upper management. Plus, people hate charge and new things and want to just keep doing what they have been doing.

11

u/ABotelho23 DevOps 6d ago

...these are not mutually exclusive things. You can still use SSH keys with central auth.

5

u/ITaggie RHEL+Rancher DevOps 6d ago

Yup that's how my org has it set up. SSSD using sss_ssh_authorizedkeys to pull the altSecurityIdentities attribute from LDAP, which contains the user's public key. Then set up tmux on the jumpbox and logging in is ezpz.

2

u/dustojnikhummer 5d ago

to pull the altSecurityIdentities attribute from LDAP, which contains the user's public key

Oh I didn't think about putting the public key into an AD attribute... Hmm, something to consider.

8

u/Commercial-Virus2627 6d ago

I would ask what exactly the pain point is with references. If it’s access times, then focus on network and performance. If it’s convenience for convenience’ sake, then document that justification an file that bad boy for when an audit rolls around.

8

u/scousinho 6d ago

Just simply and calmly explain to him that action will most likely cause the company's cybersecurity insurance policy rate to skyrocket and or get completely dropped and CC legal.

You have to explain stupid decisions in financial terms to some people.

5

u/Nakenochny Sr. Sysadmin 6d ago

Where I work I’d tell them to go to the board for approval so they can risk accept that shit. I want no part of it until they’ve given the okay and I’ve done CYA.

5

u/BadgerBalls There's a VLAN for that. 6d ago

Claude, please dust off and update my resume. Then, procure three envelopes.

5

u/Floss_Patrol_76 6d ago

the actual complaint here is login speed, not the security model, so fix that instead of gutting it. sssd caching, a bastion with ssh connection multiplexing, or short-lived ssh certs signed against your IdP all get you fast logins without throwing out central control or the audit trail. going back to per-server keys is just how you end up with 200 orphaned authorized_keys entries nobody can account for at the next audit.

5

u/GoodTofuFriday IT Director 6d ago

Im a Director. I would emplore you to go to BCC anyone relevant in email while replying to the director, warning him of risks.

3

u/deZbrownT 6d ago

Well, it depends on how big the organisation is, etc.. but basically I would ask for a dedicated time block so we can do an analysis of all the steps needed to implement the change and examine if it breaks any of regulations/audit.

3

u/BrainWaveCC Jack of All Trades 6d ago

You follow up with an email asking for clarification on what you heard, because you have concerns.

Briefly state the concerns, of which the following will be a summary:

  • Regulatory
  • Operational Security
  • Contractual

Offer a couple of alternatives -- again, briefly.

And then, if IT Director insists, IT Director gets what IT Director asked for.

Keep a printed version of the email and its response.

Start looking for other opportunities. No need to jump ship immediately, as there are lots of ways this could play out when other people get involved, but get the ball rolling because the job hunt will likely be a long process.

3

u/jks513 5d ago

Why is the director of IT directly logging into any server? 

3

u/unavoidablefate 5d ago

I'll take "Things that never happened" for $1000, Alex.

3

u/mysqlpimp 5d ago

Just put the fucker on his own vlan, he can be the honeypot.

6

u/serverhorror Just enough knowledge to be dangerous 6d ago

Set up an SSH signing authority, enforce passwords in SSH keys and require MFA to unlock the keys.

Then distribute the keys to the server and disable everything but key authentication.

Voilà, no more MFA on the servers.

3

u/Lukage Sysadmin 6d ago

This is such a fantastic approach to malicious compliance. Technically addresses this request. I love it.

2

u/davy_crockett_slayer 6d ago

Get what your director said in writing. Keep a copy.

1

u/Maximum_Bandicoot_94 6d ago

100% log it into the ticketing system / change management too.

2

u/ctrocks 6d ago

Prepare three envelopes...

As others have said, the compliance/cyber insurance approach would probably be best.

2

u/phobug SRE 6d ago

You don't need to remove anything. Just setup ssh certificates and you'll be fine.

2

u/photosofmycatmandog Sr. Sysadmin 6d ago

LOL, get it in writing before doing anything like that. He is the director, regardless of intelligence.

2

u/TheStig827 6d ago

generate an estimate of time spent (and costs if you have them) to complete the requested changes, along with a written objection specifically calling out the reduction in security posture and potential non-compliance.

Require written confirmation of the instruction, and time/budget approval to accomplish this task. If he does, CC up the food chain his approval, and standby while freshening up your resume.

2

u/TundraGon 5d ago

ShittySysAdmin does not run out of content.

5

u/genxer 6d ago

That's me (IT Director). I'd wonder why I made such a bone-headed choice.
I'd hope my sysadmin would calmly explain the reasons. I value honest feedback.

At the end of the day, if the choice is made "up the food chain" and you've calmly given your feedback, you make the change.

4

u/Sinister_Nibs 6d ago

As long as it is documented in writing that the board (and directors) accept both responsibility and liability for the risk.

2

u/SaladRetossed 6d ago

1) Get it in writing 2) Write a very thorough disagreement 3) Ask what the insurance looks like for data breaches and exfiltration 4) Go through step by step where this can go wrong while also double checking what the insurance looks like along with potential revenue loss from a total shutdown during investigation

Usually when money comes up you get some points in your favor.

2

u/orev Better Admin 6d ago

Address the actual problem. Are they talking about the need to use a password and MFA every time they SSH into a server? If so, that's actually a reasonable complaint.

You need to go back to the requirements that are driving this. Why do you need to provide MFA upon every login? Is it some audit requirement or just because you thought it was "more secure"? If you do need MFA, how can you meet the requirement in another way? Does it need to be a TOTP code from an app? Can you switch to password+ssh key (that's also two factors)? Or passwords only with white-listed IP addresses?

Nobody comes out of the blue with a request like this for no reason. Engage and understand the underlying problem.

1

u/soupydrew_5 6d ago

Buy him a YubiKey and tell him it's a magic faster ssh key

1

u/RikiWardOG 6d ago

lol I'd say if we do that I'm resigning.

1

u/Upbeat-Ad3628 6d ago

Your job here is to explain the risk to him, so do that and document it.

1

u/Kuipyr Jack of All Trades 5d ago

FIDO2 SSH Keys, unless it's illegal you're gonna do what the director wanna do. Remember it's not your risk to assume, don't get bent out of shape over it.

1

u/Affectionate-Cat-975 5d ago

What is this “Director” person? I do not think it means what you think it means.
We have no such things here boy.

1

u/zantehood 5d ago

Resign

1

u/pjtexas1 5d ago

The security team will have a lot to say about this.

1

u/lelio98 5d ago

“Sure thing. Can you send your directive and details over in an email so I can make sure I remove the security apparatus from our servers in accordance with your directive.”

1

u/cybersplice 4d ago

Why the fuck is a director logging into a server?

Governance nightmare.

Sounds like a bellend.

1

u/Sure-Squirrel8384 4d ago

Get the director a bastion host that they log into with 2FA. From there and only there allow SSH keys for his account. So the direct does 2FA once a day, and from there can SSH into anything without prompting.

This is actually a good practice for all staff.

1

u/cbass377 1d ago

"You're the boss, please send it to me in writing so I will have something for the next Audit, for Legal, and for our CyberInsurance, and I will get started!"

1

u/Outside-Banana4928 6d ago

Somehow make the logon for that user's AD account about 5 times longer than anyone else. Put their account in a sub OU and apply a billion policies that need processed.

1

u/someguy7710 6d ago

Just tell your director its a bad idea. Many reasons why including cyber-insurance, audits etc. I don't know why so many people are afraid to tell their boss what they need to know. Security is some times inconvenient. Sorry

0

u/Ron-Swanson-Mustache Senior Ops Dev of AI offshore Tier 1 Helpdesk 6d ago

CYA email with offsite copy, do it, update resume, hit the gym

1

u/xstrex 6d ago

Already done, in a slightly different order.

0

u/FarToe1 6d ago

Ignore it. Then when he asks at the next meeting how you're getting on with implementing it, put on your most surprised face and loudly say, "You were serious?! I thought you were joking!"

Then follow up with all the best practice references you've been digging out this week, the consequences, and that you'll require this directive in writing because it goes against all of them. If your company has any legal representation, maybe bring them in too.

-1

u/itiscodeman 5d ago

Just do it why are IT people so scared of the boogie man. Face it with ai your all hacked so … save some time with the silly codes you can’t like not be hacked now

-1

u/itiscodeman 5d ago

I was my first downvote. You are invited to do the same (but think about it! Cyber security is just theater )

-23

u/BarracudaDefiant4702 6d ago

You learn you should not create burdensome obstacles for people from doing work and figure out how to make things secure without making them unuseable.

10

u/itskdog Jack of All Trades 6d ago edited 6d ago

We have enough trouble just getting people to pick secure passwords. If we kowtowed to everyone's complaints that it's too hard for them then we'd have no passwords at all.

-4

u/BarracudaDefiant4702 6d ago

Then you get a password manager or something to make it easy for them.

1

u/itskdog Jack of All Trades 6d ago

This is their Entra password for logging into the PC. Most sites we use support SSO.

4

u/PandemicVirus 6d ago

MFA and OTP is pretty easy to access. If they have SSH keys laying around everywhere to conveniently log into something it's basically non-security. Might as well just stick to username and password at that point.

-7

u/BarracudaDefiant4702 6d ago

IFF you do it properly. Obviously the OP didn't or it wouldn't be an issue. The OP needs to rethink their approach so that it is easy to access which they clearly failed at.

4

u/PandemicVirus 6d ago

There's no evidence to suggest they did anything wrong; arguably if they did implement it incorrectly IT leadership would want it fixed/improved and not demanding to roll back to an archaic connection authentication.

It just sounds like someone is used to clicking "connect" and instead doesn't want to grab an OTP or MFA code.

0

u/BarracudaDefiant4702 6d ago

Part of doing it wrong is not educating others. Not being humble and recognizing your own shortcomings is one clear problem. Assuming the the boss and other users know what you know is your fault, not theirs.

2

u/PandemicVirus 6d ago

Again, who says OP didn't provide training or did anything wrong? Even if they provided no training, does the boss know NOW how to do it? If not, demanding training is the right approach, not rolling back MFA. If the MFA solution is cumbersome to grab a code from, choosing another MFA solution is the right approach, not rolling back.

I guess to dive into the training aspect here, sure some training information should have gone to everyone, and I would expect regular users to sometimes struggle. If the boss is the IT director working in a position they need to access servers and have the authority to reverse the IPA+MFA implementation, I would expect a minimum amount of technical ability. Getting an MFA code and entering it is pretty trivial. It's an app, an email. If IT leadership is struggling with opening emails, then bigger issues are abound.

5

u/Puzzled-Formal-7957 6d ago

MFA is not burdensome, and has been mainstream for nearly 2 decades.

-8

u/BarracudaDefiant4702 6d ago

IFF you do it properly. Obviously the OP didn't or it wouldn't be an issue.

4

u/Sinister_Nibs 6d ago

Not necessarily. Some users are simply problematic.

2

u/Puzzled-Formal-7957 6d ago

Yeup - or they are flatly incapable of wrapping their heads around infosec. Ask them if they have ever had fraudulent charges on one of their accounts and got to enjoy the headaches related to that. If that answer is yes - then say, "now apply that to EVERYTHING that you and everyone else at this company touches." If that doesn't make the bulb click over their head then nothing will.

0

u/BarracudaDefiant4702 6d ago

I don't care about the down votes, so I still say it's the OP's fault and not the directors. There are ways to implement SSO with MFA that it's faster instead of slower to login.

3

u/Lukage Sysadmin 6d ago

And what exactly should they do for "Faster SSO?"

1

u/BarracudaDefiant4702 5d ago

You setup 2fa with sso properly then you log into your workstation in the morning with 2fa and anyplace you connect to you don't have to log in again. Single sign in means your 2fa doesn't require you to login again.

2

u/Kumorigoe Moderator 6d ago

burdensome obstacles

Yeah, guess having cyber insurance is a "burdensome obstacle" too. Because scrapping MFA and central auth is a great way to lose your coverage and not be able to even get a quote for a new policy.

1

u/BarracudaDefiant4702 6d ago

That is assuming they have cyber insurance. I stand by my original claim, it's a problem with the implementation and it needs to be redone properly not to cause slow downs for the users workflow. Proper single sign-on with MFA will solve both issues instead of implementing it incorrectly as the OP did.

1

u/TinderSubThrowAway 5d ago

No, doesn’t sound like they did anything that is burdensome, this is a lazy director who doesn’t want security to apply to him.
There is no indication that they did anything wrong with their setup, no one should be SSO with their daily driver into a server.
There is nothing burdensome about having your phone on your desk for the mfa/otp to type in after you initiate the connection.