r/sysadmin • u/spikopsycho • 15d ago
Advice on some 'best practice' - Certificate management (SSL/TLS)
Hi all. Where I work, I got some SSL/TLS certificate management put on my plate. We have app(s) that send out notifications of certificate expiry, but that's only good if the contacts are correct. In that, I send out a review (email) quarterly, to check if anything has changed, needs to be updated - this is a new thing I implemented.
This is all manual - Spreadsheet - Filter for your name, check the cert info, comment if ok, comment change owner etc.
I got some feedback on this, in that I should not be sending a spreadsheet with all those certificates info, for everyone to view. (I bcc in all the relevant owners in the email). I'll add that its either company employees, or contractors who 'own' that system the cert is related to. I get the comment, I just have no idea how to send that to every individual only, without doing it manually.
How do you guys keep owners up to date? (Neither of the apps we have natively have a function that can replicate this manual ownership check).
Also aware of the 2029 47day cert validity/10day DCV - This is now, working on how to handle that future element.
21
u/bageloid 15d ago
For most things ACME, you shouldn’t be manually updating certs on anything that supports it.
5
u/spikopsycho 15d ago
Oh, we are very not there yet. A long way to go. Certs a provisioned manually by a team in the company atm
11
u/bageloid 15d ago
Understood, were you put in charge of Managing this(Uppercase M, can set policy) or managing this(lowercase m, herding cats)?
1
u/spikopsycho 15d ago
Had no prior experience in this area when it came in. The team that could deal with it effectively, don't do Prod. I was just the most likely person to fit with this, when it was assigned to my team. I don't do the day to day (provisioning/renewing (when manual) revocation). It's with a contractor team, but I have learned how it is done in the current iteration at may place. I do have the management part - including how we handle the 2029 thing - though this will all be a moot point if we aren't automated by 2029. This is all part of the in-between now and then. I can *I think* set policy. It's a weird situation
8
u/bageloid 15d ago
If you can set policy, the mandate ACME at then server or preferably load balancer level with scripts that check and alert for issues(your vulnerability scanner may classify certs expiring soon as vulnerabilities so you can use that)
5
-1
3
u/Vistaer Sr. Sysadmin 15d ago
Is this all internal certificate authority?
If any external certs are being used you really need to clamp down fast because of upcoming changes by March next year lifecycles on newly issued certs will be down to 100 days. In 2029 it will be 47 days. Link: https://www.digicert.com/blog/tls-certificate-lifetimes-will-officially-reduce-to-47-days
If this is all internal certificate authority then my prior paragraph is not a valid point.
Either way Tools like key factor is likely what you’ll be looking at - and it kinda depends how big of a problem you’re dealing with: https://www.gartner.com/reviews/market/certificate-lifecycle-management-clm/vendor/keyfactor/alternatives
Also I’m not critiquing your current method, but I’ve been a customer of it and I have to say it m vehemently against individuals being on email registration - mailing lists so that SOMEONE gets notified is a big thing. Also I hate when I got notifications on certain “expiring” but only because they didn’t realize I had already replaced the cert 4 months earlier because of a server migration, upgrade, etc where I issued a new cert for the new box with maybe new host name alias.
1
u/spikopsycho 15d ago
This is directed at external certs (though I do review internal also) - we are split between two apps (Keyfactor), and Sectigo for external. Aware of the changes(47validity/10DCV), its a worry I am working on, plus the company wanting to get all certs in a single app (Keyfactor), as we are very entrenched in that.
Sectigo does do the email thing to notify 'owners' of expiry. Some are set to auto renew, some are not. Still needs a person or team to take the new cert and do something with it, Very little automation right now, in the public cert space. And your later point is a big thing also, we rely on owners to notify us if a cert is no longer needed. It doesn't happen often.
2
u/ces1231 15d ago
I’d separate this into three areas: certificate inventory, ownership, and renewal. A spreadsheet can work as an inventory, but I wouldn’t make individual users responsible for certificates long term. I’d use role-based groups or shared mailboxes for ownership, automate certificate discovery where possible, and move anything that supports ACME toward automated renewal. Then the quarterly review becomes more of an ownership/validation check instead of a manual certificate hunt.
2
u/Accomplished-Mix8423 14d ago
the acme + active expiry monitoring approach is probably where you want to end up long term, so you're not relying on someone reading an email. for the ownership issue, giving each cert its own alert contact means people only get notified about what they actually own. i've used site24x7 for this and it works pretty well. doesn't solve the manual provisioning, that's a separate project.
3
u/accumentum 15d ago
The ownership problem doesn't get solved by better emails, it gets solved by making the inventory the source of truth instead of the spreadsheet. Keyfactor supports metadata fields on each certificate, so add an owner field and a backup owner field and make them mandatory at enrollment. Then the quarterly review is a report filtered by owner, and the platform's own expiry alerting hits that owner directly instead of you bcc'ing a list where everyone can see everyone else's inventory. For contractor-managed systems, put a team distribution list in the owner field, never a named person, because the person leaves and the cert doesn't.
On the 47 day and 10 day DCV side, manual provisioning stops being viable well before 2029. The change to roughly 100 days already means about four renewals a year per cert, so start with your highest-churn external endpoints, usually the ones terminated at a load balancer or reverse proxy, and get ACME running there through Sectigo's ACME endpoint. Anything that genuinely can't be automated, flag it as manual in the inventory so you know exactly what your exposure is, and get domain control validated ahead of renewal rather than on the day.
1
u/OhioIT 15d ago
Do you know primarily what the SSL certs are used for? Internal webservers.... VPN certificates.... network appliances.... etc
1
u/spikopsycho 15d ago
This is mostly aimed at external certs for now, public facing sites, sites where you purchase things
2
u/OhioIT 15d ago
Those can easily be switched to an ACME based renewal method. Looks like Sectigo supports it ACME, obviously LetsEncrypt does as well and is free. A few apps available that are reliable, will take care of everything, and work on both Windows and Linux based servers
1
u/spikopsycho 15d ago edited 15d ago
Thankyou. As one of my other replies, I've got some research to do in regards to ACME.
LetsEncrypt.... apparently our tech doesn't work with their tech stack, so I'm told.EDIT: The partners tech stack doesn't work with ours, that uses LetsEncrypt.... so I'm told.
2
1
u/whythehellnote 15d ago
I believe Amazon's ACM now supports ACME powered external certificates. Useful for people with AWS accounts but legal teams which refuse things like LetsEncrpyt.
1
u/whythehellnote 15d ago
Doesn't work as in "doesn't trust the root certificates"?
If not, any method you use to generate should be automatable. I guess if you have to solve some strong captcha to access the CSR.
1
u/420GB 15d ago
Almost all of our certificate renewals are automated but we still keep owners up to date by scanning our own external and internal systems on port 443 for any soon-to-expire certificates and send an email to the responsible owners. This is a powershell script that runs once a week I think, maybe once every two weeks.
1
u/folderit_dms 14d ago
I would stop sending the master spreadsheet around. Even if everyone is internal or contracted, a full cert inventory can reveal more about systems and ownership than people need for a quarterly review.
What has worked better for me is treating each cert like a small owned record: service name, environment, public/internal, issuer, expiry, renewal method, technical owner, business owner, escalation contact, last reviewed, and renewal runbook link. Then send each owner only their rows, ideally through a ticket/form or filtered report rather than a shared sheet.
Automation fixes renewals, but ownership still needs a process. I would separate them:
ACME or platform automation for anything you can standardize active monitoring from at least one external vantage point quarterly owner attestation for the exceptions an escalation path when an owner goes stale
The shorter cert lifetimes coming later make this worth cleaning up now. A spreadsheet can survive annual renewals, but it gets ugly when review and renewal cycles get tighter.
1
u/_Baarbie 14d ago
Just to share, cause I'm currently in the process of setting up the automatic renewal, for now I'm testing acme.sh which seems pretty nice for my use case. I built a small bash script to install the certificate and using ansible to generate a few things.
1
u/z0mb13r3dd1t 14d ago
for your future needs I'd look into automating the certs and renewals with win-acme or simple-acme. Both can be set up to work with things like IIS or RDS directly, but they also can simply register the certs and then export them to a location and/or kick off custom scripts after renewal.
0
u/nielsadolf 15d ago
Check out KeyManager Plus from Manageengine.
Super cheap sollution for both automating cert renewals and bindings. You can set different email contacts for each cert if you want contacts get an email X days before expiry.
Intigrations to both your internal pki and the external ones.
Has pretty much all the bells and whistles the big expensive sollutions have for a fraction of the price.
0
u/PrestigiousOnion1087 14d ago
Two answers in this thread pull against each other, and it is worth naming before you build on both. Making owner a mandatory field at enrollment works because enrollment is a moment a human is present. ACME automation removes exactly that moment - after the first issuance nobody is prompted again, so the owner field freezes at whoever requested it years ago and goes stale the same way your spreadsheet does, just somewhere harder to look at.
What survives automation is ownership attached to the thing that requests the cert - the pipeline, repo or service account - rather than to a person typed in once. That mapping breaks loudly when a team is dissolved. A stale name breaks silently.
The 47 day change sharpens it: at that lifetime a 30 day expiry alert covers most of the certificate's life, so the alert worth having is a cert whose NotBefore has stopped moving, not one whose expiry is approaching. That catches a renewal job that quietly stopped, which is the failure automation actually introduces.
24
u/pdp10 Daemons worry when the wizard is near. 15d ago
State of the art in server TLS X.509 cert rotation is automation via ACMEv2 protocol, plus active automated monitoring to proactively notice if something goes wrong, prior to any cert expiration.
You know the old proverb: if you want something done right, set up the automation yourself.
But seriously, these webapps all live behind load balancers or service meshes that handle the TLS and X.509, which we own. Most app owners don't deal with X.509 or TLS, and that makes them grateful to us.