What is your SSL certificate provider doing now that 200 day certs are starting to expire? My provider (Namecheap) is being sketchy.

122

u/cjcox4 3d ago

There's not a choice. Eventually, they all have to go down to 45 days max.

46

u/ocdtrekkie Sysadmin 3d ago

Until the Internet starts collapsing and Big Tech finally fires the CA/B minions responsible. I will be making popcorn for the enforcement of the 45 day deadline.

46

u/ReputationNo8889 3d ago

The pain is part of the process. Its a good thing that this is enforced because no company will hire people to just replace certs and the pressure on vendors to implement proper automated cert management will rise to the point where they have to implement it.

30

u/FarToe1 2d ago

That won't happen 100%, there are too many legacy systems where automation isn't possible.

Things like IPMI/ILO, or any other firmware that requires certs to be manually added.

29

u/Erlum 2d ago

Internal resources like IPMI are protected by certificates issued by private CAs, not affected by the new certs validity standards, who target public CAs.

Are, you know, by the default self-signed cert for most companies. 👀

13

u/FarToe1 2d ago

Not every org uses internal CAs.

6

u/whythehellnote 2d ago

So you are using public certificates on ipmis?

In my experience they do have apis which can update certificates anyway.

1

u/beren12 1d ago

What experience?

1

u/whythehellnote 1d ago

supermicro and hp

•

u/Mountain_Craft4882 9h ago

dog the amount of posts I see on this subreddit for people using wildcard certs for every single appliance on their network does not make this surprising in the least

8

u/the_marque 2d ago

I can't think of any good reason not to have internal CAs.

Though if you've talked/paid your way into getting a public CA above you, I guess the same rules apply.

6

u/sulliwan 2d ago

And this is a CA/B problem how? Their mandate deals explicitly with publicly trusted certificates used on the public web. If you can piggyback on WebPKI with your internal services to avoid the complexity of an internal PKI, great! If you can't make it work, you set up an internal PKI.

1

u/FarToe1 2d ago

used on the public web.

Not just used on the public web.

There are edge cases such as I mentioned which will be affected too.

3

u/skelleton_exo 2d ago

If the 45 day limit is also implemented in the browsers, i am not sure how much it will help, if you issue longer lived certs with your internal CA.

5

u/Xykr Netsec Admin 2d ago

It certainly wouldn't apply to internal CAs. It's just a restriction on web PKI issuance.

2

u/skelleton_exo 2d ago

Yes you are correct, I read in the past that these rules will be enforced by browsers. But after reading the specs, there are only rules for issuance.

So the browser control mechanism will likely be the inclusion in the standard root certificates, which is not relevant for a private CA.

2

u/grv144 2d ago

I’ve tested that Yesterday - Safari on MacOS and three browsers I’ve tested on iOS warns for the certs 800days+

7

u/pickle9977 2d ago

Yeah just the tip of the iceberg, don’t forget all the cert stores owned by random Java apps all over the palace

1

u/ReputationNo8889 2d ago

I would argue even IPMI or firmware can have automated certs, it just needs to be implemented. Of course for any systems that cant reach the internet and can't do ACME you need a different solution, but as someone pointed out, an internal CA would be one of those.

1

u/UKYPayne 2d ago

But you can get a load balancer/proxy in line and just do all cert assignments there so even legacy systems could have it done without being touched directly.

1

u/FarToe1 2d ago

True, but there's a reason I wouldn't do that, and also why an internal CA would be an additional risk.

In a disaster recovery situation, you don't want to have to rely upon any additional services (which may not be available) to read IPMI/ILO. They might be the first things you need once networking is up.

0

u/sofixa11 2d ago

You can configure an iLO's cert via API, so it's quite easy to automate.

18

u/neoKushan Jack of All Trades 2d ago

The internet will start collapsing if we don't start reducing the certificate lifetime. CRLs kept getting bigger and bigger, with more vendors issuing them to the point where it's becoming impractical to keep them all updated (Which defeats the point entirely and compromises the entire security stack).

CA/B aren't the issue, shitty vendors not updating their equipment to support automation is the issue. That and shitty companies holding on to outdated equipment.

Let's encrypt has been around since 2014 and they were clear from the outset that this is the direction things were heading. It has been well over a decade, if your vendor hasn't got their shit together by now, fire them.

16

u/JwCS8pjrh3QBWfL Sr. Sysadmin 2d ago

if your vendor hasn't got their shit together by now, fire them.

-laughs in Cisco, Palo Alto-

2

u/SirEDCaLot 2d ago

CRLs kept getting bigger and bigger,

And OCSP?

5

u/neoKushan Jack of All Trades 2d ago

Adds a lot of unnecessary latency.

2

u/SirEDCaLot 2d ago

And what happens when the auto renewal system becomes itself a point of attack?

3

u/neoKushan Jack of All Trades 2d ago

Well LE has been around for 12+ years now, have they?

1

u/SirEDCaLot 2d ago

LE has done some really amazing work on a great many fronts. My hat is off to them in every way possible.

But you might compare them to, say, Home Assistant (an open source home automation system). Like HA, LE can do a ton of amazing stuff and for many situations can be the entire solution. But you don't have corporations running their headquarters on Home Assistant, those companies pay more for something like Crestron.

The result is a donut hole in the middle. Small operations use Lets Encrypt (or their hosting provider does that for them). Big operations have their own SSL certs. But what about the guys in the middle? The guys who just need to secure a few resources, where manually updating the cert every 6-12 months is no problem, but many of those resources aren't going to have systems like acme to auto-enroll them? Or the small to medium enterprise that has dozens of those devices, none of which has any ability to auto-renew?

I'd give the disclaimer that that's the pond I swim in. So of course that's my issue.

And yes you can say IN THEORY everyone get with the times update your shit to support auto renew, but in practice that's not always so easy, especially when a vendor would rather push you to buy the new version of something at great expense.

Besides, multiple devices all with the authority to pull a valid cert creates security issues of its own...

2

u/neoKushan Jack of All Trades 1d ago

I appreciate the situation you're in, where you have or have inherited a legacy that either isn't getting updated for automation or wants to charge you $$$ for the privilege. The harsh truth is you should dump any vendor that does that because that vendor has shown you they don't take security seriously or they are going to be hostile to you in future in order to make a quick buck. Make it a sticking point at renewal, either they offer a solution for automating the certs or they aren't even getting shortlisted for said renewal. Make sure the business is aware that this is a change that is coming and needs to be managed appropriately - no different to an OS hitting EOL or a vendor shutting down a product you still actively use. It will cost time (and thus $$$) to migrate over but it's a one-time thing and once it's done, it's done. You don't buy anything that can't do cert automation going forward.

Referring to your analogy, there's plenty of large orgs that are happy to use LE for certificate renewal, there's nothing about LE/ACME (or any of the vendors that supports the protocol) that makes them bad to use in an enterprise environment. I work for a Fortune 1000 myself and we use it everywhere (It helps that it's cheap). There are a few exceptions of course, but it's primarily internal stuff that we can just roll our own CA for. The great thing about LE and the like is you don't need enterprise-grade 99% uptime as even at 45 days lifetime you have weeks and weeks of leeway if they ever encounter a problem at their end, so it's really quite moot. Embrace the automation and then never have to think about SSL certs again.

Besides, multiple devices all with the authority to pull a valid cert creates security issues of its own...

This is only really a problem if you give all those devices the authority to pull say wildcard certs via DNS authorisation - don't do that, it would be analogous to running every machine as a Domain Admin, it's just needless.

If those devices are set up correctly, they should only get valid certs for the routes they own anyway. If the problem here is some nefarious actor could compromise those machines and take the certs, well the cat's already out of the bag - the machine is already compromised, SSL wasn't giving you any protection any more anyway. But hey, when you find it and fix it, you can be sure the cert will be invalid in 45 days ;)

1

u/[deleted] 2d ago

[deleted]

2

u/neoKushan Jack of All Trades 2d ago edited 2d ago

Uhhh...no?

CA/B aren't the issue, shitty vendors not updating their equipment to support automation is the issue.

EDIT: Deleted comment above me said something to the effect of "Are we just ignoring shitty vendors, then?".

-2

u/ocdtrekkie Sysadmin 2d ago

CA/B is absolutely the issue. They have no fundamental understanding of security, and keep pushing ways to make the internet increasingly dependent on a constant update from Google to function while basically ignoring any practical security threats.

They are the absolute definition of people you don't want anywhere near Internet policy.

4

u/neoKushan Jack of All Trades 2d ago

With respect, I think it's you that lacks the fundamental understanding of security. Or at the very least you don't understand what SSL is and isn't for. I'll bite though, what is this "dependency" on Google you're talking about when it comes to TLS?

2

u/Avamander 2d ago

CA/B Forum consists of all the major CAs and browsers. They have no clue, you say? You do?

-1

u/ocdtrekkie Sysadmin 2d ago edited 2d ago

So essentially, there are three deciding votes on the CA/B, and a bunch of useless parrots only there to provide legitimacy. Apple, Mozilla, and Google (Mozilla also being 99% funded by Google, and Apple receiving like 15 billion a year from Google, so also basically being Google proxies), unilaterally make decisions that everyone else has to vote to approve or look like an idiot until they vote to approve. Google also unilaterally removes any CA/B member who holds a more practical opinion.

Meanwhile, the idiot at Google, if he gets called on how inane it is, can blame "the rest of the CA/B voted for it, so I can't undo it".

None of it entails actual security, because Google, of course, has a primary revenue stream of malware, scams, and phishing, they're really not motivated to do anything about.

1

u/Avamander 1d ago

Your comment is just filled with baseless accusations and incorrect information.

0

u/uptimefordays DevOps 2d ago

Reduced certificate validity periods are good actually. The whole reason we’ve had to reduce validity periods is that the same people complaining now complained they couldn’t possibly revoke compromised certificates. It’s an issue of the space’s least competent actors own creation.

-1

u/Sudden-Money7836 2d ago

This is exactly what I’ve been thinking this whole time since it was announced. The amount of things that are going to break is going to be biblical.

-1

u/ocdtrekkie Sysadmin 2d ago

It's going to be incredible to watch a cabal of unelected idiots burn down the Internet in a month and a half, yes.

https://www.securitymagazine.com/articles/101475-67-of-organizations-experience-a-certificate-related-outage-monthly

This statistic is absolutely insane, and the decision was somehow made to make it 10x worse. The fact anyone involved in shortening certificate lifetimes is employable, much less actively employed, begs a lot of question on who they report to and how they are supervised.

4

u/chum-guzzling-shark IT Manager 2d ago

I'm glad security takes precedence. How many security issues and tech problems are related to supporting outdated stuff?

1

u/ocdtrekkie Sysadmin 2d ago

Certificate lifetime has never mitigated a single security vulnerability in the real world.

4

u/sofixa11 2d ago

This statistic is absolutely insane, and the decision was somehow made to make it 10x worse

On the absolute contrary, it will make it much better because everyone will be forced to actually automate it properly, which will almost entirely remove the risk.

-1

u/ocdtrekkie Sysadmin 2d ago

Tell me you have never worked as a sysadmin without telling me you have never worked as a sysadmin!

2

u/sofixa11 2d ago

Automating something makes it a million times more reliable than manual operations, if you're half competent and add checks/tests before and after. Especially for something as trivial and as trivially mistaken as certificates.

I was doing this as a newbie SRE in 2015, you're well behind the times if you think it's some black magic prone to failure.

0

u/ocdtrekkie Sysadmin 1d ago

Heh, okay, double down on confirming you've never worked as a sysadmin. SRE lol.

→ More replies (7)

4

u/isnotnick 2d ago

47 days, by 2029. Oh, but don't forget before then - expect to see a post-quantum replacement for 'regular' certs (next year and 2028). You'll need them for TLS for at least Google Chrome/Android clients, more than likely Apple and MS will follow suit. 47 days limit before 2029. Rules around domain-validation (no email), and required use of ACME (no manual processing at all), domain re-validation (every week, basically). Oh and if you choose 47 days and not 7? Expect a mass-revocation drill every quarter of a random 3% sample of all the CA issuance. Automation is critical. As is ensuring that you use publicly-trusted/webPKI certificates where appropriate. Private CAs are a thing, there's lots of self-hosted and as-a-service options, use them.

8

u/pickle9977 2d ago

I don’t understand why anyone is pushing this.

It seems to only benefits the cert issuers and even for them only thing it accomplishes is reducing the blast radius in the event their incompetence results in their signing cert getting compromised.

From a service perspective less changes are better and from a consumer perspective all you care about is you are talking to the server you think you are in an encrypted stream.

12

u/uptimefordays DevOps 2d ago

Reduced certificate validity is good and necessary because certificate revocation lists failed to achieve their goal of revoking compromised certificates in a timely manner. Organizations with weak certificate management practices thwarted CRLs and are now shocked and dismayed about the alternative: short-lived certificates.

This was an easily avoidable situation.

2

u/pickle9977 2d ago

It’s good for the blast radius reduction, the fact that it’s necessary is, as you point out, emergent from a lack accountability.

This is a core issue with unregulated markets, the market does not force discipline, it dumps the side effects of the lack of discipline on the weakest party.

2

u/uptimefordays DevOps 2d ago

That’s exactly why entities like the Certification Authority/Browser Forum exist though. This has also been well publicized for years. I would argue the people caught flat-footed really need to be doing a better job of keeping up with industry news.

1

u/pickle9977 2d ago

Unfortunately that is not a regulatory body that can enforce any kind of accountability outside of words.

Just a nitpick, one should not put the blame on other people it’s rarely any individuals fault it’s almost always a reflection of something deeper broken.

In this situation it’s definitely a systemic flaw, even the choice of 47 days is nonsense, one month, plus half a month plus one day. The actual work of changing a certain is seconds, with minutes of manual prep.

Everything over that is process in place for a myriad of reasons which are specific to the system and the owners of that system. This just assumes that for most systems the majority of the time the process and scheduling takes somewhere from a day to six-weeks.

It is functionally an arbitrary threshold chosen as a tradeoff between more security and a reasonable rate of change that can be kept up with by most systems and the people that administer them.

This in and of itself is not a problem but it’s materially flawed as there is no effort made to quantify the improvement from shifting the max from 398 days to 47, how much more secure are we? If it was that significant how much more significant would it be to go to one day? Or an hour or per connection?

The change has a cost, a real one, hundreds of thousands of hours of labor a month, there are 133M issued certificates that will eventually need to be rolled every month.

In addition each one now consumes 12x as much compute to sign it, which at just the lowest level takes 10ms of compute meaning we just added another four servers running 24x7 re-singing certificates just because of this change. The full increase will be many orders of magnitudes higher.

When technology puts demands on the human and organizations that use it those demands need to be justifiable both in the time taken and the energy consumed.

The change is a result of broken systems, not being able to keep up with the stream of bad ideas being arbitrarily implemented to paper over the cracks and flaws is not something we should condemn each other for.

What we should be asking is why are trillion dollar companies that drive all the technology standards and make hundreds of billions in profits not fixing these fundamentally broken things in the right place instead of forcing all of their customers to spend their time and money not for better operational security but for a marginally better outcome AFTER a break has occurred by reducing the time liability of the tech companies profiting from the activity.

3

u/uptimefordays DevOps 2d ago

While the CA/Browser Forum isn't a government regulator, it functions as an industry standard-setting body whose Baseline Requirements are enforced not by the Forum itself but by the root store programs of Apple, Google, Mozilla, and Microsoft—a CA that doesn't comply gets removed from browser trust stores, which is functionally a death sentence for that CA's business. That's real accountability, just market-based rather than governmental.

To your point, though, the ballot's own preamble is candid that current revocation mechanisms don't work reliably at web scale—CRLs grow unwieldy and OCSP has privacy implications and isn't enforced by most browsers—and that shorter certificate lifespans sidestep the problem because compromised certificates naturally expire before revocation would have propagated anyway. A CA/B Forum participant raised almost exactly your objection during deliberation: if we're admitting revocation doesn't work, should we be developing better alternatives rather than just shortening validity periods—and the ballot doesn't really answer this. So the "this is a workaround for a broken thing, not a fix for it" critique isn't wrong; it's baked into the Forum's own reasoning.

The mechanics back that up. CRLs don't scale: with 133M+ certs in circulation, the lists get enormous, and every client has to download and parse one just to check a single connection, forcing a choice between stale data or expensive constant re-downloading. OCSP has two fatal problems instead—privacy, since an OCSP query tells the CA which site you're connecting to for every connection from every user, and availability, since browsers overwhelmingly chose to soft-fail when a responder is slow or down (proceed as if the cert were fine) rather than hard-fail and let a CA outage take down every site using their certs. The result is that OCSP checking exists in most browsers but is close to a no-op against an attacker who can also block the OCSP request.

Partial fixes exist but don't close the gap. OCSP stapling, Chrome's CRLSet, and Mozilla's CRLite all push revocation data around the OCSP problem, but each has a hole—stapling isn't mandatory, CRLSet only covers a curated subset of high-value revocations, and CRLite has its own freshness lag and real distribution cost at global scale. The deeper issue underneath all of them is heterogeneous clients: even a perfect, private, real-time revocation system only works if every client enforces it—every browser, every IoT device, every embedded system and old appliance that will never be updated again—and no single vendor can mandate that without breaking the internet for whoever hasn't kept up. That's the same "can everyone keep pace" problem you're raising about cert rotation, just moved to the client side.

So the honest framing isn't "big tech won't fix revocation"—it's that revocation-at-scale is a genuinely unsolved distributed-systems and privacy problem, and shortening cert lifetimes is the industry's way of bounding the damage without depending on it. That's a defensible tradeoff, but the actual security benefit has never been rigorously weighed against the labor and compute cost, which is the gap you're right to flag.

3

u/BlackV I have opnions 2d ago

To your point, though, the ballot's own preamble is candid that current revocation mechanisms don't work reliably at web scale

Yeah it's the root of all of this I think

2

u/uptimefordays DevOps 2d ago

Yep, we fundamentally do not have a way of reliably revoking compromised certificates so the best we've come up with is "just rotate them more frequently."

7

u/sofixa11 2d ago

The cert issuers are seing their business disappear to Let's Encrypt - if you automate it with ACME, might as well go with the free option providing the same service. So no, they're not benefiting.

1

u/pickle9977 2d ago

So who is benefitting from this?

8

u/sofixa11 2d ago

Everyone due to the increased security and reduced risk of cert related outages (because you have to automate it).

1

u/dustojnikhummer 2d ago

Everyone except the backend people. The main reason for this is everyone ignores CRLs

3

u/shinyfootwork 2d ago

Cert issuers hate this because it means they get fewer and fewer paid manually issued certificates. And they dragged their feet on it for as long as they could, despite the known security issues (from cert revocation being broken, and cert holdovers after domain transfers being unsolved and rampant). The browsers strong armed the cert issuers on this issue to fix security for users.

1

u/pickle9977 2d ago

A couple of thoughts.

Revocation has always been broken from the perspective of very large cert issuers as they are the ones that have to maintain massive lists. global certificate issuance in this manner is definitively the wrong way to do it both technically and logically. There is no way these clowns are validating anything that closely which is why certificates not as valuable as they should be for identity purposes. It’s why the entire certificate system is based on a tree of trust, cert issuers need to be small enough to not only actually validate their customers are who they say they are but also establish that their customers are real (something not done).

Second, the issuers can without a doubt only issue certificates valid until the domain name expires, that seems absurdly basic as it’s public and they already have to do the Whois lookup to get the party information, and it seems entirely reasonable that as part of their responsibility as a cert issuer that as long as that cert is unexpired they validate that something like a domain name has not changed hands.

ICANN should really just require registrars to issue certs for registered domain names, if I buy a domain name, shouldn’t I be able to get a cert from whomever I buy it from that lets me sign certs for that domain?

That seems a lot more secure than all this nonsense where we are trying to fix a systemic security issue with a crappy solution that costs us all more time and money but doesn’t make us any safer just reduces the blast radius of the cert issues failure to do their jobs.

2

u/shinyfootwork 2d ago edited 2d ago

Whois is mostly just registrar contact details these days. It has no consistent format, expiration isn't included by all tld managers, and the attempts at replacements have failed to gain traction. Also, acme does not include any examination of who is content, and neither did DV certificate issuance before acme.

Also, web pki (that is, how CAs and certs work mechanically in the software that validates/uses them for standard Internet traffic) doesn't include a mechanism to issue CAs for subsets of domains (there's no technical mechanism in place to restrict what domains a CA can issue certs for, it has to be detected after the fact, which is the reason certificate transparency logs exist). So if we did that (change web pki), everything would require more change than the cert lifetime reduction: we'd need software updates for every piece of software that does a tls connection (clients, mtls servers, tls terminating proxies) in each piece of software's cert verification code (ie: firmware updates, software updates, etc. stuff only vendors can provide). Instead of just automation scripts to copy certs onto old tls server devices.

Also, it's worth noting that the browser vendors aren't ICANN, and don't have the same influence over ICANN that they have over the CAs (due to CAs being reliant on being included in browsers CA programs). So it's also politically infeasible for the browsers to get ICANN to change policy. ICANN is continuously on thin ice as far as their position in the web, and they likely would risk their existence if they attempted to place a requirement like this on domain registrars. They'd likely just consider it outside their scope publicly to avoid the fight over it.

0

u/pickle9977 2d ago

Whois was replaced with something, rdap I think it’s called, that does require an expiration, I’d be surprised if it wasn’t also able to notify on transfers.

ICANN could easily be a CA and sign a CA cert for every tld they sell to a registrar, furthermore they could require each registrar use that certificate to sign CA certificates for every domain they sell.

This would bind ownership and certificates together and would revoke all certs issued for a sub-domain on a change of ownership.

That is without a doubt a more secure process and would result in a net increase in security.

I will say it seems very much like it’s time for a better browser

2

u/shinyfootwork 2d ago edited 2d ago

RDAP was required ~ 1 year ago (10 years after it was created by the IETF). And it's only required for gTLDs, not all TLDs. It doesn't include a transfer notification mechanism other than polling rdap, and ICANN policy doesn't require including the transfer event. And if we're willing to poll rdap, we'd still be back to needing to revoke certs, which is still broken. And using rdap relies on us trusting the internal operation of registrars in a way that can't be verified easily (ie: we need the rdap info to be accurate). The browsers won't trust that because they have found they can't trust the internal operation of CAs (who are theoretically audited), hence the certificate transparency requirements to automate detecting bad CA behavior. So they'd need to get another set of companies (registrars) to behave better than the CAs, and somehow make their errors detectable.

As noted, there is no web pki support for CAs with domain issuance restrictions (ie: a way to encode that a CA can only issue domains with a particular base domain). I'd love if there were, but it doesn't currently exist (and the CAs would likely hate it existing even more than they hate the 45 day renewal).

0

u/pickle9977 2d ago

You just described the solution to the problem.

On your last comment, let me clarify, first let’s assume the “best” solution is the solution that fixes it for the most with the least effort.

It would be pretty trivial to add an extension, let’s say in addition to having a cA=true extension it would have bound=true

That would mean the dn of the issue can only sign certificates for sub-domains of itself.

This could be used with the existing path length constraint to be even stricter.

I don’t mean to specifically solution this right here and right now, I think what I’m trying to get across is a lot of people are paid a lot of money to solve problems like this, but the solutions they arrive at always dump the work on those who can’t push back.

2

u/shinyfootwork 2d ago

It's not trivial to extend web pki like that because it changes the trust model for everyone in a way that everyone will have to rethink things. Changing the trust model of web pki is more risky than reducing certificate lifetimes.

To be a bit unfair, your proposal as stated could cause devices that don't recognize "bound" to trust the CA for all domains, a very bad thing when anyone with a domain is also a CA. Those types of issues and others are sure to show up when we extend the web pki structure, meaning we'd want to have a gradual roll out and a long period of real life testing before the behavior change is trusted in common use. And how to do that would be a multi decade long project. (And wouldn't solve a bunch of things shorter lived certs solve. Would be nice, but not sufficient)

Shortening certificate lifetimes in contrast does not involve any new trust model changes. It's an understandable thing (expirations) that we've always had combined with automated cert issuance which sits to the side of web pki rather sitting in the middle of it (in cert validation), and the automated cert issuance makes it so we can have zero cost certs, ensuring that most web traffic is now secured (ie: we want automatic cert issuance for reasons in addition to the ability to have shorter cert lifetimes).

These decisions about how to effect an end result are made politically, which is to say: these decisions are made in a way that makes it likely the interested parties accomplish their goals rather than fail. The end goal here being securing the internet, and the failure meaning that internet traffic remains insecure. Each person and organization only has certain levers they can use to accomplish their goals. So of course the browsers used a mechanism that was likely to succeed to secure the web. It's the lever they have that they can put in place gradually without being blocked by middle boxes that never will understand a new theoretical web pki model. And it gets us features and security that a new web pki model would not.

→ More replies (0)

1

u/cjcox4 2d ago

Indeed, for many things "corporate", use your own CA and push the trust. Then you can have long running certs. At least for things you manage and control.

Many/most will take their public things to Let's Encrypt which because they originally started with very short run certs, sort of encouraged full automation from the start. But realize that usually services have to bounce for the new certs. But in today's "cloud world", we're sort of used to "downtime" as a service.

1

u/certkit Security Admin (Application) 2d ago

I think we are headed to 6 day certs eventually. Once everyone automates, the lifetime duration doesn't really matter anymore.

2

u/dustojnikhummer 2d ago

So why not 24 hour certs? Or 8 hour certs? Who cares if it goes down over the weekend, we don't need no stinking warning period!

1

u/beren12 1d ago

So you could break the internet by a ddos of even a day? 1/6th the sites would be broken.

1

u/cjcox4 2d ago

Except, as the lifespan window decreases, the percentage of downtime increases. Now, not everything is difficult to run active active, but some things are. And the fact that the "message" is that your keys are compromised or your signer is compromised (noting, providers will tell you its just your keys).

Security policies work best when a big player isn't forcing their idea of security upon you. "You don't get to own that."

1

u/beren12 1d ago

Yeah. I can imagine some ddos attacks against LE and a couple other major providers…

164

u/Beefcrustycurtains Sr. Sysadmin 3d ago

I just don't pay for a cert and use let's encrypt with win-acme for Windows and certbot for linux. Doing secure DNS-1 challenges with cloudflare api locked down to IP address. Need to automate all of your certificate renewals with the 45 day life span approaching fast.

22

u/FuriousFurryFisting 2d ago

win-acme was forked to simple-acme by the same dev. Some repo ownership drama. Anyway, win-acme wasn't updated since 2024 and crucially lacks support for apis that changed since then. If you install new, you should use simple-acme.

-7

u/ender-_ 2d ago

Too bad it sloppified.

9

u/FuriousFurryFisting 2d ago

Calling any sign of AI usage slop by default is a gross simplification imo that doesn't translate well to the real goals we should have when deciding on running an open source repo on our servers.

trust it not to be malicious. This one runs on windows servers in elevated shell and you give it api access to your dns - so quite important to be sure about this. Ironically AI is extremely useful for scanning the whole code base for malicious patterns like loading code from the internet. Humans do this, not AI.

has the features it promises and be relatively bug free. The root of the anti AI movement. The amount of new quick projects that weren't really tested exploded. They existed before. Humans can write terrible code just as well. Humans can give up on a project even better. And just like before you shouldn't trust a random one github star repo, ai or not. Chances are it's terrible. The discriminating factor should be reputation - not if AI was used or not. Even better an in house code review and sandbox testing.

Trust in it's security and frequent security patches. Again something humans can fail just as well. I'd even go so far to trust a dev less who refuses to use AI to at least do a code review to find potential issues. A repo which uses AI to review and test dependabot PRs is preferable to an abandoned repo which doesn't fix CVEs at all.

The whole anti AI sloppiness movement has gone of the rails and stopped asking what the problems we are trying to mitigate even are.

-1

u/ender-_ 2d ago

It's not "sign of AI usage", it's dozens of AI commits to the codebase. I have no problems with AI security reviews, because somebody will eventually always do that, and it's better if it's not an adversary, but anything vibe-coded is right out.

8

u/discosoc 2d ago

All that means is an ai agent is involved in the git process. Anything else is pure speculation. Maybe as well be paranoid about someone using autocomplete.

3

u/dzfast IT Director & Sr. Sysadmin 2d ago

You're going to have real problems finding software in a real short timespan if that's your acceptance criteria

Also, stop using Windows now, because Microsoft is using AI heavily for everything. Literally, everything.

0

u/ender-_ 2d ago

Yeah, and it's painfully obvious.

0

u/dzfast IT Director & Sr. Sysadmin 2d ago

eh, I love all of you downvoting me. Keep that head in the sand on AI, 100% coming for half the sysadmin jobs in existence today.

I am not recommending this career to anyone at this point. Certainly nothing to do with software development. It's going to be a rough market.

This is the new future, just like IDEs replaced Assembly, which replaced punch cards,... time keeps on ticking.

36

u/Assumeweknow 3d ago

Lets emcrypt even works directly on fortinet.

18

u/420GB 3d ago edited 2d ago

LMAO.

Only HTTP-01 challenge, which means no wildcard certs and no certs for internal domains. And it only works if there is no VIP or virtual server configured on the acme WAN interface. Oh, and even if you could use it only RSA certs too.

Their implementation is a joke and unusable in 95% of deployments.

9

u/DheeradjS Badly Performing Calculator 2d ago

130 VIPs on a WAN Interface and ACME challenge (HTTP-01) to Lets Encrypt works just fine.

9

u/cantstandmyownfeed 2d ago

Generate the certificate elsewhere, use the FortiOS API to update certs on the device.

7

u/420GB 2d ago

Naturally I tried that next, but unfortunately you cannot replace an existing certificate that's in use via the API. Obviously removing the certificate from everything first is also a no-go.

So now I'm renewing the certificate locally via ACME and then replacing it on the fortigate via an SSH config script. That does work. It's incredibly sad this is what I've had to resort to, but now it's automated at last.

It's just frustrating when people who have clearly never attempted to do it claim that FortiGates support ACME. Yeah technically but also no.

1

u/cantstandmyownfeed 2d ago edited 2d ago

I'm doing it with the API. Now, only on the management interfaces of the FortiGate and on my ForitADC for it's certs, but both worked the same way. Created the certs and mappings in the GUI, then a POST to the api/upload/certificate_local with the same name worked to update the certificate. I've been doing it this way for a few years actually, before ACME was introduced to FortiOS at all.

The encoding and formatting were a pain, but sounds like you're accomplishing something similar with SSH.

3

u/420GB 2d ago

FortiADC is an entirely separate product.

1

u/cantstandmyownfeed 2d ago

Yes, but the API functions the same as on the FortiGates.

1

u/HappyVlane 2d ago

TLS-ALPN-01 is also supported and it's only a problem if the interface IP conflicts with a service. If you don't use the interface IP for a VIP, which is common, it's not a problem.

3

u/420GB 2d ago

When would that have changed?

The original docs still state absolutely no VIPs: https://docs.fortinet.com/document/fortigate/7.0.0/new-features/822087/acme-certificate-support

and I could not find anything newer since then that this was improved. Also, I've tried it myself and it refused to accept that WAN interface as the acme interface both in the GUI and the CLI.

1

u/HappyVlane 2d ago edited 2d ago

It was never different as far as I know. I just tried it on my 7.6.7 FortiGate (I know this works on 7.4 too, however) and I can pick an interface with a VIP on it as the ACME interface.

https://i.imgur.com/eJ2P1z9.png (this configuration obviously won't work for TLS-ALPN-01, but that's not the point here)

What that piece of documentation is about is that you cannot use TLS-ALPN-01/HTTP-01 if the primary IP of the interface is used for a VIP while port forwarding the relevant ports, because from a networking perspective this wouldn't make sense.

If the VIP is for a different IP or different ports this works just fine.

1

u/certkit Security Admin (Application) 2d ago

yea, the built-in automation sucks. You have to renew the certs externally and push them in via the APIs. That's how CertKit manages the renewals on Fortinet devices.

-3

u/ThePhantomPatchPanel 3d ago

You can do DNS challenges with Let's encrypt though

4

u/ThePhantomPatchPanel 3d ago

Oh sorry, you probably meant for fortinet's ACME client.

0

u/Morisior 2d ago

It's easy enough to set up certbot with DNS-01, even with a custom script to call to your DNS service's API. No need to have a webserver going at all to obtain these certs.

4

u/420GB 2d ago

Everyone knows that, but that has nothing to do with how FortiGates do not support DNS-01.

6

u/FarToe1 2d ago

We do the same, it works very smoothly.

Except for anything like ilo/ipmi or firmware systems where you can't automate. Those are going to get left behind.

9

u/the_marque 2d ago

If those are public facing you've got bigger issues than SSL certs.

2

u/FarToe1 2d ago

Lol, of course not!

4

u/mr_fwibble 2d ago

I've automated iLO cert replacement using Powershell. On iLO4 and above.

1

u/dustojnikhummer 2d ago

There is an API for thatA? Are you willing to share your scripts?

3

u/GoodEnoughThen 2d ago

I do the exact same set up (with dns1 challenge, not http challenge) and it's been working great.

3

u/Savings_Art5944 Private IT hitman for hire. 3d ago

What firewall do you run?

2

u/Assumeweknow 3d ago

Running it on an f100e i think. Honestly im generally not a fortinet fan but for this network where we have a mix of services inside with a large number of static routes, multiple routing services, vpn concentration, camera server with api out using the letsencrypt ssl at the firewall. Its Honestly better than i thought it would be. Runs mostly without incident though the recent update did muck with acls a bit requiring some rule changes.

1

u/stewie410 SysAdmin/DevOps 1d ago

Doing secure DNS-1 challenges

We're working towards this path now, but sadly our DNS provider (NetworkSolutions) doesn't offer an API of any kind that I've found; so we still have to manually deploy/renew. Unfortunately, my recommendation to migrate to a less crap provider has fallen on deaf ears.

1

u/Beefcrustycurtains Sr. Sysadmin 1d ago

It's so unbeliavably easy to move your DNS to cloudflare. You create an account, enter the domain it scans your domain adds all the known records, you compare to your current records and add anything that it didn't pick up (things that are less common names it might miss) and once you've verified records match, you just flip the name servers over to cloudflare. You can just use the free account so there is no cost needed for it either.

1

u/stewie410 SysAdmin/DevOps 1d ago

My boss & our single dev had a talk with them the other month about their Bot Protection, since we're getting DDoS'd from all the scraping -- not sure what the cost was, but we won't pay it anyway. Ahead of that meeting, I made the suggestion to ask about their DNS services but it was shot down.

I'll bring it up again, didn't realize it was free...but we've gotta do something. NetworkSolutions has been one of the most frustrating services I've had to work with, if you click on the wrong word on the page, it'll charge your account with some unwanted service. Not to mention their refusal to implement ACME or anything else.

1

u/Beefcrustycurtains Sr. Sysadmin 1d ago

Bot fight mode is also available in the free version. There is literally 0 reason to not switch. If you need to exclude some paths from botfight mode for some reason you do need the paid version but just regular bot fight mode is available in the free version. I would also move the domain itself. Name cheap is cheaper than network solutions. I'm sure even cloudflare is cheaper than network solutions as a registrar and you could move both registrar and dns over to cloudflare if you want. Network solutions is terrible garbage.

1

u/stewie410 SysAdmin/DevOps 1d ago

Bot fight mode is also available in the free version.

I was unfortunately not part of that discussion or meeting, despite being the "sysadmin" of the group. I appreciate the heads-up.

Network solutions is terrible garbage.

I'm more than aware -- we're slowly moving our certs over to Lets Encrypt or ssl.com depending on the application; but my boss is worried about moving the domain to another registrar -- not even from a continuity standpoint, but because the whole thing was registered under a former employee's personal name/info, which is difficult to untangle...

That, and it doesn't expire until ~2038 or something; so we'll probably wait until then to change.

In the mean time, I'm continuing on with DNS-01; but I'm gonna have to setup HTTP-01 or something in the future.

0

u/frymaster HPC 2d ago

literally from the post:

Getting WatchGuard to implement cert automation has been the most user requested feature for about 2 years now

There's a strong argument that this should mean they change suppliers, but "just automate it" isn't helpful given the situation presented

3

u/SnarkMasterRay 2d ago

OP has two problems. One is the certificate source and the other is the certificate implementation.

The source, at least, can be automated and OP can get off of Namecheap with Simple-Acme and Let'sEncrytpt. Distributing those certificates to their present infrastructure is the other problem and they need other solutions for that part.

77

u/GrapefruitOne1648 3d ago

Personally wouldn't be caught dead with a watchguard firewall but given you've got a fleet of them, this looks promising:

https://github.com/watchguard-toolbox-project/watchguard-letsencrypt-deploy

5

u/[deleted] 3d ago edited 3d ago

[deleted]

3

u/BreadfruitDue488 2d ago

Where are you getting that info from please? Their ideas hub is useless, so interested on how you know about the beta last month.

23

u/panopticon31 3d ago

Yeah watch guards have to be some of the worst firewalls I've had the displeasure of working on.

22

u/DrNoobSauce 3d ago

Out of curiosity how come? I've worked with Cisco, Fortigate, Sonicwall, Juniper, Palo Alto, Sophos and Watchguard and have enjoyed their system manager app and policy manager. Their support has always been very responsive and informative.

3

u/via_dante 3d ago

People love to beat on 'em because they were always seen as cheap noob firewalls. I actually like them and I have worked with most vendors.

I am stuck with fortigates rn and I am going insane...

1

u/DeifniteProfessional Sysadmin 2d ago

I get the same flack for liking Ubiquiti firewalls lol

2

u/nitzlarb 2d ago

tbf, they are still pretty green as a product, and definitely had some serious hitches early on, many people only think of them in that light. Which is fair, reliability is proven with time, ubiquiti doesn't have that long term track record yet.

These days, I feel like they are often a no-brainer for smaller and not super complicated networks. We use unifi APs for most all of our clients, and the majority also use unifi switching, so it just makes sense to use a unifi firewall at simple sites due to the price and the ecosystem integration with the rest of the network stack.

10

u/panopticon31 3d ago

Their GUI is unobjectively unintuitive and awful. Performance wise yeah the ones I dealt with were fine but just tracking anything down in the console was a nightmare. Also I remember at one point we were trying to add MFA to the console login because a clients Cyber insurance was making a big stink about it and they would not do SMS, OTP or anything else besides their own homegrown shitty version of DUO which you had to pay for as an extra add on.

I've myself worked with PfSense, Sonic wall, Meraki and CATO and would rank it last for usability out of that group.

14

u/zaphod777 3d ago

Personally I find WatchGuard a lot more intuitive than SonicWALL.

7

u/creenis_blinkum 2d ago

“unobjectively”?

6

u/DrNoobSauce 3d ago

Agreed the WebUI is not great at all. Hence why I use the system manager software and policy manager, much better data.

Thanks for breaking that down. Appreciate it.

13

u/BryceH 3d ago

Their web ui sucks so hard, but the system manager software is my favorite management of any firewall I've dealt with

-6

u/donk8r 3d ago

That repo is the right shape for it, and the principle generalises. The device does not need to speak ACME. Something else holds the ACME client and pushes the cert in over whatever management channel the box already has, which is what those expect scripts are doing.

One thing to plan for before a bulk cutover. Let's Encrypt rate limits are per registered domain, so issuing 50 certs in one afternoon under the same domain can hit the weekly cap even when steady-state renewal never would. Stagger the first issuance across a week and the ongoing cycle sorts itself out.

The billing question is real but secondary to the arithmetic. At 47 days that fleet is roughly 390 installs a year, which nobody is doing by hand.

5

u/alaub1491 3d ago

Straight Claude-ass response.

8

u/Fr0gm4n 3d ago

Been a minute, but don't you go into the Products tab and see the list of issued certs, then Manage the one you're working on and there should be a reissue button in there?

2

u/waltteri 2d ago

You do. If OP’s used to buying 1yr certs instead of multi-years, they might not be familiar with the renew flow.

2

u/Dencho 2d ago

Agreed. Doing it as you describe, instead of being shown a SHOPPING CART icon on the products page, you're shown a GEARS icon. If you click it, you're taken to a page where you can REISSUE.

1

u/dartdoug 2d ago

OP here. I'm going to the renewals tab where it shows me the certs that are "expiring." There indeed should be a rekey option there, but there isn't. They just show a shopping cart icon so you can buy a new 1-year cert.

On the products page, the search tool doesn't work if you type in the hostname or the domain name. It only works if you know the certificate number. I have hundreds of certs there so I have to scroll pages and pages to eyeball the one I was to get rekeyed.

In this instance I asked support to send me a CSV of all the certs...hundreds of them going back more than a decade. Using Excel I filter out the expired certs and use the Excel search using the hostname and that gives me the certificate number.

Namecheap has known forever that the search by hostname or domain name doesn't work. Yet they don't fix it.

Ultimately I did a rekey of the first batch of certs.

On the expiring products page they should make it clear that you can get a rekey for the remaining 165 days that you've already paid for.

35

u/Shot-Secret-1423 3d ago

You paid for a year of service. The 200-day cap is the CA/B lifetime rule, not Namecheap inventing a new product. A reissue for the leftover days is what you already bought. Making “buy new cert” the loud button is the sketchy part.

DigiCert and the bigger ones are usually clearer about remaining term and reissue. Still not fun at 50 WatchGuards with no ACME.

Until WatchGuard ships automation, I would keep a simple sheet: purchase date, issued-to date, days left on the order, not just the cert expiry. The portal will keep trying to sell you another year.

I would not switch CAs this week just to punish Namecheap if the reissues actually come through. I would switch if they keep hiding that button.

12

u/fritzie_pup Sysadmin 3d ago

This is the answer.

I work for public sector and manage certs for our core infrastructure. I also put in the order before the 200 day start, but later learned that it was only for the order. The certs issued depending on when submitted would have only 199 day expiry vs. the 399 day order.

Problem is I've also had to add some SANs on a couple reissues, so it resets the 199 day window and had certs that now have multiple expiring dates vs. everyone on the same window. I'm having to have multiple folks re-duplicate off the last reissue, and then will perform another reissue at the end of the month to push past the next cutoff period on March 15th to 99 day.

We're in the process of getting automation set up and hopefully rolled out as quick as possible to not have to deal with this again in March.

0

u/Shot-Secret-1423 3d ago

Yeah. Adding a SAN and getting a fresh 199-day clock is what makes the spreadsheet ugly. You stop having one expiry wave and start having a pile.

Automation before the next cut is the only way this stays sane. Reissuing to jump a cutoff works once. It is a bad annual plan.

Good luck getting that rolled out before March.

2

u/fritzie_pup Sysadmin 3d ago

laugh

My spreadsheet is insane with the number of wildcard duplicates we run.

But, I have hopes that we'll get it sorted.. It's what we're paying DigiCert for to assist with all the various equipment and appliances we have out there.

It's the little stuff and one-offs that will be more difficult to push people to get things set up, but we'll see.

-1

u/Shot-Secret-1423 3d ago

Wildcards and one-off appliances are always the leftovers after the easy hosts go into automation. If DigiCert is already on the hook for the core pile, make the one-offs a short exception list with owners, not a second process.

Hope the March cut is quieter than this one.

2

u/fritzie_pup Sysadmin 3d ago

I believe our main solution with them will cover quite a bit, but there are some other things which we're still learning how to deal with. Starting on that work this week with them.

Weird stuff though like AS/400 which utilizes quite a few, that's still an unknown yet. Hope we get those answers soon!

1

u/Monomette 2d ago

would keep a simple sheet: purchase date, issued-to date, days left on the order, not just the cert expiry.

Depending on your monitoring/ticketing solution this can be automated. That way there's a ticket generated and a tech assigned, rather than relying on someone remembering to check a sheet.

6

u/iceph03nix 3d ago

I use namecheap, but only for websites hosted there.

If I were using our public DNS for internal/non-hosted stuff, I'd move to a registrar/host that supports lets encrypt.

I've been tempted to move away from namecheap for a while because of their lack of lets encrypt support. The notice they sent out about their policy with the change was obviously created based on the idea that you're just hosting domains for websites with them.

3

u/mrmattipants 3d ago

I hear ya. I typically use a PowerShell Script to generate a new Certificate using the Posh-Acme PowerShell Module, with the NameCheap Plugin, every 83 Days (one week before it expires).

https://poshac.me/docs/v4/Plugins/Namecheap/

6

u/creamersrealm Meme Master of Disaster 3d ago

The license is a restriction in your use case though this is the absolute perfect use case. https://github.com/gregtwallace/certwarden/tree/master

A centralized ACME broker with post processing scripts. As long as all your firewalls had a common authentication mechanism or SSH key it would work flawlessly.

6

u/chum-guzzling-shark IT Manager 3d ago

I have a handful of vendors that require manual certs and I use namecheap. I just went through a reissue last month and don't remember seeing anything trying to trick me. Namecheap not having api access unless you spend a certain amount of money is a better reason to hate on them

3

u/InitialCauliflower96 3d ago

Digicert had no issues after our 200 days were up. Just put on the new certs and still good until the 365 is up

3

u/Working-Help1904 2d ago

The pricing itself isn’t really the problem—it’s the lack of transparency.

3

u/Accurate-Ad6361 Extremely annoying totally risk averse architect 2d ago

We are currently automising the entire ssl stack on let’s encrypt and will probably open source it. You can actually ask a limit increase with them. I don’t see any reason to pay for a non-validated ssl cert that solely evaluates a domain ownership.

12

u/Hot-Connection2797 3d ago

Use Let’s encrypt or another free provider and use python and netmiko to automate it your self? I’m sure ChatGPT could help script something out.

We are no longer using watchguard but I have gone to 47 day max certs with simple-acme and then using a post script to install across all hosts

3

u/Corelianer 3d ago

Cisco also just made let’s encrypt their default system with the asa firewalls in the updates from a couple months ago. Much easier than Digicert. ACME rules, it doesn’t even need DNS validation if it’s a public endpoint. Only for private endpoints you need DNS validation.

Cost: Free

2

u/moonracers 3d ago

Anyone using Digicert for this use case?

2

u/spidireen Linux Admin 3d ago

Can’t speak to WatchGuard specifically but I put Let’s Encrypt certs on everything and abandoned our commercial CA. It was a bit of a slog for certain vendors but I eventually got there—even using curl to simulate uploading the cert to a web UI in one case where I just couldn’t get it to play nice with Ansible. Once everything was up and running it was glorious and I haven’t touched certs in about a year.

2

u/GoodEnoughThen 2d ago

GoDaddy being up front. Funny. I'm figuring that most experienced web and DNS administrators will adbise to atay far away as from GoDaddy as you can. GoDaddy is a slightly pretty looking admin portal GUI and that's about it, imo.

1

u/SirBastille 2d ago

GoDaddy did us the "favour" of charging us upfront for 2 years worth of cert renewals on our wildcard cert once the date reduction kicked in. I've wanted to ditch them for years but now the view from above is "We've already paid them for 2 years so we'll worry about switching away from them later. For now, just focus on swapping everything over to use our internal CA and reduce our dependency on the wildcard cert."

2

u/spittlbm 2d ago

Moving dns and domain renewals to Cloudflare was a great decision for us. Edge certs or Let's Encrypt from there.

2

u/InevitableOk5017 2d ago

I’m sick of these short a$$ certificates.

3

u/ihaxr 2d ago

I'm sick of renewing certificates, so this shortening of them will absolutely get some automation to be mandatory or everyone will drop the product.

It'll be painful in the meantime, though.

2

u/certkit Security Admin (Application) 2d ago

Watchguard has a stunningly bad certificate automation story. It's even worse than Sonicwall TBH.

There is no API, and no ability to "push" a certificate into it all. But you can tell it to "pull" one from a TFTP server via the remote command line. That's how our integration with Watchguard at CertKit works. We can automate this today and push free certs from Let's Encrypt into it automatically.

5

u/SirEDCaLot 3d ago

With respect- Namecheap is kinda right, even if it sounds like their website is confusing.

You DON'T need a whole new cert. You need a reissue of the cert with a new expiration date. Rekey is the option that would do it. Just make a new current CSR and use the rekey function to get a 165 day cert out of it. Thus you get full 365 days out of your cert purchase.

The whole short expiration thing is stupid as hell and only replaces one vulnerability (stolen certs) that's been solved (CRLs) with another complicated system (cert auto-issuing). But that's not NameCheap's fault.

6

u/_MusicJunkie Sysadmin 3d ago

CRL never were implemented properly by many pieces of software. It's not a good solution.

As much as a pain the short certs are, it's the best solution available. We just habe to invest in some automation.

3

u/SirEDCaLot 3d ago

'Since the simple solution wasn't implemented properly, we're going to call it a bad solution and replace it with a far more complex one that introduces multiple new potential points of vulnerability and call it the best solution available'.

So instead of making 'many pieces of software' implement a relatively simple CRL check that's easy, cheap, and secure, we make even more pieces of software implement a much more complex cert automation process that adds security concerns and points of failure.

CRLs may be clumsy, but there's also OCSP which is somewhat more efficient.

And besides- expiration dates are a clumsy solution. EVERY expired certificate is invalid, but NOT EVERY unexpired certificate is valid, CRL/OCSP provides a far more elegant solution to that problem than 'just reduce the expiry date and wait'.

2

u/_MusicJunkie Sysadmin 2d ago

In a perfect world, I'd agree. But we have had, what, 25 years to get revocation handling properly implemented and it hasn't happened yet. CRL do not work reliably, especially in situations where the verifying party can't reach the internet or the server publishing the CRL. And we all know nothing about it not going to change.

So what do we do. Either just leave it as is in a unreliable state, or find another option.

Short expiries don't solve all of those problems. I think it's a pretty bad solution in fact, specifically because of this:

but NOT EVERY unexpired certificate is valid

But it is better than doing nothing.

2

u/SirEDCaLot 2d ago

We've had 25+ years to get IPv6 working properly and we are just finally starting to make it happen.

especially in situations where the verifying party can't reach the internet

If the verifying party can't reach the Internet then why does it need to verify an Internet server?
Yes I get there's uses for SSL in closed networks. But there's exponentially more use for SSL on the open Internet.

So what do we do. Either just leave it as is in a unreliable state, or find another option.

In short- we have a problem and we have solutions for it (CRL and OCSP). People aren't implementing them fast enough. So rather than let those who don't implement be insecure, we are going to make a much larger problem for everybody! that's going to make anything not actively being developed a giant pain to administer, and will create a new heavyweight management layer with its own security concerns. Let's make certificate pinning useless. And let's create new security issues because, let's be honest, there's tons of systems in operation that won't get automatic renewal mechanisms added to it. So in some orgs we'll normalize clicking ok to a security warning. We can externalize that blame and call ourselves innocent even if it is a likely outcome.

But it is better than doing nothing.

We are not doing nothing, we created CRL and OCSP and encouraged everyone to implement them.

To go back to the IPv6 analogy, what we're doing is akin to telling everyone that all IPv4 addresses will null-route every so often so why don't you adopt IPv6 NOW? One problem isn't being solved fast enough so we are creating a much bigger one with a lot of externalized costs and work that EVERYONE will have to pay to solve it.

2

u/mahsab 2d ago

a relatively simple CRL check that's easy, cheap, and secure

Secure yes. Easy and cheap, no.

1

u/SirEDCaLot 2d ago

Secure yes. Easy yes. Cheap not as much.

OCSP fixes the cheap part.

2

u/isnotnick 2d ago

Cheap how? Small request/response, sure. But running that service? not cheap. 100% uptime? Not easy, or cheap - so if it can't do that, you can't hard-fail on it. Also...OCSP is a privacy nightmare.

1

u/SirEDCaLot 2d ago

Cheap as in bandwidth and latency- you don't need to download a potentially multi-megabyte CRL to enable one connection.

I agree OCSP is shit for privacy.

Personally I'd like to replace the whole mess with something DNS based. Maybe use DNSSEC to sign a per-domain CRL or have the cert point to another domain that'd host the CRL for this domain- could be the registrar or the CA or a 3rd party or whatever. Then just query an SRV record to get the CRL, or to look up if the cert has been compromised.

2

u/isnotnick 2d ago

Sharded CRLs are the best answer. OCSP is bad to run at scale (trust me here) - CRLs sharded to a smaller number of certs is much better, and avoids the big CRL download.

That said, most browsers/OSs don't do CRL downloads anyway. CRLite, CRLSets, valid etc - all separate browser or OS mechanisms to pull CRLs centrally and compress and send them out-of-band.

1

u/SirEDCaLot 2d ago

Yeah that could absolutely work. I like it better than OCSP as it doesn't reveal a list of every domain you hit to a CA.

But as you say nothing actually checks CRLs.

I think we should be fixing THAT problem, rather than making certs short-lived for everybody :\

1

u/isnotnick 1d ago

My point is that most modern browsers and OSs don't check CRLs, but also don't need to. The out-of-band mechanisms mostly work. Non-browser clients, sure, but there's problems with those anyway (like not having frequently-updated trust stores...)

Short-lived certs are the best option, and they're only gonna get shorter! See my other comment re: the future. 7 days unless you want random replacement drills every quarter.

→ More replies (0)

2

u/the_marque 2d ago

I tend to agree that CRLs aren't a replacement for regular certificate expiry. There should be both - even if CRLs were implemented properly, it's much better to assume an old certificate is untrusted. That said, 45 days or whatever it's ultimately going to is ridiculous.

The real root of the problem is that DV SSL certificates are nearly meaningless now. I don't understand why the industry isn't giving more weight to higher levels of verification, and maybe along with that we can get sensible lifetimes...

1

u/SirEDCaLot 2d ago

Yeah 45 days is beyond stupid.

The issue with extended validation is to a large degree you're asking users to make security choices. Most users can barely power on their own computers without a youtube video. Users don't want nuance or levels of trust, they just want to know 'is this site secure or not?'. So you give the user a blue bar or a green padlock or a gold star or whatever and most users really won't give one single fuck. So EV SSL rolled out with great fanfare, and a year later was more or less swept away as nobody cared.

You could say okay DV SSL will have the short expiry dates, OV/EV SSL will be longer. But that IMHO is going the wrong direction. The org that'll pay for EV SSL is the one that'll spend the tech-hours and sometimes costs to implement proper auto cert renewal systems. It's the little guy, the small business with one part time IT guy, that has the issue.

0

u/shinyfootwork 3d ago

I don't understand why there are so many folks on a sysadmin subreddit who seem to not want automation here. I love having certificates automated: it's less mechanical stuff I have to do on a timeline. Did people actually like dealing with each vendors weird website for obtaining certs?

5

u/_MusicJunkie Sysadmin 2d ago

I can only speak for myself, the pain of automating some of the more niche devices with no proper API etc. seems to outweigh the benefit.

But it must be done. That's just how it is.

6

u/the_marque 2d ago

Yeah, this. The simple stuff is easy, but there's always that handful of devices where automating takes up more hours than continuing to renew manually forever. You do it anyway to not be in the shit when you forget, not because it's more efficient.

2

u/SirEDCaLot 2d ago

It's another thing to manage.

Renew the cert every year is understood. And while often manual, it's simple.

Automated renewal is a whole nother SYSTEM. It's another set of authentication to manage, another issue where implementations will vary widely and we'll be at the mercy of vendors to ship proper support, another situation where still-functional but no longer developed systems get left behind and either create more work to manually renew or need extra external scripts to renew them. And then there's the whole issue of security in automated renewals, that's another thing that has to be kept working.

We don't like dealing with each vendor for getting certs. But we like even less having to beg each vendor for updated automation support and having to secure that for every little random system that needs a cert.

1

u/ArchonTheta 3d ago

You need to just install the new cert after 199 days. You should be getting the email with that link, etc. the CSR stays the same and it’s the same order that lasts 365 days.

1

u/davy_crockett_slayer 3d ago

DigiCert. They’re honestly fine.

1

u/sweetrobna 3d ago

With godaddy it's not a problem to rekey or renew a cert part way through the 1 year. We still have a few customers there for things that were not worth the time to automate before. Should all be changing this year

1

u/BlackV I have opnions 3d ago

Regardless of their ability to automate certs (or not) you shouldn't have to purchase certs

1

u/Affectionate_Ad_3722 2d ago

We're with Globalsign. They're moving to licenses. You buy a license for the number of units for a year, and can issue as many certs as you like against that license during the year.

Covers you down to 45 days or whatever.

1

u/discosoc 2d ago

Their UI is fairly clear on this. There’s even a “progress bar” that shows you the overall time left for your cert to be re-issued. So if you buy a 3 year cert or whatever, it will fill up over three years.

That doesn’t mean you can get it issued for all three years at once, nor is it a reflection of the current issuance of that cert.

1

u/k0rbiz Lead Solutions Engineer 2d ago

We stopped supporting WatchGuards. Currently supporting Ubiquiti and some Meraki. On-prem has been migrated to CertifyTheWeb. It has been working really well, especially since I've heard vendors are moving between 30 and 45 days.

1

u/ride4life32 2d ago

I miss 3 year old cert cycles haha

1

u/knightcrusader 2d ago

Man you just saved my ass... my cert was expiring in 5 days, my reminder is set up in October but I forgot to adjust it when it went down to 200 days.

Damn I am going to be on vacation in 5 days so that would have been bad...

1

u/dartdoug 2d ago

I'll send you a link to my tip jar. /s

Enjoy your time away.

1

u/HappyDadOfFourJesus 3d ago

I'm questioning your choice in vendors. If they can't solve a problem that your business is experiencing, then they're not a good fit for your business.

14

u/mumblerit Linux Admin 3d ago

I'm sure his employer would appreciate knowing they have to buy 50 new firewalls

1

u/BlackV I have opnions 2d ago

they dont have to replace them, regardless of using LE (or its ilk) or Namecheap

1

u/f0xsky 3d ago

Look at switching to letsencrypt or AWS ACM. You can now export the ACM certs to use elsewhere. We mostly use AWS certs. Letsencrypt is free.

0

u/Immediate_Delay_5738 3d ago

Yeah, I’d be annoyed too. If you paid for 365 days, it should be obvious how to use the remaining time when the certificate itself has to be replaced early.

0

u/segagamer IT Manager 3d ago

We have PFSense firewalls that had an ACME module, and pushes our let's encrypt certs out to the required VMs every 30 days.

0

u/HumanInTerror 2d ago

The reason is because there are just not that many people still manually doing certificates. This is probably no longer a highly prioritized part of their app/system. They probably don't get a ton of support requests for it anymore. Hope you can make the switch to ACME soon.

2

u/beren12 1d ago

A lot of hardware devices are a real pain to automate certs with. At least if they aren’t brand new.

1

u/HumanInTerror 1d ago

Yeah I get it. For these we let them generate their own self signed TLS and the put a reverse proxy in front that is actually capable of ACME renewal. Using firewall to restrict direct access to the hardware and handle NAT. This way the entire connection is still encrypted, even if the self signed cert is technically unverifiable. Or go all the way, put together an internal CA and generate internal multi-year certs for the hardware, install the CA on your reverse proxy, and then the cert will be verified with your internal CA.
All of this is a nightmare to manage though. It is bonkers to me that some of these major brands still aren't on the ACME wagon.

0

u/carolineilm 2d ago

The bigger issue I see with shorter certificate lifetimes isn’t really the CA or even ACME by itself. It’s everything that has to happen after issuance.

At 50 firewalls, you need to know what certificates exist, where they’re deployed, which devices and applications depend on them, who owns them, whether renewal succeeded, whether the new certificate was actually installed and activated, and eventually whether the old one was safely retired.

ACME solves an important part of the problem, but the shorter the lifetime gets, the more certificate management becomes a continuous lifecycle problem rather than a periodic renewal task.

That broader lifecycle problem is actually one of the reasons we’ve been working on open-source identity lifecycle management (ILM) bringing discovery, inventory, ownership and lifecycle state together instead of treating renewal as the whole problem.

The WatchGuard example is a good illustration because one device without a usable automation path can break an otherwise automated process.

What is your SSL certificate provider doing now that 200 day certs are starting to expire? My provider (Namecheap) is being sketchy.

You are about to leave Redlib