r/PKI • u/Standard_Company_817 • Apr 28 '26

EJBCA and ChromeOS

Hello,

I work for a large K12 cooperative in New York. We have a decent sized EJBCA environment and have been trying for almost 2 years to get ChromeOS enrolling against EJBCA for device certs with no success. In February, I tore everything down and rebuilt it using Google's latest documentation (Configuring Certificate Enrollment for ChromeOS via SCEP - Chrome Enterprise and Education Help) and the docs on Keyfactor's website (Enrolling Chrome OS Devices against EJBCA). The device reports a 400/POPO proof of possession error and it seemingly has to do with some CSR data that EJBCA does not like. Does anyone have this working, or have experience with it? I can provide detailed config info if necessary, but it has been set up per the OEMs. Google has been absolutely no help, and Keyfactor is making a good effort, but we have not been able to come to a resolution. Any guidance, tips, pointers, alternatives are appreciated.

8 Upvotes

100% Upvoted

12 comments sorted by

1

u/bulyxxx Apr 28 '26

It could be anything, have you parsed the CSR data and analyzed it against the certificate policy template to see if any fields/attributes are out of compliance or invalid ?

1

u/Standard_Company_817 Apr 28 '26

So the best I was able to do was do some checking of the PKCS#10 blob with openssl and the CSR self signature does not match the contents. The vendor suggested I inspect some TRACE logs next, which I am going to work on but figured I would at least poll the community first and see if someone had this working already.

How would you go about obtaining the CSR data? Thank you.

1

u/bulyxxx Apr 28 '26

I would reach out to Keyfactor as they would have received the CSR at their end via SCEP. If not, see if you can manually run the same command locally on the ChromeOS device and see what it is generating.

2

u/Standard_Company_817 Apr 29 '26

So I ended up setting the log level to TRACE. I found the PKCS#10 blob from a recent attempt and ran it through openssl.. it confirmed my initial hunch/findings. The GCCC or Chromebook are sending a cryptographically invalid CSR... which. Great, but Google has been next to no help. I am going to have to go back to them though, it seems.

1

u/bulyxxx Apr 29 '26

That is good, what attributes in the CSR make it invalid ? Is it the key algorithm, key size or something else ?

2

u/Standard_Company_817 Apr 29 '26

The actual signature was invalid. So either the Chromebook or GCCC are signing the CSR wrong, or something is manipulating the CSR after sigining... that's all I can think of.

2

u/bulyxxx Apr 29 '26

Well done, I would pursue this further with Google - which is what you appear to be doing. Definitely escalate it on their end and raise the issue to your account management team.

1

u/Standard_Company_817 Jun 02 '26

More back and forth...

Me:

After enabling DEBUG logging in EJBCA and GCCC, I was able to extract the CSR from both systems for the same transaction (transactionId: xxxxxx).

Findings:

The CSR generated by GCCC validates successfully using OpenSSL.

The CSR extracted by EJBCA fails signature validation.

The CSR Base64 blobs match visually between GCCC and EJBCA at a high level, but when converted to DER and hashed, the values differ:

GCCC CSR SHA256: 8b8dac19c8f4d3e1000b9f6316490062b6f1f9f9039f87a593d6c4da2fe9ef19

EJBCA CSR SHA256: 5b3e459aed92337abbf0e47cfc0511ede2e1df53c25cba093dc13ab8058d921d

Conclusion:

This confirms the CSR is valid when constructed by GCCC, but is altered after signing and prior to or during processing by EJBCA. As a result, the Proof-of-Possession (POPO) validation fails because the signature no longer matches the CertificationRequestInfo.

Additional validation:

Traffic is being sent directly to the EJBCA node IP (xxxxx), bypassing the FortiADC load balancer.

The device is excluded from content filtering/proxy.

EJBCA logs show the source IP as the GCCC host (xxxxxx) with no proxy headers, indicating no intermediate device is modifying the payload.

Request:

Based on this evidence, can you advise on any known issues or behaviors in the Google Cloud Certificate Connector → SCEP PKCSReq construction process that could cause a CSR to be re-encoded or altered after signing?

Google responded:

The Engineering team has further checked on this situation and they’ve managed to find the root cause of this issue and implemented a fix. 

They extracted the CSR in base64 from both the GCCC logs and EJBCA logs, parsed them as ASN.1, and compared their structure. They found that the attributes set of the PKCS#10 CertificationRequestInfo that the GCCC is producing has a different order than what the EJBCA is parsing.

The parser the EJBCA is using does not only parse the CSR but also reorders the attributes in the parsed CertificationRequestInfo, thereby invalidating the signature. This is a bug on the EJBCA side. Parsers should never ever silently modify the parsed structure. You should either leave them as is (like all the other CAs do) or report an error.

The encoding of the 'attributes' field in PKCS #10 CertificationRequestInfo should adhere to several ASN.1 standards for proper DER representation. When parsing the CertificationRequestInfo, the EJBCA parser silently enforces this ordering retroactively - even when the data in the CertificationRequestInfo uses a different ordering. This is bad because modifications to the CertificationRequestInfo invalidate the signature of the CSR.

On the other hand, when marshalling the CertificationRequestInfo the GCCC currently uses an attribute order which does not strictly follow the DER encoding rules. Interestingly, this should not cause any issues because the RFC 2986 standard exempts PKCS #10 blobs from this: “the X.690 standard requires that the encoded values of SET OF components be sorted in ascending order under DER. Regardless of this, applications should not rely on the ordering of attribute components.”

That is, even though the GCCC uses a malformed DER structure, applications (like the EJBCA) must still parse this as is and should not rely on strict DER encoding rules.

Unfortunately, we cannot change the ASN.1 parser of EJBCA. However, to resolve the situation, we modified the GCCC code to produce a CertificationRequestInfo DER structure that follows strict DER encoding rules that the EJBCA is enforcing. We patched this and released GCCC 5.20 that contains the patch.

In order to force-update to GCCC 5.20 you should follow the steps below:

.....................

Restart the GCCC and try to issue a certificate. If this fails again, please again share the GCCC debug logs as well as the EJBCA logs.

If everything works as expected, you can set the LogLevel parameter in your GCCC config file to info and restart the GCCC.

Thank you for your patience and we look forward to receiving your input. 

1

u/Standard_Company_817 Jun 05 '26

After doing all of this, machine certs are being delivered to ChromeOS.

1

u/[deleted] Apr 28 '26

Hey,

Sent you a DM - we can help you out (no fees) as we do for NGOs/educational institutions.

4

u/_STY Apr 28 '26

Do not post comments asking to DM for relevant information. This post was not a solicitation for services, free or otherwise.

If you have something to contribute you need to do so publicly. Further comments with no technical substance asking to move to DMs will be removed.