question

StanMorisse-8060 avatar image
1 Vote"
StanMorisse-8060 asked StanMorisse-8060 commented

What is the autorenewal procedure for multiple certificates enrolled using the same certificate template?

Hi,

At one of our customers, I think we might be running into a certificate autorenewal bug on our Windows server 2019 Webservers which are configured to use SSL certificates with autorenewal.
Automatic rebinding of the certificate is enabled in IIS (via Task Scheduler) and SNI is activated for each website we host.

Deployment Scenario

  1. On a single webserver we created a few websites.

  2. Each website was configured with SNI enabled.

  3. Certificate Autoenrollment was configured and applied via Group policy on the Webserver.

  4. In IIS the 'Certificate Rebind' feature was enabled

  5. For each website an SSL certificate was enrolled using the same custom 'Webserver' certificate template. The template had a lifetime of 1 year and a renewal at 2 months prior to end

  6. The enrollment of all SSL certificates occurred in the same timeframe (less than 30 minutes)

  7. Each certificate was bound to its proper website in IIS.

Renewal Time
At the moment of renewal, the group policy client side engine kicked off the autorenewal of the certificates. It should renew all certificates that will expire in less than 2 months as per the custom 'Webserver' template.
However, only one of all the certificates remained. And that one was used for rebinding to all websites, which was incorrect.

Investigation
When looking into the eventlog "CertificateServicesClient-Lifecycle-System", we noticed the events that took place to replace the old certificates.
we noticed the following with the first certificate:

  1. Event 1006 was logged: A new certificate has been installed.
    The Enroll action informed us of the thumbprint of the new certificate

  2. Event 1001 was logged: A certificate has been replaced.
    The renew action informed us of the old thumbprint and the new thumbprint.

  3. Event 1005 was logged: A certificate has been archived
    The log entry shows the thumbprint of the certificate which has been replaced.

The subsequent action is to renew the following certificate. However:

  1. No event 1006 is logged, we also see no certificate pending request at the CA.

  2. No event 1001 is logged.

  3. Event 1005 is logged, and the certificate is archived.

This happens for each subsequent certificate, and at the end the rebinding occurs with the single replaced certificate.

Reproducing the error
We have created a test-webserver and lowered the lifetime on the template to one day, with a renewal of 2 hours prior to expiration.
We are able to continuously reproduce the behavior.
Even when a certificate is not renewed at the same time but more than 2 hours later, no event 1006 nor 1001 is logged. The certificate not replaced but archived, and due to the lack of event 1001 automatic rebinding does not occur.
It only occurs for certificates based on the same template. During the test we also used single certificates based upon another template and those were replaced correctly.

Assumption

What we assume, based upon the tests we performed, is that in the renewal algorithm a verification occurs based upon the template oid/name.
it looks like the algorithm verifies that, if a certificate still exists in the certificate store that has a valid lifetime and is based upon the same template, other certificates that are also based upon this template do not need renewal.


Can someone have a look into this issue? Thanks!

windows-apiwindows-server-security
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Crypt32 avatar image
1 Vote"
Crypt32 answered StanMorisse-8060 commented

I think we might be running into a certificate autorenewal bug on our Windows server 2019

it is not a bug, it is by design. Autoenrollment never was designed to handle multiple certificates based on same template where autoenrollment is configured. Only first instance of certificate is automatically renewed. If there are additional certificates you need to renew -- you must do it manually, or use scripting. I would recommend to read certificate autoenrollment whitepaper I wrote 3 years ago: https://www.sysadmins.lv/blog-en/certificate-autoenrollment-in-windows-server-2016-summary.aspx. This document explains the autoenrollment behavior in details and confirms that your issue is by design.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the info Vadims!

I will read through your documentation right away. But at first instance it seems strange behavior if it is indeed 'by design'. (I always feel miserable when those 2 words (By design) are combined with the Windows OS ;-))
Nonetheless, I will get back when I have read through all of the info you sent me via the link.

0 Votes 0 ·
DaisyZhou-MSFT avatar image
0 Votes"
DaisyZhou-MSFT answered

Hello @StanMorisse-8060,

Thank you for posting here.

Hope the information provided by Crypt32 is helpful to you.


Should you have any question or concern, please feel free to let us know.


Best Regards,
Daisy Zhou

============================================
If the Answer is helpful, please click "Accept Answer" and upvote it.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

StanMorisse-8060 avatar image
0 Votes"
StanMorisse-8060 answered StanMorisse-8060 commented

Hi Vadims ( @Crypt32 ),

I have read through all the documentation you wrote, including the comments and answers.
I am already familiarized with most of it (except for some details), since I implement/configure/use this on a daily basis at my customers.

To ensure I didn't miss something, I have again re-read the article https://www.sysadmins.lv/blog-en/certificate-autoenrollment-in-windows-server-2016-part-2.aspx, which has a more depth view on certificate autoenrollment processing rules.

My findings based upon your documentation
To summarize what I have understood from that article, as per your article under chapter "Automatic certificate renewal":

  1. In the first step, autoenrollment enumerates all existing certificates that use certificate templates and checks its validity.

  2. If existing certificate passes validation checks, autoenrollment examines whether certificate template is set up for autoenrollment.

  3. autoenrollment checks passes the certificate to certificate chaining engine (CCE) to determine its validity.

  4. If existing certificate’s validity meets renewal threshold, autoenrollment will submit renewal request to CA server.

All of the above applies to my case, except for one important thing (and I assume that that is the 'by design' you are refering to ): The certificates were initially manually enrolled, as they required input for the website names.

Our case of manually enrolled certificates

When I look further into the chapter "Renew manually enrolled certificates" I've read the following:

Manually enrolled certificate renewal if none of the following conditions are true:
2. Existing valid and non-expired certificate based on this certificate template is found.

I assume that the latter is the 'by design' part of the process you are referring to. Once the first certificate has been renewed, the condition applies and therefore other certificates based on the same template are archived.

Remaining question

I certainly understand that you call it 'by design' if at all this is the exact case you're addressing me too.
However, in my opinion, a design can contain flaws. Earlier you wrote:

"Autoenrollment never was designed to handle multiple certificates based on same template where autoenrollment is configured".

Although I now certainly understand this to be true, the original document you refer to was for XP and the article does not contain the specific condition you described. So I was wondering where you got this information, regarding the conditions, from?

My 2 cents about this condition
According to me this 'by design' issue is something which MS should look into, as certificate use (TLS...) and autoenrollment becomes increasingly popular for private websites in Enterprises. Webserver SSL certificates in these webfarms are being set up with loads of certificates which should be able to be autorenewed without any manual or scriptbased solution, since that is exactly the definition of the word 'Autorenewal'.


Last but not least

  1. Vadims (@crypto), I want to already thank you for pointing me in the right direction of your documentation, so I have some more understanding of the design.

  2. Towards Microsoft: Regarding the design, I certainly would be a happy customer if Microsoft was able to have a look at this specific condition and rework it, so these certificates with different identities are not archived/deleted but renewed.

Thanks!

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I assume that the latter is the 'by design' part of the process you are referring to. Once the first certificate has been renewed, the condition applies and therefore other certificates based on the same template are archived.

that's correct. In short, once particular certificate template succeeds during renewal, no other certificates based on processed certificate template are eligible for renewal. Instead, they are archived. Say, you have 100 certificates based on particular certificate template: 1st certificate entry is attempted to be renewed. If renewal succeeds, then all remaining 99 certificates are going to be archived and not renewed.

So I was wondering where you got this information, regarding the conditions, from?

There is a reference list in autoenrollment summary blog post that contains all the links used to write this whitepaper. Specifically, manually enrolled certificate renewal details came from [MS-CAESO] specs.

According to me

LOL

this 'by design' issue is something which MS should look into

whether they should, they will not, unfortunately. Microsoft even refused to review my whitepaper, so I doubt they will review entire AE mechanism just based on forum request. You may need to open a ticket with Premier Support.

0 Votes 0 ·

Vadims (@Crypt32)

Again a big thanks for your latest reply.
The customer has a MS Premier support contract, so I will ask them to open such a case. Whether the customer will, is uncertain at this point. They might turn to scripting instead.

Also thanks for the feedback on where you got the info from. I have found the specific pdf and noticed no changes have occurred in the document since 2013.

PS. I was wondering why you wrote 'LOL' underneath 'According to me'. I assume you had to laugh, because I am just a 'little shrimp in the ever expanding world of Microsoft' and have no leverage. And right you are ;-)

Thanks for all your help and the references you posted!

0 Votes 0 ·

I assume you had to laugh, because I am just a 'little shrimp in the ever expanding world of Microsoft'

"According to XYZ" often is used as a reference to an authoritative statement, research, source, etc. And you are claiming yourself an authoritative source :)

0 Votes 0 ·
Show more comments