question

GraemeShea-5893 avatar image
0 Votes"
GraemeShea-5893 asked GraemeShea-5893 edited

Automatic certificate enrolment or renewal failes at remote site, manual enrolment ok.

Hello, I am asking for help with the following problem,
Automatic certificate enrolment/renewal works ok at our main site and manual enrolment/renewal works ok at our remote (routed wan) site but automatic enrolment/renewal suddenly started failing at our remote site with a "The RPC server is unavailable. 0x800706ba (WIN32: 1722 RPC_S_SERVER_UNAVAILABLE)" error about 4 months ago. There is nothing I can relate the failure to. The tests I have done are:

Manual enrolment or renewal works fine, it's only automatic that fails and only at the remote site.
BPA on the clients and server shows nothing of interest.
Certutil -ping -config "Ourserver\OurCA" completes successfully (in a user context) when run on a client.
certutil -tcainfo completes successfully on both "good and bad" clients with “A certification chain processed correctly, but one of the CA certificates is not trusted by the policy provider. 0x800b0112 (-2146762478 CERT_E_UNTRUSTEDCA)” error at both sites
It's not immediately obvious to me which certificate is not trusted but since the result is the same at both sites it's not likely to be the cause of this issue.
There is nothing obviously different either networking or group policy wise at the remote site that would cause any issue like this.
I have tried Ryan Ries’s TestRPC script and it reports RPC to and from the CA is ok. (when run in a user context).

I haven't tried with the firewall turned off as it would have to be off for some time to make sure I captured the event.
I haven't been able to get meaningful logs from the CA server. I have enabled object auditing in the local security policy and Auditing in the CertServ console but nothing appears in the security event log.
Has anyone stuck this before and can point me in the right direction?

Many thanks in advance.
Graeme

windows-server-security
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

LimitlessTechnology-2700 avatar image
0 Votes"
LimitlessTechnology-2700 answered

Hello @GraemeShea-5893

According to documentation, the behavior you are facing is expected, by design and cannot be turned off. Kerberos Authentication requires an RPC connection from CA to DC. What are the options for you:

-Enable RPC communication between CA and domain controller.
-Use Domain Controller Authentication certificate template instead of Kerberos Authentication template. Domain Controller Authentication template does not require RPC connection back to DC.
In fact, I didn't remember all the details and kudos to you, that you did good investigation and pointed about a failed RPC callback, this really reduced the amount of possible reasons. Full details on why this happens are below.

About the 0x800b0112 error, This issue can occur if the CA certificate is not in client's Enterprise NTAuth store. The local NTAuth store can be manually populated using the utility certutil.exe:

Certutil -enterprise -addstore NTAuth CaCertificate.cer

More info here:
https://docs.microsoft.com/en-US/troubleshoot/windows-server/windows-security/import-third-party-ca-to-enterprise-ntauth-store

Hope this helps with your query,


--If the reply is helpful, please Upvote and Accept as answer--

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GraemeShea-5893 avatar image
0 Votes"
GraemeShea-5893 answered

Thankyou @LimitlessTechnology-2700 for the reply.
The clients are using the machine template to renew the certificate, so bidirectional communication shouldn't be the problem. Just to make sure I ran Ryan Ries’s TestRPC script and it shows that RPC is available in both directions between the clients, DC and the CA. We have only a small network (by enterprise standards) and the CA is installed on a DC.

What is confusing me is that manual renewal (or enrolment) using the certificates snap-in works and It’s only automatic that fails with the RPC error.

I found the task in task scheduler that handles the renewal and tested with both firewalls disabled and also while running the task under a different account but the error is the same.
I ran Wireshark finding a lot of “spurious retransmissions” on the CA server and retransmissions on the client that don’t occur when I do a manual renew on another host at this site. I only got one duplicate Ack in this case.

I’m at a loss now, it’s not a firewall issue or routing issue. I can show our WAN provider the packet captures but since manual enrolment doesn’t show the problem they are likely to say it’s not their network causing the issue and I’m not sure they would be wrong.
After hours tonight (Melbourne time) I’ll move the CA to another Hyper-V host with different hardware (and drivers) to see if that makes a difference. otherwise I'll have to see if I can roll out a script to do a manual renew before many more clients drop off the WiFi.

Regards
Graeme

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

GraemeShea-5893 avatar image
0 Votes"
GraemeShea-5893 answered GraemeShea-5893 edited

Just an update incase it helps anyone in the future.
I found an apparent bug in Ryan Ries RPCCheckv2 script in that the Variable $Computer changes to $Computername halfway through the script and doesn’t pass the computer (target) name on. Correcting this returned multiple ports used for RPC when run from the local site but only one port when run from the remote site.
Lacking the skills to take this any further I implemented a workaround by using Group Policy to create a task to run a PowerShell script once a day that queries the certificate store to find certificates expiring in the next 180 days and renews them. The script has a limitation of only handling one certificate so job for the future is to allow for multiple certificates.
I know this workaround is a kluge but its got me going for the time being, your experience may differ. Replace Your.Domain, Your.CA.Server log path etc. with your local information.

Regards
Graeme


 #Renew the machine cert. Auto renewal at the remote campus failed and have not found the cause.
 # all thanks and original rights to the post by Alex at https://stackoverflow.com/questions/57435078/creating-a-powershell-script-to-automatically-renew-user-certificates on which this script is based.
 # modified by Graeme Shea 23/03/2022
 $ExpiringDays = 180
    
 if ((Get-ChildItem Cert:\LocalMachine\My -Recurse -ExpiringInDays $ExpiringDays | Where subject -Like "*.your.domain -ne $null ) {
     Get-Date | Out-File -FilePath C:\scripts\CertEnrol-Log.txt
    
     Get-ChildItem Cert:\LocalMachine\My -Recurse -ExpiringInDays $ExpiringDays | Where subject -Like "*.your.domain | % {
         certreq -enroll -machine -q -config your.CA.Server\Your-Server-CA -cert $($_.thumbprint) renew  | Add-Content -Path C:\scripts\CertEnrol-Log.txt
         #certreq -enroll -machine -policyserver * -cert $($_.thumbprint) renew  | Add-Content -Path C:\scripts\CertEnrol.txt
         }
     } 
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.