Secure Channel Broken - continuation of "The trust relationship between this workstation and the primary domain failed."

While there can be several reasons for AD replication to fail due to an "access denied" error (you may find more information in KB article 2002013 - Troubleshooting AD Replication error 5: Access is denied http://support.microsoft.com/kb/2002013/EN-US),
in here we will be focusing on broken secure channel issues on Domain Controllers and how to reset them.
Then as promised in my previous post we will also have a look at tools/commands you may use to verify and reset secure channels and trusts.

A replication issue may actually be the cause for a broken secure channel between DCs in the first place.
Another common cause may be restoring from backup a DC to a previous point in time (or to a system restore point in case of client machines) or time jumps which don't cause the broken secure channel directly but may seen be as a colateral damage.
(by the way take a look at http://blogs.technet.com/b/askpfeplat/archive/2012/11/19/did-your-active-directory-domain-time-just-jump-to-the-year-2000.aspx and http://blogs.technet.com/b/askpfeplat/archive/2012/11/26/fixing-when-your-domain-traveled-back-in-time-the-great-system-time-rollback-to-the-year-2000.aspx
which have been updated recently).

To reset the Secure Channel of a Domain Controller you have 2 methods, one requires reboot the other doesn't however requires more tasks to be completed.

If restarting the DC is an issue for you due to business, availability or change control requirements do the following:

1. Set the Kerberos Key Distribution Center (KDC) service startup type to Disabled, and restart the domain controller
(particularly important if you have more than one DC on the same domain, so this way you will force the affected DC to contact another DC for kerberos authentication, instead of using itself)

2. Once the DC restarts, reset the secure channel:

netdom resetpwd /server: /userd:\ /passwordd:*
(you should get the foolowing message: "The machine account password for the local machine has been successfully reset")

3. Then, set the startup type for the KDC service back to Automatic and restart the DC once again.

Note: Do not start the service manually as it will be started automatically during reboot.

The 2nd way is:

1. Delete any mapped drives that might exist on the affected DC, for that run:

net use * /d

2. Stop the KDC service.

3. Purge all kerberos tickets from the computer credentials cache (this is the memory space were the computer stores its tickets, there's nothing in common with cached credentials).

Depending on the Operating System run the following:

On Windows Server 2003:

at /interactive cmd.exe
(check current system time and add a couple of extra minutes, for example if current time is 14:14 type 14:16)

Note:
Another command prompt will launch under the System account context (SVCHost.exe )on the Console session of the problem DC at the time you entered (14:16 in the example above).
If running the command via RDP ensure that you are connected to session 0 (mstsc /console or /admin depending on mstsc version) or you won't be able to see the 2nd command prompt.

klist purge

On Windows Server 2008 or above:

klist -li 0x3e7 purge

Note:

The -li 0x3e7 gets klist into the computer credentials cache so there's no need for the "at" command.

4. Reset the Secure Channel:

netdom resetpwd /server: /userd:\ /passwordd:*
(You should get the following message "The machine account password for the local machine has been successfully reset"

 
net use \\\IPC$
(The DC will request new kerberos tickets)

5. Force the DC to replicate from/to the PDCemulator (so both ways first inbound then outbound):

repadmin /replicate <destination> <source> <NC> /force

example:
repadmin /replicate DC1 DC2 DC=Contoso,DC=com /force
(you should get the following message "sync from DC2 (the PDCe) to DC1 (the affected DC) completed successfully")
repadmin /replicate DC2 DC1 DC=Contoso,DC=com /force
(you should get the following message "sync from DC1 (the affected DC) to DC2 (the PDCe) completed successfully")

Note:
If the affected DC successfully replicates from the PDCe then start the KDC service. If not successful, you will have to use the method the requires restart.
If replication to the PDCe fails, delete the old outbound connection object and trigger KCC (repadmin /kcc) in order to rebuild them, and force outbound replication again. (this might be required for COs with other replication partners)

Done.

In order to verify the health status of secure channels you may use Nltest.exe or Netdom.exe (which is also used for trusts).

If using Nltest run:

Nltest /sc_verify:<DomainName>
You should get "Trust Verification Status = 0 0x0 NERR_Success" if not you must reset the secure channel.

Note:
This command doesn't work on a PDCe, so please don't reset the PDCe secure channel once you run it thus getting an error message.
Don't use /sc_query to verify secure channels because it doesn't verify the SC, it just tells you the information about the last established SC. (so you will get a valid answer even when you cannot reach a DC)

If using Netdom:

netdom verify /domain:<DomainName>

In order to verify Trusts:
(Trusts work in a similar way as Secure Channels, there is a TDO (Trust Domain Object) maintained in each trusting and trusted domain partition, which password has to be in sync, of not the trust gets broken).

netdom trust /Domain: /verify

Note:
Please take into account that depending on security settings between domains this command may fail, for more troubleshooting information see the article below:
Client, service, and program issues can occur if you change security settings and user rights assignments
http://support.microsoft.com/kb/823659

Hope it has been useful for you. Feel free to share your thoughts.