question

AlexMcFarland-2053 avatar image
0 Votes"
AlexMcFarland-2053 asked HannahXiong-MSFT answered

Replication issues after ungraceful DC restore/restore from backup

Hi,

I had a major issue with one of my domain controllers where it could not be gracefully demoted and had to be restored from backup. I know this is a no no but there was no other option at the time. Unfortunately I went back too far, 1 month to be precise, and since then my domain has had some big replication issues. I have been using dcdiag to try and diagnose the issues and I am receiving this error when I attempt to replicate to any of the other DCs from my FSMO master:

TEST: Authentication (Auth)
Error: Authentication failed with specified credentials
[Error details: 1326 (Type: Win32 - Description: The user name or password is incorrect.) - Add connection failed]

            TEST: Basic (Basc)
               Error: No LDAP connectivity
               Error: No WMI connectivity
               [Error details: 0x80070005 (Type: HRESULT - Facility: Win32, Description: Access is denied.) - Connection to WMI server failed]
               No host records (A or AAAA) were found for this DC

I do see host records for all of the DCs in ADS&S so I don't understand that error message. At first I believed that this had to do with KDC/Kerberos more than anything because the secure channel between my failed DC & the rest of the domain was broken. Trying to fix the secure channel has been a headache, not really sure where to go from here.

I did find this article useful and I think it pertains to me: https://support.microsoft.com/en-us/help/2002013/active-directory-replication-error-5-access-is-denied

These are the resources/guides that I have tried using:

Any leads would be appreciated as I'm really trying everything to repair this. Once I figure out one error, it leads to another, and so on... Thank you

Also, I did try posting this in TechNet and it keeps redirecting me to here... please let me know if this is incorrect.


windows-serverwindows-active-directorywindows-dhcp-dns
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DSPatrick avatar image
0 Votes"
DSPatrick answered

Restoring a domain controller in a multi DC environment is not recommended. The much safer / cleaner option is to seize roles to a healthy one (if needed)
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

then perform cleanup.
https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup

then rebuild the failed one from scratch by using dcdiag / repadmin tools to verify health correcting all errors found before starting. Then stand up the new replacement, patch it fully, license it, join existing domain, add active directory domain services, promote it also making it a GC (recommended), transfer FSMO roles over (optional), transfer pdc emulator role (optional), use dcdiag / repadmin tools to again verify health.


--please don't forget to Accept as answer if the reply is helpful--











5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HannahXiong-MSFT avatar image
0 Votes"
HannahXiong-MSFT answered AlexMcFarland-2053 commented

Hello,

Thank you so much for posting here.

According to our description, what restore did we preform, authoritative restore or nonauthoritative restore? Besides, how about our other DCs? Do they work properly? It is suggested that we could backup the healthy DC before any operations.

If there are so many issues with this DC, as Dave mentioned, we could forcefully demote the DC, and done a meta data cleanup. Then promote it as a new DC.

Hope the information is helpful. For any question, please feel free to contact us.


Best regards,
Hannah Xiong

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

We restored the failed DC from backup, it is a virtual machine running in an ESX environment.. The other DCs seem to be working fine with one another except the FSMO master. Replication between the FSMO master DC and the other 8 DCs are failing. I am worried about transferring the FSMO roles to another DC, how is this possible if replication is not working anyway? I have had this issue on subdomains before and it broke the entire domain. I need to fix replication between the FSMO master DC and the other 8 DCs, not demote my FSMO DC and make things worse. If that server goes down, nothing in our domain is going to function.

Why would restoring a DC from backup that is a week old not break replication, but going back a month broke everything? I feel like this has to do with something related to KDC/time error but I am running into error after error when I try to fix the KDC service.

Thanks, Alex

0 Votes 0 ·
DSPatrick avatar image
0 Votes"
DSPatrick answered AlexMcFarland-2053 commented

Why would restoring a DC from backup that is a week old not break replication, but going back a month broke everything?

Neither is recommended. The much safer / cleaner option is to seize roles to a healthy one (if needed)
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

then perform cleanup.
https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup

then rebuild the failed one from scratch by using dcdiag / repadmin tools to verify health correcting all errors found before starting. Then stand up the new replacement, patch it fully, license it, join existing domain, add active directory domain services, promote it also making it a GC (recommended), transfer FSMO roles over (optional), transfer pdc emulator role (optional), use dcdiag / repadmin tools to again verify health.


--please don't forget to Accept as answer if the reply is helpful--






· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I can't seize the roles on a healthy DC this is the problem, the FSMO master won't replicate so how could I seize roles elsewhere? The roles have to be taken from the FSMO master that isn't functioning right. The information on that server will not successfully copy to any other server. I'm afraid that it's going to blow up everything even more. Do you understand that the roles can't be transferred? I have to use dcdiag/repadmin on the failing FSMO master but I am running into issues doing that with error 5: access is denied.

0 Votes 0 ·
DSPatrick avatar image
0 Votes"
DSPatrick answered DSPatrick edited

What happens when you try? If the failed one happens to be the fsmo role holder then you can seize the roles to a healthy one.
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control





--please don't forget to Accept as answer if the reply is helpful--





5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HannahXiong-MSFT avatar image
0 Votes"
HannahXiong-MSFT answered AlexMcFarland-2053 edited

Hi Alex,

Thank you so much for your feedback.

May I know the current situation of our issue? As you mentioned, the FSMO roles could not be transferred. How about seizing the roles to a healthy DC?

As per my understanding, the FSMO master roles will not replicate among the DCs. You are worried about transferring the FSMO roles to another DC as the replication is not working properly. But to transfer the FSMO roles or seize the FSMO roles will make sure that our whole AD environment will always work properly. About when to transfer or seize FSMO roles, we could refer to:
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

After transferring or seizing the FSMO roles, we could run the command "netdom query FSMO" on all the DCs to verify the DC holding FSMO roles.

We understand that we would like to fix the replication between the FSMO DC and the other 8 DCs, not to demote the FSMO DC. If so, since we preformed the restore from backup, we need to figure out whether the replication failure is caused by USN rollback. If it is caused by USN rollback, it is suggested that we would remove the DC from domain. For more information about this, we could refer to:
https://support.microsoft.com/en-us/help/875495/how-to-detect-and-recover-from-a-usn-rollback-in-windows-server-dc

If there is no USN rollback, we will go next and try to fix the replication. To fix the replication issue, we need to figure out all the replication errors first. To check about this, we could run the below commands:

repadmin /showrepl * /csv >C:\showrepl.csv (run the command on one of the DCs)
repadmin /showrepl >c:\showrepl.txt (run the commands on all the DCs)
repadmin /replsum >c:\replsum.txt (run the commands on all the DCs)

For any question, please feel free to contact us.


Best regards,
Hannah Xiong

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello,

I am checking how the issue is going, if you still have any questions, please feel free to contact us.

Thank you so much for your time and support.

Best regards,
Hannah Xiong

0 Votes 0 ·

Hi,

Still getting some replication errors. I created a new FSMO master server (lets call it FSMONew) from scratch with a Win 2019 iso. I seized the roles from the original FSMO master (lets call it FSMOOld) to this brand new server successfully.

Here is what you requested, I ran this on FSMONew:

repadmin /showrepl * /csv >C:\showrepl.csv:
17801-image.png

Be right back with the rest.


0 Votes 0 ·
image.png (669.3 KiB)
DSPatrick avatar image
0 Votes"
DSPatrick answered AlexMcFarland-2053 commented

Please run;

  • Dcdiag /v /c /d /e /s:%computername% >c:\dcdiag.log

  • repadmin /showrepl >C:\repl.txt

  • ipconfig /all > C:\dc1.txt

  • ipconfig /all > C:\dc2.txt

  • (etc. as other DC's exist)

then put unzipped text files up on OneDrive and share a link.




· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Trying to get this for you. The healthy DCs are failing their Advertising, KCC, DFSRevent, and Systemlog tests

This is what I was getting at, simply transferring the FSMO roles is not going to fix the underlying issues with this. If the old FSMO master FSMOOld can't replicate properly to begin with how is it going to replicate anything to FSMONew? The entire domain is now having this issue.

0 Votes 0 ·
DSPatrick avatar image
0 Votes"
DSPatrick answered AlexMcFarland-2053 commented

Ok, sounds like you have successfully seized roles
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

then perform cleanup.
https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup

after cleanup put up the files I requested.

  • Dcdiag /v /c /d /e /s:%computername% >c:\dcdiag.log

  • repadmin /showrepl >C:\repl.txt

  • ipconfig /all > C:\dc1.txt

  • ipconfig /all > C:\dc2.txt

  • (etc. as other DC's exist)

then put unzipped text files up on OneDrive and share a link.






· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I seized the roles successfully yes. I am really worried about performing metadata cleanup, it is basically deleting the entire DC from the domain. I kept the FSMOOld as a DC, we CAN gracefully demote this server to be a member server. Should I try that first I really do not want to just delete it out of AD. Right now it's just supporting AD DS, DNS and File services however it does carry a lot of certificate information as well.

0 Votes 0 ·
DSPatrick avatar image
0 Votes"
DSPatrick answered

Please put up the files I requested.

  • Dcdiag /v /c /d /e /s:%computername% >c:\dcdiag.log

  • repadmin /showrepl >C:\repl.txt

  • ipconfig /all > C:\dc1.txt

  • ipconfig /all > C:\dc2.txt

  • (etc. as other DC's exist)

then put unzipped text files up on OneDrive and share a link.








5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HannahXiong-MSFT avatar image
0 Votes"
HannahXiong-MSFT answered

Hello Alex,

Thank you so much for your reply.

May I know the current situation of our issue? Hope our issue could be resolved soon.

According to the screenshot of the AD replication, FSMOOld DC is UnHealthyDC1? Have we got any other error messages when checking the AD replication? From the provided screenshot of replication, the replication seems to work properly for other healthy DCs. But as per the UnhealthyDC1, may I know more information about this DC, such as dcdiag, repadmin /showrepl?

Besides, we also mentioned that the healthy DCs failed some tests, such as Advertising, KCC, DFSRevent and Systemlog. All the healthy DCs have these error messages? More information will be needed to judge these error messages.

As Dave mentioned, we could help to collect the requested files. Thanks so much for your time and support.


Best regards,
Hannah Xiong

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.