question

FotS avatar image
0 Votes"
FotS asked DSPatrick commented

Replace DC server (server 2019)

We were hit by a ransomware attack. Because the company hadn't invested any money into offsite backups, we only had onsite and, yeah, for all intents and purposes they are toast right now. The 2 DCs we had were fine, until someone powered on an infected machine I had powered down. Now both DC servers are at least partially encrypted. I don't see signs of the virus itself on them (yet), but I don't want to take any chances. I'm actually afraid to logout or even disconnect from DC01 for fear I won't be able to log back into it (I can't log into DC02 anymore due to the encryption).

I've powered down DC02 and using our virtual environment, spun up another server (DC03). I've added the same roles to it as what DC01 had (AD DS, DHCP, DNS, and NPAS), promoted DC03 as a DC in AD and started some work on DNS (on DC01 I've added DC03 as a Name Server), but when following guides for steps that continue to the new server, I've noticed all the zones are already visible on DC03, so I'm not sure what to do.

I really don't have the level of expertise to do all of this, but the company lacks the money to hire outside help. I've found some guides for complete replacements of DCs, but they're old (Server 2008 or older), and I'm not sure how much has changed over the versions (sorry, should mention that all of our servers are 2019 and I know I've already run across some guides saying that even going from server 2008 to 2016 require some additional steps like upgrading to 2012, first). Frankly, I don't even know what things I should be asking/looking out for. Until I was looking at some of these guides, I had never even heard of FSMO in relation to AD.

I need to get the AD, DHCP, DNS, and NPAS services transferred over to DC03, plus whatever else in there that I may not know about. NPAS I think(?) is being used for a RADIUS connection from the site's internet firewall for VPN. That said, there's some software installed on DC01 for the firewall, too, for the purposes of the web filters, so I think I'm going to engage the firewall support people to move the software and get VPN going through that instead. There's also Azure AD Connect software running on DC01, too (primarily used to sync our AD accounts with our Office 365 email accounts). That should be it.

Once all of that is transferred over, I want to shut DC01 down, then either rename and re-IP DC03 to match DC01, or spin up another DC server to match DC01's name and IP (would that be easier?). I know there have to be a lot of references and pointers to its name and IP, so I want to make sure at least something exists with it's identical info. If it makes it any easier, I think we have like less than a half dozen PCs that are still functional at this point, anyway. XD

What a nightmare this is turning into.... I need help. :(

windows-active-directorywindows-server-2019
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DSPatrick avatar image
1 Vote"
DSPatrick answered

For the additional domain controllers I'd use dcdiag / repadmin tools to verify health correcting all errors found before starting any operations. Then stand up the new 2019, patch it fully, license it, join existing domain, add active directory domain services, promote it also making it a GC (recommended), transfer FSMO roles over (optional), transfer pdc emulator role (optional), use dcdiag / repadmin tools to again verify health, when all is good you can decommission / demote old one.

For DHCP do you have a backup or starting over?

For the NPAS question I'd start a new thread over here.
https://docs.microsoft.com/en-us/answers/topics/windows-network-access-protection.html

Do not install the vpn on a domain controller. I'd stand up a separate instance for this purpose.


--please don't forget to upvote and Accept as answer if the reply is helpful--





5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FotS avatar image
0 Votes"
FotS answered FotS edited

@DSPatrick, it's not letting me post a comment to your answer for some reason, so I'm posting my comment as an "answer".

I appreciate the response, though I've been powering ahead this whole time (didn't have a choice). XD I did run the dcdiag commands, though, and all the errors being created were related to not being able to contact DC02 while I had it powered down. Powering it up seemed to correct everything. However, I had to manually remove it from the domain as I was unable to actually log into the server anymore.

DNS was actually easier than the guides I was finding led me to believe. The person who initially set all of this up apparently used the option to sync it with AD, so merely adding the server name into DC01's DNS settings was enough for it to replicate over to DC03.

I also got the FSMO roles moved over to DC03, though schema fought me some (I eventually got it).

DHCP I was able to pull a good copy off, however it's not working. I suspect an issue with the helper IP configs on the core network switches for that, though....

At this point, I've successfully fully removed DC02, then created a new DC02 to take its place and have it setup with AD DS, DHCP, and DNS, promoted as a DC. DC01 is now also fully removed from the domain and I'm in the process of setting up a new replacement for that. Hopefully once I get DHCP moved back over to that one, DHCP related services will start working again. XD

Oh, and GC seemed to take care of itself. Each server as I set it up with AD DS and promoted it as a DC automatically got that checked off.

I think I got lucky....

As for VPN/web filter, the firewall tech support people actually say their software works best directly from a DC server?

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DSPatrick avatar image
1 Vote"
DSPatrick answered FotS commented

all the errors being created were related to not being able to contact DC02

You'll need to perform some cleanup to remove the failed ones.
https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup
https://techcommunity.microsoft.com/t5/itops-talk-blog/step-by-step-manually-removing-a-domain-controller-server/ba-p/280564

As for VPN/web filter, the firewall tech support people actually say their software works best directly from a DC server?

Never install the vpn role on a domain controller. The multi-homing will always cause no end to grief for active directory DNS

--please don't forget to upvote and Accept as answer if the reply is helpful--









· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks, yeah, that second link you posted on metadata cleanup was actually one I found and followed. Didn't need to do step 3, though. It was already cleaned out from the first 2.

0 Votes 0 ·
DSPatrick avatar image
0 Votes"
DSPatrick answered DSPatrick edited

You're welcome, sounds good then.

--please don't forget to upvote and Accept as answer if the reply is helpful--



5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

FotS avatar image
0 Votes"
FotS answered FotS edited

Ok, I'm close.... I've got a new DC01 all up, same name and IP as the old, joined to domain and promoted as DC, even got DHCP working on it and confirmed the rest of the network now works on DHCP.

However, DNS seems desync'd between them. I had to add an A record for DC01 from DC02/DC03 in order for them to resolve it when attempting to add it as a Name Server, and I've noticed as I've begun demoting and removing DC03 from everything, that DC01's DNS records are not updating with the removal of DC03. I've apparently done gone and mucked something up. :/

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DSPatrick avatar image
0 Votes"
DSPatrick answered FotS commented

Please run;

Dcdiag /v /c /d /e /s:%computername% >C:\dcdiag.log
repadmin /showrepl >C:\repl.txt
ipconfig /all > C:\dc1.txt
ipconfig /all > C:\dc2.txt
ipconfig /all > C:\dc3.txt

then put unzipped text files up on OneDrive and share a link.



· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Do the first two need to be run from all 3, or just DC01?

0 Votes 0 ·

any healthy one


0 Votes 0 ·
DSPatrick avatar image
1 Vote"
DSPatrick answered FotS commented

Looks like replication is badly broken. Not sure of the steps history here but it may be some new domain controllers were added without confirming domain health further complicating issues.

One option may be to try a non authoritative sync on the broken ones. (check the event logs for details)
https://support.microsoft.com/en-us/help/2218556/how-to-force-an-authoritative-and-non-authoritative-synchronization-fo

For DC01 I'd check that it isn't somehow firewalled off, also check the SRV records are there
https://docs.microsoft.com/en-US/troubleshoot/windows-server/networking/verify-srv-dns-records-have-been-created



--please don't forget to upvote and Accept as answer if the reply is helpful--





· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Ok, thanks. I’ll give that a try at my next opportunity. For now, though, sleep. The DC servers got hit at 11 PM my time, so I ended up having to put off sleeping to fix it. I’ve been up over 30 hrs now. :P

0 Votes 0 ·
DSPatrick avatar image
0 Votes"
DSPatrick answered FotS commented

Sounds good.

--please don't forget to upvote and Accept as answer if the reply is helpful--



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Ok, things are looking much better. A new run of the dcdiag test you had me do shows all tests passing in the summary and says that the domain passes. Took running both of the sync methods listed on that page to get there.

As for SRV, first, how they have it listed in that doc doesn't appear to be how it's listed in my DNS? Where it says to look in "Forward Lookup Zones/Domain_Name/_msdcs" for both sets of subfolders and records, mine actually stops there. No subfolders, just 2 entries for Name Servers (not 3... DC03 is missing).

Where I had to go for the _kerberos and _ldap entries it wanted me to check was actually "Forward Lookup Zones/_msdcs.Domain_Name". In there I find the appropriate folder structure it mentions and all looks good for DC01. The interesting thing in there, though, is in the dc/_tcp folder, I see 2 sets of _kerberos and _ldap entries for DC03. (Edit: This looks like it explains the dup entries. Doesn't seem to be an issue, really. dns-registers-duplicate-srv-records-for-dc)

No idea if any of this is bad. I took screen shots and uploaded to the same OneDrive link provided earlier, along with the new dcdiag log from DC02.


0 Votes 0 ·

And cause I reached my character limit....

RE: missing DC03 Name Server entry - Oh, I didn't notice before that particular entry had it's own properties to add/remove Name Servers. The guides I had been following along with before on migrating DNS only ever had me check the top level zones for that, so I thought they'd be the only ones with them. Adding DC03 in there fixes that, though considering it has no sub entries (and the fact that I was planning on decoming DC03 anyway), not sure what difference it makes.

0 Votes 0 ·
DSPatrick avatar image
0 Votes"
DSPatrick answered DSPatrick commented

Glad to hear of success and that health and replication are now back to 100%.

--please don't forget to upvote and Accept as answer if the reply is helpful--



· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Just checking if there's any progress or updates?

--please don't forget to upvote and Accept as answer if the reply is helpful--



0 Votes 0 ·

Hi @DSPatrick, uh, it was working. Then I found two other servers being actively encrypted, found evidence on a third of the virus, then found a user reference to the third server on my brand new DC01... so I ended up shutting down and deleting all 4 servers. I properly removed DC01 from everything first, though, and the dcdiag only shows a warning in the summary due to the static IP DNS settings having a reference to DC01 that doesn't exist at the moment. Already in the process of replacing it, but I think I might actually have this, now (what a way to get in practice, huh?).

I guess we can consider this thread resolved, but I'm not sure how to proceed there. I don't feel as though there was any one post that properly fills as an answer. Should I just pick one regardless, or edit the title to "[RESOLVED]"?

0 Votes 0 ·

Should I just pick one regardless,

Sure, take your best shot






0 Votes 0 ·