Exchange 2019 MetaCacheDatabase MCDB

Question

I am having some problems with configuring the MetaCacheDatabase on Exchange 2019.
I have created a test environment with two Exchange 2019 CU9 virtual servers. There are 4 Databases, 2 Active on each server. I have one DAG.

I followed the official guide at https://learn.microsoft.com/en-us/exchange/high-availability/database-availability-groups/metacachedatabase-setup?view=exchserver-2019, however my experience is that it doesn't seem to be working properly.
I have some questions.

The Guide says that you must have Raw SSD drives. I left my SSD drives as Raw and the disks were automatically formatted with the mount points added. Some articles say that you must manually format the drives (Is this old information based on older MCDB Scripts)?

If I follow the guide the MCDB for the 2 active databases on Server A get created and the MCDB is Healthy for those (Not the passive copies).
The other two databases are on Server B. If I use the MCDB command to failover from Server A the databases on server B don't change to healthy

It is hard to describe what is happening but I have been working on this for weeks and just don't seem to be seeing any consistency. I have never seen all four Databases Healthy. I have seen both servers with the MCDB Disabled or Offline at the same time even after leaving overnight.
When I run the commands I don't see any errors.

Am I missing a vital piece of information from the guide that is maybe known elsewhere?

I ran the following commands in order:
ServerA
Manage-MCDB -DagName DAG -ConfigureMCDBPrerequisite -SSDSizeInBytes 21474836480 -SSDCountPerServer 1 -verbose
Manage-MCDB -DagName DAG -ServerAllowMCDB:$true -ServerName "ServerA"
Manage-MCDB -DagName DAG -ConfigureMCDBOnServer -ServerName "ServerA" -SSDSizeInBytes 21474836480 -verbose
Manage-MCDB -DagName DAG -ServerName "ServerA" -ForceFailover $true

ServerB
Manage-MCDB -DagName DAG -ServerName "ServerB" -ForceFailover $true

Is the above order of commands correct and do I need to perform any manual intervention after running the failover such as manually formatting the SSD again?

Sorry for the ramble

Regards,

Matt

Answer 1

Hi @MatthewRidleyDigitalandICTServices-8427

Please make sure that the operating Scope of your command is correct at each step, for example:

At Step 3: Run Manage-MCDB -ServerAllowMCDB

Scope:
Server: You need to run ServerAllowMCDB on each server in the DAG.

At Step 4: Run Manage-MCDB -ConfigureMCDBOnServer

Scope:
Server: You need to run ConfigureMCDBOnServer on each server in the DAG.

Also check the health and status information for the database copies.

Get-MailboxDatabaseCopyStatus | Fl name, *meta*  

SSD volumes need to be correctly formatted and mounted before configuring MCDB, and we don't need to manually format SSD again. Here is also an article which provided detailed steps for your reference: Microsoft Exchange 2019 Metacache Database setup

---

If an Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 2

Hi Joyce,

Thank you again for your comments and yes that link does explain why I might be seeing some differences in size.

So, from what you are saying once you configure ServerAllowMCDB:$true on the failover server the status will be healthy on both servers which is currently the case.

The article that you mentioned earlier in this thread (https://community.hpe.com/t5/Around-the-Storage-Block/Microsoft-Exchange-2019-Metacache-Database-setup/ba-p/7077841#.YVMv-iuSmUl) suggested that after entering MCDB-Manage -DagName dag -Servername SERVERNAME -ForceFailover:$true, the server that was failed over would have a status of disabled, which is what I was seeing but it seems that if I understand you correctly that should not be the case.

Would you (someone) be able to provide a list of the commands used and what you should see after each command?

Answer 3

Hi Joyce,

Sorry, I don;t think I explained myself properly

I have been using the Get-MailboxDatabaseCopyStatus | Fl name, meta command which is how I have been finding out the status. It would be nice to see more information though in a log somewhere.

I had a single Unformatted SSD disk in each of the two servers.

I performed the following on both servers (Note the commands only show ServerA but were also run against ServerB)
Manage-MCDB -DagName DAG -ConfigureMCDBPrerequisite -SSDSizeInBytes 21474836480 -SSDCountPerServer 1 -verbose
Manage-MCDB -DagName DAG -ServerAllowMCDB:$true -ServerName "ServerA"

Manage-MCDB -DagName DAG -ConfigureMCDBOnServer -ServerName "ServerA" -SSDSizeInBytes 21474836480 -verbose
Note the above command formatted the SSD and created a mount point for Exchange SSD5 linked to the disk.
You mention that the SSD Volumes need to be correctly formatted and mounted before configuring MCDB. Is this my problem that I am letting the above MCDB command do the work and really I should have manually done it before hand.

In the link you mentioned, the author does say that they had to manually format the SSD, however the official Microsoft guide just says that the SSD's need to be unformatted, so I am confused as to what I need to do here.

I then ran the following ONLY on ServerA
Manage-MCDB -DagName DAG -ServerName "ServerA" -ForceFailover $true

MCDB status changes to disabled on ServerA
MCDB databases were created on ServerB as healthy relating to the Active databases on ServerA. For the passive databases the status on Server B is Offline.
I then waited for a while

ServerB
Manage-MCDB -DagName DAG -ServerName "ServerB" -ForceFailover $true
MCDB status changes to Disabled on ServerB
I would now expect (maybe wrongly) that the MCDB status should change to Healthy for all databases on ServerA as this server now has all the active databases.
This just doesn't happen and I get left with both servers either in a disabled or offline status.

I think I will:
Run Manage-MCDB -DagName DAG -ServerAllowMCDB:$false -ServerName "ServerA/B" on both servers to break everything down.
delete the volume on the SSD and run clear disk to wipe the disk.
Manually format the disk and create the mount points
run through the commands again

I must be doing something wrong as there is not much out there on customers having problems.

Answer 4

So an update and I still don't know what's happening really.

Both servers were showing the MCDB cache as disabled and the c:\ExchangeMetaCacheDbs folder was not mounted into the SSD on either Server.

On Friday I did the following to break or tear down the MCDB:
Manage-MCDB -DagName MAILDAG -ServerAllowMCDB:$false -ServerName "ServerA"
Manage-MCDB -DagName MAILDAG -ServerAllowMCDB:$false -ServerName "ServerB"

I then did the following:

Deleted the Volume on the SSD disks on both servers
Manually created the folder C:\ExchangeMCDBVolumes\ExchangeSSD4
In Disk Management I have Initialized the disk as GPT but kept as Basic
Mount into C:\ExchangeMCDBVolumes\ExchangeSSD4 and formatted as ReFS
Quick Format

Ran the following

Manage-MCDB -DagName MAILDAG -ConfigureMCDBPrerequisite -SSDSizeInBytes 21474836480 -SSDCountPerServer 1 -verbose

Manage-MCDB -DagName MAILDAG -ConfigureMCDBOnServer -ServerName "ServerA" -SSDSizeInBytes 21474836480 -verbose

Manage-MCDB -DagName MAILDAG -ConfigureMCDBOnServer -ServerName "ServerB" -SSDSizeInBytes 21474836480 -verbose

Manage-MCDB -DagName MAILDAG -ServerAllowMCDB:$true -ServerName "ServerA" -verbose

I waited but nothing changed. Still all Disabled.

I revisited the servers today (Day after the changes above) and:
On ServerA:
c:\ExchangeMetaCacheDbs folders were mounted into the SSD on the server.
The MCDB status for DB2 and DB4 was healthy. The last rest was dated the Thursday before the Friday I made the changes above.
Note DB1 and DB3 were mounted on ServerA and DB2 and DB4 were mounted on ServerB.

The MCDB status for DB1 and DB3 was Offline.

On ServerB:
c:\ExchangeMetaCacheDbs folders were NOT mounted into the SSD on the server.
The MCDB status for all databases on ServerB was disabled.

I had to install CU10 security updates today so I installed first on ServerA and then rebooted it. All the Databases were then mounted on ServerB.
At this point everything was still the same as above regarding MCDB.

I then installed the CU10 security update on ServerB and rebooted the server. All the databases were then mounted on ServerA.
At this point all MCDB for all databases on ServerA are showing as Healthy.
All databases on ServerB are showing as Disabled.

I rebalanced the DAG 1 hour ago. so the Databases are now DB1 and DB3 on ServerA and DB2 and DB4 on ServerB.
MCDB is still the same with all on ServerA Healthy and all on ServerB Disabled.

I would have though that the DB's moved back to ServerB would have MCDB cache as Healthy.
Am I doing anything wrong here?
Is the MCDB based on the server rather than the Database?
Does anyone know of a video someone/Microsoft has made showing what should happen as that would be really useful.
Sorry for all the questions.

Answer 5

Hi Joyce,

A bit of an update.
As of yesterday everything in my note above was still true and nothing had changed.
Yesterday I ran Manage-MCDB -DagName MAILDAG -ServerAllowMCDB:$true -ServerName "ServerB" (as you had suggested today)

About an hour later MCDB databases had been configured for all databases for both servers.
The mount points were enabled on both servers
MCDB status was healthy on both servers

So a quick question at this point should I see the MCDB as healthy on both servers or should it say disabled on one of them?

Also I have noticed that the size of the MCDB databases seem strange.
ServerA:
DB1 - 253,888 KB
DB2 - 8,256 KB
DB3 - 8,256 KB
DB4 - 8,256 KB

ServerB:
DB1 - 131,104 KB
DB2 - 8,256 KB
DB3 - 8,288 KB
DB4 - 8,256 KB

Should DB1 and DB2 have the same MCDB database size?

It appears to be working but I must admit I am still not sure if I am doing it right.
I am hoping that when it comes to my live environment things work a bit easier like the in the guides (just wait a few minutes to see the changes etc)