question

LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 asked ZhengqiLou-MSFT commented

exchange database have automatic switch over

the event log as below

A request to mount Active or Passive database was processed in 0.0214939 seconds.

Flags: 0x00000008
Error code: NoError (0x00000000)

Information Store - (47140,R,0,15.01.1779.002) The database engine has begun replaying logfile E:\Program Files\Microsoft\Exchange Server\V15\Mailbox\database\E04.log.


Previous Log Processing Stats:
[1] 395.168726 -2.929166 (1584) CM +J(CM:1584, PgRf:329433, Rd:3696/31, Dy:3782/520119, Lg:139787869/556120) +M(C:1341976K, Fs:237443, WS:363016K # 348824K, PF:1381028K # 1367372K, P:1381028K).


Then run a log to copy to E drive
“Information Store - MBX - Senior01 (47140,R,0,15.01.1779.002) MBX - Senior01: The database engine has begun replaying logfile E:\Program Files\Microsoft\Exchange Server\V15. “

how to avoid this status , because when failover to other database , around 1 hr the database will auto switch to original database and mounted back

is it the right not good or other possible reason

it is too trouble because the status happen the user outlook will no response and hang

[1]: /answers/storage/attachments/65207-image.png

office-exchange-server-administration
image.png (48.4 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered ZhengqiLou-MSFT commented

Get-Cluster | Get-ClusterNode | Select Name,DynamicWeight,NodeWeight,Id,State

Name : a
DynamicWeight : 1
NodeWeight : 1
Id : 1
State : Up

Name : b
DynamicWeight : 1
NodeWeight : 1
Id : 2
State : Up

Name : c
DynamicWeight : 1
NodeWeight : 1
Id : 3
State : Up


status is up , do you i can test remove one of exchange and re-add to DAG group

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @LouisRuth-3980 ,

I deleted the above post to make it looks clearer.
Sorry, do you mean you want to test this? And what do you want to know or others?
Based on my test, if you remove one of the servers, the witness server would be re-enabled.
It's not like you stopped a server(Stop-ClusterNode) but not remove it. In this case, the witness server will still be disabled.

Could you tell me our original issue: exchange database have automatic switch over, does is fixed? If so you could close this thread by accepting one answer and post a new thread for other questions. Basically we solve one issue in one thread. Thanks for understanding. Also please feel free to goon if this issue is still there, I would follow this until it's resolved:)

Regards,
Lou

0 Votes 0 ·

exchange database have automatic switch over
automatic switch over is still not solved , now i power off one server to avoid exchange failover to other exchange , so i have study have re-build DAG and cluster to solve the problem


you remove one of the servers, the witness server would be re-enabled.

83034-image.png



do you i remove one of server and add back , will make the witness enable ?

0 Votes 0 ·
image.png (8.5 KiB)

Hi @LouisRuth-3980 ,

Oh sorry for that, truly I mean.

As I know and my tests, we could see the following situations:
1. Two servers are in the DAG, then the witness server could be enabled and you could use (Get-Cluster).witnessDynamicWeight to check.
2. Three servers are in the DAG, the witness server is disabled whatever you do like stopping one or two servers with Stop-ClusterNode ServerName.
I may guess even servers could be the same with scenario 1 and odd with scenario 2.

Best regards,
Lou


0 Votes 0 ·
LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered LouisRuth-3980 commented

event 1146 The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically associated with cluster health detection and recovery of a resource. Refer to the System event log to determine which resource and resource DLL is causing the issue.

event id 1177
The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk.
Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

have make cluster node two host restart last night ?

do you advise to create a new cluster for solve this issue?



event id 1135
Cluster node 'exg01' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

· 9
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @LouisRuth-3980 ,

Sorry for the delay of this reply.
Event 1135: this says the exchange server exg01 was removed from a DAG.
Event 1177: this may caused by serval reasons, Event ID - 1177.
Event 1146: Event ID 1146 — Cluster Service Startup, seems like the DLL files caused this event.

Did you just removed a server from the DAG? I think you can ignore these events for now and remove all members then delete the DAG. After that you can recreate the DAG and add members to test if the automatically switch stops.

Regards,
Lou

0 Votes 0 ·

but this event will make the exchange server auto reboot?


DAG and cluster recreate need require estimate down time and its change will affect user?
it is most concern point.

0 Votes 0 ·

Hi,

I think this event was created after the server was rebooted or was rebooting.

No, if you just remove & recreate a DAG or remove a member of this DAG, it won't affect users.

Regards,
Lou

0 Votes 0 ·
Show more comments
LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered ZhengqiLou-MSFT commented

Scenario: GetKillBit. Failed to download killbit list from OMEX server. Exception: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at Microsoft.Exchange.Data.ApplicationLogic.Extension.BaseAsyncOmexCommand.<>c_DisplayClass18_0.<EndGetResponseCallback>b_0()


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

have any advise

0 Votes 0 ·

Hi,

Add these values to Registry may fix this error:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001

Here is a related thread: https://social.technet.microsoft.com/Forums/zh-CN/d6e295cb-e6f1-4136-af76-ebbc51859968/receiving-quotsomething-went-wrongquot-and-quotthe-app-couldnt-be-downloadedquot-messages?forum=Exch2016GD

Regards,
Lou

0 Votes 0 ·
LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered ZhengqiLou-MSFT commented

The performance counter '\\MSExchange Assistants - Per Database(msexchangemailboxassistants XX-XX )\Event Dispatchers Catching Up' sustained a value of '35,659.00', for the '30' minute(s) interval starting at '2/22/2021 1:26:00 AM'. Threshold breached since '2/17/2021 11:26 AM'. None Trigger Name:EventDispatchersCatchupQueueTrigger. Instance:msexchangemailboxassistants XX-XX

every half hour have pop up this error

how to solve

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @LouisRuth-3980 ,

Sorry for the delay, I had a leave for the previous days.

According to my research, this error "The performance counter" seems like a cosmetic error and all the resolutions are to stop it by
1. Open this file in notepad: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Microsoft.Exchange.Diagnostics.Service.exe.config
2. Change "ExchangeJobs.Triggers.EventDispatchersCatchupQueueTrigger" from "True" to "False".

re-create have how long down time

Won't be long if you don't have so many member servers, but you will have to remove the database copies first.
70810-image.png
Exchange server won't stop when removing the DAG, so you don't have to worry that.

0 Votes 0 ·
image.png (33.4 KiB)

is it need to same setup to my existing 3 exchange servers ?

0 Votes 0 ·

Hi,
Do you mean clean the cosmetic error messages? You can modify it where the error log occurs.

Regards,
Lou

0 Votes 0 ·
Show more comments
LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered

The DAG is a cluster, a node of a cluster is like a member server of a DAG.

I think, if you could, you can remove the member servers then remove the DAG and recreate one.

re-create have how long down time

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ZhengqiLou-MSFT avatar image
0 Votes"
ZhengqiLou-MSFT answered LouisRuth-3980 commented

Hi @LouisRuth-3980 ,

The DAG is a cluster, a node of a cluster is like a member server of a DAG.

I think, if you could, you can remove the member servers then remove the DAG and recreate one.

Best regards,
Lou


If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

The performance counter '\\MSExchange Assistants - Per Database(msexchangemailboxassistants XX-XX )\Event Dispatchers Catching Up' sustained a value of '35,659.00', for the '30' minute(s) interval starting at '2/22/2021 1:26:00 AM'. Threshold breached since '2/17/2021 11:26 AM'. None Trigger Name:EventDispatchersCatchupQueueTrigger. Instance:msexchangemailboxassistants XX-XX

every half hour have pop up this error

how to solve

0 Votes 0 ·
LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered

show other Exchange B mounted default on Exchange A database
Exchange A database has a most preferred copy

but event log show you before trigger database mounted to exchange B

Number of these copies show you
is 1 copy

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered LouisRuth-3980 commented

66733-image.png



have active preference ,
setup log is which folder

i only found Program Files\Microsoft\Exchange Server\V15\Mailbox , only database folder


image.png (16.7 KiB)
· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

It should be in <ExchangeInstallDrive>:\ExchangeSetupLogs\DagTask, by default it's C.
It's not like the database logs, it only record the operation on DAG.

but always show other Exchange B mounted default on Exchange A database

I'm not pretty understand this, do you mean the Exchange A database has a most preferred copy with Exchange B database?
In this case, i think you should check about the Number of these copies as i replied above.
And i will go on researching and follow this issue.

Regards,
Lou

If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

0 Votes 0 ·

Hi

i face other issue on exchange , the exchange B auto start at 4 days ago
the event log is
Cluster node 'EXG01A' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges

then EXG01B have restart , have any advise on this

0 Votes 0 ·

Hi @LouisRuth-3980 ,

Thanks for providing these information.
I found an article about this event, it could be helpful:
Troubleshooting cluster issue with Event ID 1135
Seems like it may because of several reasons like cluster or network settings.

Also, if you could, please try to remove the DAG and recreate one to test if the DAG has issues.

Bests,
Lou


If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

0 Votes 0 ·
Show more comments
LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered ZhengqiLou-MSFT commented

Get-MailboxDatabaseCopyStatus *
66374-image.png
have no failed

but always show other Exchange B mounted default on Exchange A database

after 1 hr , Database willauto change back Exchange A mounted , it's too confused

during switch database , user outlook will hang and email will show disconnect with exchange
66289-image.png


image.png (74 B)
image.png (26.4 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for sharing these information!

Please check the Database copies in EAC and check the Activation preference number of the default Copy(It should be 1).

66401-image.png
66402-image.png
Have you moved these log files to E drive? Can you find the ExchangeSetupLog folder in E ?

I will do a research about it and also test in my lab. And if there are any settings configuration of Exchange server before this issue happened?

Thanks a lot and have a nice day.

0 Votes 0 ·
image.png (24.3 KiB)
image.png (19.9 KiB)
LouisRuth-3980 avatar image
0 Votes"
LouisRuth-3980 answered ZhengqiLou-MSFT commented

the copy of database 'XX' on this server appears to be experiencing performance issues, possibly as a result of storage failure. Consult the event log on the server for other storage and "ExchangeStoreDb" events for more specific information about the failure. Recovery was not attempted.

also happen database auto switch

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

Do you mean after running the command the database still switch automatically in 1 hr? Try restarting the MS Exchange Replication service and check.

If this still happens, can you check the ExchangeStoreDB events as the error message said and share them?

Also run this command to check if there are any failed database copies in your server:

 Get-MailboxDatabaseCopyStatus * 

Regards,
Lou


If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

0 Votes 0 ·