question

MarcelSchneider-8631 avatar image
0 Votes"
MarcelSchneider-8631 asked KaelYao-MSFT commented

CU10 update problem

Hi there, 2 Exchange 2019 standard servers on server OS 2019, hostnames snmta20 and snmta21, update from CU9 to CU10 worked on snmta20. not on snmta21 despite may trials. Both are dual stack, IPv6 tested and working. Below a short summary of the findings:
Always error at end of step 12/6 Mailbox role: Mailbox service
SNMTA21 IS IN MAINTENANCE MODE, snmta20 is in normal mode
• Error message from CU10 update process: "System.InvalidOperationException: Failed to mount database "MBX-DB-02". Error: An Active Manager operation failed. Error: Operation failed with message: Error 0x5 (Access is denied) from cli_MountDatabase3 [Server: SNMTA20.SNNET.CH]
• Database redundancy: SNMTA21: The RPC to retrieve the status of active database copy 'MBX-DB-01\SNMTA20' failed. Error: A server-side administrative operation has failed. Operation failed with message: Error 0x5 (Access is denied) from RpccGetCopyStatusEx4 [Server: SNMTA20.SNNET.CH]
• All 3 MBX-DB’s on SNMTA20 as seen from SNMTA21 have status SERVICEDOWN and thus cannot be accessed. From SNMTA20 all 3 MBX-DB’s look good, 3 mounted (SNMTA20), 3 healthy (SNMTA21)
• Event 2280 IIS-W3SVC-WP: The Module DLL C:\Exchange2019\V15\FrontEnd\HttpProxy\bin\exppw.dll failed to load
• Event 5139 WAS: A listener channel for protocol 'http' in worker process '23044' serving application pool 'DefaultAppPool' reported a listener channel failure.
• Service “Microsoft Exchange Search Host Controller” is running
• All server components of SNMTA20 are active, all on SNMTA21 are inactive
• All server components of SNMTA21 cannot be set into Active state, possibly due to maintenance mode.

Assumed a permission problem and have therefore added Organization Management with Full Control to all three MBX-DB directories and MBX EDB files. Also gave Full Control to Authenticated Users. Did not help. Thank you for any suggestions, Marcel

office-exchange-server-deployment
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi folks, this CU9 to 10 update is not working. It always crashes with same error, see my original mail. It is not possible to stop or disable "MS Exchange Replication". Thus I'm stuck with this CU10 update in maintenance mode because the machine in maintenance wants to perform replication on the other and crashes with Error 0x5 (Access is denied) when it tries to mount database "MBX-DB-02. Seems to me a faulty update since the machine running with CU10 exhibits every 15-30 minutes a couple the same event 4999 MSExchangeCommon Watson reports that user have experienced in 2006 and could only be resolved in a new CU. It did that exactly from the time the DNS RR's were changed from the stuck machine to the one already working with CU10.

The only solution to the stuck machine I see is completely remove Exchange 2019 and start again with CU9 which was installed and worked before. This takes up a lot of work. Do you have other, better suggestions? If anybody needs further information to these 2 problems it will readily be supplied. Frustrated, Marcel

0 Votes 0 ·

Hi,

My apologies for the late reply.

From your information, it seems that SNMTA21 may get corrupted.

The only solution to the stuck machine I see is completely remove Exchange 2019 and start again with CU9 which was installed and worked before. This takes up a lot of work.

I totally understand your frustration.
However, since troubleshooting all these issues may also take a lot of effort, I suppose that restoring the server would be the best solution in this case.

In the meantime have completely removed DAG and tried with stopping or disabling "MS Exchange Replication". No success with all of this.

Did you successfully remove DAG?
If DAG has been removed, please first try uninstalling Exchange from SNMTA21.
If Exchange can be uninstalled successfully, please reinstall the operating system and install CU10 directly.
0 Votes 0 ·
KaelYao-MSFT avatar image
0 Votes"
KaelYao-MSFT answered MarcelSchneider-8631 commented

Hi Marcel.

Thanks for the detailed information.

Here are a few more questions:
1.Are you using the same account on both servers?
2.Will it return permission errors when running other EMS commands on SNMTA21?

If other EMS commands can be run successfully,
since all databases are currently active on SNMTA20, if possible, please remove the passive copies on SNMTA21 and remove SNMTA21 from DAG.
And retry the upgrade and see if you can upgrade successfully.


If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HI

.Are you using the same account on both servers?

Yes.

2.Will it return permission errors when running other EMS commands on SNMTA21?

On the machine in maintenance, the EMC returned errors but now HTTP Error 404, nothing found but this machine is now only in IPv4 mode, will return it to duals stack now.

remove the passive copies on SNMTA21 and remove SNMTA21 from DAG

Right, thank you, will do that and let you know.

Have sent another similar question today because couldn't find the first, please ignore the second. Best, Marcel
0 Votes 0 ·
MarcelSchneider-8631 avatar image
0 Votes"
MarcelSchneider-8631 answered MarcelSchneider-8631 commented

Hi Kael Yao,

MTA21 is now again in dual stack. Error when connecting Exchange Management Shell to SNMTA21:
VERBOSE: Connecting to SNMTA21.xxxx.
New-PSSession : [snmta21.xxxx] Connecting to remote server snmta21.xxxx failed with the following error
message : The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from
the destination computer. The content type is absent or invalid. For more information, see the
about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ConnectionURI "$connectionUri" -ConfigurationName Micr ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
gTransportException
+ FullyQualifiedErrorId : -2144108297,PSSessionOpenFailed

And when trying to open EMC these errors occur:

Server Error in '/owa' Application.


Configuration Error
Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

Parser Error Message: Could not load file or assembly 'Microsoft.Exchange.Clients.Strings, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

Source Error:

Line 67: the compiler. All assemblies in the GAC and owa\bin are referenced automatically.
Line 68: -->
Line 69: <add assembly="Microsoft.Exchange.Clients.Strings, Version=15.0.0.0, Culture=neutral, publicKeyToken=31bf3856ad364e35" />
Line 70: <add assembly="Microsoft.Exchange.Data.Directory, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
Line 71: <add assembly="Microsoft.Exchange.Clients.Common, Version=15.0.0.0,Culture=neutral, publicKeyToken=31bf3856ad364e35" />

Source File: C:\Exchange2019\V15\FrontEnd\HttpProxy\owa\web.config Line: 69

Assembly Load Trace: The following information can be helpful to determine why the assembly 'Microsoft.Exchange.Clients.Strings, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' could not be loaded.

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

Best, Marcel

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Kael Yao,

In the meantime have completely removed DAG and tried with stopping or disabling "MS Exchange Replication". No success with all of this. The CU switches replication on and the error persists. The machine in maintenance wants to do something on the other, unclear what and why. It is mandatory to disable this behavior.

Best, Marcel

0 Votes 0 ·
MarcelSchneider-8631 avatar image
0 Votes"
MarcelSchneider-8631 answered KaelYao-MSFT commented

Hi Kael Yao,

Why have my last answers disappeared. E.g. the last you have answered? You must have seen it. Also cannot see your answer but try to comment it from e-mail:

Did you successfully remove DAG?

Yes. The other machine is now the only MTA, all necessary DNS entries have been changed. But - as mentioned - it writes now every 15 minutes event 4999 MSExchangecommon i

Watson report about to be sent for process id: 8820, with parameters: E12, c-RTL-AMD64, 15.02.0922.005, MSExchangeHMWorker, M.Exchange.Diagnostics,
.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, PerformActualLog, M.E.Diagnostics.BlockedDeserializeTypeException, 1193763487, 15.02.0922.005.
ErrorReportingEnabled: True
exData=|exHResult=|exStacktrace=at Microsoft.Exchange.Diagnostics.DeserializationTypeLogger.PerformActualLog(Dictionary`2 hits)
at Microsoft.Exchange.Diagnostics.DeserializationTypeLogger.ProcessCachedHits()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.TimerQueueTimer.CallCallback()
at System.Threading.TimerQueueTimer.Fire()
at System.Threading.TimerQueue.FireNextTimers()|exTargetSite=Microsoft.Exchange.Diagnostics, Version=15.0.0.0, Culture=neutral,
PublicKeyToken=31bf3856ad364e35 PerformActualLog|exSource=|exMessage=|exComplete=

please reinstall the operating system and install CU10 directly

Why uninstall OS?

Regards. Marcel





· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

Why have my last answers disappeared. E.g. the last you have answered?

Due to the policy of Microsoft Q&A forum, it is recommended to use comment instead of answer when you would like to add information to the post.
Unless the text reaches the limitation of comment (1,600 characters).
Reference: [When to use answers or comments][1]

So I have concerted some of your answers to comments.
You may see your last answer under the main post.
![113708-11.png][2]
Sorry for the confusion caused.

Why uninstall OS?

Reinstalling OS would make the OS clean to avoid some unpredictable issues.
Usually installing on a new hardware would be more recommended.

[1]: https://docs.microsoft.com/en-us/answers/support/answers-vs-comments
[2]: /answers/storage/attachments/113708-11.png
0 Votes 0 ·
11.png (62.8 KiB)

Hi @MarcelSchneider-8631

I am writing here to confirm with you how thing going now?
Did the issue get resolved?

0 Votes 0 ·
MarcelSchneider-8631 avatar image
0 Votes"
MarcelSchneider-8631 answered KaelYao-MSFT commented

Hi Kael Yao,
Nothing has been resolved. The current situation: 3 MTA's (with get-clientaccessservice) 19, 20, 21. 21 is the one used as MX 10 and where autodiscovers are pointing to. 21 exhibited the "Failed to mount database "MBX-DB-02". Error: An Active Manager operation failed. Error: Operation failed with message: Error 0x5 (Access is denied) from cli_MountDatabase3 [Server: SNMTA20.SNNET.CH]" errors. It worked for 4.5 years, only with CU10 these error occurred and I plan to implement your most drastic solution: To completely rebuild it. No. 20 has taken over the role of 21 and 19 is a new Exchange 2019 license, built on a previously spare OS 2019 machine.

No. 19 never worked as expected. Firing up EAC shows error 500, it looks for a non-present resource. I login with Admin to 19 and assume it looks for an admin mailbox on its newly created database. But cannot add admin to this MBX because there is already one in the 3 legacy MBX'es. Have tried to create a new DAG with 19 and 20. No luck, 19 never became active. And since these efforts, get tons of 3002 events, failed MsExchange BackEndRehydration. They complain about protocols mapi/emsmdb, mapi/nspi, Autodiscover and Microsoft-Server-ActiveSync. Never seen such events. My assumption is that probably NT Authority/System service account rights are missing for ms-Exch-EPI-Token-Serialization. Do not understand what it wants to "hydrate".

The plan is now to completely decommission 21 and rebuild it. Then enclose it in a DAC with 20 and change DNS RR's to 21 as before. After that probably have to decommission and build 19 again. Exchange can be a mess and it is now one here.
Regards, Marcel

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi,

I noticed you may have asked a similar question on TechNet before:
Exchange 2019 CU3 Mailbox role Mailbox service error

I suppose the issues may have existed since that time.

Did you manage to install CU3 on 21 successfully?

0 Votes 0 ·
MarcelSchneider-8631 avatar image
0 Votes"
MarcelSchneider-8631 answered KaelYao-MSFT commented

Hi, indeed, the issue occurred sometimes with CU's but could always be resolved by just rebooting no. 20.

Did you manage to install CU3 on 21 successfully?

All CU's up to 9 with all variants were installed on 21.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the information.

I wonder if it will be a issue caused by AD replication.
Are 20 and 21 in different AD site?

0 Votes 0 ·
MarcelSchneider-8631 avatar image
0 Votes"
MarcelSchneider-8631 answered KaelYao-MSFT commented

Hi,

Are 20 and 21 in different AD site?

No, the same forest and domain.
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Marcel.

How are things going now?
Have you started the rebuilding plan?

0 Votes 0 ·
MarcelSchneider-8631 avatar image
1 Vote"
MarcelSchneider-8631 answered KaelYao-MSFT commented

Hi Kael Yao,

How are things going now?

Despite your bad advice: All 3 MTA's are running. The culprit was not no. 21, an entry in a domain controller was missing. Thus, the complete rebuild of no. 21 was a major waste of time. Because you appear as an MS-employee, I trusted you. That was a big mistake and it will never ever happen again. And, btw., the question regarding rehydration you did not even bother to answer.

Regards, Marcel


· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Marcel.

My sincere apologies for not being able to help you with this issue.
I haven't seen this error before and was not able to reproduce it in my lab.

Anyway, glad to hear that you found the root cause.
Thanks for your update.

0 Votes 0 ·