Mailbox Migration from Exchange 2010 Fails

Question

Dear All,

We have Environment as below,

Exchange 2010 (MBX) --- Exchange 2010 (Hub/CAS) --- Exchange 2016 (Hybrid) ---- Exchange online ,

I am able to migrate mailbox from exchange 2016 to Exchange online
However when we try to migrate mailbox from Exchange 2010 if fails with below message .

FailureCode : -2146233088
FailureType : TooManyTransientFailureRetriesPermanentException
FailureSide : Source
Message : Error: The job encountered too many transient failures (61) and is quitting. The most common failure is CommunicationErrorTransientException/ServerTooBusyException/WebException with the hit
count 60. --> The call to 'https://mail<domainname>/EWS/mrsproxy.svc' failed. Error details: The HTTP service located at https://mail.<domainname>/EWS/mrsproxy.svc is unavailable. This could be because
the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote
server returned an error: (503) Server Unavailable.. --> The HTTP service located at https://mail.<Domain name>/EWS/mrsproxy.svc is unavailable. This could be because the service is too busy or
because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote server returned an error:
(503) Server Unavailable.
FailureTimestamp : 10/20/2021 7:20:57 PM

Then there is retries as below

'AUXP273MB0751.AREP273.PROD.OUTLOOK.COM' (15.20.4628.16 ServerCaps:3FFFFFFF,
ProxyCaps:3FFFFFFFFFFFC7DD2DFDBF5FFFFFCB07EFFF, MailboxCaps:, legacyCaps:3FFFFFFF) is examining the request.
10/20/2021 3:04:59 PM [AUXP273MB0751] Content from the Shard mailbox (Mailbox Guid: fbae3fb3-7e1c-4de5-b975-dde5daf91c89, Database: f177eee8-1c87-4db3-b255-427c835658b6) will be merged into
the target mailbox.
10/20/2021 3:04:59 PM [AUXP273MB0751] Connected to target mailbox 'efiuae.onmicrosoft.com\3b4ac44f-ba4f-4ce0-acf6-68ee9c67c061 (Primary)', database 'AREP273DG046-db013', Mailbox server
'AUXP273MB0751.AREP273.PROD.OUTLOOK.COM' Version 15.20 (Build 4628.0).
10/20/2021 3:05:40 PM [AUXP273MB0751] Transient error CommunicationErrorTransientException has occurred. The system will retry (48/60, 48/600).
10/20/2021 3:06:13 PM [AUXP273MB0751] The Microsoft Exchange Mailbox Replication service 'AUXP273MB0751.AREP273.PROD.OUTLOOK.COM' (15.20.4628.16 ServerCaps:3FFFFFFF,
ProxyCaps:3FFFFFFFFFFFC7DD2DFDBF5FFFFFCB07EFFF, MailboxCaps:, legacyCaps:3FFFFFFF) is examining the request.
10/20/2021 3:06:13 PM [AUXP273MB0751] Content from the Shard mailbox (Mailbox Guid: fbae3fb3-7e1c-4de5-b975-dde5daf91c89, Database: f177eee8-1c87-4db3-b255-427c835658b6) will be merged into

Finally

LastFailure : <MiniFailureRec Time="2021-10-20T15:20:57.5517918Z" Type="TooManyTransientFailureRetriesPermanentException/CommunicationErrorTransientException/ServerTooBusyException/WebException" Side="1"
Fatal="true" ETypes="633 10 12 1 14 653" ExStackHash="86ce" />
AnalysisResults :
ConfigCache : Microsoft.M365.Core.ConfigurationSettings.ConfigCache

We have tested the below,

1- ensure there is no firewall between exchange 2010 and exchange 2016
2- we can migrate mailbox from exchange 2010 to exchange 2016
3- checked the iis logs on exchange 2016 , and we can see the exchange 2016 sending 503 back to Exchange online

What we are not able to find is which process from Exchange 2010 is sending the 503 back to exchange 2016.

Need Help.

regards - Hasan Reza.

Answer 1

Dear All,

we have sorted this complex issue , posting the complete solution with approach for someone with similar problem,

Extracted the error from the mailbox migration fail problem - too many transient error : 503

Step 1 : Increase the time out
E:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\exchweb\ews
<!-- Mailbox Replication Proxy Service configuration -->
<MRSProxyConfiguration
DataImportTimeout="00:10:00" />

Step -2 Look for failure message in W3SVC1 Log
C:\inetpub\logs\LogFiles\W3SVC1
2021-10-21 18:22:59 ::1 POST /powershell serializationLevel=Full;ExchClientVer=15.1.2308.8;clientApplication=ManagementShell;TargetServer=;PSVersion=5.1.14393.4583&sessionID=Version_15.1_(Build_2307.8)=rJqNiZqNgaq+urunvbq5trK9p8/O0ZGX0Z6agc7Gy83PyczMycuBzc/NztLOz9LNzqvOx8XNxsXKxg==&CorrelationID=<empty>;&cafeReqId=c724d85f-feec-4ee8-8a09-a284646f77e2; 80 domain\efio365migadm ::1 Microsoft+WinRM+Client - 500 0 0 180016
2021-10-21 18:23:05 172.16.4.210 POST /EWS/mrsproxy.svc &CorrelationID=<empty>;&cafeReqId=3ad7afe0-cfed-4b05-9e00-3082df06d358; 443 domain\efio365migadm 52.98.95.21 - - 503 0 0 27
2021-10-21 18:23:15 172.16.4.210 POST /EWS/mrsproxy.svc &CorrelationID=<empty>;&cafeReqId=3200d140-3bd4-4cae-a785-fbec1b6e7555; 443 domain\efio365migadm 52.98.95.21 - - 503 0 0 28
2021-10-21 18:23:46 172.16.4.210 POST /EWS/mrsproxy.svc &CorrelationID=<empty>;&cafeReqId=e7f64c8e-ee0e-4690-a799-18feebfb560b; 443 domain\efio365migadm 52.98.95.21 - - 503 0 0 29
2021-10-21 18:23:56 172.16.4.210 POST /EWS/mrsproxy.svc &CorrelationID=<empty>;&cafeReqId=f1023422-053b-4912-9b87-e22603db7360; 443 domain\efio365migadm 52.98.95.21 - - 503 0 0 27
2021-10-21 18:24:06 172.16.4.210 POST /EWS/mrsproxy.svc &CorrelationID=<empty>;&cafeReqId=6e3e0858-be44-4389-992e-c485e4bbc108; 443 domain\efio365migadm 52.98.95.21 - - 200 0 0 47
2021-10-21 18:

Step3 -Track the same error in below using the cafeid
E:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews
2021-10-21T18:23:46.142Z,e7f64c8e-ee0e-4690-a799-18feebfb560b,15,1,2308,8,,Ews,mail.<domain>.ae,/EWS/mrsproxy.svc,,Negotiate,true,domain\efio365migadm,,DatabaseGuid~595027c7-37e9-4ada-b723-e4c166e1617d,,52.98.95.21,UAEDXBEFIMBX01,503,NameResolutionFailure,NameResolutionFailure,POST,Proxy,uaeauhpenmbx01.<domain>.ae,15.01.1034.000,IntraForest,TargetDatabase-GUID,,UAEAUHNHGMBX01,0,651,,,,0,10,0;,0,,0,4;2;2;,8,8,,0,12,0,,,,,,,,0,0,12,0,,12,,12,12,,,,BeginRequest=2021-10-21T18:23:46.129Z;CorrelationID=<empty>;ProxyState-Run=None;ServerLocatorCall=DM:595027c7-37e9-4ada-b723-e4c166e1617d~~;ClientAccessServer=UAEAUHPENMBX01.<domain>.ae;ResolveCasLatency=0;FEAuth=BEVersion-1942062090;RoutingEntry=DatabaseGuid:595027c7-37e9-4ada-b723-e4c166e1617d%40<domain>.ae%40<domain>.ae Server:UAEAUHNHGMBX01.<domain>.ae+1937997947@0;BeginGetRequestStream=2021-10-21T18:23:46.141Z;OnRequestStreamReady=2021-10-21T18:23:46.142Z;InvalidatingBackEndServerCache=595027c7-37e9-4ada-b723-e4c166e1617d;ProxyState-Complete=ProxyRequestData;SharedCacheGuard=0;EndRequest=2021-10-21T18:23:46.142Z;S:ServiceCommonMetadata.Cookie=01e67b7474fe403c8f6b7b60b5aff825;I32:ATE.C[UAEDXBNHGDOC01.<domain>.ae]=3;F:ATE.AL[UAEDXBNHGDOC01.<domain>.ae]=0;I32:ADS.C[UAEDXBNHGDOC01]=1;F:ADS.AL[UAEDXBNHGDOC01]=3.0098;I32:ADR.C[UAEDXBNHGDOC01]=2;F:ADR.AL[UAEDXBNHGDOC01]=1.98075,WebExceptionStatus=NameResolutionFailure;WebException=System.Net.WebException: The remote name could not be resolved: 'uaeauhpenmbx01.<domain>.ae' at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult TransportContext& context) at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult) at Microsoft.Exchange.HttpProxy.ProxyRequestHandler.<>c__DisplayClass198_0.<OnRequestStreamReady>b__0();SharedCache=MailboxServerCacheEntryRemovalFailure;SharedCache=AMCacheRemovalFailure;,,|RoutingDB:595027c7-37e9-4ada-b723-e4c166e1617d,,,CafeV1
2021-10-21T18:23:56.140Z,f1023422-053b-4912-9b87-

Find the CAS Access Servers in the same site and remove it out of service
Set-ClientAccessServer -Identity UAEAUHPENMBX01 -IsOutOfService $true

In summary if was going to Exchange cas server that did not exist .

I hope some one else would find it useful,

regards - Hasan Reza

Answer 2

According the error message, it seems that there are some issue with the mailbox data. The whole settings is correct. So the solution is increasing the value of bad items or repair the mailbox before moving.

If the issue persist, since there are only two mailboxes have this issue, use PST migration is also a good option.

Answer 3

Hi @Hasan Reza

According to the error: TooManyTransientFailureRetriesPermanentException, we could consider increasing the value of baditems. Details: Set-MoveRequest

The BadItemLimit parameter specifies the maximum number of bad items that are allowed before the request fails. A bad item is a corrupt item in the source mailbox that can't be copied to the target mailbox. Also included in the bad item limit are missing items. Missing items are items in the source mailbox that can't be found in the target mailbox when the request is ready to complete.

Valid input for this parameter is an integer or the value unlimited. The default value is 0, which means the request will fail if any bad items are detected. If you are OK with leaving a few bad items behind, you can set this parameter to a reasonable value (we recommend 10 or lower) so the request can proceed. If too many bad items are detected, consider using the New-MailboxRepairRequest cmdlet to attempt to fix corrupted items in the source mailbox, and try the request again.

In Exchange 2010, if you set this value to 51 or higher, you also need to use the AcceptLargeDataLoss switch. Otherwise, the command will fail.

Set-MoveRequest -BadItemLimit 500 -LargeItemLimit 200 -AcceptLargeDataLoss  

Here is also a related thread: hybrid mailbox migration
Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

You could also move the mailboxes to a different database, by doing so exchange itself do the error check and clean up on the mailbox which has solve the problem and succeed migrating the mailboxes.

---

If an Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.