question

AdamStaub-1626 avatar image
4 Votes"
AdamStaub-1626 asked jham6509-5400 commented

Having problem backing up a file disk using DPM to the cloud.

Try to backup some flat files on a volume to the cloud using MS DPM. Other cloud backups are working fine. However, This one and a couple of others are struggling. Looking thru the logs I'm seeing this, as shown below. What is interesting to me is the log talks about ErrorCode:CloudAsyncWorkLimitReached is there some I can do on my end to adjust this or is this a MS thing? It looks like it hits this cloudAsyncWorklimitReached deal then does a retry, looping over and over. I've run a consistency check on the Volume. and also moved the volume to a different protection group. Same error. Does anybody have an Idea?


 17D0 2518 02/26 18:31:37.698 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: Executing retry policy | Params: {CurrentAttempt = 5}{MaxAttempts = 125}{BackOffIntervalInSec = 18}{OperationCode = WcfProxyAsyncCall}{Retry policy id = 8}
 17D0 0CEC 02/26 18:31:39.011 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:31:41.027 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:31:43.043 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:31:45.059 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:31:47.074 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:31:49.090 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 2518 02/26 18:31:50.751 79 WcfClient.cs(960) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.FileCatalog.Interface.IFileCatalogExternalChannel}{Description = UpdateROArgs}{TargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}
 17D0 2518 02/26 18:31:50.751 69 CBEngineWcfClientHelper.cs(396) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL Setting RequestId header for outgoing request | Params: {RequestId = 9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{TaskId = 41f5d7db-7b3e-4564-9c04-522ae6448e4d}{AgentVersion = 2.0.9197.0}
 17D0 0CEC 02/26 18:31:51.094 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING Unable to make web service call | Params: {Exception:  = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING  (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ).}
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 2/26/2021 6:31:50 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 2/26/2021 6:30:56 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
 17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
    
 17D0 0CEC 02/26 18:32:13.196 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:32:15.212 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:32:17.215 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 0CEC 02/26 18:32:19.231 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
 17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING Unable to make web service call | Params: {Exception:  = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING  (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ).}
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 2/26/2021 6:32:09 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 5bbe5f65-f6a6-4570-a5de-f12605d09947}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_b2161111-b2e7-457d-8eee-f30a7aa6939c_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 2/26/2021 6:30:56 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 5bbe5f65-f6a6-4570-a5de-f12605d09947}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_b2161111-b2e7-457d-8eee-f30a7aa6939c_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
 17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
azure-backup
· 55
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello All,

Sorry for the inconvenience this must have caused!

This is a known issue and has been mitigated. The fix has been deployed in all the regions. The issue should get resolved by EOD.

Appreciate your patience in this matter.

Thanks!

2 Votes 2 ·

confirmed UKSouth is now functioning normally

0 Votes 0 ·

@SadiqhAhmed-MSFT:
Still getting the following error in Azure region "west europe":
"WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkSubmitted, DetailedCode:0, Source:None, Message:]

MARS agent version is 2.0.9207.0

Any hint?

0 Votes 0 ·

Getting the same error with a number of recovery points on my MABS server. The three recovery points are all file-based.
72618-image.png



I'm also in a different region
https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc

1 Vote 1 ·
image.png (6.6 KiB)

Where did you get the Azure backup agent version 2.0.9205.0 from and is that the latest?

The latest I'm seeing is 9.0.9200.0 from here https://docs.microsoft.com/en-us/azure/backup/install-mars-agent#download-the-mars-agent.

0 Votes 0 ·

Guys I spoke with the Azure Backup Product manager today. As others say there is a fix being pushed which should hit all Regions by Sunday Night. East coast is scheduled for 10pm March 6th. Nothing can be done but wait for this patch to apply.

1 Vote 1 ·

Let's hope for the best! Our customers get worried because no file based online backup could be finished for a while because of these error.

0 Votes 0 ·

Hello.

Please format your post so we can read it rather than have everything together.

0 Votes 0 ·

I've now had this reported it is an Azure issue and MS are looking into it. Not sure why they haven't updated the status page though

0 Votes 0 ·

James:
How did you find out that this is a MS deal?

0 Votes 0 ·

@AdamStaub-1626 James reached out to Azure technical support team and the issue is being investigated.

0 Votes 0 ·
Show more comments
Show more comments

1 Answer

JamesWilliams-0236 avatar image
0 Votes"
JamesWilliams-0236 answered JamesWilliams-0236 published
 1E2C    1C9C    02/27    15:57:55.451    79    WcfClient.cs(830)        E35FFF34-DF88-4333-A731-0305000562DF        NORMAL    GetStatus Arguments | Params: {lasttimeClientReceivedNewHeartBeat = 27/02/2021 15:57:38}{lastNewHeatbeat = 27/02/2021 15:57:38}{statusPollTime = 27/02/2021 15:57:55}
 1E2C    1C9C    02/27    15:57:55.451    79    WcfClient.cs(960)        E35FFF34-DF88-4333-A731-0305000562DF        NORMAL    Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Protection.Interface.IProtectionServiceChannel}{Description = GetStatus}{TargetEndpoint = https://pod01-prot1.ukw.backup.windowsazure.com/ProtectionService.svc}
 1E2C    1C9C    02/27    15:57:55.451    69    CBEngineWcfClientHelper.cs(396)        E35FFF34-DF88-4333-A731-0305000562DF        NORMAL    Setting RequestId header for outgoing request | Params: {RequestId = 3f1440e2-6d52-4fc0-9915-ce50cae9550c}{WorkitemId = acb2a9ea-e8b5-4368-8bf9-2f85f0b67ae2}{TaskId = e35fff34-df88-4333-a731-0305000562df}{AgentVersion = 2.0.9205.0}
 1E2C    1C9C    02/27    15:57:55.498    79    WcfClient.cs(981)        E35FFF34-DF88-4333-A731-0305000562DF        NORMAL    Finished web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Protection.Interface.IProtectionServiceChannel}{Description = GetStatus}{TargetEndpoint = https://pod01-prot1.ukw.backup.windowsazure.com/ProtectionService.svc}
 1E2C    1C9C    02/27    15:57:55.514    79    WcfClient.cs(863)        E35FFF34-DF88-4333-A731-0305000562DF        NORMAL    GetStatus Response and Updated variables | Params: {status.LastHearbeatTime = 27/02/2021 15:57:38}{lasttimeClientReceivedNewHeartBeat = 27/02/2021 15:57:38}{lastNewHeatbeat = 27/02/2021 15:57:38}{statusPollTime = 27/02/2021 15:57:55}
 1E2C    1C9C    02/27    15:57:55.514    79    WcfClient.cs(815)        E35FFF34-DF88-4333-A731-0305000562DF        WARNING    FMBlock: Executing retry policy | Params: {CurrentAttempt = 99}{MaxAttempts = 28800}{BackOffIntervalInSec = 18}{OperationCode = WcfProxyGenericAsyncOperationPollingCall}{Retry policy id = 7}
 1E2C    1AD0    02/27    15:57:55.936    79    WcfClient.cs(960)        BD4069C5-779E-4554-82D3-2465A632F306        NORMAL    Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.FileCatalog.Interface.IFileCatalogExternalChannel}{Description = UpdateROArgs}{TargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}
 1E2C    1AD0    02/27    15:57:55.936    69    CBEngineWcfClientHelper.cs(396)        BD4069C5-779E-4554-82D3-2465A632F306        NORMAL    Setting RequestId header for outgoing request | Params: {RequestId = 88c8b94f-c433-4a08-a117-19f42f49a631}{WorkitemId = ad5657c0-67c8-472c-9734-63b05d306859}{TaskId = bd4069c5-779e-4554-82d3-2465a632f306}{AgentVersion = 2.0.9205.0}
 1E2C    099C    02/27    15:57:55.967    71    dscontext.cpp(184)    [000000071BF512D0]    E63383DC-C408-4E3E-A11E-B51218C91F28        ACTIVITY    Last completed state for Ds Id (35184717886556) is 5
 1E2C    19B0    02/27    15:57:57.061    79    WcfClient.cs(960)        E63383DC-C408-4E3E-A11E-B51218C91F28        NORMAL    Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.FileCatalog.Interface.IFileCatalogExternalChannel}{Description = UpdateROArgs}{TargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}
 1E2C    19B0    02/27    15:57:57.061    69    CBEngineWcfClientHelper.cs(396)        E63383DC-C408-4E3E-A11E-B51218C91F28        NORMAL    Setting RequestId header for outgoing request | Params: {RequestId = f5dfafe2-ae69-4339-8f0b-bec26a9262d5}{WorkitemId = d89a37cf-a3f0-4f15-aa3d-20fbc473f3a1}{TaskId = e63383dc-c408-4e3e-a11e-b51218c91f28}{AgentVersion = 2.0.9205.0}
 1E2C    1BB8    02/27    15:57:57.123    71    dscontext.cpp(184)    [0000000722B8BB40]    BD4069C5-779E-4554-82D3-2465A632F306        ACTIVITY    Last completed state for Ds Id (35184652646204) is 5
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(998)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    Unable to make web service call | Params: {Exception:  = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(998)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING     (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(998)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    ).}
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(946)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 27/02/2021 15:57:55}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(946)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    ]
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(946)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    {InitialTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = ad5657c0-67c8-472c-9734-63b05d306859}{CurrentTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_88c8b94f-c433-4a08-a117-19f42f49a631_bd4069c5-779e-4554-82d3-2465a632f306}{IdentityType = Container}{Identity = 2629e21830edda7b6e054ae781e7c3441d09d00d}
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(946)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    }
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(946)        BD4069C5-779E-4554-82D3-2465A632F306        NORMAL    FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(946)        BD4069C5-779E-4554-82D3-2465A632F306        NORMAL    FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(694)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 27/02/2021 15:53:10}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(694)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    ]
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(694)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    {InitialTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = ad5657c0-67c8-472c-9734-63b05d306859}{CurrentTargetEndpoint = https://pod01-fc1.ukw.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_88c8b94f-c433-4a08-a117-19f42f49a631_bd4069c5-779e-4554-82d3-2465a632f306}{IdentityType = Container}{Identity = 2629e21830edda7b6e054ae781e7c3441d09d00d}
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(694)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    }
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(694)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    FMBlock: Current operationTime timeout reached or will reach after backoff time. | Params: {OperationCode = WcfProxyAsyncCall}{OperationTimeoutTime = 27/02/2021 15:58:10}{BackOffTimeInSec = 18}{CurrentAttempt = 16}{MaxAttempts = 125}
 1E2C    1AD0    02/27    15:57:57.311    79    WcfClient.cs(694)        BD4069C5-779E-4554-82D3-2465A632F306        NORMAL    FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyAsyncCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudTimeoutError}
 1E2C    1AD0    02/27    15:57:57.311    69    CalExternalServiceProxy.cs(2732)        BD4069C5-779E-4554-82D3-2465A632F306        WARNING    FMBlock: FM exception caught | Params: {OperationCode = UpdateROArgs}{OperationStartTime = 27/02/2021 15:53:1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.