Try to backup some flat files on a volume to the cloud using MS DPM. Other cloud backups are working fine. However, This one and a couple of others are struggling. Looking thru the logs I'm seeing this, as shown below. What is interesting to me is the log talks about ErrorCode:CloudAsyncWorkLimitReached is there some I can do on my end to adjust this or is this a MS thing? It looks like it hits this cloudAsyncWorklimitReached deal then does a retry, looping over and over. I've run a consistency check on the Volume. and also moved the volume to a different protection group. Same error. Does anybody have an Idea?
17D0 2518 02/26 18:31:37.698 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: Executing retry policy | Params: {CurrentAttempt = 5}{MaxAttempts = 125}{BackOffIntervalInSec = 18}{OperationCode = WcfProxyAsyncCall}{Retry policy id = 8}
17D0 0CEC 02/26 18:31:39.011 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:41.027 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:43.043 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:45.059 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:47.074 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:31:49.090 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 2518 02/26 18:31:50.751 79 WcfClient.cs(960) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.FileCatalog.Interface.IFileCatalogExternalChannel}{Description = UpdateROArgs}{TargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}
17D0 2518 02/26 18:31:50.751 69 CBEngineWcfClientHelper.cs(396) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL Setting RequestId header for outgoing request | Params: {RequestId = 9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{TaskId = 41f5d7db-7b3e-4564-9c04-522ae6448e4d}{AgentVersion = 2.0.9197.0}
17D0 0CEC 02/26 18:31:51.094 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ).}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 2/26/2021 6:31:50 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 2/26/2021 6:30:56 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 87b20db0-7e29-49d4-95dd-9953fffee9ba}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_9fb0ab40-b186-47d2-90c2-7dfe0e9af1d4_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:31:52.860 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
17D0 0CEC 02/26 18:32:13.196 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:32:15.212 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:32:17.215 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 0CEC 02/26 18:32:19.231 71 dscontext.cpp(184) [000002009E62B990] 41F5D7DB-7B3E-4564-9C04-522AE6448E4D ACTIVITY Last completed state for Ds Id (1989240489) is 5
17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING Unable to make web service call | Params: {Exception: = System.ServiceModel.FaultException`1[Microsoft.Internal.CloudBackup.Common.FailureModeling.CloudServiceFault]: FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING (Fault Detail is equal to ErrorCode = CloudAsyncWorkLimitReached, DetailedErrorCode = 0, DetailedErrorSource = None/None, Message = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.861 79 WcfClient.cs(998) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ).}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyCall}{OperationStartTime = 2/26/2021 6:32:09 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 5bbe5f65-f6a6-4570-a5de-f12605d09947}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_b2161111-b2e7-457d-8eee-f30a7aa6939c_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: No retry policy specified | Params: {ErrorCode = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(946) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D NORMAL FMBlock: Mapping source error to target error | Params: {OperationCode = WcfProxyCall}{SourceError = CloudAsyncWorkLimitReached}{TargetError = CloudAsyncWorkLimitReached}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING FMBlock: FM exception caught | Params: {OperationCode = WcfProxyAsyncCall}{OperationStartTime = 2/26/2021 6:30:56 PM}{FMExceptionDetail = FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:Service/None/None, Message:FMException: [ErrorCode:CloudAsyncWorkLimitReached, DetailedCode:0, Source:None, Message:]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING ]
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING {InitialTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{Name = BasicHttpBinding_IFileCatalogExternal}{WorkitemId = 5bbe5f65-f6a6-4570-a5de-f12605d09947}{CurrentTargetEndpoint = https://pod01-fc1.cus.backup.windowsazure.com/FileCatalogExternalService.svc}{RequestId = 0000000002_b2161111-b2e7-457d-8eee-f30a7aa6939c_41f5d7db-7b3e-4564-9c04-522ae6448e4d}{IdentityType = Container}{Identity = 50aebc141d99421416176dd13c4715a0f709e7d2}
17D0 2518 02/26 18:32:19.876 79 WcfClient.cs(694) 41F5D7DB-7B3E-4564-9C04-522AE6448E4D WARNING }
