question

FV-6580 avatar image
1 Vote"
FV-6580 asked BahaMasoud-0939 answered

Microsoft.Azure.ServiceBus.ServiceBusTimeoutException issues

Hi Q&A team.
I have problem with Azure Service Bus integration and hope you can help me.
I send messages to ASB and regularly receive timeout exceptions with strange timeout value 00:00:00. I use exponetial retry on client and connection with retry starting from 1 second. I'm not sure where does ASB take this value and I can't understand if it is infrastructure regular problem or I should implement additional retry on my code side. I can't find any tips in documentation about this retry behaviour.
I use Microsoft.Azure.ServiceBus 5.1.2 package

So the question is should I fix my timeout and retry usage or everything is just an infrastructure issue?

Exception stacktrace:

 Microsoft.Azure.ServiceBus.ServiceBusTimeoutException: The operation did not complete within the allocated time 00:00:00 for object sender14579. ---> System.AggregateException: One or more errors occurred. ---> System.TimeoutException: The operation did not complete within the allocated time **00:00:00** for object sender14579.
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
    at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
    at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
    at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
 --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Microsoft.Azure.ServiceBus.Amqp.AmqpLinkCreator.<CreateAndOpenAmqpLinkAsync>d__11.MoveNext()
    --- End of inner exception stack trace ---
    --- End of inner exception stack trace ---
    at Microsoft.Azure.ServiceBus.Core.MessageSender.<OnSendAsync>d__58.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Microsoft.Azure.ServiceBus.RetryPolicy.<RunOperation>d__19.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at Microsoft.Azure.ServiceBus.RetryPolicy.<RunOperation>d__19.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at Microsoft.Azure.ServiceBus.Core.MessageSender.<SendAsync>d__45.MoveNext()
 --- End of stack trace from previous location where exception was thrown ---
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    at CCWolverine.Messaging.ServiceBusAccessor.SendMessage[T](MessageToSend`1 messageToSend)
    at CCWolverine.Model.Product.ProductNameUpdater.SendProductChangedMessages(IEntityWriter writer) in D:\a\1\s\CCWolverine.Model\Model\Product\ProductNameUpdater.cs:line 400

Code used to sent message:

  // https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#service-bus
         private readonly RetryPolicy _exponentialRetry = new RetryExponential(
             TimeSpan.FromSeconds(1),
             TimeSpan.FromSeconds(30),
             TimeSpan.FromSeconds(1.5),
             3);
    
         /// <inheritdoc />
         public MessageSender GetMessageSender(string topicName)
         {
             if (string.IsNullOrWhiteSpace(topicName))
                 throw new ArgumentException("Value cannot be null or whitespace.", nameof(topicName));
    
             EnsureConnectionIsAlive();
    
             var messageSender = new MessageSender(_connection, topicName, _exponentialRetry);
    
             return messageSender;
         }
    
         /// <summary>
         ///     We reinitialize connection in case of connection failure
         ///     Use of Double-Check-Locking to optimize
         /// </summary>
         private void EnsureConnectionIsAlive()
         {
             if (_connection == null || _connection.IsClosedOrClosing)
             {
                 lock (Locker)
                 {
                     if (_connection == null || _connection.IsClosedOrClosing)
                     {
                         var connection = new ServiceBusConnection(_serviceBus.Settings.Url)
                         {
                             RetryPolicy = _exponentialRetry
                         };
    
                         _connection = connection;
                     }
                 }
             }
         }
    
         /// <inheritdoc />
         public bool SendMessage<T>(MessageToSend<T> messageToSend) where T : WolverineMessage
         {
             if (messageToSend == null)
                 throw new ArgumentNullException(nameof(messageToSend));
    
             var topicClient = ClientFactory.GetMessageSender(messageToSend.To);
    
             var message = messageToSend.Data.ToMessage().WithLabel(_label);
    
             topicClient.SendAsync(message)
                 .ConfigureAwait(false)
                 .GetAwaiter()
                 .GetResult();
    
             topicClient.CloseAsync()
                 .ConfigureAwait(false)
                 .GetAwaiter()
                 .GetResult();
    
             return true;
         }



dotnet-standardazure-service-bus
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @FV-6580, Is this the first time you are experiencing this? If not, how frequent this happens? have you set any timeout value? Could there be any firewalls blocking the connection? I have also requested for couple of more information via private session.

0 Votes 0 ·
FV-6580 avatar image
0 Votes"
FV-6580 answered

Hi @JayaC-MSFT,
No, it's not the first time I'm experiencing this issue, it starts from the beginning of our ASB usage (maybe two or three weeks). It usually happens periodically, for example everything is fine all day but from 7 to 9 am we have dozens of timeout exceptions but I think it's related to MQ load.
I didn't set any timeout value because in Microsoft.Azure.ServiceBus 5.1.2 package timeouts on MessageSender and ServiceBusConnection marked as obsolete and it's recommended to use RetryPolicy.
I used the same RetryPolicy for connection and for sender from https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#service-bus.

 private readonly RetryPolicy _exponentialRetry = new RetryExponential(             TimeSpan.FromSeconds(1),             TimeSpan.FromSeconds(30),             TimeSpan.FromSeconds(1.5),             3);    

I don't think that there are any firewalls, because most messages are delivered without problems. I've sent you more information into a private channel.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ThiagoRizzo-4619 avatar image
0 Votes"
ThiagoRizzo-4619 answered FV-6580 commented

@FV-6580 Same problem here, have you solved it yet ?

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ThiagoRizzo-4619, no we've still met this problem. After investigation on Microsoft side they said that it's completely client's problem because they haven't received these requests with timeout on their infrastructure side. I just used our own retry policy over their retry policy. Smells but works =)

 // microsoft https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific#service-bus
 private readonly RetryPolicy _exponentialRetry = new RetryExponential(
             TimeSpan.FromSeconds(1),
             TimeSpan.FromSeconds(30),
             TimeSpan.FromSeconds(1.5),
             3);
    
 //our own
 private static readonly InternalRetryPolicy InternalRetryPolicy = InternalRetryPolicy.ExponentialRetry(TimeSpan.FromSeconds(2), 3);
    
 var topicClient = new MessageSender(_connection, topicName, _exponentialRetry);
    
 InternalRetryPolicy.Do(() => topicClient.SendAsync(message).WaitAndUnwrapException());




0 Votes 0 ·
BahaMasoud-0939 avatar image
0 Votes"
BahaMasoud-0939 answered

same problem here! it happens once in a blue moon, even if we do not have any messages send or received from the date we started the app till this happen.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.