question

VenkateshK-3729 avatar image
0 Votes"
VenkateshK-3729 asked ·

Timeout exceptions in StackExchange.Redis for Azure Redis Cache

Hi,
We have recently implemented Azure Redis Cache with .Net Framework 4.7.2. It was working fine. Suddenly we are keep getting the timeout exceptions in StackExchange.Redis for Azure Redis Cache.

Exception:
Timeout performing EVAL (5000ms), next: EVAL, inst: 22, qu: 0, qs: 18, aw: False, rs: ReadAsync, ws: Idle, in: 131072, serverEndpoint: Unspecified/KreatoCRMCache.redis.cache.windows.net:6380, mgr: 10 of 10 available, clientName: NextGenITServices_IN_0, IOCP: (Busy=0,Free=1000,Min=4,Max=1000), WORKER: (Busy=20,Free=8171,Min=4,Max=8191), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Stack Trace:
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\ConnectionMultiplexer.cs:line 2250 at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisBase.cs:line 54 at StackExchange.Redis.RedisDatabase.ScriptEvaluate(String script, RedisKey[] keys, RedisValue[] values, CommandFlags flags) in C:\projects\stackexchange-redis\src\StackExchange.Redis\RedisDatabase.cs:line 1134 at Microsoft.Web.Redis.StackExchangeClientConnection.<>c_DisplayClass7_0.b0() in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 68 at Microsoft.Web.Redis.StackExchangeClientConnection.OperationExecutor(Func`1 redisOperation) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 95 at Microsoft.Web.Redis.StackExchangeClientConnection.RetryLogic(Func`1 redisOperation) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 118 at Microsoft.Web.Redis.StackExchangeClientConnection.Eval(String script, String[] keyArgs, Object[] valueArgs) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\Shared\StackExchangeClientConnection.cs:line 68 at Microsoft.Web.Redis.RedisConnectionWrapper.TryTakeWriteLockAndGetData(DateTime lockTime, Int32 lockTimeout, Object& lockId, ISessionStateItemCollection& data, Int32& sessionTimeout) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\RedisSessionStateProvider\RedisConnectionWrapper.cs:line 184 at Microsoft.Web.Redis.RedisSessionStateProvider.GetItemFromSessionStore(Boolean isWriteLockRequired, HttpContextBase context, String id, CancellationToken cancellationToken, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\RedisSessionStateProvider\RedisSessionStateProvider.cs:line 291 at Microsoft.Web.Redis.RedisSessionStateProvider.d20.MoveNext() in C:\TeamCity\buildAgent\work\59b31e8e4035fd30\src\RedisSessionStateProvider\RedisSessionStateProvider.cs:line 206 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNet.SessionState.SessionStateModuleAsync.d74.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNet.SessionState.SessionStateModuleAsync.d_65.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.AspNet.SessionState.TaskAsyncHelper.EndTask(IAsyncResult ar) at Microsoft.AspNet.SessionState.SessionStateModuleAsync.EndAcquireState(IAsyncResult result) at System.Web.HttpApplication.AsyncEventExecutionStep.InvokeEndHandler(IAsyncResult ar) at System.Web.HttpApplication.AsyncEventExecutionStep.OnAsyncEventCompletion(IAsyncResult ar)

Plan: Basic 1 GB (C1)

Web Configuration:


azure-cache-redis
· 2
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @VenkateshK-3729 which region is your Redis deployed to and when did your timeouts start to occur? We can also work more closely with you by sending us the name of your Redis instance to AzCommunity[at]microsoft[dot]com ATTN Ryan.

0 Votes 0 · ·

Hi Ryan, I've sent a mail about Redis cache details to AzCommunity[at]microsoft[dot]com.

Redis Intance Name: KreatoCRMCache.redis.cache.windows.net
Location: Southeast Asia (Both Redis and Web Instance)

0 Votes 0 · ·
xequence avatar image
0 Votes"
xequence answered ·
· 1 · Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi, I've investigated the details using above link which you shared. We have implemented with correct manner only.
Cache Name: KreatoCRMCache.redis.cache.windows.net:6380
Cache Size: Basic 1 GB(c1)
Public Virtual IP (VIP) Address of client deployments: 23.97.51.212
Version of StackExchange.Redis: 2.0.601, Microsoft.Web.RedisSessionStateProvider: 4.0.1
Cache service and clients Region: Southeast Asia
Number of client instances: only 1 ( our clients are very less. max 100 clients)
CPU utilization on our client both before and during the incident: Its max: 25-30% only
Client for sync timeout and for connection timeout: connectionTimeoutInMilliseconds="5000", operationTimeoutInMilliseconds="1000" & retryTimeoutInMilliseconds="3000"

0 Votes 0 · ·
ryanchill avatar image
0 Votes"
ryanchill answered ·

@VenkateshK-3729, take a look at https://gist.github.com/JonCole/e65411214030f0d823cb#file-threadpool-md which actually refers to the link @xequence posted. The timeout you're encountering might be related to threadpool throttling. file-threadpool.md has a gist that recommends some best practices so that you can do to avoid hitting throttling issues.

Let me know if this helps.

· 4 · Share
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Shall we increase the "minIoThreads" 4 to 50 & "minWorkerThreads" 4 to 50? Ours is 4 core machine.

0 Votes 0 · ·
ryanchill avatar image ryanchill VenkateshK-3729 ·

That seems like a good start @VenkateshK-3729. I would make sure to do the changes incrementally and check for improvements.

0 Votes 0 · ·
ryanchill avatar image ryanchill VenkateshK-3729 ·

@VenkateshK-3729, wanted to follow up to see if you were successful in increasing thread count settings.

0 Votes 0 · ·

Hi @ryanchill , we are making the process by next week only. Now we are in vocation. Lets implement this and will check for times then will let you know by this month end(Jan20). Don't close the ticket, please keeps its open till my updates.

1 Vote 1 · ·