question

anuthereaper avatar image
0 Votes"
anuthereaper asked anuthereaper answered

Azure Function Consumption autoscale not working as expected

I was attempting to execute a Jmeter API load test on a simple Azure function API which fetches data from an Azure SQL DB and returns the response in JSON format. I hosted the function on a consumption plan with a max instance count of 200. The SQL DB was set to 10DTU (just for testing sake).
I set Jmeter to have 100 threads (users), 60s ramp up period with a duration of 5mins. I also used a csv file to have variable query parameters across the API calls.
I’ve attached a snapshot of the application insights. My observations and questions are below :
1) I noticed was the CPU usage going up to 100% and dropping down again. This kept happening for the entire 5 minutes.
2) The number of instances did not go beyond 1. I was expecting this to go up after an initial warm up period.
3) Out of 37000 samples, there was around 2% of failures in the JMeter tests with the same error message “Connection timed out”.
Any idea why the instance count is not scaling up? Why is the CPU usage dropping periodically and what is the reason for the time out errors? Am I missing some additional configuration setup?
93061-image.png


 #r "Newtonsoft.Json"
 #r "System.Configuration"
 using System;
 using System.Data;
 using System.Web;
 //using System.Net;
 using Newtonsoft.Json;
 using Dapper;
 using System.Data.SqlClient;
 using Microsoft.Extensions.Configuration;
 using System.Configuration;
 using Microsoft.AspNetCore.Mvc;
 using Microsoft.Extensions.Primitives;
 public static async Task<object> Run(HttpRequest req, ILogger log)
 {
 string responseMessage;
 //We retrieve the id field, which comes as a parameter to the function, by deserializing req.Content.
 //string jsonContent = await req.Content.ReadAsStringAsync();
 //dynamic data = JsonConvert.DeserializeObject(jsonContent);
 //Uri myUri = new Uri("http://www.example.com?param1=good&param2=bad");
 string payer = req.Query["payer"];
    
 //log.LogInformation(payer);
    
 //If there is no username, we return the error message.xx
 if (payer == null) {
 responseMessage = "ERROR!!!Please pass a Payer";
 return new OkObjectResult(responseMessage);
 }
    
 //log.LogInformation("test");
 var connectionString = Environment.GetEnvironmentVariable("SqlConnection", EnvironmentVariableTarget.Process);
 //Azure SQLDB Logx
 var logAdded = true;
 try
 {
 //We get the Connection String in the Function App Settings section we xdefined.
 using(SqlConnection connection = new SqlConnection(connectionString))
 {
 //Opens Azure SQL DB connection.
 connection.Open();
 string qs = $"SELECT TOP 10 * FROM [dbo].[MYtable] where [Payer] = '{payer}'";
    
 SqlCommand command = new SqlCommand(qs, connection);
 string queryop = "";
 using (SqlDataReader reader = command.ExecuteReader())
 {
 queryop = sqlDatoToJson(reader);
 }
 responseMessage = (queryop);
 connection.Close();
 }
 }
 catch(Exception e)
 {
 logAdded = false;
 log.LogError(e.ToString());
 responseMessage = e.ToString();
 // connection.Close();
 }
 return new OkObjectResult(responseMessage);
 }
 static String sqlDatoToJson(SqlDataReader dataReader)
 // transform the returned data to JSON
 {
 var dataTable = new DataTable();
 dataTable.Load(dataReader);
 string JSONString = string.Empty;
 JSONString = JsonConvert.SerializeObject(dataTable);
 return JSONString;
 }


Function.proj file

 <Project Sdk="Microsoft.NET.Sdk">
     <PropertyGroup>
         <TargetFramework>netstandard2.0</TargetFramework>
     </PropertyGroup>
     <ItemGroup>
         <PackageReference Include="Dapper" Version="2.0.78" />
         <PackageReference Include="Microsoft.Data.SqlClient" Version="2.1.2" />
         <PackageReference Include="Microsoft.WindowsAzure.ConfigurationManager" Version="3.2.3" />
     </ItemGroup>
 </Project>

Host.json file

 {
   "version": "2.0",
   "extensions": {
     "http": {
       "routePrefix": "api",
       "maxOutstandingRequests": 200,
       "maxConcurrentRequests": 100
     }
   },
   "extensionBundle": {
     "id": "Microsoft.Azure.Functions.ExtensionBundle",
     "version": "[1.*, 2.0.0)"
   }
 }






azure-functions
image.png (19.2 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

anuthereaper avatar image
0 Votes"
anuthereaper answered

Edit : I think this is because I was scaling up the load too soon 0 to 100 in 60s. I tried the same setup but scaled from 0-100 in 5 mins and I found new servers were automatically provisioned within a minute. I guess the scale controller needs some amount of time to gauge the load and whether to scale up or not.
1) The CPU was dropping because of how Jmeter was setup. It was set to scale from 0-100 in 60s and then loop through this 5 times. So it keeps going from 0-100 and then repeat.
2) Keeping a gradual scale up helps the scale controller to scale up properly. The number of servers provisions was 4 which was able to match the load.
3) There was still error rates of around 1-2% and all of them were time outs. We still need to investigate this and if it can be avoided.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.