Process asynchronous tasks as they complete (C#)

By using Task.WhenAny, you can start multiple tasks at the same time and process them one by one as they're completed rather than process them in the order in which they're started.

The following example uses a query to create a collection of tasks. Each task downloads the contents of a specified website. In each iteration of a while loop, an awaited call to WhenAny returns the task in the collection of tasks that finishes its download first. That task is removed from the collection and processed. The loop repeats until the collection contains no more tasks.

Create example application

Create a new .NET Core console application. You can create one by using the dotnet new console command or from Visual Studio. Open the Program.cs file in your favorite code editor.

Replace using statements

Replace the existing using statements with these declarations:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;

Add fields

In the Program class definition, add the following two fields:

static readonly HttpClient s_client = new HttpClient
{
    MaxResponseContentBufferSize = 1_000_000
};

static readonly IEnumerable<string> s_urlList = new string[]
{
    "https://docs.microsoft.com",
    "https://docs.microsoft.com/aspnet/core",
    "https://docs.microsoft.com/azure",
    "https://docs.microsoft.com/azure/devops",
    "https://docs.microsoft.com/dotnet",
    "https://docs.microsoft.com/dynamics365",
    "https://docs.microsoft.com/education",
    "https://docs.microsoft.com/enterprise-mobility-security",
    "https://docs.microsoft.com/gaming",
    "https://docs.microsoft.com/graph",
    "https://docs.microsoft.com/microsoft-365",
    "https://docs.microsoft.com/office",
    "https://docs.microsoft.com/powershell",
    "https://docs.microsoft.com/sql",
    "https://docs.microsoft.com/surface",
    "https://docs.microsoft.com/system-center",
    "https://docs.microsoft.com/visualstudio",
    "https://docs.microsoft.com/windows",
    "https://docs.microsoft.com/xamarin"
};

The HttpClient exposes the ability to send HTTP requests and receive HTTP responses. The s_urlList holds all of the URLs that the application plans to process.

Update application entry point

The main entry point into the console application is the Main method. Replace the existing method with the following:

static Task Main() => SumPageSizesAsync();

The updated Main method is now considered an Async main, which allows for an asynchronous entry point into the executable. It is expressed a call to SumPageSizesAsync.

Create the asynchronous sum page sizes method

Below the Main method, add the SumPageSizesAsync method:

static async Task SumPageSizesAsync()
{
    var stopwatch = Stopwatch.StartNew();

    IEnumerable<Task<int>> downloadTasksQuery =
        from url in s_urlList
        select ProcessUrlAsync(url, s_client);

    List<Task<int>> downloadTasks = downloadTasksQuery.ToList();

    int total = 0;
    while (downloadTasks.Any())
    {
        Task<int> finishedTask = await Task.WhenAny(downloadTasks);
        downloadTasks.Remove(finishedTask);
        total += await finishedTask;
    }

    stopwatch.Stop();

    Console.WriteLine($"\nTotal bytes returned:  {total:#,#}");
    Console.WriteLine($"Elapsed time:          {stopwatch.Elapsed}\n");
}

The method starts by instantiating and starting a Stopwatch. It then includes a query that, when executed, creates a collection of tasks. Each call to ProcessUrlAsync in the following code returns a Task<TResult>, where TResult is an integer:

IEnumerable<Task<int>> downloadTasksQuery =
    from url in s_urlList
    select ProcessUrlAsync(url, s_client);

Due to deferred execution with the LINQ, you call Enumerable.ToList to start each task.

List<Task<int>> downloadTasks = downloadTasksQuery.ToList();

The while loop performs the following steps for each task in the collection:

  1. Awaits a call to WhenAny to identify the first task in the collection that has finished its download.

    Task<int> finishedTask = await Task.WhenAny(downloadTasks);
    
  2. Removes that task from the collection.

    downloadTasks.Remove(finishedTask);
    
  3. Awaits finishedTask, which is returned by a call to ProcessUrlAsync. The finishedTask variable is a Task<TResult> where TResult is an integer. The task is already complete, but you await it to retrieve the length of the downloaded website, as the following example shows. If the task is faulted, await will throw the first child exception stored in the AggregateException, unlike reading the Task<TResult>.Result property, which would throw the AggregateException.

    total += await finishedTask;
    

Add process method

Add the following ProcessUrlAsync method below the SumPageSizesAsync method:

static async Task<int> ProcessUrlAsync(string url, HttpClient client)
{
    byte[] content = await client.GetByteArrayAsync(url);
    Console.WriteLine($"{url,-60} {content.Length,10:#,#}");

    return content.Length;
}

For any given URL, the method will use the client instance provided to get the response as a byte[]. The length is returned after the URL and length is written to the console.

Run the program several times to verify that the downloaded lengths don't always appear in the same order.

Caution

You can use WhenAny in a loop, as described in the example, to solve problems that involve a small number of tasks. However, other approaches are more efficient if you have a large number of tasks to process. For more information and examples, see Processing tasks as they complete.

Complete example

The following code is the complete text of the Program.cs file for the example.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;

namespace ProcessTasksAsTheyFinish
{
    class Program
    {
        static readonly HttpClient s_client = new HttpClient
        {
            MaxResponseContentBufferSize = 1_000_000
        };

        static readonly IEnumerable<string> s_urlList = new string[]
        {
            "https://docs.microsoft.com",
            "https://docs.microsoft.com/aspnet/core",
            "https://docs.microsoft.com/azure",
            "https://docs.microsoft.com/azure/devops",
            "https://docs.microsoft.com/dotnet",
            "https://docs.microsoft.com/dynamics365",
            "https://docs.microsoft.com/education",
            "https://docs.microsoft.com/enterprise-mobility-security",
            "https://docs.microsoft.com/gaming",
            "https://docs.microsoft.com/graph",
            "https://docs.microsoft.com/microsoft-365",
            "https://docs.microsoft.com/office",
            "https://docs.microsoft.com/powershell",
            "https://docs.microsoft.com/sql",
            "https://docs.microsoft.com/surface",
            "https://docs.microsoft.com/system-center",
            "https://docs.microsoft.com/visualstudio",
            "https://docs.microsoft.com/windows",
            "https://docs.microsoft.com/xamarin"
        };

        static Task Main() => SumPageSizesAsync();

        static async Task SumPageSizesAsync()
        {
            var stopwatch = Stopwatch.StartNew();

            IEnumerable<Task<int>> downloadTasksQuery =
                from url in s_urlList
                select ProcessUrlAsync(url, s_client);

            List<Task<int>> downloadTasks = downloadTasksQuery.ToList();

            int total = 0;
            while (downloadTasks.Any())
            {
                Task<int> finishedTask = await Task.WhenAny(downloadTasks);
                downloadTasks.Remove(finishedTask);
                total += await finishedTask;
            }

            stopwatch.Stop();

            Console.WriteLine($"\nTotal bytes returned:  {total:#,#}");
            Console.WriteLine($"Elapsed time:          {stopwatch.Elapsed}\n");
        }

        static async Task<int> ProcessUrlAsync(string url, HttpClient client)
        {
            byte[] content = await client.GetByteArrayAsync(url);
            Console.WriteLine($"{url,-60} {content.Length,10:#,#}");

            return content.Length;
        }
    }
}
// Example output:
// https://docs.microsoft.com/windows                               25,513
// https://docs.microsoft.com/gaming                                30,705
// https://docs.microsoft.com/dotnet                                69,626
// https://docs.microsoft.com/dynamics365                           50,756
// https://docs.microsoft.com/surface                               35,519
// https://docs.microsoft.com                                       39,531
// https://docs.microsoft.com/azure/devops                          75,837
// https://docs.microsoft.com/xamarin                               60,284
// https://docs.microsoft.com/system-center                         43,444
// https://docs.microsoft.com/enterprise-mobility-security          28,946
// https://docs.microsoft.com/microsoft-365                         43,278
// https://docs.microsoft.com/visualstudio                          31,414
// https://docs.microsoft.com/office                                42,292
// https://docs.microsoft.com/azure                                401,113
// https://docs.microsoft.com/graph                                 46,831
// https://docs.microsoft.com/education                             25,098
// https://docs.microsoft.com/powershell                            58,173
// https://docs.microsoft.com/aspnet/core                           87,763
// https://docs.microsoft.com/sql                                   53,362
   
// Total bytes returned: 1,249,485
// Elapsed time:          00:00:02.7068725

See also