Tutorial: Analyze videos with Media Services v3 using .NET

This tutorial shows you how to analyze videos with Azure Media Services. There are many scenarios in which you might want to gain deep insights into recorded videos or audio content. For example, to achieve higher customer satisfaction, organizations can run speech-to-text processing to convert customer support recordings into a searchable catalog, with indexes and dashboards. Then, they can obtain insights into their business such as a list of common complaints, sources of such complaints, and other useful information.

This tutorial shows you how to:

  • Download the sample app described in the topic
  • Examine the code that analyzes the specified video
  • Run the app
  • Examine the output
  • Clean up resources

If you don't have an Azure subscription, create a free account before you begin.

Prerequisites

Download and configure the sample

Clone a GitHub repository that contains the .NET sample to your machine using the following command:

git clone https://github.com/Azure-Samples/media-services-v3-dotnet-tutorials.git

The sample is located in the AnalyzeVideos folder.

Open appsettings.json in you downloaded project. Replace the values with credentials that you got from accessing APIs.

Examine the code that analyzes the specified video

This section examines functions defined in the Program.cs file of the AnalyzeVideos project.

The sample performs the following actions:

  1. Creates a Transform and a Job that analyzes your video.
  2. Creates an input Asset and uploads the video into it. The asset is used as the job's input.
  3. Creates an output asset that stores the job's output.
  4. Submits the job.
  5. Checks the job's status.
  6. Downloads the files that resulted from running the job.

Note

When using a Video or Audio Analyzer presets, use the Azure portal to set your account to have 10 S3 Media Reserved Units. For more information, see Scale media processing.

Start using Media Services APIs with .NET SDK

To start using Media Services APIs with .NET, you need to create an AzureMediaServicesClient object. To create the object, you need to supply credentials needed for the client to connect to Azure using Azure AD. In the code you cloned at the beginning of the article, the GetCredentialsAsync function creates the ServiceClientCredentials object based on the credentials supplied in local configuration file.

private static async Task<IAzureMediaServicesClient> CreateMediaServicesClientAsync(ConfigWrapper config)
{
    var credentials = await GetCredentialsAsync(config);

    return new AzureMediaServicesClient(config.ArmEndpoint, credentials)
    {
        SubscriptionId = config.SubscriptionId,
    };
}

Create an input asset and upload a local file into it

The CreateInputAsset function creates a new input Asset and uploads the specified local video file into it. This Asset is used as the input to your encoding Job. In Media Services v3, the input to a Job can either be an Asset, or it can be content that you make available to your Media Services account via HTTPS URLs. If you want to learn how to encode from a HTTPS URL, see this article.

In Media Services v3, you use Azure Storage APIs to upload files. The following .NET snippet shows how.

The following function performs these actions:

  • Creates an Asset
  • Gets a writable SAS URL to the Asset’s container in storage
  • Uploads the file into the container in storage using the SAS URL
private static async Task<Asset> CreateInputAssetAsync(
    IAzureMediaServicesClient client,
    string resourceGroupName,
    string accountName,
    string assetName,
    string fileToUpload)
{
    // In this example, we are assuming that the asset name is unique.
    //
    // If you already have an asset with the desired name, use the Assets.Get method
    // to get the existing asset. In Media Services v3, the Get method on entities returns null 
    // if the entity doesn't exist (a case-insensitive check on the name).

    // Call Media Services API to create an Asset.
    // This method creates a container in storage for the Asset.
    // The files (blobs) associated with the asset will be stored in this container.
    Asset asset = await client.Assets.CreateOrUpdateAsync(resourceGroupName, accountName, assetName, new Asset());

    // Use Media Services API to get back a response that contains
    // SAS URL for the Asset container into which to upload blobs.
    // That is where you would specify read-write permissions 
    // and the exparation time for the SAS URL.
    var response = await client.Assets.ListContainerSasAsync(
        resourceGroupName,
        accountName,
        assetName,
        permissions: AssetContainerPermission.ReadWrite,
        expiryTime: DateTime.UtcNow.AddHours(4).ToUniversalTime());

    var sasUri = new Uri(response.AssetContainerSasUrls.First());

    // Use Storage API to get a reference to the Asset container
    // that was created by calling Asset's CreateOrUpdate method.  
    CloudBlobContainer container = new CloudBlobContainer(sasUri);
    var blob = container.GetBlockBlobReference(Path.GetFileName(fileToUpload));

    // Use Strorage API to upload the file into the container in storage.
    await blob.UploadFromFileAsync(fileToUpload);

    return asset;
}

Create an output asset to store the result of the job

The output Asset stores the result of your job. The project defines the DownloadResults function that downloads the results from this output asset into the "output" folder, so you can see what you got.

private static async Task<Asset> CreateOutputAssetAsync(IAzureMediaServicesClient client, string resourceGroupName, string accountName, string assetName)
{
    // Check if an Asset already exists
    Asset outputAsset = await client.Assets.GetAsync(resourceGroupName, accountName, assetName);
    Asset asset = new Asset();
    string outputAssetName = assetName;

    if (outputAsset != null)
    {
        // Name collision! In order to get the sample to work, let's just go ahead and create a unique asset name
        // Note that the returned Asset can have a different name than the one specified as an input parameter.
        // You may want to update this part to throw an Exception instead, and handle name collisions differently.
        string uniqueness = $"-{Guid.NewGuid().ToString("N")}";
        outputAssetName += uniqueness;
        
        Console.WriteLine("Warning – found an existing Asset with name = " + assetName);
        Console.WriteLine("Creating an Asset with this name instead: " + outputAssetName);
    }

    return await client.Assets.CreateOrUpdateAsync(resourceGroupName, accountName, outputAssetName, asset);
}

Create a transform and a job that analyzes videos

When encoding or processing content in Media Services, it is a common pattern to set up the encoding settings as a recipe. You would then submit a Job to apply that recipe to a video. By submitting new Jobs for each new video, you are applying that recipe to all the videos in your library. A recipe in Media Services is called as a Transform. For more information, see Transforms and jobs. The sample described in this tutorial defines a recipe that analyzes the specified video.

Transform

When creating a new Transform instance, you need to specify what you want it to produce as an output. The required parameter is a TransformOutput object, as shown in the code above. Each TransformOutput contains a Preset. Preset describes step-by-step instructions of video and/or audio processing operations that are to be used to generate the desired TransformOutput. In this example, the VideoAnalyzerPreset preset is used and the language ("en-US") is passed to its constructor. This preset enables you to extract multiple audio and video insights from a video. You can use the AudioAnalyzerPreset preset if you need to extract multiple audio insights from a video.

When creating a Transform, you should first check if one already exists using the Get method, as shown in the code that follows. In Media Services v3, Get methods on entities return null if the entity doesn’t exist (a case-insensitive check on the name).

private static async Task<Transform> GetOrCreateTransformAsync(IAzureMediaServicesClient client, 
    string resourceGroupName, 
    string accountName, 
    string transformName, 
    Preset preset)
{
    // Does a Transform already exist with the desired name? Assume that an existing Transform with the desired name
    // also uses the same recipe or Preset for processing content.
    Transform transform = await client.Transforms.GetAsync(resourceGroupName, accountName, transformName);

    if (transform == null)
    {
        // Start by defining the desired outputs.
        TransformOutput[] outputs = new TransformOutput[]
        {
            new TransformOutput(preset),
        };

        // Create the Transform with the output defined above
        transform = await client.Transforms.CreateOrUpdateAsync(resourceGroupName, accountName, transformName, outputs);
    }

    return transform;
}

Job

As mentioned above, the Transform object is the recipe and a Job is the actual request to Media Services to apply that Transform to a given input video or audio content. The Job specifies information like the location of the input video, and the location for the output. You can specify the location of your video using: HTTPS URLs, SAS URLs, or Assets that are in your Media Service account.

In this example, the job input is a local video.

private static async Task<Job> SubmitJobAsync(IAzureMediaServicesClient client,
    string resourceGroupName,
    string accountName,
    string transformName,
    string jobName,
    JobInput jobInput,
    string outputAssetName)
{
    JobOutput[] jobOutputs =
    {
        new JobOutputAsset(outputAssetName),
    };

    // In this example, we are assuming that the job name is unique.
    //
    // If you already have a job with the desired name, use the Jobs.Get method
    // to get the existing job. In Media Services v3, Get methods on entities returns null 
    // if the entity doesn't exist (a case-insensitive check on the name).
    Job job = await client.Jobs.CreateAsync(
        resourceGroupName,
        accountName,
        transformName,
        jobName,
        new Job
        {
            Input = jobInput,
            Outputs = jobOutputs,
        });

    return job;
}

Wait for the job to complete

The job takes some time to complete and when it does you want to be notified. There are different options to get notified about the Job completion. The simplest option (that is shown here) is to use polling.

Polling is not a recommended best practice for production applications because of potential latency. Polling can be throttled if overused on an account. Developers should instead use Event Grid.

Event Grid is designed for high availability, consistent performance, and dynamic scale. With Event Grid, your apps can listen for and react to events from virtually all Azure services, as well as custom sources. Simple, HTTP-based reactive event handling helps you build efficient solutions through intelligent filtering and routing of events. See Route events to a custom web endpoint.

The Job usually goes through the following states: Scheduled, Queued, Processing, Finished (the final state). If the job has encountered an error, you get the Error state. If the job is in the process of being canceled, you get Canceling and Canceled when it is done.

private static async Task<Job> WaitForJobToFinishAsync(IAzureMediaServicesClient client,
    string resourceGroupName,
    string accountName,
    string transformName,
    string jobName)
{
    const int SleepIntervalMs = 60 * 1000;

    Job job = null;

    do
    {
        job = await client.Jobs.GetAsync(resourceGroupName, accountName, transformName, jobName);

        Console.WriteLine($"Job is '{job.State}'.");
        for (int i = 0; i < job.Outputs.Count; i++)
        {
            JobOutput output = job.Outputs[i];
            Console.Write($"\tJobOutput[{i}] is '{output.State}'.");
            if (output.State == JobState.Processing)
            {
                Console.Write($"  Progress: '{output.Progress}'.");
            }

            Console.WriteLine();
        }

        if (job.State != JobState.Finished && job.State != JobState.Error && job.State != JobState.Canceled)
        {
            await Task.Delay(SleepIntervalMs);
        }
    }
    while (job.State != JobState.Finished && job.State != JobState.Error && job.State != JobState.Canceled);

    return job;
}

Job error codes

See Error codes.

Download the result of the job

The following function downloads the results from the output Asset into the "output" folder, so you can examine the results of the job.

private static async Task DownloadOutputAssetAsync(
    IAzureMediaServicesClient client,
    string resourceGroup,
    string accountName,
    string assetName,
    string outputFolderName)
{
    const int ListBlobsSegmentMaxResult = 5;

    if (!Directory.Exists(outputFolderName))
    {
        Directory.CreateDirectory(outputFolderName);
    }

    AssetContainerSas assetContainerSas = await client.Assets.ListContainerSasAsync(
        resourceGroup,
        accountName,
        assetName,
        permissions: AssetContainerPermission.Read,
        expiryTime: DateTime.UtcNow.AddHours(1).ToUniversalTime());

    Uri containerSasUrl = new Uri(assetContainerSas.AssetContainerSasUrls.FirstOrDefault());
    CloudBlobContainer container = new CloudBlobContainer(containerSasUrl);

    string directory = Path.Combine(outputFolderName, assetName);
    Directory.CreateDirectory(directory);

    Console.WriteLine($"Downloading output results to '{directory}'...");

    BlobContinuationToken continuationToken = null;
    IList<Task> downloadTasks = new List<Task>();

    do
    {
        BlobResultSegment segment = await container.ListBlobsSegmentedAsync(null, true, BlobListingDetails.None, ListBlobsSegmentMaxResult, continuationToken, null, null);

        foreach (IListBlobItem blobItem in segment.Results)
        {
            CloudBlockBlob blob = blobItem as CloudBlockBlob;
            if (blob != null)
            {
                string path = Path.Combine(directory, blob.Name);

                downloadTasks.Add(blob.DownloadToFileAsync(path, FileMode.Create));
            }
        }

        continuationToken = segment.ContinuationToken;
    }
    while (continuationToken != null);

    await Task.WhenAll(downloadTasks);

    Console.WriteLine("Download complete.");
}

Clean up resource in your Media Services account

Generally, you should clean up everything except objects that you are planning to reuse (typically, you will reuse Transforms and persist StreamingLocators). If you want for your account to be clean after experimenting, you should delete the resources that you do not plan to reuse. For example, the following code deletes Jobs.

private static async Task CleanUpAsync(
    IAzureMediaServicesClient client,
    string resourceGroupName,
    string accountName,
    string transformName)
{

    var jobs = await client.Jobs.ListAsync(resourceGroupName, accountName, transformName);
    foreach (var job in jobs)
    {
        await client.Jobs.DeleteAsync(resourceGroupName, accountName, transformName, job.Name);
    }

    var assets = await client.Assets.ListAsync(resourceGroupName, accountName);
    foreach (var asset in assets)
    {
        await client.Assets.DeleteAsync(resourceGroupName, accountName, asset.Name);
    }
}

Run the sample app

Press Ctrl+F5 to run the AnalyzeVideos application.

When we run the program, the job produces thumbnails for each face that it finds in the video. It also produces the insights.json file.

Examine the output

The output file of analyzing videos is called insights.json. This file contains insights about your video. You can find description of elements found in the json file in the Media intelligence article.

Clean up resources

If you no longer need any of the resources in your resource group, including the Media Services and storage accounts you created for this tutorial, delete the resource group you created earlier.

Execute the following CLI command:

az group delete --name amsResourceGroup

Multithreading

The Azure Media Services v3 SDKs are not thread-safe. When working with multi-threaded application, you should generate a new AzureMediaServicesClient object per thread.

Next steps