Tutorial: Moderate e-commerce product images with Azure Content Moderator

In this tutorial, you'll learn how to use Azure Cognitive Services, including Content Moderator, to classify and moderate product images for an e-commerce scenario. You'll use Computer Vision and Custom Vision to apply tags (labels) to images, and then you will create a team review, which combines Content Moderator's machine-learning-based technologies with human review teams to provide an intelligent moderation system.

This tutorial shows you how to:

  • Sign up for Content Moderator and create a review team.
  • Use Content Moderator's image API to scan for potential adult and racy content.
  • Use the Computer Vision service to scan for celebrity content (or other Computer-Vision-detectable tags).
  • Use the Custom Vision service to scan for the presence of flags, toys, and pens (or other custom tags).
  • Present the combined scan results for human review and final decision making.

The complete sample code is available in the Samples eCommerce Catalog Moderation repository on GitHub.

If you don’t have an Azure subscription, create a free account before you begin.

Prerequisites

  • A Content Moderator subscription key. Follow the instructions in Create a Cognitive Services account to subscribe to the Content Moderator service and get your key.
  • A Computer Vision subscription key (same instructions as above).
  • Any edition of Visual Studio 2015 or 2017.
  • A set of images for each label that the Custom Vision classifier will use (in this case toys, pens, and US flags).

Create a review team

Refer to the Try Content Moderator on the web quickstart for instructions on how to sign up for the Content Moderator Review tool and create a review team. Take note of the Team ID value on the Credentials page.

Create custom moderation tags

Next, create custom tags in the Review tool (see the Tags article if you need help with this process). In this case, we will add the following tags: celebrity, USA, flag, toy, and pen. Not all of the tags need to be detectable categories in Computer Vision (like celebrity); you can add your own custom tags as long as you train the Custom Vision classifier to detect them later on.

Configure custom tags

Create Visual Studio project

  1. In Visual Studio, open the New Project dialog. Expand Installed, then Visual C#, then select Console app (.NET Framework).
  2. Name the application EcommerceModeration, then click OK.
  3. If you're adding this project to an existing solution, select this project as the single startup project.

This tutorial highlights the code that is central to the project, but it won't cover every line of code. Copy the full contents of Program.cs from the sample project (Samples eCommerce Catalog Moderation) into the Program.cs file of your new project. Then, step through the following sections to learn about how the project works and how to use it yourself.

Define API keys and endpoints

This tutorial uses three cognitive services; therefore, it requires three corresponding keys and API endpoints. See the following fields in the Program class:

public const string ContentModeratorKey = "XXXXXXXXXXXXX";
public const string ComputerVisionKey = "XXXXXXXXXXXX";
public const string CustomVisionKey = "XXXXXXXXXXX";

// All your end points based on the new account and subscriptions
public const string ImageUri = "https://westus.api.cognitive.microsoft.com/contentmoderator/moderate/v1.0/ProcessImage/Evaluate";
public const string ReviewUri = "https://westus.api.cognitive.microsoft.com/contentmoderator/review/v1.0/teams/YOUR-TEAM-ID/reviews";
public const string ComputerVisionUri = "https://westcentralus.api.cognitive.microsoft.com/vision/v1.0/analyze?details=celebrities";
public const string CustomVisionUri = "https://southcentralus.api.cognitive.microsoft.com/customvision/v1.0/Prediction/XXXXXXXXXXXXXXXX/url";

You'll need to update the ___Key fields with the values of your subscription keys (you'll get the CustomVisionKey later on), and you may need to change the ___Uri fields so they contain the correct region identifiers. Fill in the YOURTEAMID part of the ReviewUri field with the ID of the review team you created earlier. You'll fill in the final part of the CustomVisionUri field later on.

Primary method calls

See the following code in the Main method, which loops through a list of image URLs. It analyzes each image with the three different services, records the applied tags in the ReviewTags array, and then creates a review for human moderators by sending the images to the Content Moderator Review Tool. You will explore these methods in the following sections. If you wish, you can control which images are sent to review, using the ReviewTags array in a conditional statement to check which tags were applied.

// for each image URL in the file...
foreach (var Url in Urls)
{
    // Initiatize a new review tags array
    ReviewTags = new KeyValuePair[MAXTAGSCOUNT];

    // Evaluate for potential adult and racy content with Content Moderator API
    EvaluateAdultRacy(Url, ref ReviewTags);

    // Evaluate for potential presence of celebrity (ies) in images with Computer Vision API
    EvaluateComputerVisionTags(Url, ComputerVisionUri, ComputerVisionKey, ref ReviewTags);

    // Evaluate for potential presence of custom categories other than Marijuana
    EvaluateCustomVisionTags(Url, CustomVisionUri, CustomVisionKey, ref ReviewTags);

    // Create review in the Content Moderator review tool
    CreateReview(Url, ReviewTags);
}

EvaluateAdultRacy method

See the EvaluateAdultRacy method in the Program class. This method takes an image URL and an array of key-value pairs as parameters. It calls the Content Moderator's Image API (using REST) to get the Adult and Racy scores of the image. If the score for either is greater than 0.4 (the range is between 0 and 1), it sets the corresponding value in the ReviewTags array to True.

/// <summary>
/// Use Content Moderator API to evaluate for potential adult and racy content
/// </summary>
/// <param name="ImageUrl"></param>
/// <param name="ReviewTags"></param>
/// <returns>API call success or not</returns>
public static bool EvaluateAdultRacy(string ImageUrl, ref KeyValuePair[] ReviewTags)
{
    float AdultScore = 0;
    float RacyScore = 0;

    var File = ImageUrl;
    string Body = $"{{\"DataRepresentation\":\"URL\",\"Value\":\"{File}\"}}";

    HttpResponseMessage response = CallAPI(ImageUri, ContentModeratorKey, CallType.POST,
                                           "Ocp-Apim-Subscription-Key", "application/json", "", Body);

    if (response.IsSuccessStatusCode)
    {
        // {“answers”:[{“answer”:“Hello”,“questions”:[“Hi”],“score”:100.0}]}
        // Parse the response body. Blocking!
        GetAdultRacyScores(response.Content.ReadAsStringAsync().Result, out AdultScore, out RacyScore);
    }

    ReviewTags[0] = new KeyValuePair();
    ReviewTags[0].Key = "a";
    ReviewTags[0].Value = "false";
    if (AdultScore > 0.4)
    {
        ReviewTags[0].Value = "true";
    }

    ReviewTags[1] = new KeyValuePair();
    ReviewTags[1].Key = "r";
    ReviewTags[1].Value = "false";
    if (RacyScore > 0.3)
    {
        ReviewTags[1].Value = "true";
    }
    return response.IsSuccessStatusCode;
}

EvaluateComputerVisionTags method

The next method takes an image URL and your Computer Vision subscription information and analyzes the image for the presence of celebrities. If one or more celebrities are found, it sets the corresponding value in the ReviewTags array to True.

/// <summary>
/// Use Computer Vision API to evaluate for potential celebrity presence in image
/// </summary>
/// <param name="ImageUrl"></param>
/// <param name="ComputerVisionUri"></param>
/// <param name="ComputerVisionKey"></param>
/// <param name="ReviewTags"></param>
/// <returns>API call success or not</returns>
public static bool EvaluateComputerVisionTags(string ImageUrl, string ComputerVisionUri, string ComputerVisionKey, ref KeyValuePair[] ReviewTags)
{
    var File = ImageUrl;
    string Body = $"{{\"URL\":\"{File}\"}}";

    HttpResponseMessage Response = CallAPI(ComputerVisionUri, ComputerVisionKey, CallType.POST,
                                           "Ocp-Apim-Subscription-Key", "application/json", "", Body);

    if (Response.IsSuccessStatusCode)
    {
        ReviewTags[2] = new KeyValuePair();
        ReviewTags[2].Key = "cb";
        ReviewTags[2].Value = "false";

        ComputerVisionPrediction CVObject = JsonConvert.DeserializeObject<ComputerVisionPrediction>(Response.Content.ReadAsStringAsync().Result);

        if ((CVObject.categories[0].detail != null) && (CVObject.categories[0].detail.celebrities.Count() > 0))
        {                 
            ReviewTags[2].Value = "true";
        }
    }

    return Response.IsSuccessStatusCode;
}

EvaluateCustomVisionTags method

Next, see the EvaluateCustomVisionTags method, which classifies the actual products—in this case flags, toys, and pens. Follow the instructions in the How to build a classifier guide to build your own custom image classifier and detect flags, toys, and pens (or whatever you chose as your custom tags) in images. You can use the images in the sample-images folder of the GitHub repo to quickly train some of the categories in this example.

Custom Vision web page with training images of pens, toys, and flags

Once you've trained your classifier, get the prediction key and prediction endpoint URL (see Get the URL and prediction key if you need help with retrieving them), and assign these values to your CustomVisionKey and CustomVisionUri fields, respectively. The method uses these values to query the classifier. If the classifier finds one or more of the custom tags in the image, this method sets the corresponding value(s) in the ReviewTags array to True.

/// <summary>
/// Use Custom Vision API to evaluate for potential content from custom-trained categories 
/// </summary>
/// <param name="ImageUrl"></param>
/// <param name="CustomVisionUri"></param>
/// <param name="CustomVisionKey"></param>
/// <param name="ReviewTags"></param>
/// <returns>API call success or not</returns>
 public static bool EvaluateCustomVisionTags(string ImageUrl, string CustomVisionUri, string CustomVisionKey, ref KeyValuePair[] ReviewTags)
 {
     var File = ImageUrl;
     string Body = $"{{\"URL\":\"{File}\"}}";

     HttpResponseMessage response = CallAPI(CustomVisionUri, CustomVisionKey, CallType.POST,
                                            "Prediction-Key", "application/json", "", Body);

     if (response.IsSuccessStatusCode)
     {
         // Parse the response body. Blocking!
         SaveCustomVisionTags(response.Content.ReadAsStringAsync().Result, ref ReviewTags);
     }

     return response.IsSuccessStatusCode;
 }

Create reviews for Review tool

In the previous sections, you explored how the app scans incoming images for adult and racy content (Content Moderator), celebrities (Computer Vision), and various other objects (Custom Vision). Next, see the CreateReview method, which uploads the images with all of their applied tags (passed in as Metadata) to the Content Moderator Review Tool.

/// <summary>
/// Call Content Moderator's Review API to create a review with the image (URL) and the review tags as inputs
/// </summary>
/// <param name="ImageUrl"></param>
/// <param name="Metadata"></param>
/// <returns>API call success or not</returns>
public static bool CreateReview(string ImageUrl, KeyValuePair[] Metadata)
{

    ReviewCreationRequest Review = new ReviewCreationRequest();
    Review.Item[0] = new ReviewItem();
    Review.Item[0].Content = ImageUrl;
    Review.Item[0].Metadata = new KeyValuePair[MAXTAGSCOUNT];
    Metadata.CopyTo(Review.Item[0].Metadata, 0);

    //SortReviewItems(ref Review);

    string Body = JsonConvert.SerializeObject(Review.Item);

    HttpResponseMessage response = CallAPI(ReviewUri, ContentModeratorKey, CallType.POST,
                                           "Ocp-Apim-Subscription-Key", "application/json", "", Body);

    return response.IsSuccessStatusCode;
}

The images will show up in the Review tab of the Content Moderator Review tool.

Screenshot of the Content Moderator Review tool with several images and their highlighted tags

Submit a list of test images

As you can see in the Main method, this program looks for a "C:Test" directory with a Urls.txt file that contains a list of image Urls. Create this file and directory, or change the path to point to your text file. Then populate this file with the URLs of images you'd like to test.

// Check for a test directory for a text file with the list of Image URLs to scan
var topdir = @"C:\test\";
var Urlsfile = topdir + "Urls.txt";

if (!Directory.Exists(topdir))
    return;

if (!File.Exists(Urlsfile))
{
    return;
}

// Read all image URLs in the file
var Urls = File.ReadLines(Urlsfile);

Run the program

If you've followed all of the above steps, the program should process each image (querying all three services for their relevant tags) and then upload the images with tag information to the Content Moderator Review Tool.

Next steps

In this tutorial, you set up a program to analyze product images, tag them by product type, and allow a review team to make informed decisions about content moderation. Next, learn more about the details of image moderation.