Find similar images from previous searches using ImageInsightsToken

The Visual Search SDK enables you to find images online from previous searches that return an ImageInsightsToken. This application gets an ImageInsightsToken and uses the token in a subsequent search. It then sends the ImageInsightsToken to Bing and returns results that include Bing Search URLs and URLs of similar images found online.

The full source code for this tutorial can be found with additional error handling and annotations on GitHub.

Prerequisites

  • Any edition of Visual Studio 2019.
  • If you are using Linux/MacOS, you can run this application using Mono.
  • The NuGet Visual Search and Image Search packages.
    • From the Solution Explorer in Visual Studio, right-click on your project and select Manage NuGet Packages from the menu. Install the Microsoft.Azure.CognitiveServices.Search.CustomSearch package, and the Microsoft.Azure.CognitiveServices.Search.ImageSearch package. Installing the NuGet packages also installs the following:
      • Microsoft.Rest.ClientRuntime
      • Microsoft.Rest.ClientRuntime.Azure
      • Newtonsoft.Json

You must have a Cognitive Services API account with access to the Bing Search APIs. Before continuing, You will need the access key provided after starting a subscription at S9 price tier as shown in Cognitive Services Pricing - Bing Search API.

To start a subscription in Azure portal:

  1. Enter 'BingSearchV7' in the text box at the top of the Azure portal that says Search resources, services, and docs.
  2. Under Marketplace in the drop-down list, select Bing Search v7.
  3. Enter Name for the new resource.
  4. Select Pay-As-You-Go subscription.
  5. Select S9 pricing tier.
  6. Click Enable to start the subscription.

Get the ImageInsightsToken from the Bing Image Search SDK

This application uses an ImageInsightsToken obtained through the Bing Image Search SDK. In a new C# console application, create a client to call the API using ImageSearchAPI(). Then use SearchAsync() with your query:

var client = new ImageSearchAPI(new Microsoft.Azure.CognitiveServices.Search.ImageSearch.ApiKeyServiceClientCredentials(subKey));
var imageResults = client.Images.SearchAsync(query: "canadian rockies").Result;
Console.WriteLine("Search images for query \"canadian rockies\"");

Store the first search result using imageResults.Value.First(), and then store the image insight's ImageInsightsToken.

String insightTok = "None";
if (imageResults.Value.Count > 0)
{
    var firstImageResult = imageResults.Value.First();
    insightTok = firstImageResult.ImageInsightsToken;
}
else
{
    insightTok = "None found";
    Console.WriteLine("Couldn't find image results!");
}

This ImageInsightsToken is sent to Bing Visual Search in a request.

Add the ImageInsightsToken to a Visual Search request

Specify the ImageInsightsToken for a Visual Search request by creating an ImageInfo object from the ImageInsightsToken contained in responses from Bing Visual Search.

ImageInfo ImageInfo = new ImageInfo(imageInsightsToken: insightsTok);

Use Bing Visual Search to find images from an ImageInsightsToken

The VisualSearchRequest object contains information about the image in ImageInfo to be searched. The VisualSearchMethodAsync() method gets the results. You don't have to provide an image binary, as the image is represented by the token.

VisualSearchRequest VisualSearchRequest = new VisualSearchRequest(ImageInfo);

var visualSearchResults = client.Images.VisualSearchMethodAsync(knowledgeRequest: VisualSearchRequest).Result;

Iterate through the Visual Search results

Visual Search results are ImageTag objects. Each tag contains a list of ImageAction objects. Each ImageAction contains a Data field, which is a list of values that depend on the type of action. You can iterate through the ImageTag objects in visualSearchResults.Tags, for instance, and get the ImageAction tag within it. The sample below prints the details of PagesIncluding actions:

if (visualSearchResults.Tags.Count > 0)
{
    // List of tags
    foreach (ImageTag t in visualSearchResults.Tags)
    {
        foreach (ImageAction i in t.Actions)
        {
            Console.WriteLine("\r\n" + "ActionType: " + i.ActionType + " WebSearchURL: " + i.WebSearchUrl);

            if (i.ActionType == "PagesIncluding")
            {
                foreach (ImageObject o in (i as ImageModuleAction).Data.Value)
                {
                    Console.WriteLine("ContentURL: " + o.ContentUrl);
                }
            }
        }
    }
}

PagesIncluding ActionTypes

Getting the actual image URLs from action types requires a cast that reads an ActionType as ImageModuleAction, which contains a Data element with a list of values. Each value is the URL of an image. The following casts the PagesIncluding action type to ImageModuleAction and reads the values:

    if (i.ActionType == "PagesIncluding")
    {
        foreach(ImageObject o in (i as ImageModuleAction).Data.Value)
        {
            Console.WriteLine("ContentURL: " + o.ContentUrl);
        }
    }

For more information about these data types, see Images - Visual Search.

Returned URLs

The complete application returns the following URLs:

ActionType URL
MoreSizes -> WebSearchUrl
VisualSearch -> WebSearchUrl
ImageById -> WebSearchUrl
RelatedSearches -> WebSearchUrl:
DocumentLevelSuggestions -> WebSearchUrl:
TopicResults -> WebSearchUrl https://www.bing.com/cr?IG=3E32CC6CA5934FBBA14ABC3B2E4651F9&CID=1BA795A21EAF6A63175699B71FC36B7C&rd=1&h=BcQifmzdKFyyBusjLxxgO42kzq1Geh7RucVVqvH-900&v=1&r=https%3a%2f%2fwww.bing.com%2fdiscover%2fcanadian%2brocky&p=DevEx,5823.1
ImageResults -> WebSearchUrl https://www.bing.com/cr?IG=3E32CC6CA5934FBBA14ABC3B2E4651F9&CID=1BA795A21EAF6A63175699B71FC36B7C&rd=1&h=PV9GzMFOI0AHZp2gKeWJ8DcveSDRE3fP2jHDKMpJSU8&v=1&r=https%3a%2f%2fwww.bing.com%2fimages%2fsearch%3fq%3doutdoor&p=DevEx,5831.1

As shown above, the TopicResults and ImageResults types contain queries for related images. The URLs link to Bing search results.

Next steps