Create a Visual Search single-page web app

The Bing Visual Search API returns insights for an image. You can either upload an image or provide a URL to one. Insights are visually similar images, shopping sources, webpages that include the image, and more. Insights returned by the Bing Visual Search API are similar to ones shown on Bing.com/images.

This tutorial explains how to extend a single-page web app for the Bing Image Search API. To view that tutorial or get the source code used here, see Tutorial: Create a single-page app for the Bing Image Search API.

The full source code for this application (after extending it to use the Bing Visual Search API), is available on GitHub.

Prerequisites

You must have a Cognitive Services API account with access to the Bing Search APIs. Before continuing, you will need the access key provided after starting a subscription at S9 price tier as shown in Cognitive Services Pricing - Bing Search API.

To start a subscription in Azure portal:

  1. Enter 'BingSearchV7' in the text box at the top of the Azure portal that says Search resources, services, and docs.
  2. Under Marketplace in the drop-down list, select Bing Search v7.
  3. Enter Name for the new resource.
  4. Select Pay-As-You-Go subscription.
  5. Select S9 pricing tier.
  6. Click Enable to start the subscription.

Call the Bing Visual Search API and handle the response

Edit the Bing Image Search tutorial and add the following code to the end of the <script> element (and before the closing </script> tag). The following code handles a visual search response from the API, iterates through the results, and displays them:

function handleVisualSearchResponse(){
    if(this.status !== 200){
        console.log(this.responseText);
        return;
    }
    let json = this.responseText;
    let response = JSON.parse(json);
    for (let i =0; i < response.tags.length; i++) {
        let tag = response.tags[i];
        if (tag.displayName === '') {
            for (let j = 0; j < tag.actions.length; j++) {
                let action = tag.actions[j];
                if (action.actionType === 'VisualSearch') {
                    let html = '';
                    for (let k = 0; k < action.data.value.length; k++) {
                        let item = action.data.value[k];
                        let height = 120;
                        let width = Math.max(Math.round(height * item.thumbnail.width / item.thumbnail.height), 120);
                        html += "<img src='"+ item.thumbnailUrl + "&h=" + height + "&w=" + width + "' height=" + height + " width=" + width + "'>";
                    }
                    showDiv("insights", html);
                    window.scrollTo({top: document.getElementById("insights").getBoundingClientRect().top, behavior: "smooth"});
                }
            }
        }
    }
}

The following code sends a search request to the API, using an event listener to call handleVisualSearchResponse():

function bingVisualSearch(insightsToken){
    let visualSearchBaseURL = 'https://api.cognitive.microsoft.com/bing/v7.0/images/visualsearch',
        boundary = 'boundary_ABC123DEF456',
        startBoundary = '--' + boundary,
        endBoundary = '--' + boundary + '--',
        newLine = "\r\n",
        bodyHeader = 'Content-Disposition: form-data; name="knowledgeRequest"' + newLine + newLine;

    let postBody = {
        imageInfo: {
            imageInsightsToken: insightsToken
        }
    }
    let requestBody = startBoundary + newLine;
    requestBody += bodyHeader;
    requestBody += JSON.stringify(postBody) + newLine + newLine;
    requestBody += endBoundary + newLine;

    let request = new XMLHttpRequest();

    try {
        request.open("POST", visualSearchBaseURL);
    } 
    catch (e) {
        renderErrorMessage("Error performing visual search: " + e.message);
    }
    request.setRequestHeader("Ocp-Apim-Subscription-Key", getSubscriptionKey());
    request.setRequestHeader("Content-Type", "multipart/form-data; boundary=" + boundary);
    request.addEventListener("load", handleVisualSearchResponse);
    request.send(requestBody);
}

Capture insights token

Add the following code to the searchItemsRenderer object. This code adds a find similar link that calls the bingVisualSearch function when clicked. The function receives the imageInsightsToken as an argument.

html.push("<a href='javascript:bingVisualSearch(\"" + item.imageInsightsToken + "\");'>find similar</a><br>");

Display similar images

Add the following HTML code at line 601. This markup code adds an element to display the results of the Bing Visual Search API call:

<div id="insights">
    <h3><a href="#" onclick="return toggleDisplay('_insights')">Similar</a></h3>
    <div id="_insights" style="display: none"></div>
</div>

With all the new JavaScript code and HTML elements in place, search results are displayed with a find similar link. Click the link to populate the Similar section with images similar to the one you picked. You may have to expand the Similar section to show the images.

Next steps