Visual Assistant

App Service
Bot Service
Language Understanding

Solution Idea

If you'd like to see us expand this article with more information, implementation details, pricing guidance, or code examples, let us know with GitHub Feedback!

Visual assistant provides rich information based on content of the image with capabilities such as reading business card, identifying barcode, and recognizing popular people, places, objects, artworks, and monuments.

Architecture

Architecture diagram Download an SVG of this architecture.

Data flow

  1. Users interact with Bot through mobile app or web app.
  2. Bot understands user intent and conversational context by using LUIS, which is built into the application.
  3. Bot passes visual context (that is, image) to the Bing Visual Search API.
  4. Bot gets additional information from Bing Entity Search for rich context on people, place, artwork, monument, and objects.
  5. Bot gets additional information for barcodes.
  6. Optionally, Bot gets more information on barcodes/queries exclusively from your domain through the Bing Custom Search API.
  7. Visual Assistant renders similar products/destinations from your domain or provides more information around celebrity/place/monuments/artworks.

Components

  • Azure App Service: A fully managed HTTP-based service for hosting web apps, REST APIs, and mobile backends
  • Azure Bot Service: Develop intelligent, enterprise-grade bots that enrich your customer experience, all while maintaining control of your data
  • Bing Custom Search: Build customized search that fits your business needs with Bing's powerful ranking and global-scale search index
  • Bing Entity Search: Infuse a deep knowledge search into your existing content by identifying the most relevant
  • Bing Visual Search: Find visual insights from your images
  • Bing Web Search: Bring intelligent search to your apps and harness the ability to comb through billions of webpages, images, videos, and news, all with a single API call
  • Language Understanding Intelligence Service (LUIS): Build natural language into apps, bots, and IoT devices

Next steps