Visual Assistant

App Service
Bot Service
Language Understanding

Solution Idea

If you'd like to see us expand this article with more information, implementation details, pricing guidance, or code examples, let us know with GitHub Feedback!

Visual assistant provides rich information based on content of the image with capabilities such as reading business card, identifying barcode, and recognizing popular people, places, objects, artworks, and monuments.

Architecture

Architecture diagram Download an SVG of this architecture.

Data Flow

  1. Users interact with bot
  2. Bot understands context from LUIS
  3. Bot passes visual context to the Bing Visual Search API
  4. Bot gets additional information from Bing Entity Search for rich context on people, place, artwork, monument, and objects.
  5. Bot gets additional information for barcodes.
  6. Optionally Bot gets more information on barcodes/queries exclusively from your domain through the Bing Custom Search API.
  7. Assistant renders similar products/destinations from your domain or provides more information around celebrity/place/monuments/artworks.