Solution Idea
If you'd like to see us expand this article with more information, such as potential use cases, alternative services, implementation considerations, or pricing guidance, let us know with GitHub Feedback!
Visual assistant provides rich information based on content of the image with capabilities such as reading business card, identifying barcode, and recognizing popular people, places, objects, artworks, and monuments.
Architecture
Download an SVG of this architecture.
Data flow
- Users interact with Bot through mobile app or web app.
- Bot understands user intent and conversational context by using LUIS, which is built into the application.
- Bot passes visual context (that is, image) to the Bing Visual Search API.
- Bot gets additional information from Bing Entity Search for rich context on people, place, artwork, monument, and objects.
- Bot gets additional information for barcodes.
- Optionally, Bot gets more information on barcodes/queries exclusively from your domain through the Bing Custom Search API.
- Visual Assistant renders similar products/destinations from your domain or provides more information around celebrity/place/monuments/artworks.
Components
- Azure App Service: A fully managed HTTP-based service for hosting web apps, REST APIs, and mobile backends
- Azure Bot Service: Develop intelligent, enterprise-grade bots that enrich your customer experience, all while maintaining control of your data
- Bing Custom Search: Build customized search that fits your business needs with Bing's powerful ranking and global-scale search index
- Bing Entity Search: Infuse a deep knowledge search into your existing content by identifying the most relevant
- Bing Visual Search: Find visual insights from your images
- Bing Web Search: Bring intelligent search to your apps and harness the ability to comb through billions of webpages, images, videos, and news, all with a single API call
- Language Understanding Intelligence Service (LUIS): Build natural language into apps, bots, and IoT devices
Next steps
- Let your app detect context that matters to you by training your own Custom Vision model
- Explore the Bing family of search APIs
- Build in LUIS into your Bot
- Explore how LUIS works in this Learn Module
- Learn how to build with Azure Bot Service
- Create a Bot that incorporates both QnA Maker and Azure Bot Service
- Solidify your understanding of LUIS, Azure Bot Service, and Bing Visual Search with Microsoft Certified: AI Fundamentals
- Use your knowledge in Azure Cognitive Services and become a Microsoft Certified: Azure AI Engineer Associate