Azure Cognitive Services containers

Warning

On June 11, 2020, Microsoft announced that it will not sell facial recognition technology to police departments in the United States until strong regulation, grounded in human rights, has been enacted. As such, customers may not use facial recognition features or functionality included in Azure Services, such as Face or Video Indexer, if a customer is, or is allowing use of such services by or for, a police department in the United States.

Azure Cognitive Services provides several Docker containers that let you use the same APIs that are available in Azure, on-premises. Using these containers gives you the flexibility to bring Cognitive Services closer to your data for compliance, security or other operational reasons.

Container support is currently available for a subset of Azure Cognitive Services, including parts of:

Containerization is an approach to software distribution in which an application or service, including its dependencies & configuration, is packaged together as a container image. With little or no modification, a container image can be deployed on a container host. Containers are isolated from each other and the underlying operating system, with a smaller footprint than a virtual machine. Containers can be instantiated from container images for short-term tasks, and removed when no longer needed.

Cognitive Services resources are available on Microsoft Azure. Sign into the Azure portal to create and explore Azure resources for these services.

Features and benefits

  • Immutable infrastructure: Enable DevOps teams' to leverage a consistent and reliable set of known system parameters, while being able to adapt to change. Containers provide the flexibility to pivot within a predictable ecosystem and avoid configuration drift.
  • Control over data: Choose where your data gets processed by Cognitive Services. This can be essential if you can't send data to the cloud but need access to Cognitive Services APIs. Support consistency in hybrid environments – across data, management, identity, and security.
  • Control over model updates: Flexibility in versioning and updating of models deployed in their solutions.
  • Portable architecture: Enables the creation of a portable application architecture that can be deployed on Azure, on-premises and the edge. Containers can be deployed directly to Azure Kubernetes Service, Azure Container Instances, or to a Kubernetes cluster deployed to Azure Stack. For more information, see Deploy Kubernetes to Azure Stack.
  • High throughput / low latency: Provide customers the ability to scale for high throughput and low latency requirements by enabling Cognitive Services to run physically close to their application logic and data. Containers do not cap transactions per second (TPS) and can be made to scale both up and out to handle demand if you provide the necessary hardware resources.
  • Scalability: With the ever growing popularity of containerization and container orchestration software, such as Kubernetes; scalability is at the forefront of technological advancements. Building on a scalable cluster foundation, application development caters to high availability.

Containers in Azure Cognitive Services

Azure Cognitive Services containers provide the following set of Docker containers, each of which contains a subset of functionality from services in Azure Cognitive Services:

Service Supported Pricing Tier Container Description
Anomaly detector F0, S0 Anomaly-Detector (image) The Anomaly Detector API enables you to monitor and detect abnormalities in your time series data with machine learning.
Request access
Computer Vision F0, S1 Read OCR (image) The Read OCR container allows you to extract printed and handwritten text from images and documents with support for JPEG, PNG, BMP, PDF, and TIFF file formats. For more information, see the Read API documentation.
Request access
Face F0, S0 Face Detects human faces in images, and identifies attributes, including face landmarks (such as noses and eyes), gender, age, and other machine-predicted facial features. In addition to detection, Face can check if two faces in the same image or different images are the same by using a confidence score, or compare faces against a database to see if a similar-looking or identical face already exists. It can also organize similar faces into groups, using shared visual traits.
Form recognizer F0, S0 Form Recognizer Form Understanding applies machine learning technology to identify and extract key-value pairs and tables from forms.
LUIS F0, S0 LUIS (image) Loads a trained or published Language Understanding model, also known as a LUIS app, into a docker container and provides access to the query predictions from the container's API endpoints. You can collect query logs from the container and upload these back to the LUIS portal to improve the app's prediction accuracy.
Speech Service API F0, S0 Speech-to-text (image) Transcribes continuous real-time speech into text.
Speech Service API F0, S0 Custom Speech-to-text (image) Transcribes continuous real-time speech into text using a custom model.
Speech Service API F0, S0 Text-to-speech (image) Converts text to natural-sounding speech.
Speech Service API F0, S0 Custom Text-to-speech (image) Converts text to natural-sounding speech using a custom model.
Speech Service API F0, S0 Neural Text-to-speech (image) Converts text to natural-sounding speech using deep neural network technology, allowing for more natural synthesized speech.
Text Analytics F0, S Key Phrase Extraction (image) Extracts key phrases to identify the main points. For example, for the input text "The food was delicious and there were wonderful staff", the API returns the main talking points: "food" and "wonderful staff".
Text Analytics F0, S Language Detection (image) For up to 120 languages, detects which language the input text is written in and report a single language code for every document submitted on the request. The language code is paired with a score indicating the strength of the score.
Text Analytics F0, S Sentiment Analysis v3 (image) Analyzes raw text for clues about positive or negative sentiment. This version of sentiment analysis returns sentiment labels (for example positive or negative) for each document and sentence within it.
Text Analytics F0, S Text Analytics for health Extract and label medical information from unstructured clinical text.
Spatial Analysis S0 Spatial analysis Analyzes real-time streaming video to understand spatial relationships between people, their movement, and interactions with objects in physical environments.

In addition, some containers are supported in Cognitive Services All-In-One offering resource keys. You can create one single Cognitive Services All-In-One resource and use the same billing key across supported services for the following services:

  • Computer Vision
  • Face
  • LUIS
  • Text Analytics

Container availability in Azure Cognitive Services

Azure Cognitive Services containers are publicly available through your Azure subscription, and Docker container images can be pulled from either the Microsoft Container Registry or Docker Hub. You can use the docker pull command to download a container image from the appropriate registry.

Container repositories and images

The tables below are a listing of the available container images offered by Azure Cognitive Services. For a complete list of all the available container image names and their available tags, see Cognitive Services container image tags.

Generally available

The Microsoft Container Registry (MCR) syndicates all of the generally available containers for Cognitive Services. The containers are also available directly from the Docker hub.

LUIS

Container Container Registry / Repository / Image Name
LUIS mcr.microsoft.com/azure-cognitive-services/language/luis

See How to run and install LUIS containers for more information.

Text Analytics

Container Container Registry / Repository / Image Name
Sentiment Analysis v3 (English) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-en
Sentiment Analysis v3 (Spanish) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-es
Sentiment Analysis v3 (French) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-fr
Sentiment Analysis v3 (Italian) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-it
Sentiment Analysis v3 (German) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-de
Sentiment Analysis v3 (Chinese - simplified) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-zh
Sentiment Analysis v3 (Chinese - traditional) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-zht
Sentiment Analysis v3 (Japanese) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-ja
Sentiment Analysis v3 (Portuguese) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-pt
Sentiment Analysis v3 (Dutch) mcr.microsoft.com/azure-cognitive-services/textanalytics/sentiment:3.0-nl

See How to run and install Text Analytics containers for more information.

Anomaly Detector

Container Container Registry / Repository / Image Name
Anomaly detector mcr.microsoft.com/azure-cognitive-services/decision/anomaly-detector

See How to run and install Anomaly detector containers for more information.

Speech Service

Note

To use Speech containers, you will need to complete an online request form.

Container Container Registry / Repository / Image Name
Speech-to-text mcr.microsoft.com/azure-cognitive-services/speechservices/speech-to-text
Custom Speech-to-text mcr.microsoft.com/azure-cognitive-services/speechservices/custom-speech-to-text
Text-to-speech mcr.microsoft.com/azure-cognitive-services/speechservices/text-to-speech

"Ungated" preview

The following preview containers are available publicly. The Microsoft Container Registry (MCR) syndicates all of the publicly available ungated containers for Cognitive Services. The containers are also available directly from the Docker hub.

Service Container Container Registry / Repository / Image Name
Text Analytics Key Phrase Extraction mcr.microsoft.com/azure-cognitive-services/textanalytics/keyphrase
Text Analytics Language Detection mcr.microsoft.com/azure-cognitive-services/textanalytics/language

"Gated" preview

Previously, gated preview containers were hosted on the containerpreview.azurecr.io repository. Starting September 22nd 2020, these containers (except Text Analytics for health) are hosted on the Microsoft Container Registry (MCR), and downloading them doesn't require using the docker login command. To use the container you will need to:

  1. Complete a request form with your Azure Subscription ID and user scenario.
  2. Upon approval, download the container from the MCR.
  3. Use the key and endpoint from an appropriate Azure resource to authenticate the container at runtime.
Service Container Container Registry / Repository / Image Name
Computer Vision Read v3.0 mcr.microsoft.com/azure-cognitive-services/vision/read:3.0-preview
Computer Vision Read v3.1 mcr.microsoft.com/azure-cognitive-services/vision/read:3.1-preview
Computer Vision Spatial Analysis mcr.microsoft.com/azure-cognitive-services/vision/spatial-analysis
Speech Service API Custom Text-to-speech mcr.microsoft.com/azure-cognitive-services/speechservices/custom-text-to-speech
Speech Service API Language Detection mcr.microsoft.com/azure-cognitive-services/speechservices/language-detection
Speech Service API Neural Text-to-speech mcr.microsoft.com/azure-cognitive-services/speechservices/neural-text-to-speech
Text Analytics for health Text Analytics for health containerpreview.azurecr.io/microsoft/cognitive-services-healthcare

Prerequisites

You must satisfy the following prerequisites before using Azure Cognitive Services containers:

Docker Engine: You must have Docker Engine installed locally. Docker provides packages that configure the Docker environment on macOS, Linux, and Windows. On Windows, Docker must be configured to support Linux containers. Docker containers can also be deployed directly to Azure Kubernetes Service or Azure Container Instances.

Docker must be configured to allow the containers to connect with and send billing data to Azure.

Familiarity with Microsoft Container Registry and Docker: You should have a basic understanding of both Microsoft Container Registry and Docker concepts, like registries, repositories, containers, and container images, as well as knowledge of basic docker commands.

For a primer on Docker and container basics, see the Docker overview.

Individual containers can have their own requirements, as well, including server and memory allocation requirements.

Azure Cognitive Services container security

Security should be a primary focus whenever you're developing applications. The importance of security is a metric for success. When you're architecting a software solution that includes Cognitive Services containers, it's vital to understand the limitations and capabilities available to you. For more information about network security, see Configure Azure Cognitive Services virtual networks.

Important

By default there is no security on the Cognitive Services container API. The reason for this is that most often the container will run as part of a pod which is protected from the outside by a network bridge. However, it is possible to enable authentication which works identically to the authentication used when accessing the cloud-based Cognitive Services.

The diagram below illustrates the default and non-secure approach:

Container security

As an alternative and secure approach, consumers of Cognitive Services containers could augment a container with a front-facing component, keeping the container endpoint private. Let's consider a scenario where we use Istio as an ingress gateway. Istio supports HTTPS/TLS and client-certificate authentication. In this scenario, the Istio frontend exposes the container access, presenting the client certificate that is approved beforehand with Istio.

Nginx is another popular choice in the same category. Both Istio and Nginx act as a service mesh and offer additional features including things like load-balancing, routing, and rate-control.

Container networking

The Cognitive Services containers are required to submit metering information for billing purposes. The only exception, is Offline containers as they follow a different billing methodology. Failure to allow list various network channels that the Cognitive Services containers rely on will prevent the container from working.

Allow list Cognitive Services domains and ports

The host should allow list port 443 and the following domains:

  • *.cognitive.microsoft.com
  • *.cognitiveservices.azure.com

Disable deep packet inspection

Deep packet inspection (DPI) is a type of data processing that inspects in detail the data being sent over a computer network, and usually takes action by blocking, re-routing, or logging it accordingly.

Disable DPI on the secure channels that the Cognitive Services containers create to Microsoft servers. Failure to do so will prevent the container from functioning correctly.

Blog posts

Developer samples

Developer samples are available at our GitHub repository.

View webinar

Join the webinar to learn about:

  • How to deploy Cognitive Services to any machine using Docker
  • How to deploy Cognitive Services to AKS

Next steps

Learn about container recipes you can use with the Cognitive Services.

Install and explore the functionality provided by containers in Azure Cognitive Services: