Install and run Docker containers for the Speech service APIs

Docker pull for the Speech-to-text container

Use the docker pull command to download a container image from Microsoft Container registry.

docker pull mcr.microsoft.com/azure-cognitive-services/speechservices/speech-to-text:latest

Speech-to-text locales

All tags, except for latest are in the following format and are case-sensitive:

<major>.<minor>.<patch>-<platform>-<locale>-<prerelease>

The following tag is an example of the format:

2.6.0-amd64-en-us

For all of the supported locales of the speech-to-text container, please see Speech-to-text image tags.

Docker pull for the Custom Speech-to-text container

Use the docker pull command to download a container image from Microsoft Container registry.

docker pull mcr.microsoft.com/azure-cognitive-services/speechservices/custom-speech-to-text:latest

Docker pull for the Text-to-speech container

Use the docker pull command to download a container image from Microsoft Container registry.

docker pull mcr.microsoft.com/azure-cognitive-services/speechservices/text-to-speech:latest

Text-to-speech locales

All tags, except for latest are in the following format and are case-sensitive:

<major>.<minor>.<patch>-<platform>-<locale>-<voice>-<prerelease>

The following tag is an example of the format:

1.8.0-amd64-en-us-ariarus

For all of the supported locales and corresponding voices of the text-to-speech container, please see Text-to-speech image tags.

Docker pull for the Neural Text-to-speech container

Use the docker pull command to download a container image from Microsoft Container registry.

docker pull mcr.microsoft.com/azure-cognitive-services/speechservices/neural-text-to-speech:latest

Neural Text-to-speech locales

All tags, except for latest are in the following format and are case-sensitive:

<major>.<minor>.<patch>-<platform>-<locale>-<voice>

The following tag is an example of the format:

1.3.0-amd64-en-us-arianeural

For all of the supported locales and corresponding voices of the neural text-to-speech container, please see Neural Text-to-speech image tags.

Docker pull for the Custom Text-to-speech container

Use the docker pull command to download a container image from Microsoft Container registry.

docker pull mcr.microsoft.com/azure-cognitive-services/speechservices/custom-text-to-speech:latest

Docker pull for the Speech Language Detection container

Use the docker pull command to download a container image from Microsoft Container registry.

docker pull mcr.microsoft.com/azure-cognitive-services/speechservices/language-detection:latest

To run the Standard Speech-to-text container, execute the following docker run command.

docker run --rm -it -p 5000:5000 --memory 4g --cpus 4 \
mcr.microsoft.com/azure-cognitive-services/speechservices/speech-to-text \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY}

This command:

• Runs a Speech-to-text container from the container image.
• Allocates 4 CPU cores and 4 gigabytes (GB) of memory.
• Exposes TCP port 5000 and allocates a pseudo-TTY for the container.
• Automatically removes the container after it exits. The container image is still available on the host computer.

Diarization on the speech-to-text output

Diarization is enabled by default. to get diarization in your response, use diarize_speech_config.set_service_property.

1. Set the phrase output format to Detailed.
2. Set the mode of diarization. The supported modes are Identity and Anonymous.

diarize_speech_config.set_service_property(
    name='speechcontext-PhraseOutput.Format',
    value='Detailed',
    channel=speechsdk.ServicePropertyChannel.UriQueryParameter
)

diarize_speech_config.set_service_property(
    name='speechcontext-phraseDetection.speakerDiarization.mode',
    value='Identity',
    channel=speechsdk.ServicePropertyChannel.UriQueryParameter
)

Analyze sentiment on the speech-to-text output

Starting in v2.6.0 of the speech-to-text container, you should use TextAnalytics 3.0 API endpoint instead of the preview one. For example

• https://westus2.api.cognitive.microsoft.com/text/analytics/v3.0/sentiment
• https://localhost:5000/text/analytics/v3.0/sentiment

Starting in v2.2.0 of the speech-to-text container, you can call the sentiment analysis v3 API on the output. To call sentiment analysis, you will need a Text Analytics API resource endpoint. For example:

• https://westus2.api.cognitive.microsoft.com/text/analytics/v3.0-preview.1/sentiment
• https://localhost:5000/text/analytics/v3.0-preview.1/sentiment

If you're accessing a Text analytics endpoint in the cloud, you will need a key. If you're running Text Analytics locally, you may not need to provide this.

The key and endpoint are passed to the Speech container as arguments, as in the following example.

docker run -it --rm -p 5000:5000 \
mcr.microsoft.com/azure-cognitive-services/speechservices/speech-to-text:latest \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY} \
CloudAI:SentimentAnalysisSettings:TextAnalyticsHost={TEXT_ANALYTICS_HOST} \
CloudAI:SentimentAnalysisSettings:SentimentAnalysisApiKey={SENTIMENT_APIKEY}

This command:

• Performs the same steps as the command above.
• Stores a Text Analytics API endpoint and key, for sending sentiment analysis requests.

Phraselist v2 on the speech-to-text output

Starting in v2.6.0 of the speech-to-text container, you can get the output with your own phrases - either the whole sentence, or phrases in the middle. For example the tall man in the following sentence:

• "This is a sentence the tall man this is another sentence."

To configure a phrase list, you need to add your own phrases when you make the call. For example:

    phrase="the tall man"
    recognizer = speechsdk.SpeechRecognizer(
        speech_config=dict_speech_config,
        audio_config=audio_config)
    phrase_list_grammer = speechsdk.PhraseListGrammar.from_recognizer(recognizer)
    phrase_list_grammer.addPhrase(phrase)
    
    dict_speech_config.set_service_property(
        name='setflight',
        value='xonlineinterp',
        channel=speechsdk.ServicePropertyChannel.UriQueryParameter
    )

If you have multiple phrases to add, call .addPhrase() for each phrase to add it to the phrase list.

The Custom Speech-to-text container relies on a custom speech model. The custom model has to have been trained using the custom speech portal.

The custom speech Model ID is required to run the container. It can be found on the Training page of the custom speech portal. From the custom speech portal, navigate to the Training page and select the model.

Obtain the Model ID to use as the argument to the ModelId parameter of the docker run command.

The following table represents the various docker run parameters and their corresponding descriptions:

To run the Custom Speech-to-text container, execute the following docker run command:

docker run --rm -it -p 5000:5000 --memory 4g --cpus 4 \
-v {VOLUME_MOUNT}:/usr/local/models \
mcr.microsoft.com/azure-cognitive-services/speechservices/custom-speech-to-text \
ModelId={MODEL_ID} \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY}

This command:

• Runs a Custom Speech-to-text container from the container image.
• Allocates 4 CPU cores and 4 gigabytes (GB) of memory.
• Loads the Custom Speech-to-Text model from the volume input mount, for example C:\CustomSpeech.
• Exposes TCP port 5000 and allocates a pseudo-TTY for the container.
• Downloads the model given the ModelId (if not found on the volume mount).
• If the custom model was previously downloaded, the ModelId is ignored.
• Automatically removes the container after it exits. The container image is still available on the host computer.

Base model download on the custom speech-to-text container

Starting in v2.6.0 of the custom-speech-to-text container, you can get the available base model information by using option BaseModelLocale=<locale>. This option will give you a list of available base models on that locale under your billing account. For example:

docker run --rm -it \
mcr.microsoft.com/azure-cognitive-services/speechservices/custom-speech-to-text \
BaseModelLocale={LOCALE} \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY}

This command:

• Runs a Custom Speech-to-text container from the container image.
• Check and return the available base models of the target locale.

The output gives you a list of base models with the information locale, model ID, and creation date time. You can use the model ID to download and use the specific base model you prefer. For example:

Checking available base model for en-us
2020/10/30 21:54:20 [Info] Searching available base models for en-us
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2016-11-04T08:23:42Z, Id: a3d8aab9-6f36-44cd-9904-b37389ce2bfa
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2016-11-04T12:01:02Z, Id: cc7826ac-5355-471d-9bc6-a54673d06e45
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2017-08-17T12:00:00Z, Id: a1f8db59-40ff-4f0e-b011-37629c3a1a53
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2018-04-16T11:55:00Z, Id: c7a69da3-27de-4a4b-ab75-b6716f6321e5
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2018-09-21T15:18:43Z, Id: da494a53-0dad-4158-b15f-8f9daca7a412
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2018-10-19T11:28:54Z, Id: 84ec130b-d047-44bf-a46d-58c1ac292ca7
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2018-11-26T07:59:09Z, Id: ee5c100f-152f-4ae5-9e9d-014af3c01c56
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2018-11-26T09:21:55Z, Id: d04959a6-71da-4913-9997-836793e3c115
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2019-01-11T10:04:19Z, Id: 488e5f23-8bc5-46f8-9ad8-ea9a49a8efda
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2019-02-18T14:37:57Z, Id: 0207b3e6-92a8-4363-8c0e-361114cdd719
2020/10/30 21:54:21 [Info] [Base model] Locale: en-us, CreatedDate: 2019-03-03T17:34:10Z, Id: 198d9b79-2950-4609-b6ec-f52254074a05
2020/10/30 21:54:21 [Fatal] Please run this tool again and assign --modelId '<one above base model id>'. If no model id listed above, it means currently there is no available base model for en-us

Custom pronunciation on the custom speech-to-text container

Starting in v2.5.0 of the custom-speech-to-text container, you can get custom pronunciation result in the output. All you need to do is to have your own custom pronunciation rules set up in your custom model and mount the model to custom-speech-to-text container.

To run the Standard Text-to-speech container, execute the following docker run command.

docker run --rm -it -p 5000:5000 --memory 2g --cpus 1 \
mcr.microsoft.com/azure-cognitive-services/speechservices/text-to-speech \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY}

This command:

• Runs a Standard Text-to-speech container from the container image.
• Allocates 1 CPU core and 2 gigabytes (GB) of memory.
• Exposes TCP port 5000 and allocates a pseudo-TTY for the container.
• Automatically removes the container after it exits. The container image is still available on the host computer.

To run the Neural Text-to-speech container, execute the following docker run command.

docker run --rm -it -p 5000:5000 --memory 12g --cpus 6 \
mcr.microsoft.com/azure-cognitive-services/speechservices/neural-text-to-speech \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY}

This command:

• Runs a Neural Text-to-speech container from the container image.
• Allocates 6 CPU cores and 12 gigabytes (GB) of memory.
• Exposes TCP port 5000 and allocates a pseudo-TTY for the container.
• Automatically removes the container after it exits. The container image is still available on the host computer.

The Custom Text-to-speech container relies on a custom voice model. The custom model has to have been trained using the custom voice portal. The custom voice Model ID is required to run the container. It can be found on the Training page of the custom voice portal. From the custom voice portal, navigate to the Training page and select the model.

Obtain the Model ID to use as the argument to the ModelId parameter of the docker run command.

The following table represents the various docker run parameters and their corresponding descriptions:

To run the Custom Text-to-speech container, execute the following docker run command:

docker run --rm -it -p 5000:5000 --memory 2g --cpus 1 \
-v {VOLUME_MOUNT}:/usr/local/models \
mcr.microsoft.com/azure-cognitive-services/speechservices/custom-text-to-speech \
ModelId={MODEL_ID} \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY}

This command:

• Runs a Custom Text-to-speech container from the container image.
• Allocates 1 CPU core and 2 gigabytes (GB) of memory.
• Loads the Custom Text-to-speech model from the volume input mount, for example C:\CustomVoice.
• Exposes TCP port 5000 and allocates a pseudo-TTY for the container.
• Downloads the model given the ModelId (if not found on the volume mount).
• If the custom model was previously downloaded, the ModelId is ignored.
• Automatically removes the container after it exits. The container image is still available on the host computer.

To run the Speech Language Detection container, execute the following docker run command.

docker run --rm -it -p 5003:5003 --memory 1g --cpus 1 \
mcr.microsoft.com/azure-cognitive-services/speechservices/language-detection \
Eula=accept \
Billing={ENDPOINT_URI} \
ApiKey={API_KEY}

This command:

• Runs a speech language-detection container from the container image. Currently you will not be charged for running this image.
• Allocates 1 CPU cores and 1 gigabyte (GB) of memory.
• Exposes TCP port 5003 and allocates a pseudo-TTY for the container.
• Automatically removes the container after it exits. The container image is still available on the host computer.

If you're sending only Speech Language Detection requests, you will need to set the Speech client's phraseDetection value to None.

speech_config.set_service_property(
      name='speechcontext-phraseDetection.Mode',
      value='None',
      channel=speechsdk.ServicePropertyChannel.UriQueryParameter
   )

If you want to run this container with the speech-to-text container, you can use this Docker image. After both containers have been started, use this Docker Run command to execute speech-to-text-with-languagedetection-client.

docker run --rm -v ${HOME}:/root -ti antsu/on-prem-client:latest ./speech-to-text-with-languagedetection-client ./audio/LanguageDetection_en-us.wav --host localhost --lport 5003 --sport 5000

Change from using this Azure-cloud initialization call:

var config = SpeechConfig.FromSubscription("YourSubscriptionKey", "YourServiceRegion");

To using this call with the container host:

var config = SpeechConfig.FromHost(
    new Uri("ws://localhost:5000"));

Change from using this Azure-cloud initialization call:

speech_config = speechsdk.SpeechConfig(
    subscription=speech_key, region=service_region)

To using this call with the container endpoint:

speech_config = speechsdk.SpeechConfig(
    endpoint="ws://localhost:5000/speech/recognition/conversation/cognitiveservices/v1"

To configure the Speech client to use a simple format, add "Sentiment" as a value for Simple.Extensions. If you want to choose a specific Text Analytics model version, replace 'latest' in the speechcontext-phraseDetection.sentimentAnalysis.modelversion property configuration.

speech_config.set_service_property(
    name='speechcontext-PhraseOutput.Simple.Extensions',
    value='["Sentiment"]',
    channel=speechsdk.ServicePropertyChannel.UriQueryParameter
)
speech_config.set_service_property(
    name='speechcontext-phraseDetection.sentimentAnalysis.modelversion',
    value='latest',
    channel=speechsdk.ServicePropertyChannel.UriQueryParameter
)

Simple.Extensions will return the sentiment result in root layer of the response.

{
   "DisplayText":"What's the weather like?",
   "Duration":13000000,
   "Id":"6098574b79434bd4849fee7e0a50f22e",
   "Offset":4700000,
   "RecognitionStatus":"Success",
   "Sentiment":{
      "Negative":0.03,
      "Neutral":0.79,
      "Positive":0.18
   }
}

To configure the Speech client to use a detailed format, add "Sentiment" as a value for Detailed.Extensions, Detailed.Options, or both. If you want to choose a specific Text Analytics model version, replace 'latest' in the speechcontext-phraseDetection.sentimentAnalysis.modelversion property configuration.

speech_config.set_service_property(
    name='speechcontext-PhraseOutput.Detailed.Options',
    value='["Sentiment"]',
    channel=speechsdk.ServicePropertyChannel.UriQueryParameter
)
speech_config.set_service_property(
    name='speechcontext-PhraseOutput.Detailed.Extensions',
    value='["Sentiment"]',
    channel=speechsdk.ServicePropertyChannel.UriQueryParameter
)
speech_config.set_service_property(
    name='speechcontext-phraseDetection.sentimentAnalysis.modelversion',
    value='latest',
    channel=speechsdk.ServicePropertyChannel.UriQueryParameter
)

Detailed.Extensions provides sentiment result in the root layer of the response. Detailed.Options provides the result in NBest layer of the response. They can be used separately or together.

{
   "DisplayText":"What's the weather like?",
   "Duration":13000000,
   "Id":"6a2aac009b9743d8a47794f3e81f7963",
   "NBest":[
      {
         "Confidence":0.973695,
         "Display":"What's the weather like?",
         "ITN":"what's the weather like",
         "Lexical":"what's the weather like",
         "MaskedITN":"What's the weather like",
         "Sentiment":{
            "Negative":0.03,
            "Neutral":0.79,
            "Positive":0.18
         }
      },
      {
         "Confidence":0.9164971,
         "Display":"What is the weather like?",
         "ITN":"what is the weather like",
         "Lexical":"what is the weather like",
         "MaskedITN":"What is the weather like",
         "Sentiment":{
            "Negative":0.02,
            "Neutral":0.88,
            "Positive":0.1
         }
      }
   ],
   "Offset":4700000,
   "RecognitionStatus":"Success",
   "Sentiment":{
      "Negative":0.03,
      "Neutral":0.79,
      "Positive":0.18
   }
}