Azure AI Speech Service - No punctuation on Recognized return

David Strachan 26 Reputation points
2024-03-26T10:35:32.91+00:00

I am using the Azure AI speech recognition, which I think is configured correctly to punctuate en-GB. Currently, the recognized return is unpunctuated and comes out as straight text

_speechConfig = SpeechConfig.FromSubscription(Token, "uksouth");

 _speechConfig.OutputFormat = OutputFormat.Detailed;

 _speechConfig.SpeechRecognitionLanguage = "en-GB";

 _speechConfig.EnableDictation();

 _speechStream = AudioInputStream.CreatePushStream(AudioStreamFormat.GetWaveFormatPCM(16000, 16, 1));

 _audioConfig = AudioConfig.FromStreamInput(_speechStream);

 _speechRecognizer = new SpeechRecognizer(_speechConfig, _audioConfig);

 _speechRecognizer.Recognized += OnRecognized;

 _speechRecognizer.Recognizing += RecognizingAudio;

 await _speechRecognizer.StartContinuousRecognitionAsync();

The above configuration returns:

2024-03-22 18:14:34,962 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:34,964 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"04a63ad5a8a14de7aba4c3256dddb620","Text":"today i went","Offset":11500000,"Duration":10800000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:35,257 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:35,257 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"c53ea1614bff479f96df4847c306ea49","Text":"today i went to","Offset":11500000,"Duration":14000000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:35,555 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:35,556 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"f0b1bdbfea924ba2b100364d59522ce9","Text":"today i went to the","Offset":11500000,"Duration":16400000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:35,851 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:35,851 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"6081cf3af7cd4b8aa113f5b64d2b8d62","Text":"today i went to the shop","Offset":11500000,"Duration":16000000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:36,261 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:36,262 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"2923456f66544624b4adced94f6e4ca2","Text":"today i went to the shop i","Offset":11500000,"Duration":19600000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:36,461 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:36,461 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"787311418c484d57810cc19e3e1a6bbf","Text":"today i went to the shop i bought","Offset":11500000,"Duration":26000000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:36,858 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:36,858 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"da59bcd910314a0e9232bead20475b29","Text":"today i went to the shop i bought apples","Offset":11500000,"Duration":28400000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:37,206 [INFO] Namspace.Client.Windows.Transcription.TranscriptionTimer - Time elapsed: 00:00:05.0181346
2024-03-22 18:14:37,758 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:37,758 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"d27d1a08570d4454a4a010f6f7919352","Text":"today i went to the shop i bought apples pears","Offset":11500000,"Duration":35600000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:38,156 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:38,157 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"6d78a51f69cf4d90b8772f6b836a9264","Text":"today i went to the shop i bought apples pears ban","Offset":11500000,"Duration":42000000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:38,457 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:38,457 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"f97850bf361c402b84a37e64dc70133a","Text":"today i went to the shop i bought apples pears bananas","Offset":11500000,"Duration":44800000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:38,758 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:38,758 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"e34e6710aabe4e5d9178ae5e3cbedc6f","Text":"today i went to the shop i bought apples pears bananas and","Offset":11500000,"Duration":48000000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:39,059 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:39,060 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"c458e883f2a0465087fafe0555bae6b7","Text":"today i went to the shop i bought apples pears bananas and oranges","Offset":11500000,"Duration":53200000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:40,359 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:40,360 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"9c521b18be3f4dc3b12079be51307478","Text":"today i went to the shop i bought apples pears bananas and oranges do you","Offset":11500000,"Duration":62000000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:40,764 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:40,765 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"b6ff1bed23c5457caeca5da1108df74b","Text":"today i went to the shop i bought apples pears bananas and oranges do you think","Offset":11500000,"Duration":65600000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:40,960 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:40,960 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"2010d0d4c8224c4a84510354b81c4aee","Text":"today i went to the shop i bought apples pears bananas and oranges do you think i missed","Offset":11500000,"Duration":71200000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:41,362 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizingSpeech
2024-03-22 18:14:41,362 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"1cefaace3a2e4448993f4a5838349992","Text":"today i went to the shop i bought apples pears bananas and oranges do you think i missed anything","Offset":11500000,"Duration":75200000,"PrimaryLanguage":{"Language":"en-GB"},"Channel":0}
2024-03-22 18:14:41,941 [INFO] Namspace.Clients.AzureTranscriptionClient - RecognizedSpeech
2024-03-22 18:14:41,941 [INFO] Namspace.Clients.AzureTranscriptionClient - {"Id":"49510f5baedf4aaca02a3620bbc7cb38","RecognitionStatus":"Success","Offset":11500000,"Duration":75200000,"Channel":0,"DisplayText":"Today I went to the shop I bought apples pears bananas and oranges do you think I missed anything","NBest":[{"Confidence":0.8908741,"Lexical":"today i went to the shop i bought apples pears bananas and oranges do you think i missed anything","ITN":"today i went to the shop i bought apples pears bananas and oranges do you think i missed anything","MaskedITN":"today i went to the shop i bought apples pears bananas and oranges do you think i missed anything","Display":"Today I went to the shop I bought apples pears bananas and oranges do you think I missed anything","Words":[{"Word":"today","Offset":11500000,"Duration":5200000,"Confidence":0.797704},{"Word":"i","Offset":16700000,"Duration":400000,"Confidence":0.9929609},{"Word":"went","Offset":17100000,"Duration":3200000,"Confidence":0.9909313},{"Word":"to","Offset":20300000,"Duration":800000,"Confidence":0.85922694},{"Word":"the","Offset":21100000,"Duration":1200000,"Confidence":0.9740201},{"Word":"shop","Offset":22300000,"Duration":5200000,"Confidence":0.99293035},{"Word":"i","Offset":29100000,"Duration":400000,"Confidence":0.7901372},{"Word":"bought","Offset":29500000,"Duration":3200000,"Confidence":0.4641154},{"Word":"apples","Offset":32700000,"Duration":7200000,"Confidence":0.5474731},{"Word":"pears","Offset":39900000,"Duration":7200000,"Confidence":0.68669873},{"Word":"bananas","Offset":47500000,"Duration":8800000,"Confidence":0.9829177},{"Word":"and","Offset":56700000,"Duration":2800000,"Confidence":0.9852306},{"Word":"oranges","Offset":59500000,"Duration":7200000,"Confidence":0.9747303},{"Word":"do","Offset":69900000,"Duration":2000000,"Confidence":0.9733427},{"Word":"you","Offset":71900000,"Duration":1200000,"Confidence":0.96302515},{"Word":"think","Offset":73100000,"Duration":3200000,"Confidence":0.99290127},{"Word":"i","Offset":76300000,"Duration":400000,"Confidence":0.989389},{"Word":"missed","Offset":76700000,"Duration":3600000,"Confidence":0.9898689},{"Word":"anything","Offset":80300000,"Duration":6400000,"Confidence":0.97900313}]},{"Confidence":0.88955027,"Lexical":"today i went to the shop i brought apples pears bananas and oranges do you think i missed anything","ITN":"today i went to the shop i brought apples pears bananas and oranges do you think i missed anything","MaskedITN":"today i went to the shop i brought apples pears bananas and oranges do you think i missed anything","Display":"today i went to the shop i brought apples pears bananas and oranges do you think i missed anything","Words":[{"Word":"today","Offset":11500000,"Duration":5200000,"Confidence":0.797704},{"Word":"i","Offset":16700000,"Duration":400000,"Confidence":0.9929609},{"Word":"went","Offset":17100000,"Duration":3200000,"Confidence":0.9909313},{"Word":"to","Offset":20300000,"Duration":800000,"Confidence":0.85922694},{"Word":"the","Offset":21100000,"Duration":1200000,"Confidence":0.9740201},{"Word":"shop","Offset":22300000,"Duration":5200000,"Confidence":0.99293035},{"Word":"i","Offset":28700000,"Duration":400000,"Confidence":0.7901372},{"Word":"brought","Offset":29100000,"Duration":2800000,"Confidence":0.022906482},{"Word":"apples","Offset":31900000,"Duration":8000000,"Confidence":0.97443783},{"Word":"pears","Offset":39900000,"Duration":7200000,"Confidence":0.6758339},{"Word":"bananas","Offset":47500000,"Duration":8800000,"Confidence":0.98287463},{"Word":"and","Offset":56700000,"Duration":2800000,"Confidence":0.9852306},{"Word":"oranges","Offset":59500000,"Duration":7200000,"Confidence":0.9747303},{"Word":"do","Offset":69900000,"Duration":2000000,"Confidence":0.9733427},{"Word":"you","Offset":71900000,"Duration":1200000,"Confidence":0.96302515},{"Word":"think","Offset":73100000,"Duration":3200000,"Confidence":0.99290127},{"Word":"i","Offset":76300000,"Duration":400000,"Confidence":0.989389},{"Word":"missed","Offset":76700000,"Duration":3600000,"Confidence":0.9898689},{"Word":"anything","Offset":80300000,"Duration":6400000,"Confidence":0.97900313}]},{"Confidence":0.8071691,"Lexical":"the day i went to the shop i bought apples pears bananas and oranges do you think i missed anything","ITN":"the day i went to the shop i bought apples pears bananas and oranges do you think i missed anything","MaskedITN":"the day i went to the shop i bought apples pears bananas and oranges do you think i missed anything","Display":"the day i went to the shop i bought apples pears bananas and oranges do you think i missed anything","Words":[{"Word":"the","Offset":11500000,"Duration":1600000,"Confidence":0.0040036854},{"Word":"day","Offset":13100000,"Duration":3600000,"Confidence":0.0051958677},{"Word":"i","Offset":16700000,"Duration":400000,"Confidence":0.992046},{"Word":"went","Offset":17100000,"Duration":3200000,"Confidence":0.99208415},{"Word":"to","Offset":20300000,"Duration":800000,"Confidence":0.86416095},{"Word":"the","Offset":21100000,"Duration":1200000,"Confidence":0.97456455},{"Word":"shop","Offset":22300000,"Duration":5200000,"Confidence":0.99249375},{"Word":"i","Offset":29100000,"Duration":400000,"Confidence":0.7901372},{"Word":"bought","Offset":29500000,"Duration":3200000,"Confidence":0.4641154},{"Word":"apples","Offset":32700000,"Duration":7200000,"Confidence":0.5474731},{"Word":"pears","Offset":39900000,"Duration":7200000,"Confidence":0.68669873},{"Word":"bananas","Offset":47500000,"Duration":8800000,"Confidence":0.9829177},{"Word":"and","Offset":56700000,"Duration":2800000,"Confidence":0.9852306},{"Word":"oranges","Offset":59500000,"Duration":7200000,"Confidence":0.9747303},{"Word":"do","Offset":69900000,"Duration":2000000,"Confidence":0.9733427},{"Word":"you","Offset":71900000,"Duration":1200000,"Confidence":0.96302515},{"Word":"think","Offset":73100000,"Duration":3200000,"Confidence":0.99290127},{"Word":"i","Offset":76300000,"Duration":400000,"Confidence":0.989389},{"Word":"missed","Offset":76700000,"Duration":3600000,"Confidence":0.9898689},{"Word":"anything","Offset":80300000,"Duration":6400000,"Confidence":0.97900313}]},{"Confidence":0.85787326,"Lexical":"today i went to the shop and i bought apples pears bananas and oranges do you think i missed anything","ITN":"today i went to the shop and i bought apples pears bananas and oranges do you think i missed anything","MaskedITN":"today i went to the shop and i bought apples pears bananas and oranges do you think i missed anything","Display":"today i went to the shop and i bought apples pears bananas and oranges do you think i missed anything","Words":[{"Word":"today","Offset":11500000,"Duration":5200000,"Confidence":0.797704},{"Word":"i","Offset":16700000,"Duration":400000,"Confidence":0.9929609},{"Word":"went","Offset":17100000,"Duration":3200000,"Confidence":0.9909313},{"Word":"to","Offset":20300000,"Duration":800000,"Confidence":0.85922694},{"Word":"the","Offset":21100000,"Duration":1200000,"Confidence":0.9740201},{"Word":"shop","Offset":22300000,"Duration":5200000,"Confidence":0.99293035},{"Word":"and","Offset":28300000,"Duration":1600000,"Confidence":0.0032146487},{"Word":"i","Offset":29900000,"Duration":400000,"Confidence":0.64129204},{"Word":"bought","Offset":30300000,"Duration":2400000,"Confidence":0.4008528},{"Word":"apples","Offset":32700000,"Duration":7200000,"Confidence":0.98722744},{"Word":"pears","Offset":39900000,"Duration":7200000,"Confidence":0.6866962},{"Word":"bananas","Offset":47500000,"Duration":8800000,"Confidence":0.9829177},{"Word":"and","Offset":56700000,"Duration":2800000,"Confidence":0.9852306},{"Word":"oranges","Offset":59500000,"Duration":7200000,"Confidence":0.9747303},{"Word":"do","Offset":69900000,"Duration":2000000,"Confidence":0.9733427},{"Word":"you","Offset":71900000,"Duration":1200000,"Confidence":0.96302515},{"Word":"think","Offset":73100000,"Duration":3200000,"Confidence":0.99290127},{"Word":"i","Offset":76300000,"Duration":400000,"Confidence":0.989389},{"Word":"missed","Offset":76700000,"Duration":3600000,"Confidence":0.9898689},{"Word":"anything","Offset":80300000,"Duration":6400000,"Confidence":0.97900313}]},{"Confidence":0.8059115,"Lexical":"the day i went to the shop i brought apples pears bananas and oranges do you think i missed anything","ITN":"the day i went to the shop i brought apples pears bananas and oranges do you think i missed anything","MaskedITN":"the day i went to the shop i brought apples pears bananas and oranges do you think i missed anything","Display":"the day i went to the shop i brought apples pears bananas and oranges do you think i missed anything","Words":[{"Word":"the","Offset":11500000,"Duration":1600000,"Confidence":0.0040036854},{"Word":"day","Offset":13100000,"Duration":3600000,"Confidence":0.0051958677},{"Word":"i","Offset":16700000,"Duration":400000,"Confidence":0.992046},{"Word":"went","Offset":17100000,"Duration":3200000,"Confidence":0.99208415},{"Word":"to","Offset":20300000,"Duration":800000,"Confidence":0.86416095},{"Word":"the","Offset":21100000,"Duration":1200000,"Confidence":0.97456455},{"Word":"shop","Offset":22300000,"Duration":5200000,"Confidence":0.99249375},{"Word":"i","Offset":28700000,"Duration":400000,"Confidence":0.7901372},{"Word":"brought","Offset":29100000,"Duration":2800000,"Confidence":0.022906482},{"Word":"apples","Offset":31900000,"Duration":8000000,"Confidence":0.97443783},{"Word":"pears","Offset":39900000,"Duration":7200000,"Confidence":0.6758339},{"Word":"bananas","Offset":47500000,"Duration":8800000,"Confidence":0.98287463},{"Word":"and","Offset":56700000,"Duration":2800000,"Confidence":0.9852306},{"Word":"oranges","Offset":59500000,"Duration":7200000,"Confidence":0.9747303},{"Word":"do","Offset":69900000,"Duration":2000000,"Confidence":0.9733427},{"Word":"you","Offset":71900000,"Duration":1200000,"Confidence":0.96302515},{"Word":"think","Offset":73100000,"Duration":3200000,"Confidence":0.99290127},{"Word":"i","Offset":76300000,"Duration":400000,"Confidence":0.989389},{"Word":"missed","Offset":76700000,"Duration":3600000,"Confidence":0.9898689},{"Word":"anything","Offset":80300000,"Duration":6400000,"Confidence":0.97900313}]}]}
2024-03-22 18:14:41,943 [DEBUG] Namspace.Client.Windows.Transcription.TranscriptionFacade - Received transcription response

I found https://github.com/microsoft/cognitive-services-speech-sdk-js/issues/686 and used the HTML file to validate the settings. The HTML file provided here does indeed return Recognized as expected.

Expected: "Today I went to the shop. I bought apples, pears, bananas, and oranges. Do you think I missed anything?"

Output from C#: "Today I went to the shop I bought apples pears bananas and oranges do you think I missed anything"

Can anyone spot the probably obvious missing piece?

Azure AI Speech
Azure AI Speech
An Azure service that integrates speech processing into apps and services.
1,393 questions
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
947 questions
C#
C#
An object-oriented and type-safe programming language that has its roots in the C family of languages and includes support for component-oriented programming.
10,249 questions
0 comments
Accepted answer
  1. dupammi 6,390 Reputation points Microsoft Vendor
    2024-03-26T11:36:44.4833333+00:00

    Hi @David Strachan

    Thank you for using the Microsoft Q&A forum.

    To ensure better preservation of punctuation in your transcription, you can disable dictation mode and use a custom language model that is optimized for recognizing punctuation.

    By removing speech_config.enable_dictation(), the speech recognizer will no longer prioritize transcribing spoken language without necessarily preserving punctuation. This might improve punctuation recognition in your transcription.

    Below is the repro output I got by removing the enable_dictation :
    User's image

    Note: For complex use cases, you may need to post-process the recognized text to insert missing punctuation based on language rules or contextual understanding, if necessary.

    Hope this helps.

    If this answers your query, do click Accept Answer and Yes for was this answer helpful.

    1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest