question

AbdulmalikAlzubi-2301 avatar image
1 Vote"
AbdulmalikAlzubi-2301 asked PaulRyan-4569 answered

LUIS and Custom Speech C#

I've been trying to program LUIS and Custom Speech together for a few days, but it didn't work. I can only use the two separately, but that doesn't help, because I want the intentions of the already trained model to be investigated. You can find my solution below.
//Erstellt eine Instanz einer Sprachkonfiguration mit dem angegebenen Abonnementschlüssel und der Dienstregion.

            var config = SpeechConfig.FromSubscription("", "");

            //Um die Zahlen richtig abzulesen wird diese Methode der Klasse Seechkonfig aufgerufen. Dadurch wird die Ausgabe detaliert sein.

            config.SetProperty("SpeechServiceResponse_OutputFormatOption", "detailed");

 

            //Erstellt eine Instanz eines Schlüsselworterkennungsmodells.

            var model = KeywordRecognitionModel.FromFile(@"");

            //Anerkennungssprache

            config.SpeechRecognitionLanguage = "de-DE";

            config.EndpointId = "";

 

           

            var stopRecognition = new TaskCompletionSource<int>();

           

            // Creates a speech recognizer using microphone as audio input.

            // Creates a Language Understanding model using the app id, and adds specific intents from your model

            var model_LUIS = LanguageUnderstandingModel.FromAppId("");

            //Erstellt einen Spracherkenner mit Mikrofon als Audioeingang.

           

            using var recognizerf = new SpeechRecognizer(config);

            using var recognizer = new IntentRecognizer(config);

 

            recognizer.AddAllIntents(model_LUIS);

            recognizer.Recognized += (s, e) =>

                {

                    if (e.Result.Reason == ResultReason.RecognizedIntent)

                    {

                        SpeechServiceText = e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult); ;

                            //Hier wird der Text der Microsoft Azure was das Json Format enthält an die Klasse LUIS übergeben um bearbeitet zu werden

                            luis.LUISAusgabe(e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult));

                    }

                    else if (e.Result.Reason == ResultReason.RecognizedSpeech)

                    {

                        SpeechServiceText = ($"RECOGNIZED: Text={e.Result.Text}");

                    }

                    else if (e.Result.Reason == ResultReason.NoMatch)

                    {

                        SpeechServiceText = ($"NOMATCH: Speech could not be recognized.");

                    }

                };

            recognizer.Canceled += (s, e) =>

            {

                SpeechServiceText = $"CANCELED: Reason=e.Reason{e.Reason}";

 

                if (e.Reason == CancellationReason.Error)

                {

                    SpeechServiceText = $"CANCELED: ErrorCode={e.ErrorCode}";

                    SpeechServiceText = $"CANCELED: ErrorDetails={e.ErrorDetails}";

                    SpeechServiceText = $"CANCELED: Did you update the subscription info?";

                }

                stopRecognition.TrySetResult(0);

            };

            recognizer.SessionStopped += (s, e) =>

            {

                stopRecognition.TrySetResult(0);

            };

 

            // Starts continuous recognition using the keyword model.

            await recognizer.StartKeywordRecognitionAsync(model).ConfigureAwait(false);

            await recognizerf.StartKeywordRecognitionAsync(model).ConfigureAwait(false);

            // Waits for a single successful keyword-triggered speech recognition (or error).

            // Use Task.WaitAny to keep the task rooted.

            Task.WaitAny(new[] { stopRecognition.Task });

 

            // Stops recognition.

            await recognizer.StopKeywordRecognitionAsync().ConfigureAwait(false);

azure-cognitive-servicesazure-speechazure-language-understanding
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@AbdulmalikAlzubi-2301 Thanks for the question. Can you please add more details about the document/sample that you are trying.

Please follow the document for Cognitive service custom speech: https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/quickstart-custom-commands-application

0 Votes 0 ·

@ramr-msft Thank you for your answer.

First of all, I try to train a custom speech model in order to convert the speech signals into texts as well as possible. I also want to use the trained model to understand the user's intention. I would like to use the already trained Custom Speech model directly with LUIS LUIS

0 Votes 0 ·
PaulRyan-4569 avatar image
1 Vote"
PaulRyan-4569 answered AbdulmalikAlzubi-2301 commented

If I understand your question, you want to use voice with LUIS, Microsoft docs has an example (search "c# luis voice intent example")
https://docs.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-recognize-intents-from-speech-csharp


here is an example of a console app (Net 5.0) that uses continuous speech,
it captures the string and json output.
You get the appID, key region and Intent Name form LUIS

string intentResultString ="";
string intentJSON = "";
var config = SpeechConfig.FromSubscription("key", "westus");

using (var recognizer = new IntentRecognizer(config))
{
var model = LanguageUnderstandingModel.FromAppId("AppID");
recognizer.AddIntent(model, "Intent Name", "id1");

         // Subscribes to events.
         recognizer.Recognizing += (s, e) =>
         {

         };

         recognizer.Recognized += (s, e) =>
         {
             intentResultString += e.Result.Text;
             intentJSON += e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult);
         };

         recognizer.Canceled += (s, e) =>
         {
             Console.WriteLine($"\n    Canceled. Reason: {e.Reason.ToString()}, CanceledReason: {e.Reason}");
         };

         recognizer.SessionStarted += (s, e) =>
         {
             Console.WriteLine("\n    Session started event.");
         };

         recognizer.SessionStopped += (s, e) =>
         {
                
             Console.WriteLine("\n    Session stopped event.");
         };

         // Starts continuous recognition. 
         // Uses StopContinuousRecognitionAsync() to stop recognition.
         await recognizer.StartContinuousRecognitionAsync().ConfigureAwait(false);

         do
         {

         } while (Console.ReadKey().Key != ConsoleKey.Enter);

         // Stops recognition.
         await recognizer.StopContinuousRecognitionAsync().ConfigureAwait(false);
     }

 }

 #endregion

}


hope this helps



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Forgot you need to enable "Speech pinning" when publishing LUIS

0 Votes 0 ·

Hello @PaulRyan-4569
Thank you for your answer. For me, it's about training a Speech Custom, which I've already done. That helps me to get the texts of the customers right. And then use LUIS to get the layers and keywords. You could use LUIS with speech to text and I would like to use my endpoint in the trained model of Speech cusom which enables better speech to text and thus better LUIS

0 Votes 0 ·
LeandroDuarte-8358 avatar image
0 Votes"
LeandroDuarte-8358 answered

Hello @PaulRyan-4569

I have been looking for a way to improve my chatbot architectury. Currently, my code has to send the user's speech to Custom Speech's endpoint firstly, after that my code sends the text generated by Custom Speech's endpoint to Luis for intention recognizer.

I'd like to connect Luis and Custom Speech directly and avoid to convert the user's speech in text firstly and then send the generetad text to Luis for intention recognition.

Have you already got any way to do that?

Would you like to think about it togheter?

Best regards!

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PaulRyan-4569 avatar image
1 Vote"
PaulRyan-4569 answered

Hi Leandro,
I am developing an app that uses voice-text and LUIS. I am using a two-step process
1) voice to text
2) Luis (using the text form voice-text) into a JSON


I created a CLI to test Voice with LUIS for some reason the results were not as accurate as using the two-step (I did not want to spend any time on figuring why I am getting different results

If you want the program here is a link (I keep them on OneDrive)
https://1drv.ms/u/s!Ai6m7WAwpO1kpqZ9PhOyBub8K1ZALA?e=ry0tNH

When you publish make sure you enable the voice option

my email is pryannow@outlook.com I am in EDT (New York)
good luck




5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.