LUIS and Custom Speech C#

Question

I've been trying to program LUIS and Custom Speech together for a few days, but it didn't work. I can only use the two separately, but that doesn't help, because I want the intentions of the already trained model to be investigated. You can find my solution below.
//Erstellt eine Instanz einer Sprachkonfiguration mit dem angegebenen Abonnementschlüssel und der Dienstregion.

            var config = SpeechConfig.FromSubscription("", "");

            //Um die Zahlen richtig abzulesen wird diese Methode der Klasse Seechkonfig aufgerufen. Dadurch wird die Ausgabe detaliert sein.

            config.SetProperty("SpeechServiceResponse_OutputFormatOption", "detailed");

 

            //Erstellt eine Instanz eines Schlüsselworterkennungsmodells.

            var model = KeywordRecognitionModel.FromFile(@"");

            //Anerkennungssprache

            config.SpeechRecognitionLanguage = "de-DE";

            config.EndpointId = "";

 

           

            var stopRecognition = new TaskCompletionSource<int>();

           

            // Creates a speech recognizer using microphone as audio input.

            // Creates a Language Understanding model using the app id, and adds specific intents from your model

            var model_LUIS = LanguageUnderstandingModel.FromAppId("");

            //Erstellt einen Spracherkenner mit Mikrofon als Audioeingang.

           

            using var recognizerf = new SpeechRecognizer(config);

            using var recognizer = new IntentRecognizer(config);

 

            recognizer.AddAllIntents(model_LUIS);

            recognizer.Recognized += (s, e) =>

                {

                    if (e.Result.Reason == ResultReason.RecognizedIntent)

                    {

                        SpeechServiceText = e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult); ;

                            //Hier wird der Text der Microsoft Azure was das Json Format enthält an die Klasse LUIS übergeben um bearbeitet zu werden

                            luis.LUISAusgabe(e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult));

                    }

                    else if (e.Result.Reason == ResultReason.RecognizedSpeech)

                    {

                        SpeechServiceText = ($"RECOGNIZED: Text={e.Result.Text}");

                    }

                    else if (e.Result.Reason == ResultReason.NoMatch)

                    {

                        SpeechServiceText = ($"NOMATCH: Speech could not be recognized.");

                    }

                };

            recognizer.Canceled += (s, e) =>

            {

                SpeechServiceText = $"CANCELED: Reason=e.Reason{e.Reason}";

 

                if (e.Reason == CancellationReason.Error)

                {

                    SpeechServiceText = $"CANCELED: ErrorCode={e.ErrorCode}";

                    SpeechServiceText = $"CANCELED: ErrorDetails={e.ErrorDetails}";

                    SpeechServiceText = $"CANCELED: Did you update the subscription info?";

                }

                stopRecognition.TrySetResult(0);

            };

            recognizer.SessionStopped += (s, e) =>

            {

                stopRecognition.TrySetResult(0);

            };

 

            // Starts continuous recognition using the keyword model.

            await recognizer.StartKeywordRecognitionAsync(model).ConfigureAwait(false);

            await recognizerf.StartKeywordRecognitionAsync(model).ConfigureAwait(false);

            // Waits for a single successful keyword-triggered speech recognition (or error).

            // Use Task.WaitAny to keep the task rooted.

            Task.WaitAny(new[] { stopRecognition.Task });

 

            // Stops recognition.

            await recognizer.StopKeywordRecognitionAsync().ConfigureAwait(false);

Answer 1

If I understand your question, you want to use voice with LUIS, Microsoft Learn has an example (search "c# luis voice intent example")
https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/how-to-recognize-intents-from-speech-csharp

here is an example of a console app (Net 5.0) that uses continuous speech,
it captures the string and json output.
You get the appID, key region and Intent Name form LUIS

string intentResultString ="";
string intentJSON = "";
var config = SpeechConfig.FromSubscription("key", "westus");

using (var recognizer = new IntentRecognizer(config))
{
var model = LanguageUnderstandingModel.FromAppId("AppID");
recognizer.AddIntent(model, "Intent Name", "id1");

        // Subscribes to events.  
        recognizer.Recognizing += (s, e) =>  
        {  

        };  

        recognizer.Recognized += (s, e) =>  
        {  
            intentResultString += e.Result.Text;  
            intentJSON += e.Result.Properties.GetProperty(PropertyId.LanguageUnderstandingServiceResponse_JsonResult);  
        };  

        recognizer.Canceled += (s, e) =>  
        {  
            Console.WriteLine($"\n    Canceled. Reason: {e.Reason.ToString()}, CanceledReason: {e.Reason}");  
        };  

        recognizer.SessionStarted += (s, e) =>  
        {  
            Console.WriteLine("\n    Session started event.");  
        };  

        recognizer.SessionStopped += (s, e) =>  
        {  
              
            Console.WriteLine("\n    Session stopped event.");  
        };  

        // Starts continuous recognition.   
        // Uses StopContinuousRecognitionAsync() to stop recognition.  
        await recognizer.StartContinuousRecognitionAsync().ConfigureAwait(false);  

        do  
        {  

        } while (Console.ReadKey().Key != ConsoleKey.Enter);  

        // Stops recognition.  
        await recognizer.StopContinuousRecognitionAsync().ConfigureAwait(false);  
    }  

}  

#endregion  

}

hope this helps

Answer 2

Hi Leandro,
I am developing an app that uses voice-text and LUIS. I am using a two-step process

1. voice to text
2. Luis (using the text form voice-text) into a JSON

I created a CLI to test Voice with LUIS for some reason the results were not as accurate as using the two-step (I did not want to spend any time on figuring why I am getting different results

If you want the program here is a link (I keep them on OneDrive)
https://1drv.ms/u/s!Ai6m7WAwpO1kpqZ9PhOyBub8K1ZALA?e=ry0tNH

When you publish make sure you enable the voice option

my email is pryannow@harsh.com .com I am in EDT (New York)
good luck

Answer 3

Hello @Paul Ryan

I have been looking for a way to improve my chatbot architectury. Currently, my code has to send the user's speech to Custom Speech's endpoint firstly, after that my code sends the text generated by Custom Speech's endpoint to Luis for intention recognizer.

I'd like to connect Luis and Custom Speech directly and avoid to convert the user's speech in text firstly and then send the generetad text to Luis for intention recognition.

Have you already got any way to do that?

Would you like to think about it togheter?

Best regards!