MR 和 Azure 303:自然语言理解 (LUIS) MR and Azure 303: Natural language understanding (LUIS)


备注

混合现实学院教程在制作时考虑到了 HoloLens(第一代)和混合现实沉浸式头戴显示设备。The Mixed Reality Academy tutorials were designed with HoloLens (1st gen) and Mixed Reality Immersive Headsets in mind. 因此,对于仍在寻求这些设备的开发指导的开发人员而言,我们觉得很有必要保留这些教程。As such, we feel it is important to leave these tutorials in place for developers who are still looking for guidance in developing for those devices. 我们 不会 在这些教程中更新 HoloLens 2 所用的最新工具集或集成相关的内容。These tutorials will not be updated with the latest toolsets or interactions being used for HoloLens 2. 我们将维护这些教程,使之持续适用于支持的设备。They will be maintained to continue working on the supported devices. 将来会发布一系列新教程,这些教程将演示如何针对 HoloLens 2 进行开发。There will be a new series of tutorials that will be posted in the future that will demonstrate how to develop for HoloLens 2. 此通知将在发布时通过指向这些教程的链接进行更新。This notice will be updated with a link to those tutorials when they are posted.


在本课程中,你将了解如何使用 Azure 认知服务将语言理解集成到混合现实应用程序中,语言理解 API。In this course, you will learn how to integrate Language Understanding into a mixed reality application using Azure Cognitive Services, with the Language Understanding API.

实验室结果

语言理解 (LUIS) 是 Microsoft Azure 服务,它使应用程序能够从用户输入中代替用户输入,例如通过使用自己的字词提取用户可能需要的内容。Language Understanding (LUIS) is a Microsoft Azure service, which provides applications with the ability to make meaning out of user input, such as through extracting what a person might want, in their own words. 这是通过机器学习来实现的,它可以理解和学习输入信息,然后可以使用详细的相关信息进行回复。This is achieved through machine learning, which understands and learns the input information, and then can reply with detailed, relevant, information. 有关详细信息,请访问 Azure 语言理解 (LUIS) "页For more information, visit the Azure Language Understanding (LUIS) page.

完成本课程后,你将拥有一个混合现实沉浸式耳机应用程序,该应用程序将能够执行以下操作:Having completed this course, you will have a mixed reality immersive headset application which will be able to do the following:

  1. 使用连接到沉浸式耳机的麦克风捕获用户输入语音。Capture user input speech, using the Microphone attached to the immersive headset.
  2. 将捕获的听写发送到 Azure 语言理解智能服务 (LUIS) 。Send the captured dictation the Azure Language Understanding Intelligent Service (LUIS).
  3. 让 LUIS 从发送信息中提取含义,将对其进行分析,并尝试确定用户请求的意图。Have LUIS extract meaning from the send information, which will be analyzed, and attempt to determine the intent of the user’s request will be made.

开发过程包括创建一个应用程序,用户可以使用该应用程序来更改场景中对象的大小和颜色。Development will include the creation of an app where the user will be able to use voice and/or gaze to change the size and the color of the objects in the scene. 不会覆盖运动控制器的使用。The use of motion controllers will not be covered.

在您的应用程序中,您将由您来决定如何将结果与您的设计相集成。In your application, it is up to you as to how you will integrate the results with your design. 本课程旨在向您介绍如何将 Azure 服务与 Unity 项目集成。This course is designed to teach you how to integrate an Azure Service with your Unity Project. 您可以使用您在本课程中获得的知识来增强混合现实应用程序的工作。It is your job to use the knowledge you gain from this course to enhance your mixed reality application.

准备多次训练 LUIS,这将在第 12 章中介绍。Be prepared to Train LUIS several times, which is covered in Chapter 12. 更好的结果是训练 LUIS 的次数越多。You will get better results the more times LUIS has been trained.

设备支持Device support

课程Course HoloLensHoloLens 沉浸式头戴显示设备Immersive headsets
MR 和 Azure 303:自然语言理解 (LUIS) MR and Azure 303: Natural language understanding (LUIS) ✔️✔️ ✔️✔️

备注

尽管本课程主要侧重于 Windows Mixed Reality 沉浸式 (VR) 耳机,但你也可以将本课程中学习的内容应用于 Microsoft HoloLens。While this course primarily focuses on Windows Mixed Reality immersive (VR) headsets, you can also apply what you learn in this course to Microsoft HoloLens. 在本课程中,你将看到有关支持 HoloLens 时可能需要执行的任何更改的说明。As you follow along with the course, you will see notes on any changes you might need to employ to support HoloLens. 使用 HoloLens 时,可能会在语音捕获过程中注意到某些回声。When using HoloLens, you may notice some echo during voice capture.

先决条件Prerequisites

备注

本教程专为具有 Unity 和 c # 基本经验的开发人员设计。This tutorial is designed for developers who have basic experience with Unity and C#. 请注意,本文档中的先决条件和书面说明表明了编写 (2018) 时测试和验证的内容。Please also be aware that the prerequisites and written instructions within this document represent what has been tested and verified at the time of writing (May 2018). 您可以随意使用最新的软件(如 安装工具 一文中所述),但不应假定本课程中的信息将与下面列出的内容完全匹配。You are free to use the latest software, as listed within the install the tools article, though it should not be assumed that the information in this course will perfectly match what you'll find in newer software than what's listed below.

本课程建议采用以下硬件和软件:We recommend the following hardware and software for this course:

开始之前Before you start

  1. 若要避免在生成此项目时遇到问题,强烈建议你在根或近乎根文件夹中创建本教程中所述的项目 (长文件夹路径在生成时) 会导致问题。To avoid encountering issues building this project, it is strongly suggested that you create the project mentioned in this tutorial in a root or near-root folder (long folder paths can cause issues at build-time).

  2. 若要允许计算机启用听写,请转到 " Windows 设置" > 隐私 > 语音 "、" 墨迹书写 & 键入 "和" 打开语音服务并键入建议"按钮。To allow your machine to enable Dictation, go to Windows Settings > Privacy > Speech, Inking & Typing and press on the button Turn On speech services and typing suggestions.

  3. 本教程中的代码将允许你从计算机上的 默认麦克风设备 设置进行录制。The code in this tutorial will allow you to record from the Default Microphone Device set on your machine. 请确保将默认麦克风设备设置为要用于捕获语音的设备。Make sure the Default Microphone Device is set as the one you wish to use to capture your voice.

  4. 如果耳机有内置麦克风,请确保在 混合现实门户 设置中启用 "我戴上耳机,切换到耳机麦克风" 选项。If your headset has a built-in microphone, make sure the option “When I wear my headset, switch to headset mic” is turned on in the Mixed Reality Portal settings.

    设置沉浸式耳机

第1章–设置 Azure 门户Chapter 1 – Setup Azure Portal

若要在 Azure 中使用 语言理解 服务,你将需要配置服务的实例,使其可用于你的应用程序。To use the Language Understanding service in Azure, you will need to configure an instance of the service to be made available to your application.

  1. 登录到 Azure 门户Log in to the Azure Portal.

    备注

    如果还没有 Azure 帐户,则需要创建一个。If you do not already have an Azure account, you will need to create one. 如果在课堂或实验室中按照本教程进行学习,请咨询教师或 proctors,以获得设置新帐户的帮助。If you are following this tutorial in a classroom or lab situation, ask your instructor or one of the proctors for help setting up your new account.

  2. 登录后,单击左上角的 " 新建 ",搜索 " 语言理解",然后单击 " Enter"。Once you are logged in, click on New in the top left corner, and search for Language Understanding, and click Enter.

    创建 LUIS 资源

    备注

    在较新的门户中,可能已将 " 新建 " 一词替换为 " 创建资源"。The word New may have been replaced with Create a resource, in newer portals.

  3. 向右的新页将提供语言理解服务的说明。The new page to the right will provide a description of the Language Understanding service. 在此页的左下角,选择 " 创建 " 按钮以创建此服务的实例。At the bottom left of this page, select the Create button, to create an instance of this service.

    创建 LUIS 服务-法律声明

  4. 单击 "创建" 后:Once you have clicked on Create:

    1. 为此服务实例插入所需的 名称Insert your desired Name for this service instance.

    2. 选择一个“订阅” 。Select a Subscription.

    3. 选择适合于你的 定价层 ,如果这是第一次创建 LUIS 服务,) 应提供 (名为 F0 的免费层。Select the Pricing Tier appropriate for you, if this is the first time creating a LUIS Service, a free tier (named F0) should be available to you. 对于本课程,免费分配应该已经足够。The free allocation should be more than sufficient for this course.

    4. 选择一个 资源组 ,或创建一个新的资源组。Choose a Resource Group or create a new one. 资源组提供一种监视、控制访问、预配和管理 Azure 资产集合的计费的方法。A resource group provides a way to monitor, control access, provision and manage billing for a collection of Azure assets. 建议保留与单个项目关联的所有 Azure 服务 (例如,这些课程) 常用资源组) 下。It is recommended to keep all the Azure services associated with a single project (e.g. such as these courses) under a common resource group).

      若要了解有关 Azure 资源组的详细信息,请 访问资源组一文If you wish to read more about Azure Resource Groups, please visit the resource group article.

    5. 如果要创建新的资源组) ,请确定资源组 (的 位置Determine the Location for your resource group (if you are creating a new Resource Group). 此位置理想情况下会在应用程序运行所在的区域中。The location would ideally be in the region where the application would run. 某些 Azure 资产仅在特定区域提供。Some Azure assets are only available in certain regions.

    6. 还需要确认是否已了解应用于此服务的条款和条件。You will also need to confirm that you have understood the Terms and Conditions applied to this Service.

    7. 选择“创建”。Select Create.

      创建 LUIS 服务-用户输入

  5. 单击 " 创建" 后,需要等待创建服务,这可能需要一分钟时间。Once you have clicked on Create, you will have to wait for the service to be created, this might take a minute.

  6. 创建服务实例后,门户中将显示一个通知。A notification will appear in the portal once the Service instance is created.

    新的 Azure 通知映像

  7. 单击通知以浏览新服务实例。Click on the notification to explore your new Service instance.

    成功创建资源通知

  8. 单击通知中的 " 中转到资源 " 按钮以浏览新服务实例。Click the Go to resource button in the notification to explore your new Service instance. 你将转到新的 LUIS 服务实例。You will be taken to your new LUIS service instance.

    访问 LUIS 项

  9. 在本教程中,你的应用程序将需要调用你的服务,这是通过使用你的服务的订阅密钥来完成的。Within this tutorial, your application will need to make calls to your service, which is done through using your service’s Subscription Key.

  10. LUIS API 服务的 "快速启动" 页上,导航到第一步,获取你的密钥,然后单击 "密钥" (你也可以通过单击位于 "服务" 导航菜单中的蓝色超链接项,) 来表示。From the Quick start page, of your LUIS API service, navigate to the first step, Grab your keys, and click Keys (you can also achieve this by clicking the blue hyperlink Keys, located in the services navigation menu, denoted by the key icon). 这会显示你的服务 密钥This will reveal your service Keys.

  11. 复制其中一个所显示的密钥,因为稍后会在项目中使用此密钥。Take a copy of one of the displayed keys, as you will need this later in your project.

  12. 在 " 服务 " 页上,单击 " 语言理解门户 " 以重定向到在 LUIS 应用中用于创建新服务的网页。In the Service page, click on Language Understanding Portal to be redirected to the webpage which you will use to create your new Service, within the LUIS App.

第2章–语言理解门户Chapter 2 – The Language Understanding Portal

在本部分中,你将了解如何在 LUIS 门户中创建 LUIS 应用程序。In this section you will learn how to make a LUIS App on the LUIS Portal.

重要

请注意,在本章中设置 实体意向最谈话 只是构建 LUIS 服务的第一步:你还需要多次重新训练服务,以便使其更准确。Please be aware, that setting up the Entities, Intents, and Utterances within this chapter is only the first step in building your LUIS service: you will also need to retrain the service, several times, so to make it more accurate. 本课程的 最后一章 介绍了重新训练你的服务,因此请确保完成此操作。Retraining your service is covered in the last Chapter of this course, so ensure that you complete it.

  1. 进入 语言理解门户 后,你可能需要登录(如果尚未登录),其凭据与 Azure 门户相同。Upon reaching the Language Understanding Portal, you may need to login, if you are not already, with the same credentials as your Azure portal.

    LUIS 登录页

  2. 如果这是你第一次使用 LUIS,则需要向下滚动到 "欢迎" 页的底部,查找并单击 " 创建 LUIS 应用 " 按钮。If this is your first time using LUIS, you will need to scroll down to the bottom of the welcome page, to find and click on the Create LUIS app button.

    "创建 LUIS 应用" 页

  3. 登录后,单击 "我的应用 " (如果当前不在该分区中) 。Once logged in, click My apps (if you are not in that section currently). 然后,可以单击 " 创建新应用"。You can then click on Create new app.

    LUIS-我的应用图像

  4. 为应用程序 命名Give your app a Name.

  5. 如果你的应用程序应了解不同于英语的语言,则应将 区域性 更改为适当的语言。If your app is supposed to understand a language different from English, you should change the Culture to the appropriate language.

  6. 你还可以在此处添加新的 LUIS 应用的 描述Here you can also add a Description of your new LUIS app.

    LUIS-创建新应用

  7. 按下 "完成" 后,将进入新 LUIS 应用程序的 "生成" 页。Once you press Done, you will enter the Build page of your new LUIS application.

  8. 此处有几个重要概念需要了解:There are a few important concepts to understand here:

    • 意向,表示将从用户查询后调用的方法。Intent, represents the method that will be called following a query from the user. 意向 可以有一个或多个 实体An INTENT may have one or more ENTITIES.
    • 实体,是描述与 意向 相关的信息的查询组件。Entity, is a component of the query that describes information relevant to the INTENT.
    • 最谈话 是由开发人员提供的查询的示例,LUIS 将使用该查询来训练自身。Utterances, are examples of queries provided by the developer, that LUIS will use to train itself.

如果这些概念并不完全清晰,请不要担心,因为本课程将在本章中进一步阐明它们。If these concepts are not perfectly clear, do not worry, as this course will clarify them further in this chapter.

首先创建生成此课程所需的 实体You will begin by creating the Entities needed to build this course.

  1. 在页面左侧,单击 " 实体",并单击 " 创建新实体"。On the left side of the page, click on Entities, then click on Create new entity.

    创建新实体

  2. 调用新的实体 颜色,将其类型设置为 " 简单",并按 " 完成"。Call the new Entity color, set its type to Simple, then press Done.

    创建简单实体-颜色

  3. 重复此过程以创建三个 (3) 名更简单的实体:Repeat this process to create three (3) more Simple Entities named:

    • 降级upsize
    • 缩小downsize
    • 目标target

结果应如下图所示:The result should look like the image below:

实体创建的结果

此时,你可以开始创建 意向At this point you can begin creating Intents.

警告

请勿删除 " "。Do not delete the None intent.

  1. 在页面左侧,单击 "选择 ",然后 单击 " 创建新意向"。On the left side of the page, click on Intents, then click on Create new intent.

    创建新意向

  2. 调用新 意向 ChangeObjectColorCall the new Intent ChangeObjectColor.

    重要

    本课程后面的代码中使用此 意向 名称,因此为了获得最佳结果,请完全按所提供的名称使用此名称。This Intent name is used within the code later in this course, so for best results, use this name exactly as provided.

确认名称后,将定向到 "意向" 页。Once you confirm the name you will be directed to the Intents Page.

LUIS 页

你会注意到有一个文本框要求你键入5个或更多不同的 最谈话You will notice that there is a textbox asking you to type 5 or more different Utterances.

备注

LUIS 将所有最谈话都转换为小写。LUIS converts all Utterances to lower case.

  1. 在顶部文本框中插入以下 查询文本 (当前包含 大约5个示例的文本类型 ...Insert the following Utterance in the top textbox (currently with the text Type about 5 examples… ) ,然后按 enter), and press Enter:
The color of the cylinder must be red

你会注意到,新的 查询文本 将显示在下面的列表中。You will notice that the new Utterance will appear in a list underneath.

按照相同的过程,插入以下六个 (6) 最谈话:Following the same process, insert the following six (6) Utterances:

make the cube black

make the cylinder color white

change the sphere to red

change it to green

make this yellow

change the color of this object to blue

对于创建的每个查询文本,必须确定 LUIS 将哪些字词作为实体使用。For each Utterance you have created, you must identify which words should be used by LUIS as Entities. 在此示例中,需要将所有颜色标记为一个 颜色 实体,并将所有可能的引用标记为 目标实体。In this example you need to label all the colors as a color Entity, and all the possible reference to a target as a target Entity.

  1. 为此,请在第一个查询文本中单击 " 圆柱 ",然后选择 " 目标"。To do so, try clicking on the word cylinder in the first Utterance and select target.

    标识查询文本目标

  2. 现在,在第一个查询文本中单击 " 红色 ",然后选择 " 颜色"。Now click on the word red in the first Utterance and select color.

    标识查询文本实体

  3. 标记下一行,其中, cube 应为 目标黑色 应为一种 颜色Label the next line also, where cube should be a target, and black should be a color. 另请注意,我们提供了单词 "this""it""this object",因此还提供了非特定目标类型。Notice also the use of the words ‘this’, ‘it’, and ‘this object’, which we are providing, so to have non-specific target types available also.

  4. 重复上述过程,直到所有最谈话都具有标记的实体。Repeat the process above until all the Utterances have the Entities labelled. 如果需要帮助,请参阅下图。See the below image if you need help.

    提示

    选择要标记为实体的单词时:When selecting words to label them as entities:

    • 只需单击一下即可。For single words just click them.
    • 对于两个或多个单词的集合,请单击 "开始",然后在集末尾单击。For a set of two or more words, click at the beginning and then at the end of the set.

    备注

    您可以使用 " 标记视图 " 切换按钮在 实体/标记视图 之间切换!You can use the Tokens View toggle button to switch between Entities / Tokens View!

  5. 结果应如下图所示,其中显示了 " 实体/标记" 视图The results should be as seen in the images below, showing the Entities / Tokens View:

    标记 & 实体视图

  6. 此时,按页面右上角的 " 训练 " 按钮,并等待其上的小圆形指示器变为绿色。At this point press the Train button at the top-right of the page and wait for the small round indicator on it to turn green. 这表示已成功训练 LUIS 以识别此目的。This indicates that LUIS has been successfully trained to recognize this Intent.

    训练 LUIS

  7. 作为练习,使用实体 target升迁缩小 创建名为 ChangeObjectSize 的新意向。As an exercise for you, create a new Intent called ChangeObjectSize, using the Entities target, upsize, and downsize.

  8. 按照与上一意向相同的过程,插入以下八个 (8) 最谈话 大小 更改:Following the same process as the previous Intent, insert the following eight (8) Utterances for Size change:

    increase the dimensions of that
    
    reduce the size of this
    
    i want the sphere smaller
    
    make the cylinder bigger
    
    size down the sphere
    
    size up the cube
    
    decrease the size of that object
    
    increase the size of this object
    
  9. 结果应类似于下图所示:The result should be like the one in the image below:

    设置 ChangeObjectSize 令牌/实体

  10. 一旦创建并训练了 ChangeObjectColorChangeObjectSize,单击页面顶部的 " 发布 " 按钮。Once both Intents, ChangeObjectColor and ChangeObjectSize, have been created and trained, click on the PUBLISH button on top of the page.

    发布 LUIS 服务

  11. 在 " 发布 " 页上,您将完成并发布 LUIS 应用程序,以便您的代码可以访问它。On the Publish page you will finalize and publish your LUIS App so that it can be accessed by your code.

    1. 将下拉 " 发布到 " 设置为 " 生产"。Set the drop down Publish To as Production.

    2. 时区 设置为时区。Set the Timezone to your time zone.

    3. 选中 " 包括所有预测意向分数" 框。Check the box Include all predicted intent scores.

    4. 单击 " 发布到生产槽"。Click on Publish to Production Slot.

      发布设置

  12. 在 " 资源和密钥" 部分中:In the section Resources and Keys:

    1. 在 Azure 门户中选择为服务实例设置的区域。Select the region you set for service instance in the Azure Portal.
    2. 你将注意到下面的一个 Starter_Key 元素,请将其忽略。You will notice a Starter_Key element below, ignore it.
    3. 单击 " 添加密钥 ",并在创建服务实例时插入在 Azure 门户中获取的 密钥Click on Add Key and insert the Key that you obtained in the Azure Portal when you created your Service instance. 如果你的 Azure 和 LUIS 门户登录到同一用户,你将向你提供 " 租户名称"、" 订阅名称" 和你想要使用的 密钥 的下拉菜单, (会与你之前在 Azure 门户中提供的名称相同。If your Azure and the LUIS portal are logged into the same user, you will be provided drop-down menus for Tenant name, Subscription Name, and the Key you wish to use (will have the same name as you provided previously in the Azure Portal.

    重要

    在 " 终结点" 下,获取与所插入的密钥对应的终结点副本,然后在代码中立即使用该副本。Underneath Endpoint, take a copy of the endpoint corresponding to the Key you have inserted, you will soon use it in your code.

第3章–设置 Unity 项目Chapter 3 – Set up the Unity project

下面是使用混合现实进行开发的典型设置,因此,这是其他项目的一个不错的模板。The following is a typical set up for developing with the mixed reality, and as such, is a good template for other projects.

  1. 打开 Unity ,并单击 " 新建"。Open Unity and click New.

    启动新的 Unity 项目。

  2. 你现在需要提供 Unity 项目名称,插入 MR_LUISYou will now need to provide a Unity Project name, insert MR_LUIS. 请确保 "项目类型" 设置为 " 3d"。Make sure the project type is set to 3D. 将位置设置为合适的 位置 (记住,更接近根目录) 。Set the Location to somewhere appropriate for you (remember, closer to root directories is better). 然后单击 " 创建项目"。Then, click Create project.

    提供新 Unity 项目的详细信息。

  3. 当 Unity 处于打开状态时,有必要选中 "默认 脚本编辑器 " 设置为 " Visual Studio"。With Unity open, it is worth checking the default Script Editor is set to Visual Studio. 转到 "编辑 > 首选项",然后在新窗口中导航到 " 外部工具"。Go to Edit > Preferences and then from the new window, navigate to External Tools. 外部脚本编辑器 更改为 Visual Studio 2017Change External Script Editor to Visual Studio 2017. 关闭 " 首选项 " 窗口。Close the Preferences window.

    更新脚本编辑器首选项。

  4. 接下来,通过单击 "切换平台" 按钮转到 "文件 > 生成设置",并将平台切换到 通用 Windows 平台Next, go to File > Build Settings and switch the platform to Universal Windows Platform, by clicking on the Switch Platform button.

    生成设置窗口,将平台切换到 UWP。

  5. 请参阅 文件 > 生成设置 ,并确保:Go to File > Build Settings and make sure that:

    1. 目标设备 设置为 任何设备Target Device is set to Any Device

      对于 Microsoft HoloLens,将 " 目标设备 " 设置为 " hololens"。For the Microsoft HoloLens, set Target Device to HoloLens.

    2. 生成类型 设置为 D3DBuild Type is set to D3D

    3. SDK 设置为 "最新安装"SDK is set to Latest installed

    4. Visual Studio 版本 设置为 "最新安装"Visual Studio Version is set to Latest installed

    5. "生成并运行" 设置为 "本地计算机"Build and Run is set to Local Machine

    6. 保存场景并将其添加到生成中。Save the scene and add it to the build.

      1. 通过选择 " 添加打开的场景" 来执行此操作。Do this by selecting Add Open Scenes. 将显示 "保存" 窗口。A save window will appear.

        单击 "添加打开的场景" 按钮

      2. 为此创建新文件夹,并为将来的任何场景创建一个新文件夹,然后选择 " 新建文件夹 " 按钮以创建新文件夹,将其命名为 场景Create a new folder for this, and any future, scene, then select the New folder button, to create a new folder, name it Scenes.

        "创建新脚本" 文件夹

      3. 打开新创建的 场景 文件夹,然后 在 "文件名:文本" 字段 中,键入 MR_LuisScene,然后按 " 保存"。Open your newly created Scenes folder, and then in the File name: text field, type MR_LuisScene, then press Save.

        为新场景指定名称。

    7. 现在," 生成设置" 中的其余设置应保留为默认值。The remaining settings, in Build Settings, should be left as default for now.

  6. 在 " 生成设置 " 窗口中,单击 " 播放机设置 " 按钮,这会在 检查器 所在的空间中打开相关面板。In the Build Settings window, click on the Player Settings button, this will open the related panel in the space where the Inspector is located.

    打开播放机设置。

  7. 在此面板中,需要验证几项设置:In this panel, a few settings need to be verified:

    1. 在 " 其他设置 " 选项卡中:In the Other Settings tab:

      1. 脚本运行时版本稳定 ( .net 3.5 等效) 。Scripting Runtime Version should be Stable (.NET 3.5 Equivalent).

      2. 脚本编写后端 应为 .netScripting Backend should be .NET

      3. API 兼容级别 应为 .net 4.6API Compatibility Level should be .NET 4.6

        更新其他设置。

    2. 在 " 发布设置 " 选项卡的 " 功能" 下,检查:Within the Publishing Settings tab, under Capabilities, check:

      1. InternetClientInternetClient

      2. 麦克风Microphone

        正在更新发布设置。

    3. 在面板中,在 " XR 设置 " 中, () "发布设置" 下的 " 发布设置 " 下提供了 支持,请确保已添加 Windows Mixed reality SDKFurther down the panel, in XR Settings (found below Publish Settings), tick Virtual Reality Supported, make sure the Windows Mixed Reality SDK is added.

      更新 X R 设置。

  8. 返回 生成设置 Unity c # 项目不再灰显;勾选此的旁边的复选框。Back in Build Settings Unity C# Projects is no longer greyed out; tick the checkbox next to this.

  9. 关闭“生成设置”窗口。Close the Build Settings window.

  10. 保存场景和项目 (文件 > 保存场景/文件 > 保存项目) 。Save your Scene and Project (FILE > SAVE SCENE / FILE > SAVE PROJECT).

第4章-创建场景Chapter 4 – Create the scene

重要

如果希望跳过本课程的 Unity 设置 组件,并继续直接进入代码,可以下载 unitypackage,将其作为 自定义包导入项目,然后从 第5章继续。If you wish to skip the Unity Set up component of this course, and continue straight into code, feel free to download this .unitypackage, import it into your project as a Custom Package, and then continue from Chapter 5.

  1. 右键单击 " 层次结构" 面板 的空白区域中的 " 3d 对象" 下的 "添加 平面"。Right-click in an empty area of the Hierarchy Panel, under 3D Object, add a Plane.

    创建平面。

  2. 请注意,在 层次结构 中再次右键单击以创建多个对象时,如果仍选择最后一个对象,则所选对象将是新对象的父对象。Be aware that when you right-click within the Hierarchy again to create more objects, if you still have the last object selected, the selected object will be the parent of your new object. 避免在层次结构内的空白区域中左键单击,然后右键单击。Avoid this left-clicking in an empty space within the Hierarchy, and then right-clicking.

  3. 重复上述过程以添加以下对象:Repeat the above procedure to add the following objects:

    1. SphereSphere
    2. 柱状Cylinder
    3. CubeCube
    4. 3D 文本3D Text
  4. 生成的场景 层次结构 应类似于下图所示:The resulting scene Hierarchy should be like the one in the image below:

    场景层次结构设置。

  5. 摄像机 上左键单击以选择它,查看 " 检查器" 面板 ,其中包含其所有组件。Left click on the Main Camera to select it, look at the Inspector Panel you will see the Camera object with all the its components.

  6. 单击位于 "检查器" 面板 底部的 "添加组件" 按钮。Click on the Add Component button located at the very bottom of the Inspector Panel.

    添加音频源

  7. 搜索名为 " 音频源" 的组件,如上所示。Search for the component called Audio Source, as shown above.

  8. 另外,请确保将摄像机的 转换 组件设置为 (0,0,0) ,这可以通过按下相机的 转换 组件旁边的 齿轮 图标,然后选择 "重置" 来完成。Also make sure that the Transform component of the Main Camera is set to (0,0,0), this can be done by pressing the Gear icon next to the Camera’s Transform component and selecting Reset. 转换 组件如下所示:The Transform component should then look like:

    1. 位置 设置为 0,0,0Position is set to 0, 0, 0.
    2. 旋转 设置为 0,0,0Rotation is set to 0, 0, 0.

    备注

    对于 Microsoft HoloLens,你还需要更改以下内容,这是 相机 组件(位于你的 主摄像机 上)的一部分:For the Microsoft HoloLens, you will need to also change the following, which are part of the Camera component, which is on your Main Camera:

    • 清除标志: 纯色。Clear Flags: Solid Color.
    • 背景 "黑色,Alpha 0" –十六进制颜色: #00000000。Background ‘Black, Alpha 0’ – Hex color: #00000000.
  9. 平面 上单击以将其选中。Left click on the Plane to select it. 在 " 检查器" 面板 中,用以下值设置 转换 组件:In the Inspector Panel set the Transform component with the following values:

    X 轴X Axis Y 轴Y Axis Z 轴Z Axis
    00 -1-1 00
  10. 左键单击 球体 将其选中。Left click on the Sphere to select it. 在 " 检查器" 面板 中,用以下值设置 转换 组件:In the Inspector Panel set the Transform component with the following values:

    X 轴X Axis Y 轴Y Axis Z 轴Z Axis
    22 11 22
  11. 左键单击 圆柱体 以将其选中。Left click on the Cylinder to select it. 在 " 检查器" 面板 中,用以下值设置 转换 组件:In the Inspector Panel set the Transform component with the following values:

    X 轴X Axis Y 轴Y Axis Z 轴Z Axis
    -2-2 11 22
  12. 左键单击 多维数据集 以将其选中。Left click on the Cube to select it. 在 " 检查器" 面板 中,用以下值设置 转换 组件:In the Inspector Panel set the Transform component with the following values:

    转换 位置Transform - Position | 转换- 旋转Transform - Rotation
    XX Y ZZ | XX Y ZZ
    00 11 44 | 4545 4545 00
  13. 左键单击新的 文本 对象以将其选中。Left click on the New Text object to select it. 在 " 检查器" 面板 中,用以下值设置 转换 组件:In the Inspector Panel set the Transform component with the following values:

    转换 位置Transform - Position | 转换- 缩放Transform - Scale
    XX Y ZZ | XX Y ZZ
    -2-2 66 99 | 0.10.1 0.10.1 0.10.1
  14. 文本网格 组件中将 字体大小 更改为 50Change Font Size in the Text Mesh component to 50.

  15. 文本网格 对象的 名称 更改为 "听写文本"。Change the name of the Text Mesh object to Dictation Text.

    创建3D 文本对象

  16. 层次结构面板结构现在应如下所示:Your Hierarchy Panel structure should now look like this:

    场景视图中的文本网格

  17. 最终场景应如下图所示:The final scene should look like the image below:

    场景视图。

第5章–创建 MicrophoneManager 类Chapter 5 – Create the MicrophoneManager class

要创建的第一个脚本是 MicrophoneManager 类。The first Script you are going to create is the MicrophoneManager class. 接下来,您将创建 LuisManager行为 类,最后一个 (就可以自由地创建所有 这些类, 尽管在每一章) 。Following this, you will create the LuisManager, the Behaviours class, and lastly the Gaze class (feel free to create all these now, though it will be covered as you reach each Chapter).

MicrophoneManager 类负责:The MicrophoneManager class is responsible for:

  • 检测连接到耳机或计算机 (的记录设备(以默认) 为准)。Detecting the recording device attached to the headset or machine (whichever is the default one).
  • 捕获音频 (语音) 并使用听写将其存储为字符串。Capture the audio (voice) and use dictation to store it as a string.
  • 暂停语音后,将听写提交到 LuisManager 类。Once the voice has paused, submit the dictation to the LuisManager class.

若要创建此类:To create this class:

  1. 右键单击 " 项目" 面板创建 > 文件夹Right-click in the Project Panel, Create > Folder. 调用文件夹 脚本Call the folder Scripts.

    创建脚本文件夹。

  2. 创建 脚本 文件夹后,双击它以打开。With the Scripts folder created, double click it, to open. 然后,在该文件夹中,右键单击, 创建 > c # 脚本Then, within that folder, right-click, Create > C# Script. 将脚本命名为 MicrophoneManagerName the script MicrophoneManager.

  3. 双击 " MicrophoneManager " 以通过 Visual Studio 打开它。Double click on MicrophoneManager to open it with Visual Studio.

  4. 将以下命名空间添加到文件顶部:Add the following namespaces to the top of the file:

        using UnityEngine;
        using UnityEngine.Windows.Speech;
    
  5. 然后在 MicrophoneManager 类中添加以下变量:Then add the following variables inside the MicrophoneManager class:

        public static MicrophoneManager instance; //help to access instance of this object
        private DictationRecognizer dictationRecognizer;  //Component converting speech to text
        public TextMesh dictationText; //a UI object used to debug dictation result
    
  6. 现在需要添加用于 唤醒 ( # B1Start ( # B3 方法的代码。Code for Awake() and Start() methods now needs to be added. 当类初始化时,将调用以下内容:These will be called when the class initializes:

        private void Awake()
        {
            // allows this class instance to behave like a singleton
            instance = this;
        }
    
        void Start()
        {
            if (Microphone.devices.Length > 0)
            {
                StartCapturingAudio();
                Debug.Log("Mic Detected");
            }
        }
    
  7. 现在,你需要应用程序用来启动和停止语音捕获,并将其传递给 LuisManager 类的方法。Now you need the method that the App uses to start and stop the voice capture, and pass it to the LuisManager class, that you will build soon.

        /// <summary>
        /// Start microphone capture, by providing the microphone as a continual audio source (looping),
        /// then initialise the DictationRecognizer, which will capture spoken words
        /// </summary>
        public void StartCapturingAudio()
        {
            if (dictationRecognizer == null)
            {
                dictationRecognizer = new DictationRecognizer
                {
                    InitialSilenceTimeoutSeconds = 60,
                    AutoSilenceTimeoutSeconds = 5
                };
    
                dictationRecognizer.DictationResult += DictationRecognizer_DictationResult;
                dictationRecognizer.DictationError += DictationRecognizer_DictationError;
            }
            dictationRecognizer.Start();
            Debug.Log("Capturing Audio...");
        }
    
        /// <summary>
        /// Stop microphone capture
        /// </summary>
        public void StopCapturingAudio()
        {
            dictationRecognizer.Stop();
            Debug.Log("Stop Capturing Audio...");
        }
    
  8. 添加语音暂停时要调用的 听写处理程序Add a Dictation Handler that will be invoked when the voice pauses. 此方法会将听写文本传递给 LuisManager 类。This method will pass the dictation text to the LuisManager class.

        /// <summary>
        /// This handler is called every time the Dictation detects a pause in the speech. 
        /// This method will stop listening for audio, send a request to the LUIS service 
        /// and then start listening again.
        /// </summary>
        private void DictationRecognizer_DictationResult(string dictationCaptured, ConfidenceLevel confidence)
        {
            StopCapturingAudio();
            StartCoroutine(LuisManager.instance.SubmitRequestToLuis(dictationCaptured, StartCapturingAudio));
            Debug.Log("Dictation: " + dictationCaptured);
            dictationText.text = dictationCaptured;
        }
    
        private void DictationRecognizer_DictationError(string error, int hresult)
        {
            Debug.Log("Dictation exception: " + error);
        }
    

    重要

    删除 更新 ( # B1 方法,因为此类将不会使用它。Delete the Update() method since this class will not use it.

  9. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

    备注

    此时,你会注意到在 Unity 编辑器控制台面板 中出现错误。At this point you will notice an error appearing in the Unity Editor Console Panel. 这是因为代码引用了将在下一章中创建的 LuisManager 类。This is because the code references the LuisManager class which you will create in the next Chapter.

第6章–创建 LUISManager 类Chapter 6 – Create the LUISManager class

你可以创建 LuisManager 类,它将调用 Azure LUIS 服务。It is time for you to create the LuisManager class, which will make the call to the Azure LUIS service.

此类的目的是接收来自 MicrophoneManager 类的听写文本,并将其发送到要分析的 Azure 语言理解 APIThe purpose of this class is to receive the dictation text from the MicrophoneManager class and send it to the Azure Language Understanding API to be analyzed.

此类将反序列化 JSON 响应并调用 行为 类的适当方法来触发操作。This class will deserialize the JSON response and call the appropriate methods of the Behaviours class to trigger an action.

若要创建此类:To create this class:

  1. 双击 " 脚本 " 文件夹以将其打开。Double click on the Scripts folder, to open it.

  2. 右键单击 " 脚本 " 文件夹中,单击 " 创建 > c # 脚本"。Right-click inside the Scripts folder, click Create > C# Script. 将脚本命名为 LuisManagerName the script LuisManager.

  3. 双击脚本以通过 Visual Studio 打开它。Double click on the script to open it with Visual Studio.

  4. 将以下命名空间添加到文件顶部:Add the following namespaces to the top of the file:

        using System;
        using System.Collections;
        using System.Collections.Generic;
        using System.IO;
        using UnityEngine;
        using UnityEngine.Networking;
    
  5. 首先) ( ,在同一脚本 文件中LuisManager 类中创建三个类 (在同一脚本文件中,该文件将表示来自 Azure 的反序列化 JSON 响应。You will begin by creating three classes inside the LuisManager class (within the same script file, above the Start() method) that will represent the deserialized JSON response from Azure.

        [Serializable] //this class represents the LUIS response
        public class AnalysedQuery
        {
            public TopScoringIntentData topScoringIntent;
            public EntityData[] entities;
            public string query;
        }
    
        // This class contains the Intent LUIS determines 
        // to be the most likely
        [Serializable]
        public class TopScoringIntentData
        {
            public string intent;
            public float score;
        }
    
        // This class contains data for an Entity
        [Serializable]
        public class EntityData
        {
            public string entity;
            public string type;
            public int startIndex;
            public int endIndex;
            public float score;
        }
    
  6. 接下来,在 LuisManager 类中添加以下变量:Next, add the following variables inside the LuisManager class:

        public static LuisManager instance;
    
        //Substitute the value of luis Endpoint with your own End Point
        string luisEndpoint = "https://westus.api.cognitive... add your endpoint from the Luis Portal";
    
  7. 请确保现在将 LUIS 终结点置于) 门户 (。Make sure to place your LUIS endpoint in now (which you will have from your LUIS portal).

  8. 现在需要添加 唤醒 ( # B1 方法的代码。Code for the Awake() method now needs to be added. 当类初始化时,将调用此方法:This method will be called when the class initializes:

        private void Awake()
        {
            // allows this class instance to behave like a singleton
            instance = this;
        }
    
  9. 现在,你需要此应用程序用来将 MicrophoneManager 类收到的听写发送到 LUIS 的方法,然后接收并反序列化响应。Now you need the methods this application uses to send the dictation received from the MicrophoneManager class to LUIS, and then receive and deserialize the response.

  10. 一旦确定了意向值和关联实体后,就会将这些值传递给 行为 类的实例,以触发预期的操作。Once the value of the Intent, and associated Entities, have been determined, they are passed to the instance of the Behaviours class to trigger the intended action.

        /// <summary>
        /// Call LUIS to submit a dictation result.
        /// The done Action is called at the completion of the method.
        /// </summary>
        public IEnumerator SubmitRequestToLuis(string dictationResult, Action done)
        {
            string queryString = string.Concat(Uri.EscapeDataString(dictationResult));
    
            using (UnityWebRequest unityWebRequest = UnityWebRequest.Get(luisEndpoint + queryString))
            {
                yield return unityWebRequest.SendWebRequest();
    
                if (unityWebRequest.isNetworkError || unityWebRequest.isHttpError)
                {
                    Debug.Log(unityWebRequest.error);
                }
                else
                {
                    try
                    {
                        AnalysedQuery analysedQuery = JsonUtility.FromJson<AnalysedQuery>(unityWebRequest.downloadHandler.text);
    
                        //analyse the elements of the response 
                        AnalyseResponseElements(analysedQuery);
                    }
                    catch (Exception exception)
                    {
                        Debug.Log("Luis Request Exception Message: " + exception.Message);
                    }
                }
    
                done();
                yield return null;
            }
        }
    
  11. 创建名为 AnalyseResponseElements 的新方法 ( # B1 ,该方法将读取生成的 AnalysedQuery 并确定实体。Create a new method called AnalyseResponseElements() that will read the resulting AnalysedQuery and determine the Entities. 确定这些实体后,它们将被传递到要在操作中使用的 行为 类的实例。Once those Entities are determined, they will be passed to the instance of the Behaviours class to use in the actions.

        private void AnalyseResponseElements(AnalysedQuery aQuery)
        {
            string topIntent = aQuery.topScoringIntent.intent;
    
            // Create a dictionary of entities associated with their type
            Dictionary<string, string> entityDic = new Dictionary<string, string>();
    
            foreach (EntityData ed in aQuery.entities)
            {
                entityDic.Add(ed.type, ed.entity);
            }
    
            // Depending on the topmost recognized intent, read the entities name
            switch (aQuery.topScoringIntent.intent)
            {
                case "ChangeObjectColor":
                    string targetForColor = null;
                    string color = null;
    
                    foreach (var pair in entityDic)
                    {
                        if (pair.Key == "target")
                        {
                            targetForColor = pair.Value;
                        }
                        else if (pair.Key == "color")
                        {
                            color = pair.Value;
                        }
                    }
    
                    Behaviours.instance.ChangeTargetColor(targetForColor, color);
                    break;
    
                case "ChangeObjectSize":
                    string targetForSize = null;
                    foreach (var pair in entityDic)
                    {
                        if (pair.Key == "target")
                        {
                            targetForSize = pair.Value;
                        }
                    }
    
                    if (entityDic.ContainsKey("upsize") == true)
                    {
                        Behaviours.instance.UpSizeTarget(targetForSize);
                    }
                    else if (entityDic.ContainsKey("downsize") == true)
                    {
                        Behaviours.instance.DownSizeTarget(targetForSize);
                    }
                    break;
            }
        }
    

    重要

    删除 开始 ( # B1更新 ( # B3 方法,因为此类将不会使用它们。Delete the Start() and Update() methods since this class will not use them.

  12. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

备注

此时,你会注意到在 Unity 编辑器控制台面板 中出现了几个错误。At this point you will notice several errors appearing in the Unity Editor Console Panel. 这是因为代码引用了将在下一章中创建的 行为 类。This is because the code references the Behaviours class which you will create in the next Chapter.

第7章–创建行为类Chapter 7 – Create the Behaviours class

行为 类将使用 LuisManager 类提供的实体触发操作。The Behaviours class will trigger the actions using the Entities provided by the LuisManager class.

若要创建此类:To create this class:

  1. 双击 " 脚本 " 文件夹以将其打开。Double click on the Scripts folder, to open it.

  2. 右键单击 " 脚本 " 文件夹中,单击 " 创建 > c # 脚本"。Right-click inside the Scripts folder, click Create > C# Script. 将脚本命名为 行为Name the script Behaviours.

  3. 双击脚本以通过 Visual Studio 打开它。Double click on the script to open it with Visual Studio.

  4. 然后在 行为 类中添加以下变量:Then add the following variables inside the Behaviours class:

        public static Behaviours instance;
    
        // the following variables are references to possible targets
        public GameObject sphere;
        public GameObject cylinder;
        public GameObject cube;
        internal GameObject gazedTarget;
    
  5. 添加 唤醒 ( # B1 方法代码。Add the Awake() method code. 当类初始化时,将调用此方法:This method will be called when the class initializes:

        void Awake()
        {
            // allows this class instance to behave like a singleton
            instance = this;
        }
    
  6. 以下方法由之前创建的 LuisManager 类 (调用,) 确定哪个对象是查询的目标,然后触发相应的操作。The following methods are called by the LuisManager class (which you have created previously) to determine which object is the target of the query and then trigger the appropriate action.

        /// <summary>
        /// Changes the color of the target GameObject by providing the name of the object
        /// and the name of the color
        /// </summary>
        public void ChangeTargetColor(string targetName, string colorName)
        {
            GameObject foundTarget = FindTarget(targetName);
            if (foundTarget != null)
            {
                Debug.Log("Changing color " + colorName + " to target: " + foundTarget.name);
    
                switch (colorName)
                {
                    case "blue":
                        foundTarget.GetComponent<Renderer>().material.color = Color.blue;
                        break;
    
                    case "red":
                        foundTarget.GetComponent<Renderer>().material.color = Color.red;
                        break;
    
                    case "yellow":
                        foundTarget.GetComponent<Renderer>().material.color = Color.yellow;
                        break;
    
                    case "green":
                        foundTarget.GetComponent<Renderer>().material.color = Color.green;
                        break;
    
                    case "white":
                        foundTarget.GetComponent<Renderer>().material.color = Color.white;
                        break;
    
                    case "black":
                        foundTarget.GetComponent<Renderer>().material.color = Color.black;
                        break;
                }          
            }
        }
    
        /// <summary>
        /// Reduces the size of the target GameObject by providing its name
        /// </summary>
        public void DownSizeTarget(string targetName)
        {
            GameObject foundTarget = FindTarget(targetName);
            foundTarget.transform.localScale -= new Vector3(0.5F, 0.5F, 0.5F);
        }
    
        /// <summary>
        /// Increases the size of the target GameObject by providing its name
        /// </summary>
        public void UpSizeTarget(string targetName)
        {
            GameObject foundTarget = FindTarget(targetName);
            foundTarget.transform.localScale += new Vector3(0.5F, 0.5F, 0.5F);
        }
    
  7. 添加 FindTarget ( # B1 方法,以确定哪一个 Gameobject 是当前意向的目标。Add the FindTarget() method to determine which of the GameObjects is the target of the current Intent. 如果实体中未定义显式目标,则此方法默认为 "gazed" 的 GameObjectThis method defaults the target to the GameObject being “gazed” if no explicit target is defined in the Entities.

        /// <summary>
        /// Determines which object reference is the target GameObject by providing its name
        /// </summary>
        private GameObject FindTarget(string name)
        {
            GameObject targetAsGO = null;
    
            switch (name)
            {
                case "sphere":
                    targetAsGO = sphere;
                    break;
    
                case "cylinder":
                    targetAsGO = cylinder;
                    break;
    
                case "cube":
                    targetAsGO = cube;
                    break;
    
                case "this": // as an example of target words that the user may use when looking at an object
                case "it":  // as this is the default, these are not actually needed in this example
                case "that":
                default: // if the target name is none of those above, check if the user is looking at something
                    if (gazedTarget != null) 
                    {
                        targetAsGO = gazedTarget;
                    }
                    break;
            }
            return targetAsGO;
        }
    

    重要

    删除 开始 ( # B1更新 ( # B3 方法,因为此类将不会使用它们。Delete the Start() and Update() methods since this class will not use them.

  8. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

第8章–创建注视类Chapter 8 – Create the Gaze Class

完成此应用需要完成的最后一个类是 " 注视 类"。The last class that you will need to complete this app is the Gaze class. 此类将对当前用户视觉对象集中的 GameObject 的引用进行更新。This class updates the reference to the GameObject currently in the user’s visual focus.

若要创建此类:To create this Class:

  1. 双击 " 脚本 " 文件夹以将其打开。Double click on the Scripts folder, to open it.

  2. 右键单击 " 脚本 " 文件夹中,单击 " 创建 > c # 脚本"。Right-click inside the Scripts folder, click Create > C# Script. 将脚本命名为 " 注视"。Name the script Gaze.

  3. 双击脚本以通过 Visual Studio 打开它。Double click on the script to open it with Visual Studio.

  4. 为此类插入以下代码:Insert the following code for this class:

        using UnityEngine;
    
        public class Gaze : MonoBehaviour
        {        
            internal GameObject gazedObject;
            public float gazeMaxDistance = 300;
    
            void Update()
            {
                // Uses a raycast from the Main Camera to determine which object is gazed upon.
                Vector3 fwd = gameObject.transform.TransformDirection(Vector3.forward);
                Ray ray = new Ray(Camera.main.transform.position, fwd);
                RaycastHit hit;
                Debug.DrawRay(Camera.main.transform.position, fwd);
    
                if (Physics.Raycast(ray, out hit, gazeMaxDistance) && hit.collider != null)
                {
                    if (gazedObject == null)
                    {
                        gazedObject = hit.transform.gameObject;
    
                        // Set the gazedTarget in the Behaviours class
                        Behaviours.instance.gazedTarget = gazedObject;
                    }
                }
                else
                {
                    ResetGaze();
                }         
            }
    
            // Turn the gaze off, reset the gazeObject in the Behaviours class.
            public void ResetGaze()
            {
                if (gazedObject != null)
                {
                    Behaviours.instance.gazedTarget = null;
                    gazedObject = null;
                }
            }
        }
    
  5. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

第9章-完成场景设置Chapter 9 – Completing the scene setup

  1. 若要完成场景的设置,请将已创建的每个脚本从 "脚本" 文件夹拖到 "层次结构" 面板 中的 "照相机" 对象。To complete the setup of the scene, drag each script that you have created from the Scripts Folder to the Main Camera object in the Hierarchy Panel.

  2. 选择 主摄像机 ,查看 检查器面板,您应该能够看到已附加的每个脚本,您会注意到每个脚本上都有一些参数还需要设置。Select the Main Camera and look at the Inspector Panel, you should be able to see each script that you have attached, and you will notice that there are parameters on each script that are yet to be set.

    设置照相机引用目标。

  3. 若要正确设置这些参数,请遵循以下说明:To set these parameters correctly, follow these instructions:

    1. MicrophoneManagerMicrophoneManager:

      • 从 " 层次结构" 面板 中,将 " 听写文本 " 对象拖到 " 听写文本 参数值" 框中。From the Hierarchy Panel, drag the Dictation Text object into the Dictation Text parameter value box.
    2. 行为,从 " 层次结构" 面板Behaviours, from the Hierarchy Panel:

      • 球体 对象拖到 " 引用目标" 框中。Drag the Sphere object into the Sphere reference target box.
      • 圆柱体 拖到 " 圆柱形 引用目标" 框中。Drag the Cylinder into the Cylinder reference target box.
      • 多维数据集 拖到 " 多维数据集 引用目标" 框中。Drag the Cube into the Cube reference target box.
    3. 注视Gaze:

      • 如果尚未) ,请将 " 注视最大距离 " 设置为 300 (。Set the Gaze Max Distance to 300 (if it is not already).
  4. 结果应如下图所示:The result should look like the image below:

    当前设置了照相机引用目标。

第10章–在 Unity 编辑器中测试Chapter 10 – Test in the Unity Editor

测试场景设置是否已正确实现。Test that the Scene setup is properly implemented.

请确保:Ensure that:

  • 所有脚本都附加到 相机 对象上。All the scripts are attached to the Main Camera object.
  • 主相机检查器面板 中的所有字段均已正确分配。All the fields in the Main Camera Inspector Panel are assigned properly.
  1. 按下 Unity 编辑器 中的 "播放" 按钮。Press the Play button in the Unity Editor. 应用应在附加的沉浸式耳机中运行。The App should be running within the attached immersive headset.

  2. 尝试一些最谈话,例如:Try a few utterances, such as:

    make the cylinder red
    
    change the cube to yellow
    
    I want the sphere blue
    
    make this to green
    
    change it to white
    

    备注

    如果在 Unity 控制台中看到有关默认音频设备变化的错误,场景可能无法按预期方式工作。If you see an error in the Unity console about the default audio device changing, the scene may not function as expected. 这是因为混合现实门户使用内置麦克风处理随附的耳机的方式。This is due to the way the mixed reality portal deals with built-in microphones for headsets that have them. 如果看到此错误,只需停止场景并再次启动,应按预期方式工作。If you see this error, simply stop the scene and start it again and things should work as expected.

第11章–构建和旁加载 UWP 解决方案Chapter 11 – Build and sideload the UWP Solution

确保应用程序在 Unity 编辑器中运行后,便可以生成并部署了。Once you have ensured that the application is working in the Unity Editor, you are ready to Build and Deploy.

若要生成:To Build:

  1. 单击 " 文件" > "保存",保存当前场景。Save the current scene by clicking on File > Save.

  2. 请参阅 文件 > 生成设置Go to File > Build Settings.

  3. 勾选称为 Unity c # 项目 的框, (在创建 UWP 项目后查看和调试代码非常有用。Tick the box called Unity C# Projects (useful for seeing and debugging your code once the UWP project is created.

  4. 单击 " 添加打开的场景",然后单击 " 生成"。Click on Add Open Scenes, then click Build.

    "生成设置" 窗口

  5. 系统将提示您选择要在其中生成解决方案的文件夹。You will be prompted to select the folder where you want to build the Solution.

  6. 创建一个 生成 文件夹,在该文件夹中创建另一个文件夹,其中包含所选的适当名称。Create a BUILDS folder and within that folder create another folder with an appropriate name of your choice.

  7. 单击 " 选择文件夹 ",在该位置开始生成。Click Select Folder to begin the build at that location.

    创建生成文件夹  选择生成文件夹Create Builds Folder Select Builds Folder

  8. Unity 完成生成后 (可能需要一些时间) ,它应在生成的位置打开 文件资源管理器 窗口。Once Unity has finished building (it might take some time), it should open a File Explorer window at the location of your build.

在本地计算机上部署:To Deploy on Local Machine:

  1. Visual Studio 中,打开已在 上一章中创建的解决方案文件。In Visual Studio, open the solution file that has been created in the previous Chapter.

  2. 解决方案平台 中,选择 " X86本地计算机"。In the Solution Platform, select x86, Local Machine.

  3. 解决方案配置 中,选择 " 调试"。In the Solution Configuration select Debug.

    对于 Microsoft HoloLens,你可能会发现将其设置为 远程计算机 会更容易,因此你不会受限到计算机上。For the Microsoft HoloLens, you may find it easier to set this to Remote Machine, so that you are not tethered to your computer. 不过,还需要执行以下操作:Though, you will need to also do the following:

    • 了解 HoloLens 的 IP 地址 ,可在 " 设置" > 网络 & Internet > Wi-Fi "> 高级选项" 中找到;IPv4 是应使用的地址。Know the IP Address of your HoloLens, which can be found within the Settings > Network & Internet > Wi-Fi > Advanced Options; the IPv4 is the address you should use.
    • 确保 开发人员模式打开;在 "设置" 中找到 > 更新开发人员 & 安全 >Ensure Developer Mode is On; found in Settings > Update & Security > For developers.

    部署应用

  4. 请在 " 生成" 菜单 中,单击 " 部署解决方案 ",将应用程序旁加载到计算机上。Go to the Build menu and click on Deploy Solution to sideload the application to your machine.

  5. 应用现在应显示在已安装的应用列表中,可以启动了!Your App should now appear in the list of installed apps, ready to be launched!

  6. 启动后,应用会提示你授权访问 麦克风Once launched, the App will prompt you to authorize access to the Microphone. 使用 运动控制器语音输入键盘"是" 按钮。Use the Motion Controllers, or Voice Input, or the Keyboard to press the YES button.

第12章-改进 LUIS 服务Chapter 12 – Improving your LUIS service

重要

本章非常重要,可能需要多次迭代,因为这将有助于提高 LUIS 服务的准确性:请确保完成此操作。This chapter is incredibly important, and may need to be iterated upon several times, as it will help improve the accuracy of your LUIS service: ensure you complete this.

若要改进 LUIS 提供的理解级别,需要捕获新的最谈话,并使用它们重新训练 LUIS 应用。To improve the level of understanding provided by LUIS you need to capture new utterances and use them to re-train your LUIS App.

例如,你可能已经训练了 LUIS 来理解 "增加" 和 "升迁",但又不希望你的应用程序也知道 "放大" 等词呢?For example, you might have trained LUIS to understand “Increase” and “Upsize”, but wouldn’t you want your app to also understand words like “Enlarge”?

一旦你使用了应用程序几次,你所说的一切将被 LUIS 收集,并在 LUIS 门户中可用。Once you have used your application a few times, everything you have said will be collected by LUIS and available in the LUIS PORTAL.

  1. 请按照以下 链接打开门户应用程序,并登录。Go to your portal application following this LINK, and Log In.

  2. 用 MS 凭据登录后,单击你的 应用名称Once you are logged in with your MS Credentials, click on your App name.

  3. 单击页面左侧的 " 查看终结点最谈话 " 按钮。Click the Review endpoint utterances button on the left of the page.

    查看最谈话

  4. 将显示一个列表,其中包含已由混合现实应用程序发送到 LUIS 的最谈话。You will be shown a list of the Utterances that have been sent to LUIS by your mixed reality Application.

    最谈话列表

你会注意到一些突出显示的 实体You will notice some highlighted Entities.

通过将鼠标悬停在每个突出显示的单词上,你可以查看每个查询文本并确定哪些实体已正确识别,哪些实体出现错误以及哪些实体丢失。By hovering over each highlighted word, you can review each Utterance and determine which Entity has been recognized correctly, which Entities are wrong and which Entities are missed.

在上面的示例中,发现 "鱼叉式" 一词已突出显示为目标,因此需要更正此错误,这是通过将鼠标悬停在单词上并单击 " 删除标签" 来完成的。In the example above, it was found that the word “spear” had been highlighted as a target, so it necessary to correct the mistake, which is done by hovering over the word with the mouse and clicking Remove Label.

检查最谈话  删除标签图像Check utterances Remove Label Image

  1. 如果找到完全错误的最谈话,可以使用屏幕右侧的 " 删除 " 按钮删除它们。If you find Utterances that are completely wrong, you can delete them using the Delete button on the right side of the screen.

    删除错误的最谈话

  2. 或者,如果您认为 LUIS 已正确解释查询文本,则可以使用 " 添加到对齐意向 " 按钮来验证其理解。Or if you feel that LUIS has interpreted the Utterance correctly, you can validate its understanding by using the Add To Aligned Intent button.

    添加到对齐意向

  3. 对所有显示的最谈话进行排序后,请尝试重新加载页面,查看是否有更多的可用功能。Once you have sorted all the displayed Utterances, try and reload the page to see if more are available.

  4. 尽可能多地重复此过程以提高应用程序理解。It is very important to repeat this process as many times as possible to improve your application understanding.

玩得愉快!Have fun!

已完成的 LUIS 集成应用程序Your finished LUIS Integrated application

恭喜,你构建了一个使用 Azure 语言理解智能服务的混合现实应用,以了解用户所说的内容,并对该信息采取措施。Congratulations, you built a mixed reality app that leverages the Azure Language Understanding Intelligence Service, to understand what a user says, and act on that information.

实验室结果

额外练习Bonus exercises

练习1Exercise 1

使用此应用程序时,您可能会注意到,如果注视地面对象并要求更改其颜色,则会这样做。While using this application you might notice that if you gaze at the Floor object and ask to change its color, it will do so. 您可以如何阻止您的应用程序更改地面颜色?Can you work out how to stop your application from changing the Floor color?

练习2Exercise 2

尝试扩展 LUIS 和应用功能,为场景中的对象添加其他功能;例如,在注视点处创建新的对象,具体取决于用户所显示的内容,然后可以将这些对象和当前场景对象与现有的命令一起使用。Try extending the LUIS and App capabilities, adding additional functionality for objects in scene; as an example, create new objects at the Gaze hit point, depending on what the user says, and then be able to use those objects alongside current scene objects, with the existing commands.