MR 和 Azure 302b:自定义视觉MR and Azure 302b: Custom vision


备注

混合现实学院教程在制作时考虑到了 HoloLens(第一代)和混合现实沉浸式头戴显示设备。The Mixed Reality Academy tutorials were designed with HoloLens (1st gen) and Mixed Reality Immersive Headsets in mind. 因此,对于仍在寻求这些设备的开发指导的开发人员而言,我们觉得很有必要保留这些教程。As such, we feel it is important to leave these tutorials in place for developers who are still looking for guidance in developing for those devices. 我们 不会 在这些教程中更新 HoloLens 2 所用的最新工具集或集成相关的内容。These tutorials will not be updated with the latest toolsets or interactions being used for HoloLens 2. 我们将维护这些教程,使之持续适用于支持的设备。They will be maintained to continue working on the supported devices. 将来会发布一系列新教程,这些教程将演示如何针对 HoloLens 2 进行开发。There will be a new series of tutorials that will be posted in the future that will demonstrate how to develop for HoloLens 2. 此通知将在发布时通过指向这些教程的链接进行更新。This notice will be updated with a link to those tutorials when they are posted.


在本课程中,你将了解如何在混合现实应用程序中使用 Azure 自定义视觉功能,识别提供的映像中的自定义视觉对象内容。In this course, you will learn how to recognize custom visual content within a provided image, using Azure Custom Vision capabilities in a mixed reality application.

此服务允许你使用对象图像训练机器学习模型。This service will allow you to train a machine learning model using object images. 然后,你将使用训练的模型来识别类似对象,这些对象由 Microsoft HoloLens 的相机捕获或连接到电脑以便沉浸式 (VR) 耳机提供。You will then use the trained model to recognize similar objects, as provided by the camera capture of Microsoft HoloLens or a camera connected to your PC for immersive (VR) headsets.

课程结果

Azure 自定义视觉是一种 Microsoft 认知服务,允许开发人员构建自定义映像分类器。Azure Custom Vision is a Microsoft Cognitive Service which allows developers to build custom image classifiers. 然后,可以将这些分类器与新图像一起使用,以识别或分类该新图像中的对象。These classifiers can then be used with new images to recognize, or classify, objects within that new image. 此服务提供简单易用的联机门户来简化流程。The Service provides a simple, easy to use, online portal to streamline the process. 有关详细信息,请访问 Azure 自定义影像服务页For more information, visit the Azure Custom Vision Service page.

完成本课程后,你将拥有一个混合现实应用程序,可以在两种模式下工作:Upon completion of this course, you will have a mixed reality application which will be able to work in two modes:

  • 分析模式:通过以下方式设置自定义影像服务:上传图像,创建标记,并培训服务,以识别在此情况下鼠标和键盘) (不同对象。Analysis Mode: setting up the Custom Vision Service manually by uploading images, creating tags, and training the Service to recognize different objects (in this case mouse and keyboard). 然后,将创建一个将使用相机捕获图像的 HoloLens 应用,并尝试识别真实环境中的对象。You will then create a HoloLens app that will capture images using the camera, and try to recognize those objects in the real world.

  • 定型模式:将实现在应用中启用 "定型模式" 的代码。Training Mode: you will implement code that will enable a "Training Mode" in your app. 训练模式允许使用 HoloLens 相机捕获图像,将捕获的图像上传到服务,并训练自定义视觉模型。The training mode will allow you to capture images using the HoloLens' camera, upload the captured images to the Service, and train the custom vision model.

本课程将介绍如何将自定义影像服务中的结果获取到基于 Unity 的示例应用程序。This course will teach you how to get the results from the Custom Vision Service into a Unity-based sample application. 您可以将这些概念应用到您可能生成的自定义应用程序。It will be up to you to apply these concepts to a custom application you might be building.

设备支持Device support

课程Course HoloLensHoloLens 沉浸式头戴显示设备Immersive headsets
MR 和 Azure 302b:自定义视觉MR and Azure 302b: Custom vision ✔️✔️ ✔️✔️

备注

尽管本课程主要侧重于 HoloLens,但你也可以将本课程中学习的内容应用于 Windows Mixed Reality 沉浸式 (VR) 耳机。While this course primarily focuses on HoloLens, you can also apply what you learn in this course to Windows Mixed Reality immersive (VR) headsets. 由于沉浸式 (VR) 耳机没有可访问的相机,因此你需要连接到电脑的外置相机。Because immersive (VR) headsets do not have accessible cameras, you will need an external camera connected to your PC. 在本课程中,您将看到有关在支持沉浸式 (VR) 耳机时可能需要执行的任何更改的说明。As you follow along with the course, you will see notes on any changes you might need to employ to support immersive (VR) headsets.

必备条件Prerequisites

备注

本教程专为具有 Unity 和 c # 基本经验的开发人员设计。This tutorial is designed for developers who have basic experience with Unity and C#. 请注意,本文档中的先决条件和书面说明表示在) 2018 年7月 (撰写本文时已测试和验证的内容。Please also be aware that the prerequisites and written instructions within this document represent what has been tested and verified at the time of writing (July 2018). 你可以随意使用最新的软件(如 安装工具 一文中所述),但不应假定本课程中的信息将与下面列出的内容完全匹配。You are free to use the latest software, as listed within the install the tools article, though it should not be assumed that the information in this course will perfectly match what you will find in newer software than what is listed below.

本课程建议采用以下硬件和软件:We recommend the following hardware and software for this course:

开始之前Before you start

  1. 若要避免在生成此项目时遇到问题,强烈建议你在根或近乎根文件夹中创建本教程中所述的项目 (长文件夹路径在生成时) 会导致问题。To avoid encountering issues building this project, it is strongly suggested that you create the project mentioned in this tutorial in a root or near-root folder (long folder paths can cause issues at build-time).
  2. 设置并测试你的 HoloLens。Set up and test your HoloLens. 如果需要支持设置 HoloLens,请 确保访问 hololens 设置一文If you need support setting up your HoloLens, make sure to visit the HoloLens setup article.
  3. 在开始开发新的 HoloLens 应用程序时,最好执行校准和传感器调整 (有时,它可以帮助为每个用户) 执行这些任务。It is a good idea to perform Calibration and Sensor Tuning when beginning developing a new HoloLens app (sometimes it can help to perform those tasks for each user).

有关校准的帮助信息,请单击此链接,了解 到 HoloLens 校准文章For help on Calibration, please follow this link to the HoloLens Calibration article.

有关传感器优化的帮助,请单击 "HoloLens 传感器优化" 一文For help on Sensor Tuning, please follow this link to the HoloLens Sensor Tuning article.

第1章-自定义影像服务门户Chapter 1 - The Custom Vision Service Portal

若要在 Azure 中使用 自定义影像服务 ,你将需要配置服务的实例,使其可用于你的应用程序。To use the Custom Vision Service in Azure, you will need to configure an instance of the Service to be made available to your application.

  1. 首先, 导航到 自定义影像服务 主页First, navigate to the Custom Vision Service main page.

  2. 单击 " 开始 " 按钮。Click on the Get Started button.

    自定义影像服务入门

  3. 登录到 自定义影像服务 门户。Sign in to the Custom Vision Service Portal.

    登录到门户

    备注

    如果还没有 Azure 帐户,则需要创建一个。If you do not already have an Azure account, you will need to create one. 如果在课堂或实验室中按照本教程进行学习,请咨询教师或 proctors,以获得设置新帐户的帮助。If you are following this tutorial in a classroom or lab situation, ask your instructor or one of the proctors for help setting up your new account.

  4. 首次登录后,系统会提示 " 服务 " 面板。Once you are logged in for the first time, you will be prompted with the Terms of Service panel. 单击相应的复选框以同意条款。Click on the checkbox to agree to the terms. 然后单击 " 我同意"。Then click on I agree.

    服务条款

  5. 同意这些条款后,你将会导航到门户的 " 项目 " 部分。Having agreed to the Terms, you will be navigated to the Projects section of the Portal. 单击 " 新建项目"。Click on New Project.

    创建新项目

  6. 右侧将显示一个选项卡,该选项卡将提示你为项目指定某些字段。A tab will appear on the right-hand side, which will prompt you to specify some fields for the project.

    1. 插入项目的 名称Insert a Name for your project.

    2. (可选) 中插入项目的 描述Insert a Description for your project (optional).

    3. 选择一个 资源组 ,或创建一个新的资源组。Choose a Resource Group or create a new one. 资源组提供一种监视、控制访问、预配和管理 Azure 资产集合的计费的方法。A resource group provides a way to monitor, control access, provision and manage billing for a collection of Azure assets. 建议保留与单个项目关联的所有 Azure 服务 (例如,这些课程) 常用资源组) 下。It is recommended to keep all the Azure Services associated with a single project (e.g. such as these courses) under a common resource group).

    4. 项目类型 设置为 分类Set the Project Types to Classification

    5. 设置为 " 常规"。Set the Domains as General.

      设置域

      若要了解有关 Azure 资源组的详细信息,请 访问资源组一文If you wish to read more about Azure Resource Groups, please visit the resource group article.

  7. 完成后,单击 " 创建项目",你会被重定向到自定义影像服务,"项目" 页。Once you are finished, click on Create project, you will be redirected to the Custom Vision Service, project page.

第2章-培训自定义视觉项目Chapter 2 - Training your Custom Vision project

在自定义视觉门户中,你的主要目标是训练你的项目以识别图像中的特定对象。Once in the Custom Vision portal, your primary objective is to train your project to recognize specific objects in images. 对于你希望应用程序识别的每个对象,你需要至少5个 (5) 映像,但最好是10个 (10) 。You need at least five (5) images, though ten (10) is preferred, for each object that you would like your application to recognize. 您可以使用本课程附带的图像 (计算机鼠标和键盘) You can use the images provided with this course (a computer mouse and a keyboard).

训练自定义影像服务项目:To train your Custom Vision Service project:

  1. 单击 " + 标记" 旁边的按钮 Click on the + button next to Tags.

    添加新标记

  2. 添加要识别的对象的 名称Add the name of the object you would like to recognize. 单击“保存” 。Click on Save.

    添加对象名称并保存

  3. 你将注意到已添加 标记 (你可能需要重新加载页面以使其) 显示。You will notice your Tag has been added (you may need to reload your page for it to appear). 单击新标记旁边的复选框(如果尚未选中)。Click the checkbox alongside your new tag, if it is not already checked.

    启用新标记

  4. 单击页中心的 " 添加图像 "。Click on Add Images in the center of the page.

    添加图像

  5. 单击 " 浏览本地文件",搜索,然后选择要上传的映像,最小为 5 (5) 。Click on Browse local files, and search, then select, the images you would like to upload, with the minimum being five (5). 请记住,这些映像应该包含您正在训练的对象。Remember all of these images should contain the object which you are training.

    备注

    你可以一次选择多个图像来上传。You can select several images at a time, to upload.

  6. 在选项卡中看到图像后,请在 " 我的标记 " 框中选择相应的标记。Once you can see the images in the tab, select the appropriate tag in the My Tags box.

    选择标记

  7. 单击 "上 传文件"。Click on Upload files. 文件将开始上传。The files will begin uploading. 确认上传后,单击 " 完成"。Once you have confirmation of the upload, click Done.

    上传文件

  8. 重复相同的过程,创建名为 "键盘" 的新 标记,并为其上传适当的照片。Repeat the same process to create a new Tag named Keyboard and upload the appropriate photos for it. 请确保在创建新标记后 *取消选中***鼠标,以便显示 "添加映像" 窗口。Make sure to uncheck Mouse once you have created the new tags, so to show the Add images window.

  9. 设置两个标记后,单击 " 训练",第一次训练迭代将开始生成。Once you have both Tags set up, click on Train, and the first training iteration will start building.

    启用定型迭代

  10. 构建后,可以看到两个称为 " 创建默认值 " 和 " 预测 URL" 的按钮。Once it's built, you'll be able to see two buttons called Make default and Prediction URL. 先单击 " 设为默认值 ",然后单击 " 预测 URL"。Click on Make default first, then click on Prediction URL.

    设置默认 URL 和预测 URL

    备注

    从此提供的端点 URL 设置为标记为默认值的任何 迭代The endpoint URL which is provided from this, is set to whichever Iteration has been marked as default. 这种情况下,如果您以后生成新的 迭代 并将其更新为默认值,则无需更改代码。As such, if you later make a new Iteration and update it as default, you will not need to change your code.

  11. 单击 " 预测 URL" 后,打开 " 记事本",复制并粘贴该 url预测密钥,以便稍后在代码中需要时进行检索。Once you have clicked on Prediction URL, open Notepad, and copy and paste the URL and the Prediction-Key, so that you can retrieve it when you need it later in the code.

    复制并粘贴 URL 和 Prediction-Key

  12. 单击屏幕右上角的 齿轮Click on the Cog at the top right of the screen.

    单击 "齿轮" 图标打开 "设置"

  13. 复制 定型密钥 ,并将其粘贴到 记事本 中供以后使用。Copy the Training Key and paste it into a Notepad, for later use.

    复制定型密钥

  14. 还应复制 项目 Id,并将其粘贴到 记事本 文件中供以后使用。Also copy your Project Id, and also paste it into your Notepad file, for later use.

    复制项目 id

第3章-设置 Unity 项目Chapter 3 - Set up the Unity project

下面是用于使用混合现实进行开发的典型设置,因此,这是其他项目的一个不错的模板。The following is a typical set up for developing with mixed reality, and as such, is a good template for other projects.

  1. 打开 Unity ,并单击 " 新建"。Open Unity and click New.

    创建新 Unity 项目

  2. 现在需要提供 Unity 项目名称。You will now need to provide a Unity project name. 插入 AzureCustomVision。Insert AzureCustomVision. 请确保将项目模板设置为 3dMake sure the project template is set to 3D. 将位置设置为合适的 位置 (记住,更接近根目录) 。Set the Location to somewhere appropriate for you (remember, closer to root directories is better). 然后单击 " 创建项目"。Then, click Create project.

    配置项目设置

  3. 当 Unity 处于打开状态时,有必要选中 "默认 脚本编辑器 " 设置为 " Visual Studio"。With Unity open, it is worth checking the default Script Editor is set to Visual Studio. 转到 " 编辑 > 首选项 ",然后在新窗口中导航到 "外部工具"。Go to Edit > Preferences and then from the new window, navigate to External Tools. 外部脚本编辑器 更改为 Visual Studio 2017Change External Script Editor to Visual Studio 2017. 关闭 " 首选项 " 窗口。Close the Preferences window.

    配置外部工具

  4. 接下来,转到 " 文件 > 生成设置 ",选择 " 通用 Windows 平台",然后单击 " 切换平台 " 按钮以应用所选内容。Next, go to File > Build Settings and select Universal Windows Platform, then click on the Switch Platform button to apply your selection.

    配置生成设置Configure build settings

  5. 尽管仍处于 文件 > 生成设置 ,但请确保:While still in File > Build Settings and make sure that:

    1. 目标设备 设置为 HoloLensTarget Device is set to HoloLens

      对于沉浸式耳机,将 " 目标设备 " 设置为 " 任何设备"。For the immersive headsets, set Target Device to Any Device.

    2. 生成类型 设置为 D3DBuild Type is set to D3D

    3. SDK 设置为 "最新安装"SDK is set to Latest installed

    4. Visual Studio 版本 设置为 "最新安装"Visual Studio Version is set to Latest installed

    5. "生成并运行" 设置为 "本地计算机"Build and Run is set to Local Machine

    6. 保存场景并将其添加到生成中。Save the scene and add it to the build.

      1. 通过选择 " 添加打开的场景" 来执行此操作。Do this by selecting Add Open Scenes. 将显示 "保存" 窗口。A save window will appear.

        将打开的场景添加到生成列表

      2. 为此创建新文件夹,并为将来的任何场景创建一个新文件夹,然后选择 " 新建文件夹 " 按钮以创建新文件夹,将其命名为 场景Create a new folder for this, and any future, scene, then select the New folder button, to create a new folder, name it Scenes.

        创建新的场景文件夹

      3. 打开新创建的 场景 文件夹,然后在 "文件名 文本" 字段中,键入 CustomVisionScene,然后单击 " 保存"。Open your newly created Scenes folder, and then in the File name: text field, type CustomVisionScene, then click on Save.

        命名新的场景文件

        请注意,必须将 Unity 场景保存在 " 资产 " 文件夹中,因为它们必须与 Unity 项目相关联。Be aware, you must save your Unity scenes within the Assets folder, as they must be associated with the Unity project. 创建场景文件夹 (和其他类似文件夹) 是构建 Unity 项目的典型方式。Creating the scenes folder (and other similar folders) is a typical way of structuring a Unity project.

    7. 现在," 生成设置" 中的其余设置应保留为默认值。The remaining settings, in Build Settings, should be left as default for now.

      默认生成设置

  6. 在 " 生成设置 " 窗口中,单击 " 播放机设置 " 按钮,这会在 检查器 所在的空间中打开相关面板。In the Build Settings window, click on the Player Settings button, this will open the related panel in the space where the Inspector is located.

  7. 在此面板中,需要验证几项设置:In this panel, a few settings need to be verified:

    1. 在 " 其他设置 " 选项卡中:In the Other Settings tab:

      1. 脚本运行时版本( .net 4.6 等效) 试验,这会触发重新启动编辑器的需要。Scripting Runtime Version should be Experimental (.NET 4.6 Equivalent), which will trigger a need to restart the Editor.

      2. 脚本编写后端 应为 .netScripting Backend should be .NET

      3. API 兼容级别 应为 .net 4.6API Compatibility Level should be .NET 4.6

      设置 API compantiblity

    2. 在 " 发布设置 " 选项卡的 " 功能" 下,检查:Within the Publishing Settings tab, under Capabilities, check:

      1. InternetClientInternetClient

      2. 网络摄像头Webcam

      3. 麦克风Microphone

      配置发布设置

    3. 在面板中,在 " XR 设置 " 中, () "发布设置" 下的 " 发布设置 " 下提供了 支持,请确保已添加 Windows Mixed reality SDKFurther down the panel, in XR Settings (found below Publish Settings), tick Virtual Reality Supported, make sure the Windows Mixed Reality SDK is added.

    配置 XR 设置

  8. 返回 生成设置 Unity C # 项目 不再灰显; 勾选此的旁边的复选框。Back in Build Settings Unity C# Projects is no longer greyed out; tick the checkbox next to this.

  9. 关闭“生成设置”窗口。Close the Build Settings window.

  10. 保存场景和项目 (文件 > 保存场景/文件 > 保存项目) 。Save your Scene and project (FILE > SAVE SCENE / FILE > SAVE PROJECT).

第4章-导入 Unity 中的 Newtonsoft.json DLLChapter 4 - Importing the Newtonsoft DLL in Unity

重要

如果你想要跳过本课程的 Unity 设置 组件,并继续直接进入代码,欢迎下载此 302b. unitypackage,将其作为 自定义包导入到你的项目,然后继续 第6章If you wish to skip the Unity Set up component of this course, and continue straight into code, feel free to download this Azure-MR-302b.unitypackage, import it into your project as a Custom Package, and then continue from Chapter 6.

本课程需要使用 newtonsoft.json 库,可将其作为 DLL 添加到资产中。This course requires the use of the Newtonsoft library, which you can add as a DLL to your assets. 可以从此链接下载包含此库的包。The package containing this library can be downloaded from this Link. 若要将 Newtonsoft.json 库导入项目,请使用本课程附带的 Unity 包。To import the Newtonsoft library into your project, use the Unity Package which came with this course.

  1. 使用 " 资产 > 导入**包 > 自定义**包 " 菜单选项将 unitypackage 添加到 Unity。Add the .unitypackage to Unity by using the Assets > Import Package > Custom Package menu option.

  2. 在弹出的 " 导入 Unity 包 " 框中,确保选择 (下的所有内容,包括) 的 插件In the Import Unity Package box that pops up, ensure everything under (and including) Plugins is selected.

    导入所有包项

  3. 单击 " 导入 " 按钮,将项添加到项目。Click the Import button to add the items to your project.

  4. 在项目视图中,在 "插件" 下,单击 " newtonsoft.json " 文件夹,然后选择 " Newtonsoft.Js" 插件Go to the Newtonsoft folder under Plugins in the project view and select the Newtonsoft.Json plugin.

    选择 Newtonsoft.json 插件

  5. 选择 " Newtonsoft.Json" 插件 后,请确保 未选中****任何平台,然后确保 WSAPlayer 未被 选中,然后单击 "应用"。With the Newtonsoft.Json plugin selected, ensure that Any Platform is unchecked, then ensure that WSAPlayer is also unchecked, then click Apply. 这只是为了确认已正确配置文件。This is just to confirm that the files are configured correctly.

    配置 Newtonsoft.json 插件

    备注

    标记这些插件会将它们配置为仅在 Unity 编辑器中使用。Marking these plugins configures them to only be used in the Unity Editor. 在从 Unity 导出项目后,将使用 WSA 文件夹中的一组不同的组。There are a different set of them in the WSA folder which will be used after the project is exported from Unity.

  6. 接下来,需要在 newtonsoft.json 文件夹中打开 WSA 文件夹。Next, you need to open the WSA folder, within the Newtonsoft folder. 你将看到刚才配置的同一文件的副本。You will see a copy of the same file you just configured. 选择该文件,然后在 "检查器" 中,确保Select the file, and then in the inspector, ensure that

    • 未选中****任何平台Any Platform is unchecked
    • 仅****检查 WSAPlayeronly WSAPlayer is checked
    • 检查****进程Dont process is checked

    配置 Newtonsoft.json 插件平台设置

第5章-照相机设置Chapter 5 - Camera setup

  1. 在 "层次结构" 面板中,选择 " 摄像机"。In the Hierarchy Panel, select the Main Camera.

  2. 选择后,你将能够在 "检查器" 面板 中看到 主相机 的所有组件。Once selected, you will be able to see all the components of the Main Camera in the Inspector Panel.

    1. 照相机 对象必须命名为 "主相机" (记下拼写! ) The camera object must be named Main Camera (note the spelling!)

    2. 必须将主相机 标记 设置为 " MainCamera ", (记下拼写! ) The Main Camera Tag must be set to MainCamera (note the spelling!)

    3. 请确保将 转换位置 设置为 0,0,0Make sure the Transform Position is set to 0, 0, 0

    4. 将 " 清除标志 " 设置为 纯色 (为沉浸式头戴式耳机) 忽略此标志。Set Clear Flags to Solid Color (ignore this for immersive headset).

    5. 将相机组件的 背景 色设置为 黑色、Alpha 0 (十六进制代码: #00000000) (为沉浸式耳机) 忽略此颜色。Set the Background Color of the camera Component to Black, Alpha 0 (Hex Code: #00000000) (ignore this for immersive headset).

    配置照相机组件属性

第6章-创建 CustomVisionAnalyser 类。Chapter 6 - Create the CustomVisionAnalyser class.

此时,您可以编写一些代码。At this point you are ready to write some code.

你将从 CustomVisionAnalyser 类开始。You will begin with the CustomVisionAnalyser class.

备注

在下面显示的代码中对 自定义影像服务 所做的调用是使用 自定义视觉 REST API 进行的。The calls to the Custom Vision Service made in the code shown below are made using the Custom Vision REST API. 通过使用此方法,你将了解如何实现和使用此 API (有助于了解如何实现类似于你自己的) 的内容。Through using this, you will see how to implement and make use of this API (useful for understanding how to implement something similar on your own). 请注意,Microsoft 提供了一个 自定义影像服务 SDK ,该 SDK 还可用于对服务进行调用。Be aware, that Microsoft offers a Custom Vision Service SDK that can also be used to make calls to the Service. 有关详细信息,请访问 自定义影像服务 SDK 文章。For more information visit the Custom Vision Service SDK article.

此类负责:This class is responsible for:

  • 加载作为字节数组捕获的最新映像。Loading the latest image captured as an array of bytes.

  • 将字节数组发送给 Azure 自定义影像服务 实例以进行分析。Sending the byte array to your Azure Custom Vision Service instance for analysis.

  • 接收 JSON 字符串形式的响应。Receiving the response as a JSON string.

  • 反序列化响应并将结果 预测 传递给 SceneOrganiser 类,这将负责显示响应的方式。Deserializing the response and passing the resulting Prediction to the SceneOrganiser class, which will take care of how the response should be displayed.

若要创建此类:To create this class:

  1. 右键单击位于 "项目" 面板 中的 资产文件夹,然后单击 "创建 > 文件夹Right-click in the Asset Folder located in the Project Panel, then click Create > Folder. 调用文件夹 脚本Call the folder Scripts.

    创建脚本文件夹

  2. 双击刚创建的文件夹以将其打开。Double-click on the folder just created, to open it.

  3. 右键单击文件夹内,然后单击 "创建 > C # 脚本"。Right-click inside the folder, then click Create > C# Script. 将脚本命名为 CustomVisionAnalyserName the script CustomVisionAnalyser.

  4. 双击新的 CustomVisionAnalyser 脚本以通过 Visual Studio 打开它。Double-click on the new CustomVisionAnalyser script to open it with Visual Studio.

  5. 更新文件顶部的命名空间,以匹配以下内容:Update the namespaces at the top of your file to match the following:

    using System.Collections;
    using System.IO;
    using UnityEngine;
    using UnityEngine.Networking;
    using Newtonsoft.Json;
    
  6. CustomVisionAnalyser 类中,添加以下变量:In the CustomVisionAnalyser class, add the following variables:

        /// <summary>
        /// Unique instance of this class
        /// </summary>
        public static CustomVisionAnalyser Instance;
    
        /// <summary>
        /// Insert your Prediction Key here
        /// </summary>
        private string predictionKey = "- Insert your key here -";
    
        /// <summary>
        /// Insert your prediction endpoint here
        /// </summary>
        private string predictionEndpoint = "Insert your prediction endpoint here";
    
        /// <summary>
        /// Byte array of the image to submit for analysis
        /// </summary>
        [HideInInspector] public byte[] imageBytes;
    

    备注

    请确保将 预测密钥 插入到 predictionKey 变量中,并将 预测终结点 插入到 predictionEndpoint 变量中。Make sure you insert your Prediction Key into the predictionKey variable and your Prediction Endpoint into the predictionEndpoint variable. 您在本课程的前面部分将它们复制到 记事本You copied these to Notepad earlier in the course.

  7. 现在需要添加用于 唤醒 ( # B1 的代码以初始化实例变量:Code for Awake() now needs to be added to initialize the Instance variable:

        /// <summary>
        /// Initialises this class
        /// </summary>
        private void Awake()
        {
            // Allows this instance to behave like a singleton
            Instance = this;
        }
    
  8. 删除方法 开始 ( # B1更新 ( # B3Delete the methods Start() and Update().

  9. 接下来,将协同程序 (与静态 GetImageAsByteArray ( # B2 方法一起添加) ,该方法将获得 ImageCapture 类捕获的映像的分析结果。Next, add the coroutine (with the static GetImageAsByteArray() method below it), which will obtain the results of the analysis of the image captured by the ImageCapture class.

    备注

    AnalyseImageCapture 协同程序中,有一个对你还需要创建的 SceneOrganiser 类的调用。In the AnalyseImageCapture coroutine, there is a call to the SceneOrganiser class that you are yet to create. 因此,请 暂时将这些行注释为Therefore, leave those lines commented for now.

        /// <summary>
        /// Call the Computer Vision Service to submit the image.
        /// </summary>
        public IEnumerator AnalyseLastImageCaptured(string imagePath)
        {
            WWWForm webForm = new WWWForm();
            using (UnityWebRequest unityWebRequest = UnityWebRequest.Post(predictionEndpoint, webForm))
            {
                // Gets a byte array out of the saved image
                imageBytes = GetImageAsByteArray(imagePath);
    
                unityWebRequest.SetRequestHeader("Content-Type", "application/octet-stream");
                unityWebRequest.SetRequestHeader("Prediction-Key", predictionKey);
    
                // The upload handler will help uploading the byte array with the request
                unityWebRequest.uploadHandler = new UploadHandlerRaw(imageBytes);
                unityWebRequest.uploadHandler.contentType = "application/octet-stream";
    
                // The download handler will help receiving the analysis from Azure
                unityWebRequest.downloadHandler = new DownloadHandlerBuffer();
    
                // Send the request
                yield return unityWebRequest.SendWebRequest();
    
                string jsonResponse = unityWebRequest.downloadHandler.text;
    
                // The response will be in JSON format, therefore it needs to be deserialized    
    
                // The following lines refers to a class that you will build in later Chapters
                // Wait until then to uncomment these lines
    
                //AnalysisObject analysisObject = new AnalysisObject();
                //analysisObject = JsonConvert.DeserializeObject<AnalysisObject>(jsonResponse);
                //SceneOrganiser.Instance.SetTagsToLastLabel(analysisObject);
            }
        }
    
        /// <summary>
        /// Returns the contents of the specified image file as a byte array.
        /// </summary>
        static byte[] GetImageAsByteArray(string imageFilePath)
        {
            FileStream fileStream = new FileStream(imageFilePath, FileMode.Open, FileAccess.Read);
    
            BinaryReader binaryReader = new BinaryReader(fileStream);
    
            return binaryReader.ReadBytes((int)fileStream.Length);
        }
    
  10. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

第7章-创建 CustomVisionObjects 类Chapter 7 - Create the CustomVisionObjects class

你现在将创建的类是 CustomVisionObjects 类。The class you will create now is the CustomVisionObjects class.

此脚本包含其他类用于序列化和反序列化对 自定义影像服务 进行的调用的许多对象。This script contains a number of objects used by other classes to serialize and deserialize the calls made to the Custom Vision Service.

警告

请务必记下自定义影像服务提供的终结点,因为下面的 JSON 结构已设置为使用 自定义视觉预测v2.0。It is important that you take note of the endpoint that the Custom Vision Service provides you, as the below JSON structure has been set up to work with Custom Vision Prediction v2.0. 如果你的版本不同,可能需要更新以下结构。If you have a different version, you may need to update the below structure.

若要创建此类:To create this class:

  1. 右键单击 "脚本" 文件夹内,然后单击 "创建 > C # 脚本"。Right-click inside the Scripts folder, then click Create > C# Script. 调用脚本 CustomVisionObjectsCall the script CustomVisionObjects.

  2. 双击新的 CustomVisionObjects 脚本以通过 Visual Studio 打开它。Double-click on the new CustomVisionObjects script to open it with Visual Studio.

  3. 将以下命名空间添加到文件顶部:Add the following namespaces to the top of the file:

    using System;
    using System.Collections.Generic;
    using UnityEngine;
    using UnityEngine.Networking;
    
  4. 删除 CustomVisionObjects 类中的 开始 ( # B1更新 ( # B3 方法;此类现在应为空。Delete the Start() and Update() methods inside the CustomVisionObjects class; this class should now be empty.

  5. 将以下类添加到 CustomVisionObjects 类的 外部Add the following classes outside the CustomVisionObjects class. Newtonsoft.json 库使用这些对象序列化和反序列化响应数据:These objects are used by the Newtonsoft library to serialize and deserialize the response data:

    // The objects contained in this script represent the deserialized version
    // of the objects used by this application 
    
    /// <summary>
    /// Web request object for image data
    /// </summary>
    class MultipartObject : IMultipartFormSection
    {
        public string sectionName { get; set; }
    
        public byte[] sectionData { get; set; }
    
        public string fileName { get; set; }
    
        public string contentType { get; set; }
    }
    
    /// <summary>
    /// JSON of all Tags existing within the project
    /// contains the list of Tags
    /// </summary> 
    public class Tags_RootObject
    {
        public List<TagOfProject> Tags { get; set; }
        public int TotalTaggedImages { get; set; }
        public int TotalUntaggedImages { get; set; }
    }
    
    public class TagOfProject
    {
        public string Id { get; set; }
        public string Name { get; set; }
        public string Description { get; set; }
        public int ImageCount { get; set; }
    }
    
    /// <summary>
    /// JSON of Tag to associate to an image
    /// Contains a list of hosting the tags,
    /// since multiple tags can be associated with one image
    /// </summary> 
    public class Tag_RootObject
    {
        public List<Tag> Tags { get; set; }
    }
    
    public class Tag
    {
        public string ImageId { get; set; }
        public string TagId { get; set; }
    }
    
    /// <summary>
    /// JSON of Images submitted
    /// Contains objects that host detailed information about one or more images
    /// </summary> 
    public class ImageRootObject
    {
        public bool IsBatchSuccessful { get; set; }
        public List<SubmittedImage> Images { get; set; }
    }
    
    public class SubmittedImage
    {
        public string SourceUrl { get; set; }
        public string Status { get; set; }
        public ImageObject Image { get; set; }
    }
    
    public class ImageObject
    {
        public string Id { get; set; }
        public DateTime Created { get; set; }
        public int Width { get; set; }
        public int Height { get; set; }
        public string ImageUri { get; set; }
        public string ThumbnailUri { get; set; }
    }
    
    /// <summary>
    /// JSON of Service Iteration
    /// </summary> 
    public class Iteration
    {
        public string Id { get; set; }
        public string Name { get; set; }
        public bool IsDefault { get; set; }
        public string Status { get; set; }
        public string Created { get; set; }
        public string LastModified { get; set; }
        public string TrainedAt { get; set; }
        public string ProjectId { get; set; }
        public bool Exportable { get; set; }
        public string DomainId { get; set; }
    }
    
    /// <summary>
    /// Predictions received by the Service after submitting an image for analysis
    /// </summary> 
    [Serializable]
    public class AnalysisObject
    {
        public List<Prediction> Predictions { get; set; }
    }
    
    [Serializable]
    public class Prediction
    {
        public string TagName { get; set; }
        public double Probability { get; set; }
    }
    

第8章-创建 VoiceRecognizer 类Chapter 8 - Create the VoiceRecognizer class

此类将识别用户的语音输入。This class will recognize the voice input from the user.

若要创建此类:To create this class:

  1. 右键单击 "脚本" 文件夹内,然后单击 "创建 > C # 脚本"。Right-click inside the Scripts folder, then click Create > C# Script. 调用脚本 VoiceRecognizerCall the script VoiceRecognizer.

  2. 双击新的 VoiceRecognizer 脚本以通过 Visual Studio 打开它。Double-click on the new VoiceRecognizer script to open it with Visual Studio.

  3. 将以下命名空间添加到 VoiceRecognizer 类的上方:Add the following namespaces above the VoiceRecognizer class:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using UnityEngine;
    using UnityEngine.Windows.Speech;
    
  4. 然后,将以下变量添加到 VoiceRecognizer 类中的 Start ( # B1 方法之上:Then add the following variables inside the VoiceRecognizer class, above the Start() method:

        /// <summary>
        /// Allows this class to behave like a singleton
        /// </summary>
        public static VoiceRecognizer Instance;
    
        /// <summary>
        /// Recognizer class for voice recognition
        /// </summary>
        internal KeywordRecognizer keywordRecognizer;
    
        /// <summary>
        /// List of Keywords registered
        /// </summary>
        private Dictionary<string, Action> _keywords = new Dictionary<string, Action>();
    
  5. 添加 唤醒 ( # B1开始 ( # B3 方法,后者将设置在将标记关联到图像时要识别的用户 关键字Add the Awake() and Start() methods, the latter of which will set up the user keywords to be recognized when associating a tag to an image:

        /// <summary>
        /// Called on initialization
        /// </summary>
        private void Awake()
        {
            Instance = this;
        }
    
        /// <summary>
        /// Runs at initialization right after Awake method
        /// </summary>
        void Start ()
        {
    
            Array tagsArray = Enum.GetValues(typeof(CustomVisionTrainer.Tags));
    
            foreach (object tagWord in tagsArray)
            {
                _keywords.Add(tagWord.ToString(), () =>
                {
                    // When a word is recognized, the following line will be called
                    CustomVisionTrainer.Instance.VerifyTag(tagWord.ToString());
                });
            }
    
            _keywords.Add("Discard", () =>
            {
                // When a word is recognized, the following line will be called
                // The user does not want to submit the image
                // therefore ignore and discard the process
                ImageCapture.Instance.ResetImageCapture();
                keywordRecognizer.Stop();
            });
    
            //Create the keyword recognizer 
            keywordRecognizer = new KeywordRecognizer(_keywords.Keys.ToArray());
    
            // Register for the OnPhraseRecognized event 
            keywordRecognizer.OnPhraseRecognized += KeywordRecognizer_OnPhraseRecognized;
        }
    
  6. 删除 ( # B1 方法的更新。Delete the Update() method.

  7. 添加以下处理程序,只要识别语音输入,就会调用该处理程序:Add the following handler, which is called whenever voice input is recognized:

        /// <summary>
        /// Handler called when a word is recognized
        /// </summary>
        private void KeywordRecognizer_OnPhraseRecognized(PhraseRecognizedEventArgs args)
        {
            Action keywordAction;
            // if the keyword recognized is in our dictionary, call that Action.
            if (_keywords.TryGetValue(args.text, out keywordAction))
            {
                keywordAction.Invoke();
            }
        }
    
  8. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

备注

不要担心可能出现错误的代码,因为您很快就会提供更多的类,这将修复这些问题。Do not worry about code which might appear to have an error, as you will provide further classes soon, which will fix these.

第9章-创建 CustomVisionTrainer 类Chapter 9 - Create the CustomVisionTrainer class

此类将链接一系列 web 调用以训练 自定义影像服务This class will chain a series of web calls to train the Custom Vision Service. 每次调用都将在代码的上方详细说明。Each call will be explained in detail right above the code.

若要创建此类:To create this class:

  1. 右键单击 "脚本" 文件夹内,然后单击 "创建 > C # 脚本"。Right-click inside the Scripts folder, then click Create > C# Script. 调用脚本 CustomVisionTrainerCall the script CustomVisionTrainer.

  2. 双击新的 CustomVisionTrainer 脚本以通过 Visual Studio 打开它。Double-click on the new CustomVisionTrainer script to open it with Visual Studio.

  3. 将以下命名空间添加到 CustomVisionTrainer 类的上方:Add the following namespaces above the CustomVisionTrainer class:

    using Newtonsoft.Json;
    using System.Collections;
    using System.Collections.Generic;
    using System.IO;
    using System.Text;
    using UnityEngine;
    using UnityEngine.Networking;
    
  4. 然后,将以下变量添加到 CustomVisionTrainer 类中的 Start ( # B1 方法之上。Then add the following variables inside the CustomVisionTrainer class, above the Start() method.

    备注

    此处使用的训练 URL 在 自定义视觉培训 1.2 文档中提供,并具有以下结构: https://southcentralus.api.cognitive.microsoft.com/customvision/v1.2/Training/projects/{projectId}/The training URL used here is provided within the Custom Vision Training 1.2 documentation, and has a structure of: https://southcentralus.api.cognitive.microsoft.com/customvision/v1.2/Training/projects/{projectId}/
    有关详细信息,请访问 自定义视觉培训 v1.0 参考 APIFor more information, visit the Custom Vision Training v1.2 reference API.

    警告

    请务必记下自定义影像服务为你提供培训模式的终结点,因为在 CustomVisionObjects) 类中使用的 JSON 结构已设置为与 自定义视觉培训v1.0 一起使用 (。It is important that you take note of the endpoint that the Custom Vision Service provides you for the training mode, as the JSON structure used (within the CustomVisionObjects class) has been set up to work with Custom Vision Training v1.2. 如果有不同的版本,则可能需要更新 对象 结构。If you have a different version, you may need to update the Objects structure.

        /// <summary>
        /// Allows this class to behave like a singleton
        /// </summary>
        public static CustomVisionTrainer Instance;
    
        /// <summary>
        /// Custom Vision Service URL root
        /// </summary>
        private string url = "https://southcentralus.api.cognitive.microsoft.com/customvision/v1.2/Training/projects/";
    
        /// <summary>
        /// Insert your prediction key here
        /// </summary>
        private string trainingKey = "- Insert your key here -";
    
        /// <summary>
        /// Insert your Project Id here
        /// </summary>
        private string projectId = "- Insert your Project Id here -";
    
        /// <summary>
        /// Byte array of the image to submit for analysis
        /// </summary>
        internal byte[] imageBytes;
    
        /// <summary>
        /// The Tags accepted
        /// </summary>
        internal enum Tags {Mouse, Keyboard}
    
        /// <summary>
        /// The UI displaying the training Chapters
        /// </summary>
        private TextMesh trainingUI_TextMesh;
    

    重要

    请确保将你的 服务密钥 添加 (定型密钥) 值和 项目 Id 值,你之前记下;这是你 在本课程前面部分中收集的值 (第2章 "第10步")Ensure that you add your Service Key (Training Key) value and Project Id value, which you noted down previous; these are the values you collected from the portal earlier in the course (Chapter 2, step 10 onwards).

  5. 添加以下 开始 ( # B1唤醒 ( # B3 方法。Add the following Start() and Awake() methods. 这些方法在初始化时调用,并包含设置 UI 的调用:Those methods are called on initialization and contain the call to set up the UI:

        /// <summary>
        /// Called on initialization
        /// </summary>
        private void Awake()
        {
            Instance = this;
        }
    
        /// <summary>
        /// Runs at initialization right after Awake method
        /// </summary>
        private void Start()
        { 
            trainingUI_TextMesh = SceneOrganiser.Instance.CreateTrainingUI("TrainingUI", 0.04f, 0, 4, false);
        }
    
  6. 删除 ( # B1 方法的更新。Delete the Update() method. 此类不需要此类。This class will not need it.

  7. 添加 RequestTagSelection ( # B1 方法。Add the RequestTagSelection() method. 此方法是在捕获图像并将其存储在设备中,并且现在已准备好提交到 自定义影像服务 的第一个调用。This method is the first to be called when an image has been captured and stored in the device and is now ready to be submitted to the Custom Vision Service, to train it. 此方法在定型 UI 中显示一组关键字,用户可以使用这些关键字标记已捕获的映像。This method displays, in the training UI, a set of keywords the user can use to tag the image that has been captured. 它还提醒 VoiceRecognizer 类开始倾听用户的语音输入。It also alerts the VoiceRecognizer class to begin listening to the user for voice input.

        internal void RequestTagSelection()
        {
            trainingUI_TextMesh.gameObject.SetActive(true);
            trainingUI_TextMesh.text = $" \nUse voice command \nto choose between the following tags: \nMouse\nKeyboard \nor say Discard";
    
            VoiceRecognizer.Instance.keywordRecognizer.Start();
        }
    
  8. 添加 VerifyTag ( # B1 方法。Add the VerifyTag() method. 此方法将接收 VoiceRecognizer 类识别的语音输入并验证其有效性,然后开始训练过程。This method will receive the voice input recognized by the VoiceRecognizer class and verify its validity, and then begin the training process.

        /// <summary>
        /// Verify voice input against stored tags.
        /// If positive, it will begin the Service training process.
        /// </summary>
        internal void VerifyTag(string spokenTag)
        {
            if (spokenTag == Tags.Mouse.ToString() || spokenTag == Tags.Keyboard.ToString())
            {
                trainingUI_TextMesh.text = $"Tag chosen: {spokenTag}";
                VoiceRecognizer.Instance.keywordRecognizer.Stop();
                StartCoroutine(SubmitImageForTraining(ImageCapture.Instance.filePath, spokenTag));
            }
        }
    
  9. 添加 SubmitImageForTraining ( # B1 方法。Add the SubmitImageForTraining() method. 此方法将开始自定义影像服务训练过程。This method will begin the Custom Vision Service training process. 第一步是从服务中检索与用户的经验证的语音输入相关联的 标记 IdThe first step is to retrieve the Tag Id from the Service which is associated with the validated speech input from the user. 标记 Id 随后将连同图像一起上传。The Tag Id will then be uploaded along with the image.

        /// <summary>
        /// Call the Custom Vision Service to submit the image.
        /// </summary>
        public IEnumerator SubmitImageForTraining(string imagePath, string tag)
        {
            yield return new WaitForSeconds(2);
            trainingUI_TextMesh.text = $"Submitting Image \nwith tag: {tag} \nto Custom Vision Service";
            string imageId = string.Empty;
            string tagId = string.Empty;
    
            // Retrieving the Tag Id relative to the voice input
            string getTagIdEndpoint = string.Format("{0}{1}/tags", url, projectId);
            using (UnityWebRequest www = UnityWebRequest.Get(getTagIdEndpoint))
            {
                www.SetRequestHeader("Training-Key", trainingKey);
                www.downloadHandler = new DownloadHandlerBuffer();
                yield return www.SendWebRequest();
                string jsonResponse = www.downloadHandler.text;
    
                Tags_RootObject tagRootObject = JsonConvert.DeserializeObject<Tags_RootObject>(jsonResponse);
    
                foreach (TagOfProject tOP in tagRootObject.Tags)
                {
                    if (tOP.Name == tag)
                    {
                        tagId = tOP.Id;
                    }             
                }
            }
    
            // Creating the image object to send for training
            List<IMultipartFormSection> multipartList = new List<IMultipartFormSection>();
            MultipartObject multipartObject = new MultipartObject();
            multipartObject.contentType = "application/octet-stream";
            multipartObject.fileName = "";
            multipartObject.sectionData = GetImageAsByteArray(imagePath);
            multipartList.Add(multipartObject);
    
            string createImageFromDataEndpoint = string.Format("{0}{1}/images?tagIds={2}", url, projectId, tagId);
    
            using (UnityWebRequest www = UnityWebRequest.Post(createImageFromDataEndpoint, multipartList))
            {
                // Gets a byte array out of the saved image
                imageBytes = GetImageAsByteArray(imagePath);           
    
                //unityWebRequest.SetRequestHeader("Content-Type", "application/octet-stream");
                www.SetRequestHeader("Training-Key", trainingKey);
    
                // The upload handler will help uploading the byte array with the request
                www.uploadHandler = new UploadHandlerRaw(imageBytes);
    
                // The download handler will help receiving the analysis from Azure
                www.downloadHandler = new DownloadHandlerBuffer();
    
                // Send the request
                yield return www.SendWebRequest();
    
                string jsonResponse = www.downloadHandler.text;
    
                ImageRootObject m = JsonConvert.DeserializeObject<ImageRootObject>(jsonResponse);
                imageId = m.Images[0].Image.Id;
            }
            trainingUI_TextMesh.text = "Image uploaded";
            StartCoroutine(TrainCustomVisionProject());
        }
    
  10. 添加 TrainCustomVisionProject ( # B1 方法。Add the TrainCustomVisionProject() method. 提交并标记该映像后,将调用此方法。Once the image has been submitted and tagged, this method will be called. 它将创建一个新的迭代,该迭代将使用提交到服务的所有之前的图像以及刚刚上传的图像进行训练。It will create a new Iteration that will be trained with all the previous images submitted to the Service plus the image just uploaded. 完成训练后,此方法将调用方法将新创建的 迭代 设置为 默认值,以便用于分析的终结点是最新的定型迭代。Once the training has been completed, this method will call a method to set the newly created Iteration as Default, so that the endpoint you are using for analysis is the latest trained iteration.

        /// <summary>
        /// Call the Custom Vision Service to train the Service.
        /// It will generate a new Iteration in the Service
        /// </summary>
        public IEnumerator TrainCustomVisionProject()
        {
            yield return new WaitForSeconds(2);
    
            trainingUI_TextMesh.text = "Training Custom Vision Service";
    
            WWWForm webForm = new WWWForm();
    
            string trainProjectEndpoint = string.Format("{0}{1}/train", url, projectId);
    
            using (UnityWebRequest www = UnityWebRequest.Post(trainProjectEndpoint, webForm))
            {
                www.SetRequestHeader("Training-Key", trainingKey);
                www.downloadHandler = new DownloadHandlerBuffer();
                yield return www.SendWebRequest();
                string jsonResponse = www.downloadHandler.text;
                Debug.Log($"Training - JSON Response: {jsonResponse}");
    
                // A new iteration that has just been created and trained
                Iteration iteration = new Iteration();
                iteration = JsonConvert.DeserializeObject<Iteration>(jsonResponse);
    
                if (www.isDone)
                {
                    trainingUI_TextMesh.text = "Custom Vision Trained";
    
                    // Since the Service has a limited number of iterations available,
                    // we need to set the last trained iteration as default
                    // and delete all the iterations you dont need anymore
                    StartCoroutine(SetDefaultIteration(iteration)); 
                }
            }
        }
    
  11. 添加 SetDefaultIteration ( # B1 方法。Add the SetDefaultIteration() method. 此方法会将以前创建和训练的迭代设置为 默认值This method will set the previously created and trained iteration as Default. 完成后,此方法将不得不删除服务中的上一个迭代。Once completed, this method will have to delete the previous iteration existing in the Service. 撰写本课程时,在服务中允许同时存在的最大 10 (10) 迭代数限制。At the time of writing this course, there is a limit of a maximum ten (10) iterations allowed to exist at the same time in the Service.

        /// <summary>
        /// Set the newly created iteration as Default
        /// </summary>
        private IEnumerator SetDefaultIteration(Iteration iteration)
        {
            yield return new WaitForSeconds(5);
            trainingUI_TextMesh.text = "Setting default iteration";
    
            // Set the last trained iteration to default
            iteration.IsDefault = true;
    
            // Convert the iteration object as JSON
            string iterationAsJson = JsonConvert.SerializeObject(iteration);
            byte[] bytes = Encoding.UTF8.GetBytes(iterationAsJson);
    
            string setDefaultIterationEndpoint = string.Format("{0}{1}/iterations/{2}", 
                                                            url, projectId, iteration.Id);
    
            using (UnityWebRequest www = UnityWebRequest.Put(setDefaultIterationEndpoint, bytes))
            {
                www.method = "PATCH";
                www.SetRequestHeader("Training-Key", trainingKey);
                www.SetRequestHeader("Content-Type", "application/json");
                www.downloadHandler = new DownloadHandlerBuffer();
    
                yield return www.SendWebRequest();
    
                string jsonResponse = www.downloadHandler.text;
    
                if (www.isDone)
                {
                    trainingUI_TextMesh.text = "Default iteration is set \nDeleting Unused Iteration";
                    StartCoroutine(DeletePreviousIteration(iteration));
                }
            }
        }
    
  12. 添加 DeletePreviousIteration ( # B1 方法。Add the DeletePreviousIteration() method. 此方法将查找并删除以前的非默认迭代:This method will find and delete the previous non-default iteration:

        /// <summary>
        /// Delete the previous non-default iteration.
        /// </summary>
        public IEnumerator DeletePreviousIteration(Iteration iteration)
        {
            yield return new WaitForSeconds(5);
    
            trainingUI_TextMesh.text = "Deleting Unused \nIteration";
    
            string iterationToDeleteId = string.Empty;
    
            string findAllIterationsEndpoint = string.Format("{0}{1}/iterations", url, projectId);
    
            using (UnityWebRequest www = UnityWebRequest.Get(findAllIterationsEndpoint))
            {
                www.SetRequestHeader("Training-Key", trainingKey);
                www.downloadHandler = new DownloadHandlerBuffer();
                yield return www.SendWebRequest();
    
                string jsonResponse = www.downloadHandler.text;
    
                // The iteration that has just been trained
                List<Iteration> iterationsList = new List<Iteration>();
                iterationsList = JsonConvert.DeserializeObject<List<Iteration>>(jsonResponse);
    
                foreach (Iteration i in iterationsList)
                {
                    if (i.IsDefault != true)
                    {
                        Debug.Log($"Cleaning - Deleting iteration: {i.Name}, {i.Id}");
                        iterationToDeleteId = i.Id;
                        break;
                    }
                }
            }
    
            string deleteEndpoint = string.Format("{0}{1}/iterations/{2}", url, projectId, iterationToDeleteId);
    
            using (UnityWebRequest www2 = UnityWebRequest.Delete(deleteEndpoint))
            {
                www2.SetRequestHeader("Training-Key", trainingKey);
                www2.downloadHandler = new DownloadHandlerBuffer();
                yield return www2.SendWebRequest();
                string jsonResponse = www2.downloadHandler.text;
    
                trainingUI_TextMesh.text = "Iteration Deleted";
                yield return new WaitForSeconds(2);
                trainingUI_TextMesh.text = "Ready for next \ncapture";
    
                yield return new WaitForSeconds(2);
                trainingUI_TextMesh.text = "";
                ImageCapture.Instance.ResetImageCapture();
            }
        }
    
  13. 要添加到此类中的最后一个方法是 GetImageAsByteArray ( # B1 方法,用于在 web 调用上将捕获的图像转换为字节数组。The last method to add in this class is the GetImageAsByteArray() method, used on the web calls to convert the image captured into a byte array.

        /// <summary>
        /// Returns the contents of the specified image file as a byte array.
        /// </summary>
        static byte[] GetImageAsByteArray(string imageFilePath)
        {
            FileStream fileStream = new FileStream(imageFilePath, FileMode.Open, FileAccess.Read);
            BinaryReader binaryReader = new BinaryReader(fileStream);
            return binaryReader.ReadBytes((int)fileStream.Length);
        }
    
  14. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

第10章-创建 SceneOrganiser 类Chapter 10 - Create the SceneOrganiser class

此类将:This class will:

  • 创建要附加到主相机的 Cursor 对象。Create a Cursor object to attach to the Main Camera.

  • 创建一个 标签 对象,该对象将在服务识别现实世界对象时显示。Create a Label object that will appear when the Service recognizes the real-world objects.

  • 通过向主摄像机附加相应的组件来设置它。Set up the Main Camera by attaching the appropriate components to it.

  • 分析模式 下,将在运行时、相对于主摄像机位置的适当世界空间中生成标签,并显示从自定义影像服务收到的数据。When in Analysis Mode, spawn the Labels at runtime, in the appropriate world space relative to the position of the Main Camera, and display the data received from the Custom Vision Service.

  • 处于 定型模式 时,将生成 UI,该 UI 将显示训练过程的不同阶段。When in Training Mode, spawn the UI that will display the different stages of the training process.

若要创建此类:To create this class:

  1. 右键单击 "脚本" 文件夹内,然后单击 "创建 > C # 脚本"。Right-click inside the Scripts folder, then click Create > C# Script. 将脚本命名为 SceneOrganiserName the script SceneOrganiser.

  2. 双击新的 SceneOrganiser 脚本以通过 Visual Studio 打开它。Double-click on the new SceneOrganiser script to open it with Visual Studio.

  3. 你只需要一个命名空间,并从 SceneOrganiser 类的上方删除其他命名空间:You will only need one namespace, remove the others from above the SceneOrganiser class:

    using UnityEngine;
    
  4. 然后,将以下变量添加到 SceneOrganiser 类中的 Start ( # B1 方法之上:Then add the following variables inside the SceneOrganiser class, above the Start() method:

        /// <summary>
        /// Allows this class to behave like a singleton
        /// </summary>
        public static SceneOrganiser Instance;
    
        /// <summary>
        /// The cursor object attached to the camera
        /// </summary>
        internal GameObject cursor;
    
        /// <summary>
        /// The label used to display the analysis on the objects in the real world
        /// </summary>
        internal GameObject label;
    
        /// <summary>
        /// Object providing the current status of the camera.
        /// </summary>
        internal TextMesh cameraStatusIndicator;
    
        /// <summary>
        /// Reference to the last label positioned
        /// </summary>
        internal Transform lastLabelPlaced;
    
        /// <summary>
        /// Reference to the last label positioned
        /// </summary>
        internal TextMesh lastLabelPlacedText;
    
        /// <summary>
        /// Current threshold accepted for displaying the label
        /// Reduce this value to display the recognition more often
        /// </summary>
        internal float probabilityThreshold = 0.5f;
    
  5. 删除 开始 ( # B1更新 ( # B3 方法。Delete the Start() and Update() methods.

  6. 在变量的下方,添加 唤醒 ( # B1 方法,这将初始化类并设置场景。Right underneath the variables, add the Awake() method, which will initialize the class and set up the scene.

        /// <summary>
        /// Called on initialization
        /// </summary>
        private void Awake()
        {
            // Use this class instance as singleton
            Instance = this;
    
            // Add the ImageCapture class to this GameObject
            gameObject.AddComponent<ImageCapture>();
    
            // Add the CustomVisionAnalyser class to this GameObject
            gameObject.AddComponent<CustomVisionAnalyser>();
    
            // Add the CustomVisionTrainer class to this GameObject
            gameObject.AddComponent<CustomVisionTrainer>();
    
            // Add the VoiceRecogniser class to this GameObject
            gameObject.AddComponent<VoiceRecognizer>();
    
            // Add the CustomVisionObjects class to this GameObject
            gameObject.AddComponent<CustomVisionObjects>();
    
            // Create the camera Cursor
            cursor = CreateCameraCursor();
    
            // Load the label prefab as reference
            label = CreateLabel();
    
            // Create the camera status indicator label, and place it above where predictions
            // and training UI will appear.
            cameraStatusIndicator = CreateTrainingUI("Status Indicator", 0.02f, 0.2f, 3, true);
    
            // Set camera status indicator to loading.
            SetCameraStatus("Loading");
        }
    
  7. 现在,添加用于创建和定位主相机光标的 CreateCameraCursor ( # B1 方法,并添加 CreateLabel ( # B3 方法,该方法创建 分析标签 对象。Now add the CreateCameraCursor() method that creates and positions the Main Camera cursor, and the CreateLabel() method, which creates the Analysis Label object.

        /// <summary>
        /// Spawns cursor for the Main Camera
        /// </summary>
        private GameObject CreateCameraCursor()
        {
            // Create a sphere as new cursor
            GameObject newCursor = GameObject.CreatePrimitive(PrimitiveType.Sphere);
    
            // Attach it to the camera
            newCursor.transform.parent = gameObject.transform;
    
            // Resize the new cursor
            newCursor.transform.localScale = new Vector3(0.02f, 0.02f, 0.02f);
    
            // Move it to the correct position
            newCursor.transform.localPosition = new Vector3(0, 0, 4);
    
            // Set the cursor color to red
            newCursor.GetComponent<Renderer>().material = new Material(Shader.Find("Diffuse"));
            newCursor.GetComponent<Renderer>().material.color = Color.green;
    
            return newCursor;
        }
    
        /// <summary>
        /// Create the analysis label object
        /// </summary>
        private GameObject CreateLabel()
        {
            // Create a sphere as new cursor
            GameObject newLabel = new GameObject();
    
            // Resize the new cursor
            newLabel.transform.localScale = new Vector3(0.01f, 0.01f, 0.01f);
    
            // Creating the text of the label
            TextMesh t = newLabel.AddComponent<TextMesh>();
            t.anchor = TextAnchor.MiddleCenter;
            t.alignment = TextAlignment.Center;
            t.fontSize = 50;
            t.text = "";
    
            return newLabel;
        }
    
  8. 添加 SetCameraStatus ( # B1 方法,该方法将处理旨在提供相机状态的文本网格的消息。Add the SetCameraStatus() method, which will handle messages intended for the text mesh providing the status of the camera.

        /// <summary>
        /// Set the camera status to a provided string. Will be coloured if it matches a keyword.
        /// </summary>
        /// <param name="statusText">Input string</param>
        public void SetCameraStatus(string statusText)
        {
            if (string.IsNullOrEmpty(statusText) == false)
            {
                string message = "white";
    
                switch (statusText.ToLower())
                {
                    case "loading":
                        message = "yellow";
                        break;
    
                    case "ready":
                        message = "green";
                        break;
    
                    case "uploading image":
                        message = "red";
                        break;
    
                    case "looping capture":
                        message = "yellow";
                        break;
    
                    case "analysis":
                        message = "red";
                        break;
                }
    
                cameraStatusIndicator.GetComponent<TextMesh>().text = $"Camera Status:\n<color={message}>{statusText}..</color>";
            }
        }
    
  9. PlaceAnalysisLabel ( # B1SetTagsToLastLabel ( # B3 方法,该方法将生成数据并将其显示自定义影像服务到场景中。Add the PlaceAnalysisLabel() and SetTagsToLastLabel() methods, which will spawn and display the data from the Custom Vision Service into the scene.

        /// <summary>
        /// Instantiate a label in the appropriate location relative to the Main Camera.
        /// </summary>
        public void PlaceAnalysisLabel()
        {
            lastLabelPlaced = Instantiate(label.transform, cursor.transform.position, transform.rotation);
            lastLabelPlacedText = lastLabelPlaced.GetComponent<TextMesh>();
        }
    
        /// <summary>
        /// Set the Tags as Text of the last label created. 
        /// </summary>
        public void SetTagsToLastLabel(AnalysisObject analysisObject)
        {
            lastLabelPlacedText = lastLabelPlaced.GetComponent<TextMesh>();
    
            if (analysisObject.Predictions != null)
            {
                foreach (Prediction p in analysisObject.Predictions)
                {
                    if (p.Probability > 0.02)
                    {
                        lastLabelPlacedText.text += $"Detected: {p.TagName} {p.Probability.ToString("0.00 \n")}";
                        Debug.Log($"Detected: {p.TagName} {p.Probability.ToString("0.00 \n")}");
                    }
                }
            }
        }
    
  10. 最后,添加 CreateTrainingUI ( # B1 方法,该方法将生成在应用程序处于定型模式时显示训练过程的多个阶段的 UI。Lastly, add the CreateTrainingUI() method, which will spawn the UI displaying the multiple stages of the training process when the application is in Training Mode. 此方法也将伴随创建相机状态对象。This method will also be harnessed to create the camera status object.

        /// <summary>
        /// Create a 3D Text Mesh in scene, with various parameters.
        /// </summary>
        /// <param name="name">name of object</param>
        /// <param name="scale">scale of object (i.e. 0.04f)</param>
        /// <param name="yPos">height above the cursor (i.e. 0.3f</param>
        /// <param name="zPos">distance from the camera</param>
        /// <param name="setActive">whether the text mesh should be visible when it has been created</param>
        /// <returns>Returns a 3D text mesh within the scene</returns>
        internal TextMesh CreateTrainingUI(string name, float scale, float yPos, float zPos, bool setActive)
        {
            GameObject display = new GameObject(name, typeof(TextMesh));
            display.transform.parent = Camera.main.transform;
            display.transform.localPosition = new Vector3(0, yPos, zPos);
            display.SetActive(setActive);
            display.transform.localScale = new Vector3(scale, scale, scale);
            display.transform.rotation = new Quaternion();
            TextMesh textMesh = display.GetComponent<TextMesh>();
            textMesh.anchor = TextAnchor.MiddleCenter;
            textMesh.alignment = TextAlignment.Center;
            return textMesh;
        }
    
  11. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

重要

继续之前,请打开 CustomVisionAnalyser 类,并在 AnalyseLastImageCaptured 中 ( # B1 方法, 取消注释 以下行:Before you continue, open the CustomVisionAnalyser class, and within the AnalyseLastImageCaptured() method, uncomment the following lines:

  AnalysisObject analysisObject = new AnalysisObject();
  analysisObject = JsonConvert.DeserializeObject<AnalysisObject>(jsonResponse);
  SceneOrganiser.Instance.SetTagsToLastLabel(analysisObject);

第11章-创建 ImageCapture 类Chapter 11 - Create the ImageCapture class

要创建的下一个类是 ImageCapture 类。The next class you are going to create is the ImageCapture class.

此类负责:This class is responsible for:

  • 使用 HoloLens 相机捕获映像,并将其存储在 App 文件夹中。Capturing an image using the HoloLens camera and storing it in the App Folder.

  • 处理用户的敲击手势。Handling Tap gestures from the user.

  • 维护用于确定应用程序是否将在 分析 模式或 定型 模式下运行的 枚举 值。Maintaining the Enum value that determines if the application will run in Analysis mode or Training Mode.

若要创建此类:To create this class:

  1. 中转到前面创建的 " 脚本 " 文件夹。Go to the Scripts folder you created previously.

  2. 右键单击文件夹内,然后单击 " 创建 > C # 脚本"。Right-click inside the folder, then click Create > C# Script. 将脚本命名为 ImageCaptureName the script ImageCapture.

  3. 双击新的 ImageCapture 脚本以通过 Visual Studio 打开它。Double-click on the new ImageCapture script to open it with Visual Studio.

  4. 将文件顶部的命名空间替换为以下内容:Replace the namespaces at the top of the file with the following:

    using System;
    using System.IO;
    using System.Linq;
    using UnityEngine;
    using UnityEngine.XR.WSA.Input;
    using UnityEngine.XR.WSA.WebCam;
    
  5. 然后,将以下变量添加到 ImageCapture 类中的 Start ( # B1 方法之上:Then add the following variables inside the ImageCapture class, above the Start() method:

        /// <summary>
        /// Allows this class to behave like a singleton
        /// </summary>
        public static ImageCapture Instance;
    
        /// <summary>
        /// Keep counts of the taps for image renaming
        /// </summary>
        private int captureCount = 0;
    
        /// <summary>
        /// Photo Capture object
        /// </summary>
        private PhotoCapture photoCaptureObject = null;
    
        /// <summary>
        /// Allows gestures recognition in HoloLens
        /// </summary>
        private GestureRecognizer recognizer;
    
        /// <summary>
        /// Loop timer
        /// </summary>
        private float secondsBetweenCaptures = 10f;
    
        /// <summary>
        /// Application main functionalities switch
        /// </summary>
        internal enum AppModes {Analysis, Training }
    
        /// <summary>
        /// Local variable for current AppMode
        /// </summary>
        internal AppModes AppMode { get; private set; }
    
        /// <summary>
        /// Flagging if the capture loop is running
        /// </summary>
        internal bool captureIsActive;
    
        /// <summary>
        /// File path of current analysed photo
        /// </summary>
        internal string filePath = string.Empty;
    
  6. 现在需要添加用于 唤醒 ( # B1Start ( # B3 方法的代码:Code for Awake() and Start() methods now needs to be added:

        /// <summary>
        /// Called on initialization
        /// </summary>
        private void Awake()
        {
            Instance = this;
    
            // Change this flag to switch between Analysis Mode and Training Mode 
            AppMode = AppModes.Training;
        }
    
        /// <summary>
        /// Runs at initialization right after Awake method
        /// </summary>
        void Start()
        {
            // Clean up the LocalState folder of this application from all photos stored
            DirectoryInfo info = new DirectoryInfo(Application.persistentDataPath);
            var fileInfo = info.GetFiles();
            foreach (var file in fileInfo)
            {
                try
                {
                    file.Delete();
                }
                catch (Exception)
                {
                    Debug.LogFormat("Cannot delete file: ", file.Name);
                }
            } 
    
            // Subscribing to the HoloLens API gesture recognizer to track user gestures
            recognizer = new GestureRecognizer();
            recognizer.SetRecognizableGestures(GestureSettings.Tap);
            recognizer.Tapped += TapHandler;
            recognizer.StartCapturingGestures();
    
            SceneOrganiser.Instance.SetCameraStatus("Ready");
        }
    
  7. 实现一个处理程序,该处理程序将在出现分流手势时调用。Implement a handler that will be called when a Tap gesture occurs.

        /// <summary>
        /// Respond to Tap Input.
        /// </summary>
        private void TapHandler(TappedEventArgs obj)
        {
            switch (AppMode)
            {
                case AppModes.Analysis:
                    if (!captureIsActive)
                    {
                        captureIsActive = true;
    
                        // Set the cursor color to red
                        SceneOrganiser.Instance.cursor.GetComponent<Renderer>().material.color = Color.red;
    
                        // Update camera status to looping capture.
                        SceneOrganiser.Instance.SetCameraStatus("Looping Capture");
    
                        // Begin the capture loop
                        InvokeRepeating("ExecuteImageCaptureAndAnalysis", 0, secondsBetweenCaptures);
                    }
                    else
                    {
                        // The user tapped while the app was analyzing 
                        // therefore stop the analysis process
                        ResetImageCapture();
                    }
                    break;
    
                case AppModes.Training:
                    if (!captureIsActive)
                    {
                        captureIsActive = true;
    
                        // Call the image capture
                        ExecuteImageCaptureAndAnalysis();
    
                        // Set the cursor color to red
                        SceneOrganiser.Instance.cursor.GetComponent<Renderer>().material.color = Color.red;
    
                        // Update camera status to uploading image.
                        SceneOrganiser.Instance.SetCameraStatus("Uploading Image");
                    }              
                    break;
            }     
        }
    

    备注

    分析 模式下, TapHandler 方法充当开始或停止照片捕获循环的开关。In Analysis mode, the TapHandler method acts as a switch to start or stop the photo capture loop.

    定型 模式下,它将从相机捕获图像。In Training mode, it will capture an image from the camera.

    如果光标为绿色,则表示相机可用于拍摄映像。When the cursor is green, it means the camera is available to take the image.

    如果光标为红色,则表示相机处于繁忙状态。When the cursor is red, it means the camera is busy.

  8. 添加应用程序用于启动映像捕获过程并存储映像的方法。Add the method that the application uses to start the image capture process and store the image.

        /// <summary>
        /// Begin process of Image Capturing and send To Azure Custom Vision Service.
        /// </summary>
        private void ExecuteImageCaptureAndAnalysis()
        {
            // Update camera status to analysis.
            SceneOrganiser.Instance.SetCameraStatus("Analysis");
    
            // Create a label in world space using the SceneOrganiser class 
            // Invisible at this point but correctly positioned where the image was taken
            SceneOrganiser.Instance.PlaceAnalysisLabel();
    
            // Set the camera resolution to be the highest possible
            Resolution cameraResolution = PhotoCapture.SupportedResolutions.OrderByDescending((res) => res.width * res.height).First();
    
            Texture2D targetTexture = new Texture2D(cameraResolution.width, cameraResolution.height);
    
            // Begin capture process, set the image format
            PhotoCapture.CreateAsync(false, delegate (PhotoCapture captureObject)
            {
                photoCaptureObject = captureObject;
    
                CameraParameters camParameters = new CameraParameters
                {
                    hologramOpacity = 0.0f,
                    cameraResolutionWidth = targetTexture.width,
                    cameraResolutionHeight = targetTexture.height,
                    pixelFormat = CapturePixelFormat.BGRA32
                };
    
                // Capture the image from the camera and save it in the App internal folder
                captureObject.StartPhotoModeAsync(camParameters, delegate (PhotoCapture.PhotoCaptureResult result)
                {
                    string filename = string.Format(@"CapturedImage{0}.jpg", captureCount);
                    filePath = Path.Combine(Application.persistentDataPath, filename);          
                    captureCount++;              
                    photoCaptureObject.TakePhotoAsync(filePath, PhotoCaptureFileOutputFormat.JPG, OnCapturedPhotoToDisk);              
                });
            });   
        }
    
  9. 添加在捕获照片时要调用的处理程序,并将其添加到已准备好进行分析的时间。Add the handlers that will be called when the photo has been captured and for when it is ready to be analyzed. 然后,将结果传递给 CustomVisionAnalyserCustomVisionTrainer ,具体取决于设置了代码的模式。The result is then passed to the CustomVisionAnalyser or the CustomVisionTrainer depending on which mode the code is set on.

        /// <summary>
        /// Register the full execution of the Photo Capture. 
        /// </summary>
        void OnCapturedPhotoToDisk(PhotoCapture.PhotoCaptureResult result)
        {
                // Call StopPhotoMode once the image has successfully captured
                photoCaptureObject.StopPhotoModeAsync(OnStoppedPhotoMode);
        }
    
    
        /// <summary>
        /// The camera photo mode has stopped after the capture.
        /// Begin the Image Analysis process.
        /// </summary>
        void OnStoppedPhotoMode(PhotoCapture.PhotoCaptureResult result)
        {
            Debug.LogFormat("Stopped Photo Mode");
    
            // Dispose from the object in memory and request the image analysis 
            photoCaptureObject.Dispose();
            photoCaptureObject = null;
    
            switch (AppMode)
            {
                case AppModes.Analysis:
                    // Call the image analysis
                    StartCoroutine(CustomVisionAnalyser.Instance.AnalyseLastImageCaptured(filePath));
                    break;
    
                case AppModes.Training:
                    // Call training using captured image
                    CustomVisionTrainer.Instance.RequestTagSelection();
                    break;
            }
        }
    
        /// <summary>
        /// Stops all capture pending actions
        /// </summary>
        internal void ResetImageCapture()
        {
            captureIsActive = false;
    
            // Set the cursor color to green
            SceneOrganiser.Instance.cursor.GetComponent<Renderer>().material.color = Color.green;
    
            // Update camera status to ready.
            SceneOrganiser.Instance.SetCameraStatus("Ready");
    
            // Stop the capture loop if active
            CancelInvoke();
        }
    
  10. 在返回到 Unity 之前,请务必保存 Visual Studio 中所做的更改。Be sure to save your changes in Visual Studio before returning to Unity.

  11. 现在,所有脚本都已完成,请返回 Unity 编辑器,然后在 "层次结构" 面板 中单击 "脚本" 文件夹中的 SceneOrganiser 类,并将其拖到 主相机 对象。Now that all the scripts have been completed, go back in the Unity Editor, then click and drag the SceneOrganiser class from the Scripts folder to the Main Camera object in the Hierarchy Panel.

第12章-生成之前Chapter 12 - Before building

若要对应用程序进行全面测试,需要将其旁加载到 HoloLens。To perform a thorough test of your application you will need to sideload it onto your HoloLens.

在执行此操作之前,请确保:Before you do, ensure that:

  • 第2章中提到的所有设置都已正确设置。All the settings mentioned in the Chapter 2 are set correctly.

  • 主相机"检查器" 面板中的所有字段均已正确分配。All the fields in the Main Camera, Inspector Panel, are assigned properly.

  • 脚本 SceneOrganiser 已附加到 摄像机主 对象。The script SceneOrganiser is attached to the Main Camera object.

  • 请确保将你的 预测密钥 插入到 predictionKey 变量。Make sure you insert your Prediction Key into the predictionKey variable.

  • 已在 predictionEndpoint 变量中插入了 预测终结点You have inserted your Prediction Endpoint into the predictionEndpoint variable.

  • 已将 定型密钥 插入 CustomVisionTrainer 类的 trainingKey 变量中。You have inserted your Training Key into the trainingKey variable, of the CustomVisionTrainer class.

  • 已在 CustomVisionTrainer 类的 projectId 变量中插入了 项目 IDYou have inserted your Project ID into the projectId variable, of the CustomVisionTrainer class.

第13章-构建并旁加载应用程序Chapter 13 - Build and sideload your application

开始 生成 过程:To begin the Build process:

  1. 请参阅 文件 > 生成设置Go to File > Build Settings.

  2. 勾选 Unity C # 项目Tick Unity C# Projects.

  3. 单击“生成”。Click Build. Unity 将启动 文件资源管理器 窗口,在该窗口中,需要创建一个文件夹,然后选择要在其中生成应用的文件夹。Unity will launch a File Explorer window, where you need to create and then select a folder to build the app into. 立即创建该文件夹并将其命名为 应用Create that folder now, and name it App. 选择 应用 文件夹后,单击 " 选择文件夹"。Then with the App folder selected, click on Select Folder.

  4. Unity 将开始向 应用 文件夹生成项目。Unity will begin building your project to the App folder.

  5. Unity 完成生成后 (可能需要一些时间) ,它将在你的生成的位置上打开 " 文件资源管理器 " 窗口 (检查任务栏,因为它可能不会始终出现在 windows 上,但会通知你添加了新的窗口) 。Once Unity has finished building (it might take some time), it will open a File Explorer window at the location of your build (check your task bar, as it may not always appear above your windows, but will notify you of the addition of a new window).

在 HoloLens 上部署:To deploy on HoloLens:

  1. 需要为远程部署) 提供 HoloLens (的 IP 地址,并确保 HoloLens 处于 开发人员模式You will need the IP Address of your HoloLens (for Remote Deploy), and to ensure your HoloLens is in Developer Mode. 具体方法为:To do this:

    1. 在戴上 HoloLens 的同时,请打开 设置Whilst wearing your HoloLens, open the Settings.

    2. 中转到 网络 & Internet > wi-fi > 高级选项Go to Network & Internet > Wi-Fi > Advanced Options

    3. 记下 IPv4 地址。Note the IPv4 address.

    4. 接下来,导航回 "设置",然后为 Update & Security > 开发人员 更新 & 安全性Next, navigate back to Settings, and then to Update & Security > For Developers

    5. 设置 开发人员模式Set Developer Mode On.

  2. 导航到新的 Unity 生成 (应用 文件夹) 并通过 Visual Studio 打开解决方案文件。Navigate to your new Unity build (the App folder) and open the solution file with Visual Studio.

  3. 解决方案配置 中,选择 " 调试"。In the Solution Configuration select Debug.

  4. 解决方案平台 中,选择 " X86,远程计算机"。In the Solution Platform, select x86, Remote Machine. 系统将提示你插入远程设备的 IP 地址 (HoloLens,在本例中,你记) 。You will be prompted to insert the IP address of a remote device (the HoloLens, in this case, which you noted).

    设置 IP 地址

  5. 中转到 " 生成 " 菜单,然后单击 " 部署解决方案 ",将应用程序旁加载到 HoloLens。Go to Build menu and click on Deploy Solution to sideload the application to your HoloLens.

  6. 应用现在应显示在你的 HoloLens 上已安装的应用列表中,可以启动了!Your app should now appear in the list of installed apps on your HoloLens, ready to be launched!

备注

若要部署到沉浸式耳机,请将 " 解决方案平台 " 设置为 " 本地计算机",并将 配置 设置为 " 调试",将 " x86 " 设置为 平台To deploy to immersive headset, set the Solution Platform to Local Machine, and set the Configuration to Debug, with x86 as the Platform. 然后,使用 " 生成 " 菜单项选择 " 部署解决方案",将部署到本地计算机。Then deploy to the local machine, using the Build menu item, selecting Deploy Solution.

使用应用程序:To use the application:

若要在 定型 模式和 预测 模式间切换应用功能,需要更新 AppMode 变量,该变量位于 ImageCapture 类中的 "唤醒 ( # B1 " 方法中。To switch the app functionality between Training mode and Prediction mode you need to update the AppMode variable, located in the Awake() method located within the ImageCapture class.

        // Change this flag to switch between Analysis mode and Training mode 
        AppMode = AppModes.Training;

or

        // Change this flag to switch between Analysis mode and Training mode 
        AppMode = AppModes.Analysis;

定型 模式下:In Training mode:

  • 查看 鼠标键盘 ,并使用 点击手势Look at Mouse or Keyboard and use the Tap gesture.

  • 接下来,将显示文本,要求你提供标记。Next, text will appear asking you to provide a tag.

  • 鼠标键盘Say either Mouse or Keyboard.

预测 模式下:In Prediction mode:

  • 查看对象并使用 点击动作Look at an object and use the Tap gesture.

  • 文本将显示,并显示检测到的对象,并且最高的概率 (这是标准化) 。Text will appear providing the object detected, with the highest probability (this is normalized).

第14章-评估并改善自定义视觉模型Chapter 14 - Evaluate and improve your Custom Vision model

若要使您的服务更准确,需要继续训练用于预测的模型。To make your Service more accurate, you will need to continue to train the model used for prediction. 这是通过将新应用程序与 培训预测 模式结合使用来实现的,后者要求你访问门户,本章将介绍这一点。This is accomplished through using your new application, with both the training and prediction modes, with the latter requiring you to visit the portal, which is what is covered in this Chapter. 准备多次重新访问您的门户,以不断改善您的模型。Be prepared to revisit your portal many times, to continually improve your model.

  1. 再次转到 Azure 自定义视觉门户,在项目中,从页面顶部中心 (选择 " 预测 " 选项卡) :Head to your Azure Custom Vision Portal again, and once you are in your project, select the Predictions tab (from the top center of the page):

    选择预测选项卡

  2. 在应用程序运行时,你将看到所有已发送到服务的映像。You will see all the images which were sent to your Service whilst your application was running. 如果你将鼠标悬停在这些图像上,它们将为你提供对该映像进行的预测:If you hover over the images, they will provide you with the predictions that were made for that image:

    预测图像列表

  3. 选择其中一个映像,将其打开。Select one of your images to open it. 打开后,您将看到对该图像的预测。Once open, you will see the predictions made for that image to the right. 如果预测是正确的,并且您希望将此图像添加到服务的训练模型,请单击 " 我的标记 " 输入框,然后选择您要关联的标记。If you the predictions were correct, and you wish to add this image to your Service's training model, click the My Tags input box, and select the tag you wish to associate. 完成后,单击右下角的 " 保存并关闭 " 按钮,然后继续转到下一张图像。When you are finished, click the Save and close button to the bottom right, and continue on to the next image.

    选择要打开的图像

  4. 返回到图像网格后,你会注意到,已将标记添加到 (并保存) 的映像将被删除。Once you are back to the grid of images, you will notice the images which you have added tags to (and saved), will be removed. 如果你发现你认为没有标记项的任何图像,则可以删除它们,方法是单击该图像上的勾选标记 (可以对几个图像执行此操作) 然后单击网格页右上角的 " 删除 "。If you find any images which you think do not have your tagged item within them, you can delete them, by clicking the tick on that image (can do this for several images) and then clicking Delete in the upper right corner of the grid page. 在下面的弹出窗口中,可以单击 "是"、"删除 " 或 " " 以确认删除或取消。On the popup that follows, you can click either Yes, delete or No, to confirm the deletion or cancel it, respectively.

    删除映像

  5. 准备好继续时,单击右上方的绿色 训练 按钮。When you are ready to proceed, click the green Train button in the top right. 你的服务模型将使用你现在提供 (的所有映像定型,使其更准确) 。Your Service model will be trained with all the images which you have now provided (which will make it more accurate). 训练完成后,请确保再次单击 "设置 默认值 " 按钮,使 预测 URL 继续使用最新的服务迭代。Once the training is complete, make sure to click the Make default button once more, so that your Prediction URL continues to use the most up-to-date iteration of your Service.

    开始培训服务模型  选择 "设置默认值" 选项Start training service model Select make default option

已完成的自定义视觉 API 应用程序Your finished Custom Vision API application

恭喜,你构建了一个使用 Azure 自定义视觉 API 来识别真实世界对象的混合现实应用,为服务模型定型,并显示对所见内容的信心。Congratulations, you built a mixed reality app that leverages the Azure Custom Vision API to recognize real world objects, train the Service model, and display confidence of what has been seen.

完成的项目示例

额外练习Bonus exercises

练习1Exercise 1

训练 自定义影像服务 来识别更多对象。Train your Custom Vision Service to recognize more objects.

练习2Exercise 2

若要在了解的情况下进行扩展,请完成以下练习:As a way to expand on what you have learned, complete the following exercises:

在识别对象时播放声音。Play a sound when an object is recognized.

练习3Exercise 3

使用 API 来重新训练你的服务,使其与你的应用程序正在分析的映像相同,以便使服务更准确 (同时) 的预测和培训。Use the API to re-train your Service with the same images your app is analyzing, so to make the Service more accurate (do both prediction and training simultaneously).