Xamarin 中的 ARKit 2ARKit 2 in Xamarin.iOS

自去年的 iOS 11 年起,ARKit 已大幅成熟。ARKit has matured considerably since its introduction last year in iOS 11. 首先,您可以检测垂直和水平平面,这大大提高了室内增加的现实体验的实用性。First and foremost, you can now detect vertical as well as horizontal planes, which greatly improves the practicality of indoor augmented reality experiences. 此外,还有一些新功能:In addition, there are new capabilities:

  • 识别引用图像和对象作为现实世界和数字图像之间的接合Recognizing reference images and objects as the junction between the real world and digital imagery
  • 模拟现实世界照明的新照明模式A new lighting mode that simulates real-world lighting
  • 共享和保留 AR 环境的功能The ability to share and persist AR environments
  • 首选存储 AR 内容的新文件格式A new file format preferred for storing AR content

识别引用对象Recognizing reference objects

ARKit 2 中的一个展示功能是识别引用图像和对象的能力。One showcase feature in ARKit 2 is the ability to recognize reference images and objects. 可以从普通图像文件(稍后讨论)加载引用映像,但必须使用以开发人员为中心的ARObjectScanningConfiguration来扫描引用对象。Reference images can be loaded from normal image files (discussed later), but reference objects must be scanned, using the developer-focused ARObjectScanningConfiguration.

示例应用:扫描和检测3D 对象Sample app: Scanning and detecting 3D objects

扫描和检测3D 对象示例是Apple 项目的一个端口,该端口演示:The Scanning and Detecting 3D Objects sample is a port of an Apple project that demonstrates:

扫描引用对象的同时,消耗电池和处理器的旧设备在实现稳定跟踪时通常会遇到问题。Scanning a reference object is battery- and processor- intensive and older devices will often have trouble achieving stable tracking.

使用 NSNotification 对象的状态管理State management using NSNotification objects

此应用程序使用状态机,该计算机在以下状态之间过渡:This application uses a state machine that transitions between the following states:

  • AppState.StartARSession
  • AppState.NotReady
  • AppState.Scanning
  • AppState.Testing

此外,在 AppState.Scanning时,还会使用嵌入的一组状态和转换:And additionally uses an embedded set of states and transitions when in AppState.Scanning:

  • Scan.ScanState.Ready
  • Scan.ScanState.DefineBoundingBox
  • Scan.ScanState.Scanning
  • Scan.ScanState.AdjustingOrigin

此应用使用反应体系结构,该体系结构将状态转换通知发布到NSNotificationCenter并订阅这些通知。The app uses a reactive architecture that posts state-transition notifications to NSNotificationCenter and subscribes to these notifications. 此设置与 ViewController.cs类似于以下代码片段:The setup looks like this snippet from ViewController.cs:

// Configure notifications for application state changes
var notificationCenter = NSNotificationCenter.DefaultCenter;

notificationCenter.AddObserver(Scan.ScanningStateChangedNotificationName, State.ScanningStateChanged);
notificationCenter.AddObserver(ScannedObject.GhostBoundingBoxCreatedNotificationName, State.GhostBoundingBoxWasCreated);
notificationCenter.AddObserver(ScannedObject.GhostBoundingBoxRemovedNotificationName, State.GhostBoundingBoxWasRemoved);
notificationCenter.AddObserver(ScannedObject.BoundingBoxCreatedNotificationName, State.BoundingBoxWasCreated);
notificationCenter.AddObserver(BoundingBox.ScanPercentageChangedNotificationName, ScanPercentageChanged);
notificationCenter.AddObserver(BoundingBox.ExtentChangedNotificationName, BoundingBoxExtentChanged);
notificationCenter.AddObserver(BoundingBox.PositionChangedNotificationName, BoundingBoxPositionChanged);
notificationCenter.AddObserver(ObjectOrigin.PositionChangedNotificationName, ObjectOriginPositionChanged);
notificationCenter.AddObserver(NSProcessInfo.PowerStateDidChangeNotification, DisplayWarningIfInLowPowerMode);

典型的通知处理程序将更新 UI,并可能修改应用程序状态,例如,在扫描对象时更新此处理程序:A typical notification handler will update the UI and possibly modify the application state, such as this handler that updates as the object is scanned:

private void ScanPercentageChanged(NSNotification notification)
{
    var pctNum = TryGet<NSNumber>(notification.UserInfo, BoundingBox.ScanPercentageUserKey);
    if (pctNum == null)
    {
        return;
    }
    double percentage = pctNum.DoubleValue;
    // Switch to the next state if scan is complete
    if (percentage >= 100.0)
    {
        State.SwitchToNextState();
    }
    else
    {
        DispatchQueue.MainQueue.DispatchAsync(() => navigationBarController.SetNavigationBarTitle($"Scan ({percentage})"));
    }
}

最后,Enter{State} 方法根据新状态修改模型和 UX:Finally, Enter{State} methods modify the model and UX as appropriate to the new state:

internal void EnterStateTesting()
{
    navigationBarController.SetNavigationBarTitle("Testing");
    navigationBarController.ShowBackButton(false);
    loadModelButton.Hidden = true;
    flashlightButton.Hidden = false;
    nextButton.Enabled = true;
    nextButton.SetTitle("Share", UIControlState.Normal);

    testRun = new TestRun(sessionInfo, sceneView);
    TestObjectDetection();
    CancelMaxScanTimeTimer();
}

自定义可视化效果Custom visualization

应用显示投影到所检测到的水平平面的边界框内的对象的低级 "点云"。The app shows the low-level “point cloud” of the object contained within a bounding box projected onto a detected horizontal plane.

此点云适用于ARFrame.RawFeaturePoints属性中的开发人员。This point cloud is available to developers in the ARFrame.RawFeaturePoints property. 有效地可视化点云可能是一个棘手的问题。Visualizing the point cloud efficiently can be a tricky problem. 遍历点,然后为每个点创建和放置新的 SceneKit 节点会终止帧速率。Iterating over the points, then creating and placing a new SceneKit node for each point would kill the frame rate. 或者,如果异步完成,则会有延迟。Alternatively, if done asynchronously, there would be a lag. 该示例通过由三个部分构成的策略来维护性能:The sample maintains performance with a three-part strategy:

internal static SCNGeometry CreateVisualization(NVector3[] points, UIColor color, float size)
{
  if (points.Length == 0)
  {
    return null;
  }

  unsafe
  {
    var stride = sizeof(float) * 3;

    // Pin the data down so that it doesn't move
    fixed (NVector3* pPoints = &amp;points[0])
    {
      // Important: Don't unpin until after `SCNGeometry.Create`, because geometry creation is lazy

      // Grab a pointer to the data and treat it as a byte buffer of the appropriate length
      var intPtr = new IntPtr(pPoints);
      var pointData = NSData.FromBytes(intPtr, (System.nuint) (stride * points.Length));

      // Create a geometry source (factory) configured properly for the data (3 vertices)
      var source = SCNGeometrySource.FromData(
        pointData,
        SCNGeometrySourceSemantics.Vertex,
        points.Length,
        true,
        3,
        sizeof(float),
        0,
        stride
      );

      // Create geometry element
      // The null and bytesPerElement = 0 look odd, but this is just a template object
      var template = SCNGeometryElement.FromData(null, SCNGeometryPrimitiveType.Point, points.Length, 0);
      template.PointSize = 0.001F;
      template.MinimumPointScreenSpaceRadius = size;
      template.MaximumPointScreenSpaceRadius = size;

      // Stitch the data (source) together with the template to create the new object
      var pointsGeometry = SCNGeometry.Create(new[] { source }, new[] { template });
      pointsGeometry.Materials = new[] { Utilities.Material(color) };
      return pointsGeometry;
    }
  }
}

结果如下所示:The result looks like this:

point_cloud

复杂手势Complex gestures

用户可以缩放、旋转和拖动环绕目标对象的边界框。The user can scale, rotate, and drag the bounding box that surrounds the target object. 关联的笔势识别器中有两个有趣的事情。There are two interesting things in the associated gesture recognizers.

首先,所有手势识别器仅在达到阈值之后才会激活;例如,手指拖动了太多的像素,或者旋转超出了某个角度。First, all of the gesture recognizers activate only after a threshold has been passed; for example, a finger has dragged so many pixels or the rotation exceeds some angle. 此方法是累积移动,直到超过阈值,然后增量应用:The technique is to accumulate the move until the threshold has been exceeded, then apply it incrementally:

// A custom rotation gesture recognizer that fires only when a threshold is passed
internal partial class ThresholdRotationGestureRecognizer : UIRotationGestureRecognizer
{
    // The threshold after which this gesture is detected.
    const double threshold = Math.PI / 15; // (12°)

    // Indicates whether the currently active gesture has exceeded the threshold
    private bool thresholdExceeded = false;

    private double previousRotation = 0;
    internal double RotationDelta { get; private set; }

    internal ThresholdRotationGestureRecognizer(IntPtr handle) : base(handle)
    {
    }

    // Observe when the gesture's state changes to reset the threshold
    public override UIGestureRecognizerState State
    {
        get => base.State;
        set
        {
            base.State = value;

            switch(value)
            {
                case UIGestureRecognizerState.Began :
                case UIGestureRecognizerState.Changed :
                    break;
                default :
                    // Reset threshold check
                    thresholdExceeded = false;
                    previousRotation = 0;
                    RotationDelta = 0;
                    break;
            }
        }
    }

    public override void TouchesMoved(NSSet touches, UIEvent evt)
    {
        base.TouchesMoved(touches, evt);

        if (thresholdExceeded)
        {
            RotationDelta = Rotation - previousRotation;
            previousRotation = Rotation;
        }

        if (! thresholdExceeded && Math.Abs(Rotation) > threshold)
        {
            thresholdExceeded = true;
            previousRotation = Rotation;
        }
    }
}

与手势相关的第二个有趣的事情就是相对于检测到的实际平面移动边界框的方式。The second interesting thing being done in relation to gestures is the way that the bounding box is moved in relation to detected real-world planes. Xamarin 博客文章中讨论了这一点。This aspect is discussed in this Xamarin blog post.

ARKit 2 中的其他新功能Other new features in ARKit 2

更多跟踪配置More tracking configurations

现在,可以使用以下任何一项作为混合现实体验的基础:Now, you can use any of the following as the basis for a mixed-reality experience:

本博客文章F#和示例中讨论的 AROrientationTrackingConfiguration是最有限的,并提供不良的混合现实体验,因为它只与设备的移动相对应的数字对象,而不尝试将设备和屏幕与真实world.AROrientationTrackingConfiguration, discussed in this blog post and F# sample, is the most limited and provides a poor mixed-reality experience, as it only places digital objects in relation to the device's motion, without trying to tie the device and screen into the real world.

ARImageTrackingConfiguration 允许识别真实的2D 图像(气急败坏、徽标等),并使用这些图像来定位数字图像:The ARImageTrackingConfiguration allows you to recognize real-world 2D images (paintings, logos, etc.) and use those to anchor digital imagery:

var imagesAndWidths = new[] {
    ("cover1.jpg", 0.185F),
    ("cover2.jpg", 0.185F),
     //...etc...
    ("cover100.jpg", 0.185F),
};

var referenceImages = new NSSet<ARReferenceImage>(
    imagesAndWidths.Select( imageAndWidth =>
    {
      // Tuples cannot be destructured in lambda arguments
        var (image, width) = imageAndWidth;
        // Read the image
        var img = UIImage.FromFile(image).CGImage;
        return new ARReferenceImage(img, ImageIO.CGImagePropertyOrientation.Up, width);
    }).ToArray());

configuration.TrackingImages = referenceImages;

此配置有两个有趣的方面:There are two interesting aspects to this configuration:

  • 这种方法很有效,可以用于可能很多的引用映像It's efficient and can be used with a potentially large number of reference images
  • 即使图像在现实世界中移动,数字图像仍会锚定(例如,如果已识别书的封面,则会跟踪书籍,因为它会从托架上向下排列,等等)。The digital imagery is anchored to the image, even if that image moves in the real world (for example, if the cover of a book is recognized, it will track the book as it is pulled off the shelf, laid down, etc.).

之前已讨论 ARObjectScanningConfiguration,它是用于扫描3d 对象的以开发人员为中心的配置。The ARObjectScanningConfiguration was discussed previously and is a developer-centric configuration for scanning 3D objects. 它会占用大量处理器和电池,不应在最终用户应用程序中使用。It is highly processor and battery intensive and should not be used in end-user applications. 扫描和检测三维对象的示例演示如何使用此配置。The sample Scanning and Detecting 3D Objects demonstrates the use of this configuration.

ARWorldTrackingConfiguration 的最后一个跟踪配置是大多数混合现实体验的主力。The final tracking configuration, ARWorldTrackingConfiguration , is the workhorse of most mixed-reality experiences. 此配置使用 "可视惯性 odometry" 将现实世界 "功能点" 与数字图像相关联。This configuration uses "visual inertial odometry" to relate real-world "feature points" to digital imagery. 数字几何或子画面相对于实际的水平和垂直平面定位,或相对于检测到的 ARReferenceObject 实例。Digital geometry or sprites are anchored relative to real-world horizontal and vertical planes or relative to detected ARReferenceObject instances. 在此配置中,世界原点是相机在空间中相对于重心的原始位置,数字对象相对于现实世界中的对象保持不变。In this configuration, the world origin is the camera's original position in space with the Z-axis aligned to gravity, and digital objects "stay in place" relative to objects in the real world.

环境纹理Environmental texturing

ARKit 2 支持使用捕获的图像来估算光照,甚至将反射高光应用于光亮对象的 "环境纹理"。ARKit 2 supports "environmental texturing" that uses captured imagery to estimate lighting and even apply specular highlights to shiny objects. 环境立方体贴图是动态构建的,一旦相机进入所有方向,就会产生极其的现实体验:The environmental cubemap is built up dynamically and, once the camera has looked in all directions, can produce an impressively realistic experience:

环境纹理演示图像

使用环境纹理:In order to use environmental texturing:

var sphere = SCNSphere.Create(0.33F);
sphere.FirstMaterial.LightingModelName = SCNLightingModel.PhysicallyBased;
// Shiny metallic sphere
sphere.FirstMaterial.Metalness.Contents = new NSNumber(1.0F);
sphere.FirstMaterial.Roughness.Contents = new NSNumber(0.0F);

// Session configuration:
var configuration = new ARWorldTrackingConfiguration
{
    PlaneDetection = ARPlaneDetection.Horizontal | ARPlaneDetection.Vertical,
    LightEstimationEnabled = true,
    EnvironmentTexturing = AREnvironmentTexturing.Automatic
};

尽管上述代码段中所示的完美反射纹理在示例中很有趣,但环境纹理可能更好地用于避免,这会触发 "特别低谷" 响应(纹理只是基于照相机录制)。Although the perfectly reflective texture shown in the preceding code snippet is fun in a sample, environmental texturing is probably better used with restraint lest it trigger an "uncanny valley" response (the texture is only an estimate based on what the camera recorded).

共享和持久的 AR 体验Shared and persistent AR experiences

ARKit 2 的另一个主要功能是ARWorldMap类,这允许您共享或存储世界上的跟踪数据。Another major addition to ARKit 2 is the ARWorldMap class, which allows you to share or store world-tracking data. 您可以ARSession.GetCurrentWorldMapAsyncGetCurrentWorldMap(Action<ARWorldMap,NSError>)获取当前世界地图:You get the current world map with ARSession.GetCurrentWorldMapAsync or GetCurrentWorldMap(Action<ARWorldMap,NSError>) :

// Local storage
var PersistentWorldPath => Environment.GetFolderPath(Environment.SpecialFolder.Personal) + "/arworldmap";

// Later, after scanning the environment thoroughly...
var worldMap = await Session.GetCurrentWorldMapAsync();
if (worldMap != null)
{
    var data = NSKeyedArchiver.ArchivedDataWithRootObject(worldMap, true, out var err);
    if (err != null)
    {
        Console.WriteLine(err);
    }
    File.WriteAllBytes(PersistentWorldPath, data.ToArray());
}

共享或还原世界地图:To share or restore the world map:

  1. 从文件加载数据,Load the data from the file,
  2. 将其取消存档到 ARWorldMap 对象中,Unarchive it into an ARWorldMap object,
  3. 使用它作为ARWorldTrackingConfiguration.InitialWorldMap属性的值:Use that as the value for the ARWorldTrackingConfiguration.InitialWorldMap property:
var data = NSData.FromArray(File.ReadAllBytes(PersistentWorldController.PersistenWorldPath));
var worldMap = (ARWorldMap)NSKeyedUnarchiver.GetUnarchivedObject(typeof(ARWorldMap), data, out var err);

var configuration = new ARWorldTrackingConfiguration
{
    PlaneDetection = ARPlaneDetection.Horizontal | ARPlaneDetection.Vertical,
    LightEstimationEnabled = true,
    EnvironmentTexturing = AREnvironmentTexturing.Automatic,
    InitialWorldMap = worldMap
};

ARWorldMap 仅包含不可见的世界跟踪数据和ARAnchor对象,而_不_包含数字资产。The ARWorldMap only contains non-visible world-tracking data and the ARAnchor objects, it does not contain digital assets. 若要共享几何或图像,你必须开发自己的适用于你的使用情况的策略(可能只是通过存储/传输几何图形的位置和方向,并将其应用于静态 SCNGeometry 或可能是通过存储/传输序列化的对象)。To share geometry or imagery, you'll have to develop your own strategy appropriate to your use-case (perhaps by storing/transmitting only the location and orientation of the geometry and applying it to static SCNGeometry or perhaps by storing/transmitting serialized objects). ARWorldMap 的好处是,在相对于共享 ARAnchor的情况下,资产将在设备或会话之间一致地显示。The benefit of the ARWorldMap is that assets, once placed relative to a shared ARAnchor, will appear consistently between devices or sessions.

通用场景描述文件格式Universal Scene Description file format

ARKit 2 的最终大字功能是 Apple 采用 Pixar 的通用场景说明文件格式。The final headline feature of ARKit 2 is Apple's adoption of Pixar's Universal Scene Description file format. 此格式将 Collada 的 DAE 格式替换为用于共享和存储 ARKit 资产的首选格式。This format replaces Collada's DAE format as the preferred format for sharing and storing ARKit assets. 在 iOS 12 和 Mojave 中内置了对可视化资源的支持。Support for visualizing assets is built into iOS 12 and Mojave. USDZ 文件扩展名是包含 USD 文件的未压缩和未加密的 zip 存档。The USDZ file extension is an uncompressed and unencrypted zip archive containing USD files. Pixar提供了用于处理 USD 文件的工具,但尚不支持很多第三方支持。Pixar provides tools for working with USD files but there is not yet much third-party support.

ARKit 编程提示ARKit programming tips

手动资源管理Manual resource management

在 ARKit 中,手动管理资源非常重要。In ARKit, it's crucial to manually manage resources. 这不仅能实现高帧率,而且确实_需要_避免 "屏幕冻结"。Not only does this allow high frame-rates, it actually is necessary to avoid a confusing "screen freeze." ARKit 框架对提供新的摄像帧(ARSession.CurrentFrame很懒惰。The ARKit framework is lazy about supplying a new camera frame (ARSession.CurrentFrame. 在当前ARFrame Dispose() 调用之前,ARKit 将不提供新帧!Until the current ARFrame has had Dispose() called on it, ARKit will not supply a new frame! 这会导致视频 "冻结",即使应用程序的其余部分都有响应。This causes the video to "freeze" even though the rest of the app is responsive. 解决方法是始终使用 using 块访问 ARSession.CurrentFrame,或者手动对其调用 Dispose()The solution is to always access ARSession.CurrentFrame with a using block or manually call Dispose() on it.

派生自 NSObject 的所有对象都 IDisposable 并且 NSObject 实现Dispose 模式,因此,通常应遵循此模式实现派生类上的 DisposeAll objects derived from NSObject are IDisposable and NSObject implements the Dispose pattern, so you should typically follow this pattern for implementing Dispose on a derived class.

操作转换矩阵Manipulating transform matrices

在任何3D 应用程序中,都将处理4x4 变换矩阵,简洁地介绍如何通过三维空间移动、旋转和切变对象。In any 3D application, you're going to be dealing with 4x4 transformation matrices that compactly describe how to move, rotate, and shear an object through 3D space. 在 SceneKit 中,这些对象是SCNMatrix4的对象。In SceneKit, these are SCNMatrix4 objects.

SCNNode.Transform属性返回SCNNode _按_行-主 simdfloat4x4 类型支持的 SCNMatrix4 变换矩阵。The SCNNode.Transform property returns the SCNMatrix4 transform matrix for the SCNNode as backed by the row-major simdfloat4x4 type. 例如:So, for instance:

var node = new SCNNode { Position = new SCNVector3(2, 3, 4) };  
var xform = node.Transform;
Console.WriteLine(xform);
// Output is: "(1, 0, 0, 0)\n(0, 1, 0, 0)\n(0, 0, 1, 0)\n(2, 3, 4, 1)"

正如您所看到的,在末行的前三个元素中对位置进行编码。As you can see, the position is encoded in the bottom row's first three elements.

在 Xamarin 中,操作转换矩阵的通用类型是 NVector4的,通过约定以主要方式进行解释。In Xamarin, the common type for manipulating transformation matrices is NVector4, which by convention is interpreted in a column-major way. 也就是说,转换/位置组件应在 M14、M24、M34、not M41、M42、M43 中。That is to say, the translation/position component is expected in M14, M24, M34, not M41, M42, M43:

行-主要与列-主要

与所选的矩阵解释保持一致对于正确的行为至关重要。Being consistent with the choice of matrix interpretation is vital to proper behavior. 由于3D 变换矩阵是4x4,因此一致性错误不会生成任何种类的编译时甚至是运行时异常,只是操作会意外执行。Since 3D transform matrices are 4x4, consistency mistakes will not produce any kind of compile-time or even run-time exception — it's just that operations will act unexpectedly. 如果 SceneKit/ARKit 对象看似粘滞、飞出或抖动,则不正确的转换矩阵是一种很好的做法。If your SceneKit / ARKit objects seem to be stuck, fly away, or jitter, an incorrect transform matrix is a good possibility. 解决方案很简单: NMatrix4.Transpose将执行元素的就地调换。The solution is simple: NMatrix4.Transpose will perform an in-place transposition of elements.