個案研究-以混合現實表示人類Case study - Representing humans in mixed reality

使用 light 的 James Turrell 設計。James Turrell designs with light. 逐步執行他的工作,使其深度和焦點都有意義。Stepping into his work blurs one’s sense of depth and focus. 牆壁看起來都是封閉且無限的,亮度提供了陰影的方式。Walls seem both close and infinite, brightness gives way to shadows. 藉由仔細平衡光線的色彩和擴散來設計陌生的認知。Unfamiliar perceptions designed by carefully balancing the light’s color and diffusion. Turrell 將這些 sensations 描述 為「 感受您的眼睛 」,這是一種擴充其實際瞭解的方法。Turrell describes these sensations as ‘feeling with your eyes’ , a way of extending one’s understanding of reality. 很棒的世界(像是 Turrell imagines)是強大的工具,可利用我們的想法,而不像現今混合現實的沉浸式環境。Fantastic worlds, like the ones Turrell imagines, are powerful tools to exploit our senses, not unlike the immersive environments of mixed reality today.

Wide Out-James Turrell (1998)

您如何在混合現實中代表複雜的真實世界環境?How do you represent complex real-world environments in mixed reality?

以沉浸式體驗表示 Turrell 的工作,讓您令人信服的挑戰。Representing Turrell’s work in an immersive experience makes for a compelling challenge. 光源、尺規和空間音訊帶來了表示工作的機會。Lighting, scale, and spatial audio present opportunities to represent his work. 雖然展示的幾何周圍會需要相當簡單的3D 模型,但它們是演出者焦點的輔助:光對檢測的影響。While the exhibit’s geometric surroundings would require relatively simple 3D modeling, they are secondary to the artist’s focus: the light’s impact on the senses.

Turrell 的 stark、surreal minimalism 是其工作的特徵,但如果我們想要以混合現實中的複雜資料呈現展示,該怎麼辦?Turrell’s stark, surreal minimalism is the hallmark of his work, but what if we wanted to represent an exhibit with more complex materials in mixed reality?

驚嘆號-Ai Weiwei (2013)

在2013中,「演出者 Ai Weiwei 會揭曉 tangling 在 Venice Biennale 上 具有 886 古典 stools 的作品。In 2013, the artist Ai Weiwei unveiled a tangling work of art featuring 886 antique stools at the Venice Biennale. 每個木制 stool 都來自中國熱情技術具高度價值的時代,而這些 stools 會在世代之間傳遞。Each wooden stool came from an era where Chinese craftsmanship was highly valued, where these stools would have been passed down between generations. Stools 本身—在 Ai 對新式文化的評論方面來說,像是木材的複雜部分,也就是他們的審慎定位。The stools themselves — the intricacies of the wood, the precision of the pieces, their careful placement — are critical to Ai’s commentary on modern culture.

古典 stools 會透過其真實性來提供演出者的訊息。The antique stools deliver the artist’s message through their authenticity. 實際的標記法對於體驗來說很重要,因為這是一項技術挑戰: Sculpting 每個 886 stools,都是大幅的詳盡且昂貴。Their realistic representation is critical to the experience, creating a technical challenge: Sculpting each of the 886 stools by hand would be enormously exhaustive and expensive. 模型和定位需要多久的時間?How long would it take to model and position? 您要如何維護材質的真實性?How would you maintain the authenticity of the material? 從頭開始重新建立這些物件,將會以許多方式來轉譯作品本身。Recreating these objects from scratch becomes, in many ways, an interpretation of the artwork itself. 您要如何保存演出者的意圖?How can you preserve the artist’s intent?

捕獲混合現實資產的方法Methods of capturing mixed reality assets

從頭開始建立某個事物的替代方法,就是捕捉真正的事物。The alternative to creating something from scratch is capturing the real thing. 透過一組向前推進的捕獲方法,我們可以在混合現實 (環境、物件和人員) 中,開發出每個核心資產類型的真實標記法。Through an ever-advancing set of capture methods, we can develop authentic representations of each of the core asset types found in mixed reality (environments, objects, and people).

廣泛的分類範圍從妥善建立的2D 影片到最新形式的體積型影片。The broad categories range from well-established 2D video to the newest forms of volumetric video. 在 Ai Weiwei 的表現案例中,掃描 (通常是由其基本技巧來參考,攝影測量) 可在建立展示時採用,並掃描每個 stools 本身。In the case of Ai Weiwei’s exhibit, scanning (often referred to by its fundamental technique, photogrammetry) could be employed during the creation of the exhibit, scanning each of the stools themselves. 360的相片和影片捕捉是另一種虛擬化體驗的方法,在整個展示中使用高品質的全方向攝影機。360° photo and video capture is another method for virtualizing the experience utilizing a high-quality omni-directional camera positioned throughout the exhibit. 使用這些技術,一開始就能瞭解規模的意義,理想的細節是查看每個部分的熱情技術。With these techniques, one begins to understand the sense of scale, ideally with enough detail to see each piece’s craftsmanship. 這一切都是以數位形式提供,但實際上並不可能有新的絢爛和觀點。All this while existing in a digital form that allows for new vistas and perspectives not possible in reality.

2D 至體積型影片

當我們無法只建立絕佳的元素,但在混合現實中使用最實際的環境、物件和人員的功能時,會出現何種機會?What kind of opportunities emerge when we cannot only create fantastic elements, but utilize the most realistic captures of environments, objects, and people in mixed reality? 探索這些方法之間的重迭,有助於照亮媒體的趨勢。Exploring the overlap between these methods help illuminate where the medium is headed.

針對環境和物件,360°映射軟體正在發展,以包含攝影測量的元素。For environments and objects, 360° imaging software is evolving to include elements of photogrammetry. 從場景隔離深度資訊,advanced 360 °影片可協助您在尋找虛擬場景時,減輕 fishbowl 的感覺。Isolating depth information from scenes, advanced 360° videos help alleviate the feeling of having your head stuck in a fishbowl when looking around a virtual scene.

對於人們而言,新的方法是結合及擴充動作捕捉和掃描的新方法:「移動捕捉」已經成為基礎,可將詳細的人力移動帶入視覺效果和 cinematic 的字元,而掃描也有先進的經驗,可捕獲臉部和手等詳細的人類視覺效果。For people, new methods are emerging that combine and extend motion capture and scanning: Motion capture has been foundational to bringing detailed human movement to visual effects and cinematic characters, while scanning has advanced to capture detailed human visuals like faces and hands. 有了轉譯技術的進步,稱為體積型 video 的新方法會建立這些技術,並結合視覺效果和深度資訊,以建立新一代的3D 人類捕捉。With advancements in rendering technology, a new method called volumetric video builds off these techniques, combining visual and depth information, to create the next generation of 3D human captures.

體積型影片,並追求真正的人工捕捉Volumetric video and the pursuit of authentic human capture

人類是故事行銷的核心,最常說的意義:人們說、執行或作為故事的主旨。Humans are central to storytelling — in the most literal sense: a human speaking, performing, or as the story’s subject. 現今早期的沉浸式體驗的一些最吸引人和引人注目的時刻是社交的。Some of the most immersive and eye-opening moments of today’s early immersive experiences are social. 在您的客廳共用混合現實體驗,以在 unbelievable 新環境時看到您的朋友。From sharing a mixed reality experience together in your living room, to seeing your friends in unbelievable new environments. 人力元素甚至是最棒的事實。The human element makes even the most fantastic reality, a reality.

在 VR 中 Mindshow

虛擬人偶在沉浸式體驗中,可在故事行銷中啟用一種新的 embodiment。Avatars in immersive experiences enable a new kind of embodiment in storytelling. 最新的應用程式會重新思考虛擬主體擁有權的概念,並設定新一代的飛躍,以消除人們之間的距離。The latest apps are rethinking the concept of virtual body ownership and setting up a generational leap in eliminating the distance between people. Mindshow 這樣的公司開發了可利用虛擬人偶的創意工具,讓使用者完全採用新的角色和字元。Companies like Mindshow are developing creative tools that leverage avatars, letting users take on entirely new personas and characters. 其他人正在探索 藝術運算式的方法,這是一項可能無限制的創意機會,可以探索本質 (以及類似人類的屬性) 。Others are exploring methods of artistic expression, a potentially limitless creative opportunity to explore the nature (and necessity) of human-like attributes. 目前,這種缺乏的真實性有助於避免 人類 likeness 的 uncanny ,以及針對日常開發人員提供的技術問題。Today, this absence of realism helps avoid the uncanny valley of human likeness along with a host of technical issues for everyday developers. 基於這些理由 (和更) ,很可能非實際的虛擬人偶會成為可預見的未來的預設值。For these reasons (and more) it is very likely that non-realistic avatars will become the default for the foreseeable future. 此外,雖然真實性對於混合的現實有很大的挑戰,但還是 需要在3d 空間中真正表示人類的重要案例And yet, while realism poses an enormous challenge for mixed reality, there are key scenarios that require authentic representation of humans in 3D space .

在 Microsoft,從 Microsoft Research 中散播的小型團隊花了過去幾年的時間,透過體積型影片的形式來開發學習人類的方法。At Microsoft, a small team borne out of Microsoft Research has spent the past several years developing a method for capturing humans through a form of volumetric video. 今日的程式類似于影片生產: sculpted 資產是完整的3D 錄製。The process today is similar to video production: rather than applying movement to a sculpted asset it is a full, 3D recording. 效能和影像會以即時方式進行,這並不是演出者的工作,而是真實的標記法。The performance and the image are captured in real-time — it’s not the work of an artist, it’s an authentic representation. 雖然這項技術剛開始擴展到商業應用程式,但體積型影片的含意對 Microsoft 對更個人化運算的願景很重要。And while the technology is just beginning to expand into commercial applications, the implications of volumetric video are critical to Microsoft’s vision of More Personal Computing.

體積型 video SIGGRAPH 2015

真實的人工捕捉會在混合現實中解除新的獨特體驗類別。Authentic human capture unlocks new unique categories of experiences in mixed reality. 看看您所辨識的人,不論是名人、同事或愛,都能在數位媒體中建立熟悉度的深度。Seeing someone you recognize, whether it’s a celebrity, a colleague, or a loved one, creates a depth of intimacy never before possible in a digital medium. 他們的臉部、其運算式、其移動中的細微之處,都屬於他們的身分。Their face, their expressions, the nuance in their movements are all part of who they are. 當我們可以在3D 空間中抓住這些人類品質時,有哪些機會解除鎖定?What opportunities unlock when we can capture these human qualities in 3D space?

現今團隊將把重點放在娛樂和教育之類的磁區,來推動體積型影片的範圍: Actiongram 有創意的字元和 名人 來建立混合的現實故事。Today the team is pushing the bounds of volumetric video by focusing on sectors like entertainment and education: Actiongram features creative characters and celebrities to create mixed reality stories. 目的地: Mars 展示,現在在 NASA 的甘迺迪空間中心,有體積型的傳說太空人影片 Aldrin 影片。Destination: Mars exhibit, now at NASA’s Kennedy Space Center, features a volumetric video of legendary astronaut Buzz Aldrin. 這項體驗可讓訪客在 Mars 的表面上四處解說,因為他在 Mars 上引進了人類 colonization 的追求。The experience allows visitors to walk around the surface of Mars with Buzz as he introduces the pursuit of human colonization on Mars.

人類是混合現實的基礎Humans are fundamental to mixed reality

設計讓這些影片看起來很自然的方法是一項挑戰,但團隊卻發現很大的潛力。Designing ways to make these videos seem natural poses a challenge but one in which the team sees enormous potential. 當技術變得更容易存取,並從錄音移至即時捕獲時,這些商機將會擴充。And these opportunities will expand as the technology becomes more accessible and moves from recordings to real-time capture.

Holoportation 是以相同的基本技術為基礎的研究工作,authentically 捕獲視覺效果和深度資訊,以及即時轉譯結果。Holoportation is a research effort that builds upon the same fundamental technology, authentically capturing visual and depth information, and rendering the result in real-time. 小組正在探索實際人類表示的能力,對於交談和分享體驗的未來有何意義。The team is exploring what the power of realistic human representation means for the future of conversations and shared experiences. 當您從世界各地的某個人可以加入至您的環境時,會發生什麼事?What happens when a three-dimensional capture of someone, from anywhere in the world, can be added into your environment?

交談的未來

從將新的深度層級分層到每日應用程式(例如 Skype),到徹底改變數字會議和企業旅遊的概念—體積型影片開啟了獨特的案例:一位專家,是一位各地的大陸或數位朋友,坐在您生活中的 couches 和椅。From layering a new level of immersion onto everyday apps like Skype, to radically reshaping the concept of digital meetings and business travel — volumetric video opens unique scenarios: A specialist virtually training doctors on a far-away continent or digital friends sitting on the couches and chairs in your living room. 將真實的人類標記法新增至混合現實體驗,將會徹底改變數字會議和企業旅遊的概念。Adding authentic human representations to mixed reality experiences will radically reshape the concept of digital meetings and business travel.

就像 James Turrell 的抽象藝術和 Ai Weiwei 的重大真實感一樣,它也提供自己獨特的技術挑戰,因此,將人類視為創意的虛擬人偶和實際的捕獲方法。Just as the abstract art of James Turrell and the critical realism of Ai Weiwei offer their own unique technical challenges, so do the methods to represent humans as creative avatars and realistic captures. 您無法在另一種情況下略過,並探索每一項的潛能,以協助我們瞭解這個新空間中的人為互動。One cannot be ignored in light of the other and exploring the potential of each will help us understand human interaction in this new space.

關於作者About the author

Picture of Mark Vitazko Mark VitazkoMark Vitazko
UX 設計工具 @MicrosoftUX Designer @Microsoft