The developers behind the upcoming mobile game for The Walking Dead, Our World, are using the iPhone X’s TrueDepth’s camera system as a motion capture tool for facial expressions for characters in the game. The developers record actors in front of the iPhone X, using a specially-built app to export the facial motion into their 3D character modelling apps.
Going beyond simple cartoon figures like Animoji, the developers feel the iPhone X sensors are good enough that they can use them to animate realistic face rigs of human characters.
The workflow starts on a desktop Mac, where the artists create a 3D model of a character. They hook up the skeleton to outlets for the 50 blend shapes that the iPhone X TrueDepth camera can track.
A special app that runs on an iPhone X loads up the face model, and the game makes have an actor record the desired human expressions. The iPhone X can detect these movements at 60 frames per seconds, and the data is saved by the custom app.
The animator can shoot multiple takes and review in realtime how the tracking will be applied to the 3D model. The data can then be exported back to the Mac for final tweaking in the 3D rendering software. You can see a video demo of how this all works in The Verge’s coverage.
By using this iPhone X centric workflow, the creation of assets can be achieved much more quickly and much more cheaply compared to setting up a full motion capture recording suite.
The Walking Dead: Our World combines cinematic cut scenes with augmented reality battles, powered by ARKit.
When the game launches, the company plans to package up this feature as a way for players to animate their own characters. It’s pretty novel for them to also use the same domestic technology for the animations of the scripted sequences.
The final game should be available to play in the next few months.