Facial AR Remote is real-time facial motion capture solution, allowing the user to record their expressions using an iPhone X mounted on their face. This doesn’t come without its DIY tricks. We ended up using an Airsoft helmet with a universal attachment located on the front and sides of the helmet, this let us create a 3-point arm out of various mounting kits with a sturdy mountain bike phone mount on the end. We had to use a counterweight on the back of the helmet, as the weight of the iPhone X when held a foot or so from your face gets quite high! We owe a lot to the excellent Kite & Lightning for the design of the helmet. They are doing some incredible stuff with the tech – Real-Time Full-Performance Motion Capture with an iPhone X, Xsens, IKINEMA, and Unreal Engine in fact.

We downloaded the toolkit from a GitHub repo and followed their instructions for settings the project up. The project included a sloth face with a few pre-made blendshapes in poses such as smiling, frowning, blinking, winking, jaw open/closed etc, but we wanted to do more.

We began constructing our own low-poly facial rig and created a total of 52 blendshapes with it in a wide range of poses. These blendshapes had to have identical names to their prefab otherwise the FacialARRemote plugin would reject them, these names differ from the standard blendshape naming convention making this step critical.

After much troubleshooting and tweaking, we finally got our model and all the blendshapes into Unity, and it worked! We had live data streaming from the iPhone X app straight into our face rig within Unity. The plugin then deciphered this data and combined all the blendshapes we had made to create the most accurate representation of our current expression. Winking, smiling, frowning, surprised, sad, suspicious, completely asymmetric poses (E.g. one eyebrow up one down), you name it we got it.

These actions could then be recorded within unity as a standard animation clip, and can be played back on the mesh at any time, making it fantastic for extreme detail within game cutscenes or any use case you have really.