Turn iPhone Videos Into Motion-Capture Animation With Epic Games’ New Generator

In 2021, game developer Epic Games released MetaHuman, a creator tool under its graphics engine Unreal Engine that aided animators in conjuring up realistic-looking human characters with lifelike facial features.
 
Since then, it has generated millions of MetaHumans and streamlined many artists’ work process. However, one of its main problems is that it still takes humans tweaking to produce game-usable animation. Often, professional studios will still have to use 4D capture. 
 
Video games are often shot with motion capture (mocap). Performance artists and voice actors will be outfitted in mocap suits with tracking dots and balls placed all over them as they act out cut scenes and certain animations like running and jumping. However, these results are just a visualization of the final product rather than what will be shown in the game.

Epic Games’ new MetaHuman Animator removes the need for motion capture for facial movement and swaps it out for a simple video shot on your iPhone. It currently only works with headshots and is mainly used to capture the tiny details of a person’s face as they think or talk.
 
In a demo video, the gaming company produced a sample featuring Ninja Theory performance artist Melina Juergens, most known for her role as Senua in Hellblade: Senua’s Sacrifice. A closeup of her was shot as she glared into the camera, displaying a range of emotions as the video switched to her being bruised and battered.

The machine-learning program took this simple video of her and churned out a 3D animation of her in less than a minute. What was most impressive was how it caught all the little tics of her face as she moved her eyes and glowered into the lens. It even had realistic tongue movement as she spoke, made possible through an “audio to tongue” algorithm.
 
It is clear that indie game developers will benefit most from such a tool as most will not have the time, resources, or labor to produce such high-quality content.

The uptake of artificial intelligence in the game sectors is sweeping over it swiftly. For example, Unreal Engine has also just released its new 5.2 version that allows visual artists to create a background scene for open-world games and have the model expand on it without having to design everything themselves.
 
On the other hand, Ubisoft has announced its dialogue generator that writes out scenes for non-playable characters (NPCs).
 

While the advancements allow developers to get back to focusing on art direction instead of spending too much time on the backend of the final product, the question of artists’ livelihoods cannot be avoided. Most companies tout the integration of these models as co-pilots to humans, but at the rate they are progressing, it could only be a matter of time before they have complete control of the ship.

 
 
 

[via Ars Technica and BGR, images via Epic Games]