Nvidia’s New ‘Neuralangelo’ Effortlessly Sculpts 3D Objects From Phone Videos

Michelangelo’s marble sculptures are revered for their realism. Now, over 500 years later, researchers at Nvidia are hoping to emulate his precision.

The trillion-dollar graphics processing and artificial intelligence giant has introduced ‘Neuralangelo’, its latest AI model that promises to generate high-fidelity 3D objects seamlessly. All you need is 2D footage, which you can record from your smartphone.

“Like Michelangelo sculpting stunning, lifelike visions from blocks of marble, Neuralangelo generates 3D structures with intricate details and textures,” Nvidia tells DesignTAXI in an email.

The tool is able to produce lifelike “digital twins” of sculptures, buildings, and other real-world objects—mimicking even complex textures of materials like roof shingles, smooth marble, and panes of glass.

Creators can then use these assets in design projects ranging from video games to industrial mockups, robotics applications, and virtual-reality software.

Powered by the same neural technology that drives ‘Instant NeRF’, Nvidia’s program that translates 2D imagery into 3D scenarios, Neuralangelo studies various angles in a 2D video of an object or scene, and cherry-picks frames to generate the right depth, size, and shape.

Then, it creates a rough 3D sketch of the scene, “like a sculptor starting to chisel the subject’s shape,” Nvidia explains in the press release. This reproduction will guide the engine to develop a precise render, “just as a sculptor painstakingly hews stone to mimic the texture of fabric or a human figure.”

A video demonstration shows how Neurangelo can reconstruct intricate sculptures, like Michelangelo’s David, as well as everyday items like a flatbed truck, and building interiors and exteriors.

Nvidia says the technique “significantly surpasses prior methods” of 3D object generators, which often have trouble re-enacting colors and repetitive textures to a T.

Plus, “the high fidelity makes [Neuralangelo’s] 3D reconstructions easier for developers and creative professionals to rapidly create usable virtual objects for their projects using footage captured by smartphones,” says the company.

Ming-Yu Liu, senior director of research at Nvidia Research and a co-author of a paper outlining this technology, hails Neuralangelo’s capabilities as a potentially “huge benefit to creators,” as it bridges the real and digital worlds.

“This tool will eventually enable developers to import detailed objects—whether small statues or massive buildings—into virtual environments for

video games or industrial digital twins,” Liu adds.

Nvidia’s researchers will reveal more about Neuralangelo, alongside nearly 30 other projects, later this month at the Conference on Computer Vision and Pattern Recognition (CVPR) in Vancouver.

[via Nvidia, images courtesy]