Nvidia Creates ‘Instant’ Imaging AI That Zaps Regular Photos Into 3D Scenes

Image via Nvidia

Although capturing a simple photo would usually suffice, if you could wait a few seconds more to make that picture “walkable,” would you do it?

Graphics processing tech firm Nvidia has built an AI tool that significantly quickens the process of converting 2D imagery into 3D scenarios (known in the industry as NeRF, short for neural radiance fields)—completing the step 1,000 times faster than existing programs, it says. The results are so impressive and immediate, it’s even tagging the term “Instant” in front of it and branding it Instant NeRF.

Essentially, NeRFs rely on neural networks to look at a selection of 2D images, decipher them, and then stitch them into 360-degree visuals that can be viewed from various perspectives. While traditional 3D conversion techniques can take hours, existing NeRF models can do the job within minutes.

Nvidia’s technology accelerates this process and transforms still photos into 3D scenarios “almost instantly,” thanks to a proprietary method called “multi-resolution hash grid encoding” that does the heavy lifting in mere milliseconds.

It’s worth noting that Instant NeRF, as with the usual NeRF techniques, requires a few dozen photos snapped from several camera positions to produce a precise 3D image. Even so, it doesn’t need a ridiculous amount of data to work with, acknowledging the added challenge of movements from capturing too many photos, which could create blur.

“In a scene that includes people or other moving elements, the quicker these shots are captured, the better,” the team explains in a blog post. “If there’s too much motion during the 2D image capture process, the AI-generated 3D scene will be blurry.”

As such, Instant NeRF connects the dots between missing scenes, predicting lighting and possible color changes from all directions, to generate a full, realistic 3D image.

To illustrate the innovation, Nvidia’s team members recreated a famous photo of Andy Warhol with his instant camera. The tribute commemorates the 75th anniversary of the first Polaroid instant photo.

Video via Nvidia

As we move into 3D environments, Nvidia foresees Instant NeRF evolving into an essential format akin to JPEGs for photography. Potential applications include creating avatars and designing architecture for virtual worlds, in addition to training self-driving cars and robots to better perceive measurements of real-world objects.

“If traditional 3D representations like polygonal meshes are akin to vector images, NeRFs are like bitmap images: they densely capture the way light radiates from an object or within a scene,” describes David Luebke, vice president for graphics research at NVIDIA. “In that sense, Instant NeRF could be as important to 3D as digital cameras and JPEG compression have been to 2D photography — vastly increasing the speed, ease and reach of 3D capture and sharing.”

[via PetaPixel and Tom’s Hardware, video and images via Nvidia]