Neat AI Can ‘Predict The Future’ And Turn A Single Photo Into A Believable Video
Image via Shutterstock
While the iPhone’s Live Photos feature automatically captures the immediate moments before and after you take a picture, a new deep learning algorithm by researchers at the University of Washington doesn’t need those—it simply “predicts” what happens during those instances.
The team’s research, which will be presented at the Conference on Computer Vision and Pattern Recognition on June 22, details an AI that can create realistic, high-resolution looping videos from a single photo.
What sets the method apart from other tools for cinemographs and animations is that it needs nothing more than a photo to get going. “It doesn’t require any user input or extra information,” explains Paul G. Allen School of Computer Science & Engineering doctoral student Aleksander Hołyński. He adds, “It effectively requires you to predict the future. And in the real world, there are nearly infinite possibilities of what might happen next.”
Hołyński and his team fed a neural network with thousands of clips depicting flowing motions, including videos of waterfalls, clouds, fire, rivers, and oceans, to train it to predict movements in imagery. They’d show it the first frame of a video and get the algorithm to guess the clip’s motion, and then compare its answer with the full video so that it could refine its predictions.
Turning an image into a video can be helped with a method called “splatting,” which involves the moving of the photo’s pixels to follow the predicted motions—but that idea sounds better on paper. “Think about a flowing waterfall,” Hołyński describes. “If you just move the pixels down the waterfall, after a few frames of the video, you’ll have no pixels at the top!”
To make up for the missing pixels, the team also incorporated “symmetric splatting,” which asks the AI to predict the “past” of a photo too.
Due to its dataset, the tool is currently best at creating “flowing” videos of parts of nature like waterfalls, rivers, smoke, and clouds. In time, the researchers hope to delve further with animations of other objects, like the movement of hair in the wind.
Video via Hołyński et al / CVPR
Image via Hołyński et al / CVPR
Video via Hołyński et al / CVPR
Video via Hołyński et al / CVPR
Video via Hołyński et al / CVPR
[via DIY Photography, videos via Hołyński et al / CVPR, images via various sources]
Also check out these recent news
