SoylentNews

upstart writes:

Breakthrough AI Technique Enables Real-Time Rendering of Scenes in 3D From 2D Images:

Humans are pretty good at looking at a single two-dimensional image and understanding the full three-dimensional scene that it captures. Artificial intelligence agents are not.

Yet a machine that needs to interact with objects in the world — like a robot designed to harvest crops or assist with surgery — must be able to infer properties about a 3D scene from observations of the 2D images it's trained on.

While scientists have had success using neural networks to infer representations of 3D scenes from images, these machine learning methods aren't fast enough to make them feasible for many real-world applications.

A new technique demonstrated by researchers at MIT and elsewhere is able to represent 3D scenes from images about 15,000 times faster than some existing models.

The method represents a scene as a 360-degree light field, which is a function that describes all the light rays in a 3D space, flowing through every point and in every direction. The light field is encoded into a neural network, which enables faster rendering of the underlying 3D scene from an image.

The light-field networks (LFNs) the researchers developed can reconstruct a light field after only a single observation of an image, and they are able to render 3D scenes at real-time frame rates.

"The big promise of these neural scene representations, at the end of the day, is to use them in vision tasks. I give you an image and from that image you create a representation of the scene, and then everything you want to reason about you do in the space of that 3D scene," says Vincent Sitzmann, a postdoc in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and co-lead author of the paper.

[...] "Neural rendering has recently enabled photorealistic rendering and editing of images from only a sparse set of input views. Unfortunately, all existing techniques are computationally very expensive, preventing applications that require real-time processing, like video conferencing. This project takes a big step toward a new generation of computationally efficient and mathematically elegant neural rendering algorithms," says Gordon Wetzstein, an associate professor of electrical engineering at Stanford University, who was not involved in this research. "I anticipate that it will have widespread applications, in computer graphics, computer vision, and beyond."

Project Website:
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering

Journal Reference:
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, et al. Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering 4 June 2021, Computer Science Computer Vision and Pattern Recognition. arXiv:2106.02634

Original Submission