Not one, but both light field video papers got accepted for Picture Coding Symposium 2018! Both papers discuss parts of Steered Mixture-of-Experts (SMoE). The main take-away messages are that

• We are able to model and approximate 5-D light field videos up to high objective quality
• We introduced a effective novel way to model SMoE models: robust and less artefacts
• We have shown that given such models, we can render views with hard real-time constraints with compromising some quality if necessary
• Furthermore, we can render pixel-level parallel! Which is very desirable, and not possible with most state-of-the-art

Progressive Modeling of Steered Mixture-of-Experts for Light Field Video Approximation

by Ruben Verhack, Glenn Van Wallendael, Martijn Courteaux, Peter Lambert, Thomas Sikora

Steered Mixture-of-Experts (SMoE) is a novel framework for the approximation, coding, and description of image modalities. The future goal is to arrive at a representation for Six Degrees-of-Freedom (6DoF) image data. The goal of this paper is to introduce SMoE for 4D light field videos by including the temporal dimension. However, these videos contain vast amounts of samples due to the large number of views per frame. Previous work on static light field images mitigated the problem by hard subdividing the modeling problem. However, such a hard subdivision introduces visually disturbing block artifacts on moving objects in dynamic image data. We propose a novel modeling method that does not result in block artifacts while minimizing the computational complexity and which allows for a varying spread of kernels in the spatio-temporal domain. Experiments validate that we can progressively model light field videos with increasing objective quality up to 0.97 SSIM.

Some examples (original left, reconstruction right)

Hard Real-Time, Pixel-Parallel Rendering of Light Field Videos using Steered Mixture-of-Experts

by Ignace Saenen, Ruben Verhack, Vasileios Avramelos, Glenn Van Wallendael, Peter Lambert

Steered Mixture-of-Experts (SMoE) is a novel framework for the approximation, coding, and description of image modalities such as light field images and video. The future goal is to arrive at a representation for Six Degrees-of-Freedom (6DoF) image data. Previous research has shown the feasibility of real-time pixel-parallel rendering of static light field images. Each pixel is independently reconstructed by kernels that lay in its vicinity. The number of kernels involved forms the bottleneck on the achievable framerate. The goal of this paper is twofold. Firstly, we introduce pixel-level rendering of light field video, as previous work only rendered static content. Secondly, we investigate rendering using a predefined number of most significant kernels. As such, we can deliver hard real-time constraints by trading off the reconstruction quality.