On April 28th, 2020, Ruben Verhack (IDLab-MEDIA) successfully defended his joint PhD in an entirely virtual setting. This joint PhD is a collaboration between the Communication Systems Lab, TU Berlin and IDLab, Ugent. This research was performed under the supervision of Prof. Peter Lambert (Ugent) and Prof. Thomas Sikora (TU Berlin)

Steered Mixture-of-Experts for Image and Light Field Representation, Processing and Coding:

A universal approach for immersive experiences of camera-captured scenes

Abstract

Imagine walking around in a movie, just as we are used to in 3D games. However, virtual reality based on camera-captured content is fundamentally different compared to VR in games. The main difference is that there is no knowledge of the geometry of the scene. At the moment, 360° degree video is the most prominent form of VR for camera-captured content. Nevertheless, the viewer is still limited as they can not walk around the scene themselves.

In the dissertation, a novel method is proposed for representing and coding a wide range of image modalities (images, 360° video, light fields, light field video, …) in a way that can scale towards these future camera-captured VR experiences. This is done nor by relying on 3D graphics, nor does it rely on standard techniques from video coding. In this work, the concept of pixels is abandoned and instead replaced by multi-dimensional kernels, i.e. “the pixel 2.0”. The dissertation shows that such kernel-based representation can scale towards such future applications.

Presentation

In the presentation below, Ruben Verhack sketches the context and the problems that are currently there for representing and coding scenes for image-based virtual reality. The presentation is purposefully kept light and it does not dive deep into the mathematics or the fine details of the solution. The presentation is rather intended to give a notion of the needs and the proposed solution in a way that is understandable by the broader audience.

The jury decided to award Ruben Verhack with the titles:

• Doctor of Computer Science Engineering (Ghent University)
• Doktor der Ingenieurwissenschaften (TU Berlin) with “summa cum laude” grade

The presented work was previously awarded the following awards:

• Top 10% paper - IEEE International Conference on Image Processing, 2014, Paris, FR
• Highly Commended Student Paper - 31st Picture Coding Symposium, 2015, Carines, AU
• Google Faculty Research Award 2016 (granted to Prof. Sikora, TU Berlin) for the joint research
• Best Student Paper Award - IEEE International Conference on Multimedia and Expo, 2017, Hong Kong
• Finalist Best Paper Award, IEEE International Conference on Multimedia and Expo, 2017, Hong Kong

The dissertation can be downloaded at http://phd.rubenverhack.be/