ILLUMINATE - Interactive streaming and representation for totally immersive virtual reality applications
Recent breakthroughs in capture and display technologies are leveraging highly immersive Virtual Reality (VR) applications. These emerging applications offer more Degrees-of-Freedom (DOF) to the users and thus make the experience much more immersive compared to traditional 2D visual content. This is largely pushed by the dream to achieve the same level of immersion for camera captured scenes as it is currently available for CGI content. In that matter, 6DOF can be considered the ultimate immersive experience as the displayed scene or environment can be viewed from any viewpoint and in any direction. Providing 6DOF experiences requires research efforts and non-incremental solutions in order to generate, compress, and stream the data, as well as to visualize and measure the quality of experience of the end-user depending on its device. In this project, we aim to take a first leap towards 6DOF by addressing and focusing on the key technical challenges of the interactive streaming of camera-captured 6DOF virtual reality content in a confined space (‘windowed 6DOF’). The project will valorize and deliver contributions in windowed 6DOF content generation, representation, streaming, visualization, layered interaction, and quality of experience.
ILLUMINATE will take a first leap towards 6DOF by addressing and focusing on the key technical challenges of the interactive streaming of camera-captured 6DOF virtual reality content
In ILLUMINATE, in-depth research will be done along the following directions:
- Design and rendering of dynamic hyper-realistic Computer-Generated Imagery (CGI) scenes (6DOF VR data set) taking into account novel storytelling aspects for windowed 6DOF interactive experiences.
- Design and comparison of representation & compression techniques for dynamic windowed 6DOF content able to be streamed towards the end-users with a focus on both data reduction and interactive functionalities.
- Design of the next-gen content delivery network (CDN) functionalities and streaming protocols to enable the efficient streaming of windowed 6DOF virtual reality experiences towards the end-users.
- Design of solutions able to render and display the windowed 6DOF virtual reality experience to the end-user while guiding the user along the storyline.
- Design of methods and procedures to assess quality of experience (QoE) for windowed 6DOF virtual reality experiences providing reproducibility and statistical significance amongst subjects.
With this research focus in mind, two main use cases will be demonstrated and evaluated. The first use case focuses on individual windowed 6DOF virtual reality experiences by bringing it to head mounted devices (HMDs). The second use case targets to bring interactive 6DOF experiences to immersive installations (caves or domes). Both use cases will be translated into tangible proof of concepts (PoC) demonstrating the integration of the entire end-to-end windowed 6DOF chain.
All research aspects within this project will be validated through either expert testing, user validation, or both to optimize the QoE for the envisioned applications.
The research consortium
Our industrial partners within the ILLUMINATE project are: Nokia, THEO Technologies, VRT, and fellow imec research groups imec-VUB-ETRO and imec-UGent-mict.
