Project co-funded by the European Fund for Regional Developement (FEDER) and the AVANZA2 program from Ministerio de Industria, Energía y Turismo (2008-2011) With Reference Number TSI-020302-2010-61
The purpose of this project is to research on the immersive digital contents production, and all the involved value chain (production, transmission and reception), developing the interactivity and immersivity further than current 3D, merely ‘bidimensional’. Thus, it is expected that final user will experiment the feeling of belonging to the environment, being able to interact with it from any spatial direction.
Innovation in media consumption has been a constant evolution in the last years and decades: from color television, to interactive television, media digitalization, recent commercial development of High Definition, and the development of three-dimensional contents. One of the key aspects in the convergence lived in the last years has been the evolution of virtual reality in an immersive way (generally focused on videogames) to conventional contents.
As a result of the project and for validation purposes, it has been accomplished the design and implementation of two use case scenarios: CAVETM and LIVING ROOM.
The CAVETM scenario is an immersive experience based on a panoramic view of the city of Barcelona, using the I-Space system available at CeDInt for its visualization. In this experience, the user is able to interact in real time with the scenario through 3D synthetic objects. This way, it is allowed to position and merge them freely in the video scenario taking into account depth information and possible occlusions. As part of the experience, it has been included also additional multimedia information such as 3D images and audio related to the interactive content.
Panoramic view of Barcelona in the CAVETM scenario
Merging of synthetic 3D objects within the video
Regarding the LIVING ROOM scenario, a pilot test has been designed concerning all the work done within each module of the chain value for immersive television content. The content in this case involves the visualization of a high definition football match recorded in stereoscopic format, coded and broadcasted live using a DVB modulator. The video stream can be received at the main visualization terminal (high definition TV) or any additional device (tablet, smartphone, etc.) connected through the local area network. The video signal, include other additional content intended to enhance the user immersive experience like associated statistics, actuation commands over domotic devices and interactive 3D models synchronized with specific events of the match.
Pilot test of the LIVING ROOM exhibited during the JVRC 2012
The research work at the Virtual Reality laboratory in CeDInt has been focused mainly in the inclusion and merging of synthetic 3D content within the high definition video. For this purpose, the next list summarizes different technologies and algorithms that have been studied and implemented:
- Depth map estimation using stereoscopic video sources
- Panoramic video generation from multiple overlapped captures, each corresponding to different views of the same scenery.
- Real time merging of interactive 3D elements considering depth information for the computation of occlusions.
Depthmap map computation for a video frame in the LIVING ROOM scenario
Merging of synthetic 3D objects in the LIVING ROOM scenario
Video Demonstration of results in CAVE environment
Video demonstration of results in living room environment.