Groupe Audio Acoustique

Wiki du groupe de recherche en audio et acoustique du LIMSI

Outils pour utilisateurs

Outils du site


Ghost Orchestra

People @ LIMSI:

Project in partnership with Conservatoire de Paris (CNSMDP)

  • Concert recording supervisor - Jean-Marc Lyzwa

This project was funded in part by the FUI-BiLi project on Binaural Listening and the ANR-ECHO project concerning Digital Heritage and Historic Auralizations.

Project Description

The project investigates our perception of space through sound and images. The journey starts with a concert, organized to celebrate the 850th birthday of the Notre-Dame de Paris cathedral, staging a symphonic orchestra accompanying soloists and choirs through J. Massenet oratorio “La Vierge” (The Virgin). Although brilliant, this performance was nonetheless ephemeral, offering this 19th century piece augmented by this edifice of such peculiar acoustics to a privileged few.

In conjunction with the concert, the Conservatoire National Supérieure recorded the event, with each instrument section and soloists being carefully recorded. In an attempt to explore the possibility of recreating this concert event for future spectators, providing a spatially accurate rendition of the concert and allowing one to navigate within the cathedral and experience different acoustic perspectives, the Ghost Orchestra Project was launched.

Combining research efforts in binaural audio of the FUI-BiLi project, digital heritage acoustic recreations of the ANR-ECHO project, and the development of interactive virtual reality environments in the BlenderVR project, this work was a means to join forces and attempt a monumental work.

Overview of the project

The following is a presentation of the project that was prepared for the FISM conference 2015. While the conference was cancelled, we provide the presentation here.

Link to the PDF!

The basic premise, as far as the audio element is concerned, was to utilize the close-mic'd audio tracks from the different musicians and to replay them in a virtual acoustic reconstruction of the Cathedral, in order to recreate the proper spatial information regarding instrument positions and directivities and their relation to the room acoustics of the space. This effort required the construction of a geometrical acoustic model, which was calibrated according to measurements also carried out in the cathedral. The goal being to make sure the sound of an instrument playing at position A and heard by a listener at B would be perceived exactly as in the original building (reverb, position, presence, timbre, amplitude, etc.).

The 3-dimension room impulse response is then numerically computed for each and every instrument and every potential listener position. These responses are then *convolved* with the corresponding audio track and appropriately combined to create the Ghost Orchestra performance.

Audio example of the simulated environment

The virtual reconstruction of the instruments in the cathedral during the performance can be compared to one of the recorded tracks at the conductor's position. If the acoustics and balance are perceived appropriate at this position, we can have confidence that listening at other positions which were not recorded will still provide a realistic virtual reconstruction.

These audio extracts are mono-phonic, representing the recording position microphone placed over the conductor.

Visual 3D model

To accompany the virtual acoustic reconstruction, a 3D model of Notre-Dame was created (3DS Max / Blender), mirroring the visual appearance of the place. Then, the various instruments themselves were reproduced in the 3D environment and positioned in the virtual cathedral so that visitors could visualize the different components of the orchestra that now played in front of them.

The 3D virtual reconstruction can be explored interactively with an Occulus Rift Head-Mounted Display (HMD). This real-time rendering is possible with BlenderVR.

Alternatively, the virtual world can be pre-rendered at a higher resolution as a 360° video.

Demos / Videos

The immersive experience was adapted to different medias allowing for different degrees of interaction.

Free navigation on a predefined trajectory

Free navigation in the scene equipped with a head mounted display + binaural acoustic rendering (3rd order Ambisonic, decoded over 16 virtual speakers, rendered binaurally)

The following video presents a recording of this navigation. The head orientation of the user is taken into consideration for updating both the visual and auditory orientation. Speed of the flying carpet can be controlled using a joystick.

Free rotation along a predefined navigation path

Equirectangular 360° video played on a tablet or on Youtube. This off-line 360° video rendering allows for real-time orientation with a suitable viewer, providing a “steerable window” looking into the virtual world. The video is combined with either Ambisonic or decoded Ambisonic virtual speaker tracks. The 360° audio is then converted to binaural in real-time as a function of the window orientation with an appropriate player. Players have been developed in the FUI-BiLi project by partners Orange Labs and Arkamys.

  • B. N. Postma and B. F. Katz, “Creation and calibration method of virtual acoustic models for historic auralizations,” Virtual Reality, vol. 19, no. SI: Spatial Sound, pp. 161–180, 2015, doi:10.1007/s10055-015-0275-3.
  • B. N. Postma, A. Tallon, and B. F. Katz, “Calibrated auralization simulation of the abbey of Saint-Germain-des-Prés for historical study,” in Intl. Conf. on Auditorium Acoustics, vol. 37, (Paris), pp. 190–197, Institute of Acoustics, Oct. 2015, URL.
  • B. F. Katz, D. Q. Felinto, D. Touraine, D. Poirier-Quinot, and P. Bourdot, “BlenderVR: Open-source framework for interactive and immersive VR,” in IEEE Virtual Reality (IEEE VR), (Arles), pp. 203–204, Mar. 2015, URL.
  • J-M. Lyzwa and A. Baskind, “Exemple d'une production en multicanal 5.1 utilisant des techniques mixtes binaurales et transaurales de spatialisation : la Vierge de Jules Massenet,” presented at les Journées de la Recherhce, Conservatoire National Supérieur de Musique et de Danse de Paris, 18-20 Mar 2014.

projets/ghostorch.txt · Dernière modification: 2016/10/24 22:20 par David Poirier-Quinot