The current state of Audio in entertainment
In cinema and games the use of visual and audio has been thoroughly studied and applied over the years. It is commonly known that the power of atmospheric songs and sound effects are crucial in horror films. I am sure that many of you have watched scary movies in great fear and have noticed that turning the sound off gives a totally different experience. Without sound we usually tend to be able to watch movies with a lower stress level and a lower heart rate, meaning we become less prone to being scared in typical jump scare situations.
Sound in computer games are a key component for creating an immersive experience where your actions match the visual queue. Traditional games played on a monitor or a TV have sound that usually emerge from a left and right speaker. This makes it possible to position sound over a horizontal scale when the game features stereo sound. Many games use e.g. 5.1 surround sound to play sound effects around you based on a more advanced speaker (or headphone) setup. Such a setup allow for more complex sound positioning using audio mixing to enhance your experience.
The rebirth and reintroduction of Binaural Audio
Binaural Audio recordings were first performed in the late 1800s and uses one microphone fitted inside each ear of a dummy head. This makes it possible to record how the sound sounds outside of your ear. When later played back to a listener the actual originating location of the recorded sound can be localised be your brain.
Developments of software algorithms and an increase in computational power have made it possible to simulate how a sound should be played back to the listener in computer game when she is equipped with normal stereo head phones.
When wearing a Virtual Reality Head Mounted Display it is possible for the player to visually look around your whole surrounding. Here, positional audio plays an important role allowing the player to hear in which tree a bird is singing or from where a car is approaching. A specific power of the use of Binaural Audio is when there are multiple human voices in the virtual room and the player wants to focus on a specific voice. With a correlation between the visual and audio representations the player is able to filter out and focus on the important characters in specific areas of the room.
An example of a binaural recording and playback is the Virtual Barbershop which you can listen to here:
Binaural Audio routines for Game developers
Many integrated game development engines have advanced routines for combining visual and audio effects. There are several third party plugins for binaural audio that extend the functionality of the game engines. Below is a short list of some of these plugins.
Realspace 3D AUDIO by Visisonics
Based in Washington the Visisonics team has developed routines that can be used in game development engines such as Unity. Being used in games – for example Technolust from IrisVR – it allows the player to define zones with individual properties that affect the characteristics of the sound.
Another demo that uses RealSpace3D Audio is the “Fixing Incus” demo that you can run on the Oculus DK2: Fixing Incus – Demo by Nick LaMartina (YouTube)
3DCeption by Two Big Ears
With the introduction of soon to be released pathfinding the 3DCeption for Unity plugin enables occluded geometry to affect the sound landscape. The user will be able to get a notion where sound is coming from such as from behind corners or walls.
The latest version targeting Unity (at the time of writing) is v1.2.1, it was released on the 17th December 2015. The team is based in Edinburgh and are now hiring for multiple positions.
Audio in the future of VR
Some say sound constitutes half of the experience. In the case of VR this is likely to be true – if not only even more true. A lot of research has gone into developing advanced routines for graphics rendering with specialized curcuits to go with it. Audio has not received the same treatment. Due to the fact that most of the computational power in games are being dedicated to the graphics render pipeline there is less cycles available for audio. With the emerging interest in VR and with the next generation of head sets (2017 and beyond) – possibly having native support for eye tracking and foveated rendering – I hope that some of the CPU and GPU cycles will be dedicated to more advanced audio processing (Audio Processing Unit).
Ideally audio processing algorithms will be able to take into account the physical space more accurately and more user friendly compared to today, giving game developers the ability to entertain players even more. However they need to be aware of the so-called “uncanney valley” which might come into play for audio as in the case of attempts of photorealistic rendering of humans or AI robots.