Firelight Technologies FMOD Studio API

Spatial Audio

Introduction

Historically audio spatialization (the process of taking an audio file and making it sound "in the world") has been all about positioning sound in speakers arranged on a horizontal plane. This arrangement is often seen in the form of 5.1 or 7.1 surround. With the advancement of VR technology more emphasis has been put on making sound as immersive as the visuals. This is achieved by more advanced processing of the audio signals for the traditional horizontal plane as well as the introduction of height spatialization. This has given the rise of the term "spatial audio" which focuses on this more realistic approach to spatialization.

Within FMOD there are several ways you can achieve a more immersive spatialization experience, depending on your target platform some may or may not apply. The following sections outline a few general approaches with specific implementation details contained within.

Channel based approach

The most traditional way to approach spatialization is by panning signal into virtual speakers, so with the introduction of 7.1.4 (7 horizontal plane speakers, 1 sub-woofer, 4 roof speakers) you can do just this.

• Set your FMOD::System to the appropriate speaker mode by calling System::setSoftwareFormat(0, FMOD_SPEAKERMODE_7POINT1POINT4, 0).
• Select an output mode capable of rendering 7.1.4 content System::setOutput(FMOD_OUTPUTTYPE_WINSONIC).
• Specify a buffer size compatible with Windows Sonic System::setDSPBufferSize(480, 4). NOTE: This requirement will be removed in a future version.

You can now System::createSound and System::playSound content authored as 7.1.4. If you have the necessary sound system setup (i.e. Dolby Atmos) you will hear the sound play back including the ceiling speakers. If you have a headphone based setup (i.e. Windows Sonic for Headphones or Dolby Atmos for Headphones) you will hear an approximation of ceiling speakers.

To take an existing horizontal plane signal and push it into the ceiling plane you can create an FMOD spatializer and adjust the height controls.

• Create the spatializer with System::createDSPByType(FMOD_DSP_TYPE_PAN).
• Add it to an FMOD::Channel or FMOD::ChannelGroup with ChannelControl::addDSP.
• Control the height by setting FMOD_DSP_PAN_2D_HEIGHT_BLEND via DSP::setParameterFloat.

Not only will this let you blend to the 0.0.4 ceiling speakers by setting the value between 0.0 and 1.0, it will also let you blend from the 0.0.4 ceiling speakers to the ground plane 7.1.0 by setting the value between 0.0 and -1.0.

Object based approach

To get more discrete spatialization of an audio signal you can use the FMOD object spatializer, so named because the audio signal is packaged with the spatialization information (position, orientation, etc) and sent to an object mixer. Often used to highlight important sounds with strong localization to add interest to a scene, usually used in-conjunction with the channel based approach, be that 7.1.4 or even simply 5.1 / 7.1.

• Set your FMOD::System to an object ready output plugin by calling System::setOutput(FMOD_OUTPUTTYPE_WINSONIC) or System::setOutput(FMOD_OUTPUTTYPE_AUDIO3D).
• Create an object spatializer with System::createDSPByType(FMOD_DSP_TYPE_OBJECTPAN).
• Provide 3D position information with FMOD_DSP_OBJECTPAN_3D_POSITION via DSP::setParameterData.

There is no limit to how many FMOD_DSP_TYPE_OBJECTPAN DSPs you create, however there is a flexible limit on a per-platform basis for how many can be processed. When there are more object spatializers in use than there is available resources FMOD will virtualize the least significant sounds by processing with a tradition channel based mix.

An important consideration when using object spatializers is signal flow, unlike most DSPs, after the signal enters the DSP it is then sent out to the object mixer. The object mixer could be a software library or a physical piece of hardware, in all cases though you no longer have access to that signal. Any processing you would like to perform must be done before that point. However (to assist mixing) the object spatializer will automatically apply any "downstream" ChannelGroup volume settings so it behaves similarly to the standard FMOD spatializer.

Third party plugins

In addition to the built-in channel and object based approaches there are third party plugins available that can assist too. The FMOD DSP plugin API (see FMOD_DSP_DESCRIPTION) allows any developer to produce an interface for their spatial audio technology and provide it across all FMOD platforms. Additionally the FMOD output plugin API (see FMOD_OUTPUT_DESCRIPTION) allows developers to implement a renderer for the FMOD object spatializer extending the functionality to more platforms and more technologies.

Resonance Audio Spatializer

Once such third party is the Resonance Audio cross-platform suite of plugins that comes bundled with FMOD. Resonance Audio offers a "Source" plugin which behaves much like the FMOD object spatializer in that audio is sent out to an object mixer, however the final signal returns as binaural output at the "Listener" plugin. Resonance Audio also offers a "Soundfield" plugin for playing back first order Ambisonic sound fields. For more details about the usage of Resonance Audio please check out the user guide.

Oculus Spatializer

Another cross-platform suite of spatial audio plugins is that offered by Oculus as part of their Audio SDK. You can find instructions and downloads for these available on their website.

Coming Soon

Additional plugins available in the future include:

• dearVR
• Steam Audio
• G'Audio Craft
• OSSIC
• RealSpace3D

Usage in FMOD Studio

All of the functionality above has been presented in the context of the lowlevel FMOD API, however it is also all available (including the plugins) within FMOD Studio. For more details about the individual components and their visual representations please read the FMOD Studio manual, for a quick reference of where each feature applies see below.

• Output mode selection: Edit -> Preferences -> Output Device, set Windows Sonic.
• 7.1.4 output: Window -> Mixer, select "Master Bus", right click "out" on the deck, set Surround 7.1.4.
• Height control: Use the "height" slider that is part of the deck panner on any bus configured as 7.1.4.
• Object spatialization: Right click the deck for any event, Add effect -> FMOD Object Spatializer.
• Resonance Audio spatialization: Right click the deck for any event, Add effect -> Plug-in effects -> Google -> Resonance Audio Source.

Note: When using Windows Sonic output you must first be running Windows 10 Creators Update. You must also configure it for your audio device. Right click the speaker icon in the system tray -> Playback devices -> Right click your default device -> Properties -> Spatial sound -> Spatial sound format, now choose your desired spatial technology. FMOD will use your default output device with the technology you select here.