In VR development, in addition to graphics and visual rendering, audio processing is an important part. Good audio processing can deceive users’ hearing and achieve immersive effects. This paper mainly introduces how WebVR audio is developed.

VR Audio

The output hardware of VR audio is mainly headset. According to the relationship between audio source and scene, VR audio can be divided into two categories: static audio and Audio spatialization.

Static audio

This kind of audio works on the whole VR scene, which can be simply understood as background music. The audio output is static, such as the sound of breeze raindrops, the sound of downtown and other background sound that fills the whole scene. For environmental sound development, we can simply use the <audio> tag to loop.

Spatial audio

Audio acts on the entity of space and has the position relationship between the speaker and the listener. The audio output will change dynamically according to the distance and direction between the speaker and the user. It simulates the propagation mode of sound in reality and has a sense of space.

Realization principle: In the virtual scene, the distance between the speaker and the speaker is described by adjusting the amplitude of the audio, and then the speaker output of the left and right headphones is controlled by adjusting the difference between the audio channel to describe the position of the speaker relative to the listener.

  • From the perspective of the distance between the voicing body and the user’s two points, for example, the farther the distance is, the smaller the audio volume (amplitude) should be;
  • From the direction of the speaker and the user, if the speaker is to the left of the listener, the left channel of the audio output should be louder than the right channel.
Principle of 3D stereo sound

Slightly more complex Audio processing, such as Audio spatialization, can be done through the Web Audio API.

Introduction to the Web Audio API

The Web Audio API provides a powerful Audio processing system that allows us to control and process Audio in real time, such as Audio visualization, Audio mixing, and so on, through JS in the browser.

Web Audio process

The Processing process of Web Audio can be likened to the processing of the sound source by a processing plant, which is connected by multiple processing modules AudioNode. After a series of processing, the Audio source is transmitted to the speaker.

AudioContext

The Canvas context represents the control center for the audio processing plant, which is responsible for the creation and composition of audionodes, created by new AudioContext().

AudioNode

AudioNode AudioNode is the processing module of the processing plant, which can be divided into three types according to functions: input node, processing node and output node. Each node has a connect method to connect to the next node, which outputs the audio to the next module.

  • Input nodes are mainly responsible for loading decoded audio sources, such as retrieving binary audio sourcesBufferSourceNodeGet <audio> audio sourceMediaElementSourceNodeAnd so on;
  • Processing nodes mainly perform computational processing on audio data, such as audio amplitude processingGainNodeAnd so on;
  • Output nodes output audio to speakers or headphones,AudioContext.destinationIs the default output node.

A simple audio processing process can be divided into four steps:

  1. Creating an Audio Context
  2. Create and initialize input nodes and process nodes
  3. The input node, processing node and output node are connected
  4. Dynamically modify node properties to output different sound effects

Refer to the following code:

const myAudio = document.querySelector('audio'); const audioCtx = new AudioContext(); // Create audio context // create input node, decode audio tag audio source; Create processing nodes, processing audio const source = audioCtx. CreateMediaElementSource (myAudio); const gainNode = audioCtx.createGain(); Source.connect (GainNode); // create GainNode to control audio amplitude Gainnode.connect (audioctx.destination); // Connect the input node to gainNode. // Connect gainNode to destination output node // Dynamically change node attributes to generate different sound source.start(0); Gainnode.gain. value = val; // Set the volumeCopy the code

Now that you understand the development process of Web Audio, let’s look at how to implement Audio Spatialization in WebVR. Here, the VR scenario is developed using Three.js.

Realize spatial audio

Audio Spatialization is realized through the coordination of AudioListener and PannerNode nodes. These two objects can dynamically process Audio sources according to spatial orientation information and output left and right sound channels.

  • AudioListenerObject represents a listener (user) in three dimensions, throughAudioContext.listenerAttribute acquisition;
  • PannerNodeObject refers to the voicing body in three-dimensional space, throughAudioContext.createPanner()To create.

    We need to initialize these two objects and dynamically pass them spatial orientation information as an input parameter.

Set the PannerNode

const myAudio = document.querySelector('audio'); const audioCtx = new AudioContext(); / / create audio context const source = audioCtx. CreateMediaElementSource (myAudio); / / set the position PannerNode const panner = audioCtx. CreatePannerNode (); panner.setPosition(speaker.position.x, speaker.position.y, speaker.position.z); // Pass the voice coordinates to PannerNode source.connect(panner); // Connect the input node to the PannerNode processing node panner.connect(Audioctx.destination); // PannerNode connects to the output node source.start(0); // Play the audioCopy the code

Set the AudioListener

VR user head display has at most 6-DOF: 3-Dof system of position position and 3-Dof system of orientation direction. We need to transmit the 6-Dof information to AudioListener, which will process audio data for us. For user location data, the AudioListener provides three location attributes: PositionX, positionY, positionZ, it respectively represent the listener to the current position of the xyz coordinate, we can be the position of the user in the scenario (usually with a camera position) assigned to these three properties.

// Set position const listener = Audioctx.listener; listener.positionX = camera.position.x; listener.positionY = camera.position.y; listener.positionZ = camera.position.z;Copy the code

In addition to passing in the user’s location, we also need to pass in the user’s perspective direction to the AudioListener, Specific for AudioListener Forward vector forwardX three components, forwardY, three component upX forwardZ and the Up vector, upY, upZ assignment.

  • The Forward vector points Forward along the nose and defaults to (0,0,-1);
  • The Up vector points Up overhead and defaults to (0,1,0).
The Forward vector and the Up vector
  • In a VR scene, when the user moves his head to change his perspective, the up or forward vectors change, but they are always vertical.

Up vector = Camera. Rotation matrix × [0,1,0]


Rotation matrix × [0,0,-1]

By referring to the above formula, camera here is the camera of three.js and refers to the user’s head. The camera rotation (quaternion) matrix is obtained by Camera.

// calculate the current listener's forward vector let forward = new THREE.Vector3(0,0,-1); forward.applyQuaternion(camera.quaternion); // Multiply the camera quaternion matrix to get the current forward vector forward.normalize(); Value = forward. X; listener.forwardx. value = forward. listener.forwardY.value = forward.y; listener.forwardZ.value = forward.z; Let up = new three.vector3 (0,1,0); up.applyQuaternion(camera.quaternion); // The up vector is multiplied by camera quaternion matrix to get the current up vector up.normalize(); // The up component of the AudioListener listener.upx.value = up.x; // The up component of the AudioListener listener.upx.value = up.x; listener.upY.value = up.y; listener.upZ.value = up.z;Copy the code

WebVR implements audio characters

In A VR scenario, audio can be divided into two roles according to the initiator and receiver: Speaker and Listener, that is, user.

Listener-speaker one-to-many relationship

A VR scene audio role consists of a Listener and multiple speakers, so the author encapsulates PannerNode and AudioListener independently and integrates them into Speaker class and Listener object.

PS: We follow the previous three. Js development method of WebVR, please refer to itWebVR Development — Standard Tutorial

Speaker implementation

The Speaker class represents the Speaker and does the following:

  1. The initialization phase loads and parses the audio source, creates and connects input nodes, processing nodes, and output nodes
  2. provideupdatePublic method that updates the PannerNode position every frame.
class Speaker { constructor(ctx,path) { this.path = path; this.ctx = ctx; this.source = ctx.createBufferSource(); this.panner = ctx.createPanner(); this.source.loop = true; // Set the audio loop to play this.source.connect(this.panner); // Connect the input node to PannerNode this.anner.connect (ctx.destination); // connect PannerNode to output this._processAudio(); Update (position) {const {panner} = this; panner.setPosition(position.x, position.y, position.z); } _loadAudio(path) {return fetch(path).then(res => res.arrayBuffer()); } async _processAudio() { const { path, ctx, source } = this; try { const data = await this._loadAudio(path); Const buffer = await ctx.decodeAudioData(data); // Decode audio data source.buffer = buffer; // Assign the decoded data to the BufferSourceNode input node source.start(0); } catch(err) {console.err(err); }}}Copy the code

The initialization process is slightly different here, where the fetch is used to request the audio file and parse the audio data through the BufferSourceNode input node. The update method passes in the voice body position and sets the PannerNode position.

The Listener implementation

The Listener object represents the Listener and provides the update common method, passing in the position and direction of the AudioListener in each frame.

Const Listener = {init(CTX) {this.ctx = CTX; this.listener = this.ctx.listener; }, update(position,quaternion) { const { listener } = this; listener.positionX = position.x; listener.positionY = position.y; listener.positionZ = position.z; // calculate the current listener's forward vector let forward = new THREE.Vector3(0,0,-1); forward.applyQuaternion(quaternion); forward.normalize(); listener.forwardX.value = forward.x; listener.forwardY.value = forward.y; listener.forwardZ.value = forward.z; Let up = new three.vector3 (0,1,0); up.applyQuaternion(quaternion); up.normalize(); listener.upX.value = up.x; listener.upY.value = up.y; listener.upZ.value = up.z; }}Copy the code

The update method is passed to the position and quaternion matrix of the camera to set the position and direction of the AudioListener.

Next, introduce the Listener and Speaker into the WebVR application. The following example describes a crude scenario: a car passing you with its horn blaring and heading off into the distance.

class WebVRApp { ... start() { const { scene, camera } = this; . // Create the light, the ground // Create a humble car const material = new THREE.MeshLambertMaterial({ color: 0xef6500 }); This. Car = new THREE. this.car.position.set(-12, 2, -100); scene.add(this.car); const ctx = new AudioContext(); // Create AudioContext listener.init (CTX); // initialize listener this.car_speaker = new Speaker(CTX,'audio/horn.wav'); // Create speaker, pass context and audio path}}Copy the code

First create the car in the start method, then initialize the Listener and create a Speaker.

class WebVRApp { ... update() { const { scene, camera, renderer} = this; // Start rendering this.car.position.z += 0.4; this.car_speaker.update(this.car.position); // Update speaker position listener. update(camera. Position, camera. Quaternion); // Update the Listener position and head towards renderer.render(scene, camera); } } new WebVRApp();Copy the code

In the animation render UPDATE method, update the position of the car, and call the Update methods of the Speaker and Listener, passing in the position of the car, the position of the user, and the rotation matrix to update the audio space information.

Example address: yonechen.github. IO/webvr-hello… (need to support es7 browser, such as the new chrome, didn’t do too lazy package compiled � �) source address: https://github.com/YoneChen/WebVR-helloworld/blob/master/3d-audio.html

summary

This paper mainly explains the realization steps of WebVR application Audio spatial, the core is the use of Web Audio API PannerNode and AudioListener two objects to process Audio sources, the end of the paper shows a simple code example of VR Audio. Three. Js itself also provides complete audio spatial support, see PositinalAudio.

I’m currently implementing WebVR multiplayer chat rooms, and my next article will focus on this. More articles can pay attention to WebVR development tutorial — interactive events (two) using Gamepad WebVR development tutorial — in-depth analysis of the development and debugging scheme and principle mechanism of WebVR development tutorial — standard entry How to use Three.js to develop WebVR scenarios