How can I apply audio effects and filters to a video stream in Javascript?

Objective

To apply an audio filter or effect on an inbound video stream/subscriber.

Applies To

• Vonage Video API
• Javascript SDK
• Subscriber Audio

Procedure

Audio and Video are normally provided as a single MediaStream to a <VIDEO> element in your HTML page.  These will need to be separated in order to apply audio effects or filters.  In this example, we are going to use the StereoPanner object to allow subscribers to pan individual streams to the left or right.  There are many other features in the Web Audio API that could be used in this way.

Below is the basic process.  Please find an example Panner class in the additional information below.

1. Get a reference to your video element
2. Get the MediaStream object from the video element
3. Get the AudioTrack from the MediaStream
4. Clone the AudioTrack and insert it back into the MediaStream
5. Disable the original AudioTrack in the MediaStream
6. Create an AudioSource from the MediaStream
7. Create a StereoPanner object
8. Connect the AudioSource to the Panner
9. Connect the Panner to the Destination
10. Set a pan value from -1 -> 1 in the Panner to pan audio left to right

Additional Information

Here is an example Panner class that can be used to pan audio on an inbound video stream.

class Panner {
  constructor(element){
    // This is the VIDEO element you are receiving the stream into
    this.element = element
    // Create an AudioContext
    const audioContext = new (window.AudioContext || window.webkitAudioContext)();
    // The "stream" is the audio and video from the element
    let stream = element.captureStream();
    // The origAudioTrack is the original audio from the stream.
    let origAudioTrack = stream.getAudioTracks()[0]; // There should only be 1
    let newAudioTrack = origAudioTrack.clone(); // Then we create a clone of the audio track
    
    // Then we add the new audio track back into the original stream and disable the old one
    stream.addTrack(newAudioTrack);
    origAudioTrack.enabled = false;

    // We create an audio source from the stream and a new StereoPanner
    let source = audioContext.createMediaStreamSource(stream);
    this.panner = audioContext.createStereoPanner();

    // We set a value for the panner.  Default is 0 (zero) for centered
    this.panner.pan.value = 1;  // --> sound panned to the right

    // Finally we connect the panner to the output destination and the source to the panner
    //     source --> panner --> destination
    this.panner.connect(audioContext.destination);
    source.connect(this.panner);
  }

  // Calling set(-1) will pan left and set(1) will pan right
  set (value){
    this.panner.pan.value = Number(value).toFixed(1);
  }
}

To use this code you need to create a new Panner object passing your video element to the constructor.  You then need to call Panner.set() with a value of -1 to 1 to pan from left to right.