Browser Dance Party

HTML5 Audio Visualizers

Jordan Santell / @jsantell

Who is this guy

• Mozilla Addon-SDK
• web audio components
• dancer.js creator

Awesome Example

A Dive In Music by Benoit Damien

Visualizers

• Procedurally generated using audio data
• Deterministic or non-deterministic

Web Audio API

Web Audio Caveats

• One AudioContext per doc
• Prefixed AudioContexts
• Deprecated APIs
• Browser Codec Wars

Support

• v25+
• prefixed
• prefixed, deprecated APIs
• prefixed, deprecated APIs
• Not in v11, future plans

Web Audio API Nodes

• Source
• Media Element
• Media Stream
• Oscillator
• Buffer

• Utility
• Script Proc.
• Analyser
• Chan. Splitter
• Chan. Merger

• Effect
• Gain
• Biquad Filter
• Delay
• Convolver
• Compressor
• WaveShaper

Web Audio API Nodes

• Source
• Media Element
• Media Stream
• Oscillator
• Buffer

• Utility
• Script Proc.
• Analyser
• Chan. Splitter
• Chan. Merger

• Effect
• Gain
• Biquad Filter
• Delay
• Convolver
• Compressor
• WaveShaper

Web Audio API

Web Audio API

Simple Route

var ctx = new AudioContext();
var dest = ctx.destination;
var osc = ctx.createOscillator();
var gain = ctx.createGain();
osc.connect(gain);
gain.connect(dest);
osc.start(0);
					

Buffer Source

• Getting array buffers via XHR
• Audio files are binary!

Array Buffers

function getBuffer (url, callback) {
  var xhr = new XMLHttpRequest();
  xhr.open('GET', url, true);
  xhr.responsetype = 'arraybuffer';
  xhr.onload = function () {
    callback(xhr.response);
  }
  xhr.send();
}
					

Decoding Buffers

function decode (url, cb) {
  getBuffer(url, function (buffer) {
    ctx.decodeAudioData(buffer, cb);
  });
}
					

Playing Buffers

var ctx = new AudioContext();
var source = ctx.createBufferSource();
decode('f.mp3', function (decoded) {
  source.buffer = decoded;
  source.connect(ctx.destination);
  source.start(0);
});
					

DSP

• Interpretting Discrete Signals
• Audio is just values over time!

Sample Rate

Bit Depth

Quality

Time Domain

Time Domain

ENHANCE

ENHANCE

ENHANCE

ENHANCE

Time Domain

Frequency Domain

Frequency Domain

Analyser Node

• analyser.getFloatFrequencyData(array)
• analyser.getByteFrequencyData(array)
• analyser.getByteTimeDomainData(array)

Frequency Data

• nyquist: Half of sample rate
• binCount: half of "FFT size"
• Results in frequency data typed array of binCount length
• Frequency data contains magnitude data for frequencies 0 to nyquist

Frequency Data

• sampleRate = 44100
• nyquist = 44100 / 2
• bins = 1024 / 2
• frequencies per bin: nyquist / bin
• Each bin (element) contains a range of 43.066 frequencies

data[0]; // 0-43hz
data[1]; // 44-87hz
data[bins-1]; // 22.00khz-22.05khz
					

Frequency Data

Fill Uint8Array with 8-bit values of fftSize/2

var a = ctx.createAnalyser();
a.fftSize = 512;
// a.frequencyBinCount == 256
var freq = new Uint8Array(256);
setInterval(function () {
  a.getByteFrequencyData(freq)
}, 20);
					

Time Data

Fill Uint8Array with 8-bit values of fftSize

var a = ctx.createAnalyser();
a.fftSize = 512;
var data = new Uint8Array(a.fftSize);
setInterval(function () {
  a.getByteTimeDomainData(data)
}, 20);
					

Script Processor

var BUFFER_SIZE = 2048;
var p = ctx.createScriptProcessor(
  BUFFER_SIZE
);
p.onaudioprocess = function (e) {
  // Called SAMPLE_RATE / BUFFER_SIZE
  // times a second
};
					

Analyser+Proc

var a = ctx.createAnalyser();
var p = ctx.createScriptProcessor(
  2048
);
a.fftSize = 1024;
var fft = new Uint8Array(512);
					

Analyser+Proc

p.onaudioprocess = render;
function render () {
  a.getByteFrequencyData(fft);
  // `fft` has 512 elements
  // containing 8 bit values
  // every 46ms
}
					

More Visualizers

Loop Waveform Visualizers by Felix Turner

Example of using both Time and Frequency Domain data

Beat Detection

• Beats are drastic changes in sound energy

Beat Detection

Average Magnitude

function getMag(data, min, max) {
  var total = 0;
  for (var i = min; i < max; i++) {
    total += data[i];
  }
  return total / (max - min + 1);
}
					

Is it a beat?

function process (d) {
  var mag = getMag(d, 0, d.length);
  if (mag > threshold) beat(mag);
  else {
    threshold -= decayRate;
  }
}
					

Beat Detection

Thanks

• Jordan Santell
• @jsantell
• slides | source