A combination of web-gl with web-audio technologies resulted in a very engaging music interaction experience

The following video is present for documentation purposes, and I encourage to visit the actual online toy before seeing the video. Play with it

Find the code in Github

Context and motivation

I started to understand that current music making is slowly merging with music listening, as the boundaries of both activities are being blurred by new interaction technologies and new music distribution technologies. I have also observed a trend where music making tools take stakes more and more towards having a unique, distinctive sound. This trend is odd in the sense that such tools are less versatile; but it makes a lot of sense when we thing of this tools as active music listening tools. This experience is very specific. It consists of a simple musical state machine, with pre-recorded tracks that are looped in sync; in a similar way to how Ableton works or also the Korg Electribes when configured in a certain way.

Process

This project emerged organically from my experimentation with two web technologies in parallel: Web-audio, and Web-gl (thanks to Three.js). On one side, I was experimenting with different performative possibilities that web browsers offer (for example, Ms Compose 95), and in the other side I was working with Three.js for some client-related work (e.g. Tzina: Symphony of longing).

I started experimenting with the concept of a musical state machine, which harness the logic of Ableton's logic of looping to present users with an intuitive way to direct a musical set.

A multi-track looping interface experiment using Web-audio api.

The instruments were modeled in Blender, in low poly. It looks like a mere aesthetic decision, but this way is also compatible with lower end computers.

The initial idea was to be surrounded by an abstract scenery with floating instruments. For me, that the track has some relation with the real presence of music, so I built instead an abstract scenery representing a performance scenario.

The different musical states among which this experience switches, gives an idea of a story line. This is why I decided to add distinct decorations to each different musical state, and optional "ending" to this story, where the user is suggested about a reflection where this experience took place: inside a head.

Reflection

This was an interesting example of a project that grew organically from some explorations in Javascript programming. In this experience I decided to take it until it feels finished. This was an interesting decision, which led me to end up with a piece that I can include in my portfolio. This work also taught me some aspects of making modular and manageable code, since there were so many instances of different types of modules that needed to intercommunicate among each other (e.g. the sound loopers with the three-dimensional models, and the user interface).