Triangle

Tue, 01 Nov 2022 00:00:00 +0000

“Triangle” is my second virtual reality experience. It is the first piece of a three part series named “Primitives”. Each piece in the series explores using a basic shape to create with.

The six degrees of freedom experience works in the Oculus Quest 2 browser and most desktop browsers.

The three degrees of freedom experience works on most mobile devices.

Overview

The triangle is everywhere in computer graphics, which is why I chose it as the first shape in the “Primitives” series. I used triangles a lot in it, as you can see. What you can’t see is I used triangle waves in the music, too. This draws an even deeper connection between the graphics and the audio than is visible on the surface.

The experience is pretty basic (primitive?) but that’s the idea for now as I continue building out my creative workflow. I spent some of my coding time for this project getting the DAW synced with the graphics engine to make editing easier. This was one of the biggest issues I ran into while making my last piece. I had no way of jumping to any specific time in the music while tweaking the animation and spatial audio settings. To see how a change affected something at the end of the piece, I had to play the whole thing from the start each time. Not fun. For this piece I used OSC to keep the graphics engine synced with the DAW, and setup all the animations so they update correctly when the playback time in the DAW changes. It’s a lot more fun, now.

I also worked on some ideas for sharing animation curves between the synths and the graphics. One of those ideas worked really well and I was able to use it to do some cool doppler effects.

The music for this experience has a lot more tracks in it than the last one I did, so I spent a lot of time making new synths and finding ways to optimize the real-time audio generation so it would still work on the Quest 2.

I had hoped to get this all done in a few months, but it took twelve. Hopefully, the next one will take less time.

Instructions

It takes a little time for the music get going so don’t turn up the volume too soon. Also note that the closer you get to the objects the music is coming from, the louder the music will get.

When the scene first loads you will see two buttons available for selecting which experience level you would like to use.

The three degrees of freedom (3dof) experience is for mobile devices with no keyboard. This level lets you look around but does not let you move around.
The six degrees of freedom (6dof) experience is for VR headsets and desktop computers. This level lets you move around and explore.

After the experience level is selected, if a VR device is available you’ll see a button in the bottom right to enter VR mode.

To look around on desktop and mobile devices, click and drag the mouse or drag your finger on the screen.

On desktop devices, press the WASD or arrow keys to move. Hold down the Shift or Space key to move faster, or use Caps Lock.

On VR devices, use the thumbstick to teleport.

Technical details

Javascript

I used two Javascript libraries: Csound for audio and Babylon.js for graphics.

Library	Version	Size
Csound	6.17.0-beta5	1.9 MB
Babylon.js	5.30.0	766 kB

My custom Javascript compresses to 153 kB. Source code is here.

Audio

The six degrees of freedom audio is generated in realtime using the WASM version of Csound. For the realtime graphics people out there, Csound is kind of like shaders for audio. In a browser it runs on the CPU in a separate AudioWorklet thread.

For the immersive audio I reimplemented Google’s Omnitone library in Csound. It works pretty well and allows me to use a lot more sound sources than I can with other methods.

My composing workflow went like this:

Sequence music in Reaper DAW with plugins generated by Cabbage from my Csound scripts.
Bounce DAW settings and plugins to a monolithic Csound file using the steps listed here. This bounce-down process also generates a JSON file with all the data needed to synchronize the animations.
Use the monolithic Csound file directly in the six degrees of freedom experience.
Bounce the monolithic Csound file to two stereo .mp3 files that are combined to create the three degrees of freedom first order ambisonic audio.

Graphics

The graphics are handled by Babylon.js. The JSON file generated by the bounced Csound script contains info needed for synchronizing the graphics to the audio.


Meshes	64
Draw calls	32
Lights	8
Vertices	17095
Materials	49
Textures	3

ProofOfConcept

Mon, 25 Oct 2021 00:00:00 +0000

“ProofOfConcept” is my first virtual reality scene. I made it to prove out the tech stack I chose for making real-time music-driven VR experiences for web browsers.

It works in the Oculus Quest 2 browser and most desktop browsers.

Instructions

It takes a little time for the music get going so don’t turn up the volume too soon. Also note that the closer you get to the objects the music is coming from, the louder the music will get.

When the scene first loads you might see a button in the top left to unmute the sound. Click it to start the music.

If a VR device is available you’ll see a button in the bottom right for entering VR mode.

To look around on desktop and mobile devices, click and drag the mouse or drag your finger on the screen.

On desktop devices, press the WASD or arrow keys to move. Hold down the Shift or Space key to move faster, or use Caps Lock.

On VR devices, use the thumbstick to teleport.

Technical details

Javascript

I used two Javascript libraries: Csound for audio and Babylon.js for graphics.

Library	Version	Size
Csound	6.17.0-beta5	1.9 MB
Babylon.js	4.2.0	749 kB

My custom Javascript compresses to 116 kB. Source code is here.

Audio

The audio is generated in realtime using the WASM version of Csound. For the realtime graphics people out there, Csound is kind of like shaders for audio. In a browser it runs on the CPU in a separate AudioWorklet thread.

For the positional audio I reimplemented Google’s Omnitone library in Csound. It works pretty well and allows me to use a lot more sound sources than I can with other methods.

My composing workflow went like this:

Sequence music in Reaper DAW with plugins generated by Cabbage from my Csound scripts.
Bounce DAW settings and plugins to a monolithic Csound file using the steps listed here.
Copy/paste the monolithic Csound file and its corresponding JSON file into the Babylon.js project.

Graphics

The graphics are handled by Babylon.js. The JSON file generated by the bounced Csound script contains info needed for synchronizing the graphics to the audio. I tried sending messages directly from Csound to the graphics engine in realtime but they got bottlenecked so I embedded them into the project’s Javascript as JSON instead.


Meshes	59
Draw calls	7
Lights	1
Vertices	26953
Materials	13
Textures	7

docEdub

Triangle

Overview

Instructions

Technical details

Javascript

Audio

Graphics

ProofOfConcept

Instructions

Technical details

Javascript

Audio

Graphics