Here is another post about Refik Anadol's WDHC DREAMS Exhibition, which I had the opportunity to realize and install in Los Angeles Philharmonic.

The user is able to explore both the image and the audio archive of LA Phil.

Creating the interactive audio cloud

It is the first time that I worked with this interesting concept of exploring a vast cloud of 500.000 audio samples, each 10 seconds long.

The cloud was again sorted by machine learning algorithms (UMAP) and I received the data for visualization and to implement the interactive exploration of the cloud.

For the visualization I improved my particle engine. It uses defered PBR shading. A custom ambient occlusion technique, Depth of Field, Glare, Color Vibrancy and other small tweaks to make the cloud look realy intense. For performance improvements I used Level of Detail Meshes, View Frustum Culling and Depth based "Combing". Combing is a technique that I use for dense clouds. It basically discards random particles with a higher likelihood the further a particle is away. So the cloud gets combed out to the farther end of the view depth, this way reducing the total amount of shaded particles. When this combing is set up correctly you don't see that there are less particles and the clusters are visually preserved.

Interaction with an large 3d audio particle cloud is something that was not done before to my knowledge, so I had to come up with my own solution.

The way how i did it is that I use a compute shader to extract the particle IDs that are in a sphere in front of the camera. The application then plays back the audio snippets in this sphere in a random order. It plays a sample for 10 seconds with 2 second fade time in the beginning and end.

Flying through this audio cloud is pure joy. You discover real gems of music history and they tonally and thematically fit together nicely in the mix.

For example there is one section in the cloud that only plays back different parts of the Star Wars imperial march, recorded live from the Orchestra. Another part plays different A Capella samples. Fly further and you will encounter wild drum solos all in one place.