How VR and interactive media are exploring the musical experience


A still from the Los Angeles Philharmonic’s music video of a Beethoven Performance at Walt Disney Concert Hall, Los Angeles. Virtual reality was used to remotely broadcast the concert with visualizations.

Have you seen the new virtual reality (VR) music video for “Ogilala” by 90s rocker Billy Corgan (Smashing Pumpkins)? Or pop songstress Bjork’s exhibition of virtual and augmented reality videos promoting her new album? Or perhaps the LA Philharmonic’s VR music video filmed at a live Beethoven performance? These are just a few examples of how a once elusive idea in technology—interactive media—is quickly becoming a new frontier for creative production in music.

And interactive media isn’t serving solely as music accompanist; rather, it’s helping us better understand the musical experience as a whole and re-purpose academic research in music into something far more wondrous and accessible. Today, we’re highlighting a few instances at the intersection of music and interactive media that have caught our attention for their creative ingenuity and captivating approach. Perhaps most telling, they help us explore what it means to hear, interpret and love music from an entirely new vantage point. So grab your headphones, enter full screen mode on each video and immerse yourself in these sonorous experiences.

This Is Your Brain on Art, by The Washington Post

Through a beautifully illustrated series of videos, music and animation, this interactive piece from The Washington Post sheds light on fascinating research findings in neuroaesthetics, a burgeoning field that investigates the relationship between art and the brain. Take for example, the neural underpinnings of why we feel connected to something larger when we witness live performance—apart from a live audience providing a rich sensory and social environment, empathy plays a key role in understanding why some performances truly speak to us. In addition, our attraction to intense movement and good storytelling can all be traced to our brain’s predisposition to seek deeper meaning in art. This interactive brings these concepts to life by accompanying its narration with clips of ballet performers, who quite literally demonstrate the dazzling effects of live music and dance on our brain in real time. Click here to enter the full experience.

Sensations of Sound, by The New York Times

Deaf and hearing disabled individuals’ experience of music is profoundly different than the general population. As this New York Times VR piece explains, it isn’t just about the actual “sound” of the music for these people; rather, music is a holistic experience that is visual, tactile and far more detailed than one might think. Based on the story of Rachel Kolb—who spent her formative years deaf and later received a cochlear implant—this piece is a moving insight into how differently abled people enjoy music. From the distorted sounds mimicking a cochlear implant to the deep bass vibrations of electronic music, you can immerse yourself in Rachel’s world thanks to clever, nuanced sound editing and effects.

Song Exploder presents: Inside Music

Song Exploder, hosted by Hrishikesh Hirway, is a podcast that explores and breaks down how songs are created with the help of the musicians who made them. Often, the songwriting process is deconstructed, with individual parts of songs highlighted in isolation, allowing listeners a rare look at the building blocks of their favorite tunes. Inside Music, created using Google’s open-source VR platform WebVR, is essentially a visual manifestation of Song Exploder that lets you immerse yourself in song structure. Currently offering a handful of songs by artists like Phoenix, Ibeyi and Perfume Genius, the simulation places you inside a black space where each instrument of the song is highlighted in an orb. By looking around the space and turning orbs on and off, features of the song that may not have been as apparent before start to shine through. Click here to enter the full experience.

“Crackle Pop”, a 360 VR music video by Human Pyramids

Synesthesia is a rare neurological condition where individuals process sensory information through multiple senses simultaneously. Many individuals with this condition experience a specific form called chromesthesia, or “sound to sight”, allowing them to actually see the music they’re hearing, often as colorful, graphic visualizations. The Scottish orchestral pop group, Human Pyramids, gathered a group of such individuals and asked them to describe what they saw when hearing their song “Crackle Pop” (off their upcoming album, Home). Using 360 VR technology, these descriptions were transformed into vivid animations that bring the synesthetic experience to life. Brilliant visualizations intensify as more and more instruments are layered, each with their own distinct color and quality. Thanks to spatial audio, the VR experience in this music video is accompanied by a “surround sound” feeling; it’s as if you’re standing right there in the room, while the music plays. Human Pyramids have used VR to showcase yet another uniquely different perspective in music listening.

With VR and interactive media further expanding into areas like medical research and therapy, it’s clear that this medium has infinite potential to teach us a great deal about the human experience at large. By understanding music’s powerful psychological and social effects, the relationship between music and physical challenges, or simply how a piece of music is created, VR is allowing us to gain never-before-seen insights into human experiences. And it’s just getting started!

Leave a Reply

Your email address will not be published. Required fields are marked *