We walked away from this past SXSW feeling very excited about the state of VR and immersive technologies. Almost every VR-related talk was a packed house, and many of the VR showcases were constantly busy with people. One of our favorite talks was the “Master Class with Felix & Paul.”
Throughout the talk, the Felix & Paul team took us on “a journey, diving deep into specific lessons learned through different projects in their library and the innovations that were developed for each experience, as well as the challenges they believe are next for the community to solve.”
This cinematic virtual reality studio creates some of the most innovative and well-produced live action VR films to date. You might remember them for experiences they’ve made for the movie “Wild,” “Jurassic World,” and even a tour of the White House with the Obamas.
The talk was moderated by the director of sound design Jean-Pascal Beaudoin and included Felix Lajeunesse, Paul Raphel, and Ryan Horrigan. In the hour-long discussion, the team dived into their creative strategies for the entire production workflow, diving into everything from the writing and directing, cinematography, and of course – audio.
To give a bit of background, the Felix & Paul team outlined their vision of what they call “presence-based storytelling,” meaning truly thinking through how the viewer would perceive the story if they were actually there. This mantra has guided all of their work, which they showcased and referenced throughout their talk. Here’s our quick recap of the talk:
Writing for 360 Video
Like many creative efforts, the story begins with the writing. While 360 video is no different in that respect, many other aspects have to be reconsidered with the VR story in mind.
Writing for VR is different from traditional film
Everything starts with a great story, but in the sense of VR you truly need to search for a VR-native way to tell it. Why is the user in the experience? Why are they there? These are some questions that need to be asked when crafting the storyboard.
Avoid camera direction
When writing your story try to avoid calling out specific camera directions, as you truly never know where the user will be looking. Instead, think of how someone would naturally take in a scene if they were truly there as well as other ways to guide them through the story.
Hint: one way is guiding the viewer is with something you hear...
Make the viewer a co-protagonist
In their VR experience, Miyubi, your perspective is from the eyes of a toy robot set within a suburban home in the 80’s. Throughout the story, you are essentially the co-protagonist, acting through scenes and interacting with other actors as a semi-character in the movie. Although you can’t react, simply by shifting your perspective into the eyes of the character makes for a more compelling story and sense of presence.
Directing in 360 Video
While directing in VR, you are constantly engaged in a balancing act between telling a story and crafting the necessary presence of VR.
Carving out a place for the viewer
In their film debut with “Strangers with Patrick Watson,” you are placed in a casual apartment setting with an artist Patrick Watson sitting at the piano. Your perspective is that of a friend, almost feeling like you’ve been asked to sit down and hang out while the artist is in this creative musical session. The team asked themselves, where would a person sit if an artist would truly be comfortable in this position?
Know the camera is not a camera
With “Strangers with Patrick Watson,” it was important to think of the camera perspective as not a camera, but a person. This gives an extra level of intimacy and realism to the experience, where as a viewer you feel more comfortable in the scene.
Cinematography plays a crucial role in telling a story for VR. After all, VR is a visual narrative as much as it is an immersive one, so paying attention to certain visual elements can really make a difference with how the story is presented.
Design in a visual space
In each scene, it is important to think about creating intimacy with the viewer. Similar to the directing strategy, you want to position the viewer in a way that helps them truly feel like a native in the scene. If they were a person, where would they actually be standing or sitting in that environment? What would be around them?
Think about depth in 3D
Like a painting, a compelling scene has things in the near field, as well as the far field. This is the same for what you see everyday which means that everything in the VR scene should be curated. What you put in the foreground will help ground the viewer in the scene, while the mid and far-field objects need to also be curated to create that sense of immersion that feels natural, not forced.
While in “Nomads”, the Felix & Paul team curated everything around the viewer — from the immediate foreground of being in a boat, to having the certain boats pass by in the middle-ground, to the mountains in the background.
Compositing + Technical Production
As with many film projects, compositing often plays a key role in the final visuals. In VR, however, this is often taken to an entirely new level. Because the camera is capturing 360 degrees of footage at all times, the supplemental lights, props, and often the director himself are often “in scene”. This requires substantial compositing work
Be in the scene
The key thing to think about is that the director should often find himself in certain scenes in order to truly understand what the viewer is seeing. With Miyubi, this required the director to literally be next to the camera (from Mubi’s perspective) in order to be able to truly know what he was shooting.
Audio - A Key Element
Audio, we can agree, is one of the most crucial elements needed for VR storytelling. We’re all familiar with the visual bias, but in VR any break in the immersion can lead to a less-than-ideal experience. For many of their projects, the Felix & Paul team worked hand-in-hand with the talented audio team at Headspace Studio to develop engaging sound experiences.
Work on the audio track from the start
As the Creative Director at Headspace Studio, Jean-Pascal Beaudoin has worked alongside the Felix & Paul team on many of their productions, from Strangers to Miyubi. He stresses the importance of having the audio planning from the very beginning of the creative process. Bringing the 3D audio discussion into the mix so early has allowed for a deeper planning of it’s impact on both immersion and experience – and arguably leading to a much better film overall.
Sound can be subtle, yet powerful
The impact of sound can be demonstrated in subtle, but powerful ways. For example, in the Strangers experience the Headspace team decided to leave the window to Patrick’s apartment slightly open during filming. The microphone was able to capture the actual street noise from filtered in from the busy city below, and while it was not the main audio cue, it gave the experience a subtle sense of unscripted realism that you might actually have if you were sitting in the character's apartment.
The soundtrack and sounds need to work together
Beaudoin went on to mention that it isn’t just 3D sound that plays role in VR storytelling, music is just as important. He calls music a “presence-accelerator” that not only drives the emotion of the story, but also bringing the viewer in.
Audio is ripe for innovation
What is amazing about spatial audio is that the rules and best practices for creation are still being written, especially in the VR space. Down-mixing the spatial audio mix to binaural stereo is currently a standard practice in VR Films. This is due to the dominant nature of the stereo headphones for playback, but new methods are on the horizon.
“It’s exciting to be engaging with such a nascent industry,” said Sally Kellaway, Creative Director at OSSIC Studios. “Content creators like Headspace and Felix and Paul are able to experiment and create the language of craft of 3D Audio practice. We’re excited to be working on the OSSIC X at this time because it means we can solve content creation challenges from the outset.”
At OSSIC we believe the future of audio for VR can go beyond stereo to multi-channel, ambisonics, and even object-based 3D playback with headphones. This ultimately requires a slightly different perspective with regards to the audio production pipeline, but it is one that will result in a more immersive audio experience for the end viewer.
With their unique perspective on filmmaking in VR, Felix & Paul have played a huge role in the awareness of the medium as a whole. By leveraging the concept of “presence-based” storytelling to it’s fullest potential, they’ve not only created great films but they have truly taken us to new places. We’re excited to see what they develop next, because at the end of the day a great VR experience starts with a great story – and telling great stories is something they do best.
Header Image: Amanda Marsalis (People)