An interview with Robert Thomas, adaptive sound design veteran

Robert Thomas has had quite the audio career creating experiences that completely change the way we think about sound. He's worked with the likes of Hans Zimmer and Imogen Heap and developed everything from traditional game audio to iPhone applications with downloads in the millions. We sat down with the man who takes sound design to new levels to learn more about his work in adaptive sound design, some of the challenges of working with new mediums, and his views on the future of sound for virtual reality.

_______

Would you mind giving our readers a bit of background on yourself and your work?

Sure. I’m basically a composer and sound designer. But in addition to making music and sound, I make intelligent behaviours for music and sound. This means that instead of experiencing recordings of my work, people experience a version of it which adapts to their unique situation or interaction. Sometimes this might be how they are moving, what their brain waves are like or perhaps what the weather is like that day, or if they are in a particular place.

What got you interested in audio as a profession?

I’ve been obsessed with sound and music for as long as I can remember. I played violin from a very early age and was quickly trying to write my own music. The usual stories really - obsessively listening to my father and grandfathers records, mainly classical, romantic, jazz, 60s / 70s rock. I played in a lot of orchestras and was always more interested in rehearsals than performance - hearing inside the arrangement. Later I learnt guitars and keys and played in many bands. I gravitated towards writing, arrangement and production. This lead me into writing music for commercials, radio and video games, which then lead on to making much more unusual adaptive music systems for apps.

You mention the word "adaptive" and "algorithmic" quite often when referring to your type of work with sound design. Do you mind describing what this means, and how it might differ from, say, other types of sound design?

Most distributed music or sound design ( for film, games or indeed any music ) is made in a way which has been common over the last 100 years or so : recordings. Someone in a studio does a load of work and outputs a fixed linear arrangement / mix. This used to be outputted as a piece of vinyl, or a tape or CD. More recently its been an audio file of some kind like a mp3. Almost all of my work is fundamentally different to this.

The systems I make incorporate many different approaches. But perhaps one overarching way of describing them is that they have some level of intelligence. They all involve various types of algorithms - rules about how to reconstruct the music in realtime.

Some are adaptive or reactive, in that they change based on what the listener is doing or their situation. This could mean the music changes when you move, or if you are excited and have a high heart rate, or if you are wearing an EEG headset and you are concentrating, or if its raining that day. I’ve made music like this for all kinds of applications, ranging from adaptive running music, to adaptive driving music, to music for meditation.

I also use generative systems which have a set of rules which generate complex outcomes over time without any intervention from outside influences like the listener or the environment.

Some projects are more explicit, for instance might involve the user tilting devices to control music. I would refer to these as interactive music systems.

As part of the above to approaches I sometimes use AI techniques, including neural networks and markov chains so that the system can learn about user behaviour and alter its behaviour at runtime.

A final element of what I do is often referred to as procedural. This normally means that the audio is generated at runtime from pure synthesis instead of samples, and normally means some kind of behavioural model. For example - instead of playing a fixed sound file when a user spins around in a game - I would generate that sound live from synthesis, translating the users movement into sound. Or in a musical project I might make a synth play a melody which imitates a passing traffic siren.

What tools do you often use for these projects, and what does your creative process look like?

I do use conventional audio production DAW’s. Specifically Ableton and Cubase. However the large proportion of my work is done in the visual programming environment Pure data. 

What kind of challenges do you face when it comes to creating unique sound experiences? 

Although higher level than conventional programming, my work is much more technical than most sound design or music making. This means that I am often operating at a much lower, more detailed level than the average sound designer or composer works. This can be very powerful but also intimidating and bewildering. It’s taken me many years to develop a workflow and set of tools which allows me to create fast enough. 

Because I deploy code, I need to have the rights to release any live effects or synths I am using. A lot of the time this means building those effects from scratch. My work often involves complex musical / sonic sequences or events. In a DAW you can draw all this out on a timeline by hand visually. But I tend to work with algorithms that describe these behaviours - which can get very intricate. 

But perhaps the biggest challenge , but also the most exciting thing, is learning to let go of conventional ways of structuring music and sound experiences. Allowing myself to think about new types of experiences, using new technology, perhaps delivering music / sound in a way which is conceptually totally different to anything I have done before. Then making a new system to allow that to happen.

You worked on traditional sound projects for advertising, music and games before joining the innovative RjDj audio team.

Yes I did a lot of conventional music composition. I think at this time I learnt how to write music to a brief in a strict timeframe. Also I learnt how to develop a brief with a client who doesn’t really know what they want yet.

Just before RjDj between 2005 and 2009 I was doing some adaptive music work in virtual reality. The RjDj guys saw me blogging about some of that work and hired me via Twitter, which was still pretty unusual in 2009! I had actually already begun using RjDj prior to its release as a very enthusiastic beta user. I loved it!

What was it like going from the more traditional sound work, to developing sound-based experiences for a new platform (the iPhone)?

So my work at RjDj was mostly focused within the Pure data code itself. There is a great team at RjDj which handles the app ( iOS level ) development and also Michael Breidenbrucker the founder of RjDj has very significant input and ultimate control over the apps. 

I joined RjDj after he had released the initial app and many people had made lots of great experiences for it. So things had already come a long way. I guess my involvement was mainly focused around producing a lot of the artist based apps we made, and also developing the UX of the apps into more coherent dramatic experiences.

I remember the first work I did on iPhones being like a huge expansion of possibility. The app store in 09/10 really felt like a wave like that. It actually feels similar now in 2015 with wearables, VR / AR coming along.

Hans Zimmer discusses The Dark Knight Rises Z+ app which he created with RjDj.

What role will sound play in the next generation of immersive experiences (virtual reality, augmented reality, etc)? 

Since leaving RjDj I have done a number of projects with the current wave of VR technology. I think its a fascinating medium which requires very new ways of thinking about designing the UX. 

For audio, it obviously means that considered use of 3d binaural mixing is a critical part of maintaining presence. The fidelity of the end users sound experience is also super important. I have to say I was very disappointed when I saw the Oculus Rift built in headphones..

Following up on that, what should one consider when designing sound for virtual reality and other immersive experiences? How can the envelope be pushed?

Its clear to me that this poses some challenges. Maintaining CPU efficiency, not making the sound field too complex or dense and maintaining punch and fidelity in the mix are some of them. In all the VR projects I have done so far I have used a combination of mono, stereo and binaurally positioned sound. The mix fidelity issue is probably even more critical for music in 3d. 

As always, each new medium offers new possibilities. The funny thing is how people always start out making a clone of an old medium within the new one. I was listening to Brian Eno talking about the classic example of the transition from theatre to film. Initially people just stuck a camera in front of the stage. It took time for adopt ideas like editing and cuts.

In music and sound today - we have the opportunity to move from recordings to software.

Smartphones / the App store, was a new medium. VR already is and AR will be too. With wearables, the increased reach of biometric analysis of human behaviour is already becoming one. People are already making conventional games, apps and films squeezed into VR. I think the interesting stuff which pushes the envelope often involves doing things you simply cannot do in previous media.

Any projects that we should be on the look out for?

Yes, I wish I could tell you about them, but then I’d have to kill you!

Your twitter handle is @dizzybanjo. Just have to ask, do you play? :)

I do own one. Its probably the instrument I play worst, no maybe thats the Ukulele. 

________

You can find out more about Robert via his website: https://dizzybanjo.wordpress.com/ or follow him on twitter @dizzybanjoHave comments? Feel free to share them below. 

Share with your friends!