Author: Rev. Dr. Brad

GDC 2015 has ended and those who weren’t there have gotten their information solely in 144 character fragments. I wanted to write up a quick post of my experiences at the conference, key takeaways, etc. to give those who weren’t able to there a (slightly) more comprehensive idea of what transpired.

To be clear, there was no drinking, having fun, or gallavanting because we’re all professionals and don’t have time for such shenanigans.

The big takeaway from the week as a whole is that everyone is interested in VR and 3d audio, but we’re still figuring out what to do with it.

I arrived Monday night, not to hit audio bootcamp on Tuesday, but because I’m lucky enough to work for Sony and have the opportunity to be a part of their Game Technology Conference before GDC in which I sit in a room with some of the most talented game audio developers in the world and talk about game audio. We heard talks from Evolution studios, some of the Morpheus team, and others from SCEE and SCEA. Talented folks. Here I am sitting in a room with guys from Naughty Dog, Sony Santa Monica, Bend Studio, Sony Cambridge, London, Evolution, Sony Japan, Insomniac. It’s a very humbling experience being surrounded by such incredible, inspiring talent, all the while having great discussions to further inspire and innovate.

I actually cut out of conference for an hour to catch the beginning of the Audio Bootcamp and Jay Weinland’s talk on Weapon Design in Destiny.  I always enjoy talks where people share some screenshots of their Wwise projects. I find it fascinating how we all use the tool in different ways to sometimes do similar things and other times create totally innovative concepts. The two big “that was cool” moments from Jay’s talk which have been done elsewhere, but were elegantly implemented: the notion of silence duckers: using a Wwise silence plugin with 0.1 second length played with an explosion to duck most other sounds by 12dB for the .1 seconds with a .2 second recovery time to carve out some space for the explosion without being detectable. Also their passby solution for rockets in which they created several sounds of the same length with the pass by the listener at the same spot in each file. Based on velocity of the object they trigger the sound based on when the midpoint should go by the listener and if it’s too late to start the sample from the beginning, they seek into it.

One final comment about Audio Bootcamp: since the beginning it’s been more of an “introduction to game audio” day. This year it seemed far more like an extra day of the audio track. So many interesting speakers and talks on music, technical sound design, VO, etc. Pretty cool that audio has so many compelling topics that it takes more than the 3 day conference to cover all pertinent info.

Wednesday started with a talk from Jim Fowler of Sony talking about using Orchestral colors in Interactive music. While it was a bit esoteric for non-music people, Jim did a fantastic job of presenting a great concept in regards to working with music stems: rather than arrange music by instrument or section, arrange it by function within the score. He then showed how he marks up charts for an orchestra so they can tell what they need to play when. Really neat concept, and some lovely dry, British humor to boot.

I then headed over to my one non-Audio talk by Alistair Hope from Creative Assembly on building fear in Alien Isolation. Unfortunately it was only a half hour talk, but somehow he managed to get through all of his content. The key takeaways here were how they used prototyping to figure out their concept and then stayed true to the concept through further testing. These guys really get the meaning of the term “grounded” in regards to design and how something is grounded when it makes sense in the world you are building, rather than the real world. Interestingly they toyed with making it a 3rd person game at one point due to the fact that most other survival horror games have been 3rd person and there was also the conflict at the time with FPS Aliens Colonial Marines. In the end they found that 3rd person felt like an Alien game, while 1st person felt like Alien, so they stuck to their guns. The last, most important thing, which should apply to all projects were their Key Universal Learnings. The seem so self-explanatory, but are definitely worth reminding ourselves (and our teams) of when working on a project:

• Have a Strong Vision
• Everything should work together to support that vision
• Deliver strongly on the vision
• Believe in what your doing

Next up for me was a talk from Harmonix on creating the interactive musical experience of Fantasia. My one wish for this talk was that they brought a Kinect along because it was cool to see some movies of their prototypes in Max/MSP and Live, but watching the movies of gameplay made we want to see how the 3d motion of the user caused various changes in the music. Still it seems like everyone that works at Harmonix is a musical tech wizard and they definitely have a lot of fun developing their gameplay.

Wednesday concluded with a talk from Monolith about Shadows of Mordor which was really great. Brian Pimantuan, the audio director as well as his programmer and staff composer did a really good job of showing how they set out to maximize emotional resonance in the open world environment of the game.  Some of the interesting things they did were moving the listener back to the player to make things more intimate and tie things closer to the player. Similar to Condemned, they added music stingers to impacts on Uruk Captains. Really nice, subtle touch of integrating music into sound design and increasing intensity. Also really dug they way they took a few queues for the Nemesis Orcs and made each one unique and reinforced each Orcs character by chanting the Orc Chieftains name over the music cue. Really slick. Also of note, though they barely touched on it, was how great the mix of this game is. So much going on and just a fantastic job of keeping everything balanced and sounding good.

Thursday was the (almost painfully) long day. The morning began with Oculus’ Brian Hook and Tom Smurdon talking about their experiences thus far with audio and virtual reality. They had some interesting perspectives on how we need to handle audio for VR including all mono sounds and a very judicious use of music. Gone are the days of simple tagging of anim roots with sounds to be replaced with a joint-based animation tagging system since the immediacy of virtual reality means we need greater spatialization of near-field sounds. They provided a great, early insight to playing with audio in VR games. It also made me very excited and encouraged about the work Sony is doing on the same front. The Oculus programmer, Brian Hook, made a VST plugin of their 3d audio SDK implementation which allows Tom, the audio lead, to easily audition 3d audio sounds before getting them in the game. A nice touch and one we should (hopefully) expect to see for other 3d audio solutions soon.

I had plans to troll the expo floor for a bit after the Oculus talk. I tried to see Nuendo’s implementation into Wwise 2015.1, but the line was too long, so I started to wander and ran into Mike Niederquell of Sony Santa Monica and Rob Krekel from Naughty Dog. We spent the next hour chatting about a gamut of topics including best practice uses for the PS4 speaker controller (perhaps a future blog post). Before I knew it, it was time for the next talk, which was Joanna Orland of SCEE talking about how to get a team on board and understanding your audio vision. Using the Book of Spells project she introduced the concept of creating a common language with the rest of the team so they could provide feedback to audio without being obtuse. In the Book of Spells example, each spell type was given an elemental name derived from natural sounds. If the rest of the team wanted changes to a specific sound they would use these elemental descriptions to help describe to Joanna the exact aesthetic they were looking for.

Rob Bridgett gave a very compelling talk on adaptive loudness and dynamics in mobile games next. His talk was arguably about much more than mobile games and easily spills into handhelds and also has implications for consoles. Rob is doing some supercool stuff out at Clockwork Fox. Not only does he do different mixes and loudness settings via compression based on whether the user has headphones connected or not, he also uses the device microphone to measure the noise floor of the room to help determine optimal loudness for the game mix. Brilliant adaptive techniques which, given the availability of a microphone, should be used in consoles as well.

Next up, Martin Foxton presented a talk on modular sound design using the Frostbite engine. His concept was essentially the notion of building sound events or in-game sound effects from smaller building blocks of sounds which can be reused as necessary and also creating templates for these sounds a la prefabs in Unity where you can create a script to carry over various settings for a sound. If you’re not already using modular sound design, it’s a great way to achieve variety while still maintaining sane bank sizes. It’s the reason every time you fire an R2 smoke bolt in inFAMOUS Second Son there’s 1024 possible derivations of the sound that can play!

The final talk of Thursday was a mind blowing presentation by Zak Belica of Epic and Seth Horowitz of Neuropop, a neuroscience research company. Seth was pretty damn hilarious and I only wish they had another hour or two to discuss their concepts. The takeaway here was that sound was one of the fastest of your six senses (yes, there are six. Seeing dead people is not one of them, but balance is). Anyway, because audio is such a fast sense, especially compared to vision, it means there are fewer possible illusions we can play on the auditory sense. However there are some neat tricks: For example the sound of bacon frying makes most non-vegetarians salivate, especially when you show an image of bacon with the sound. Speed the sound up and show a picture of bees, and people think they’re hearing bees and feel a little more uncomfortable. They showed us a few other really neat tricks including modulating a sound at 18.1 – 22kHz to make the eyeball vibrate and create a discomforting feeling. Using infrasonic distortion panned alternatively left and right to create unease. And even why fingernails scratching on a blackboard used to give everyone such shivers when blackboards existed (the envelope of the sound is identical to a child screaming in pain). Seriously, we all need to do more research into neuroscience and how it affects or manipulates audio perception. There’s a lot we can play with there.

By Friday, brains and livers were full, but there were still a couple more good talks to attend. Before these final sessions, I walked the expo floor and was finally able to check out Nuendo’s implementation into Wwise. It’s not fully realized yet (you can only import audio files, not folders which you can templatize into containers), but it’s a great start which I hope other DAWs will follow suit with. Needless to say, I’m starting to evaluate Nuendo now and hope they come to their senses, realize the opportunity they’re creating for themselves, and offer competitive crossgrades. There’s some great forthcoming features in Wwise 2015 besides Nuendo as well: calling events from other events, batch rename tool, profiler enhancements, optimizations, incremental bank building, advanced cache streaming and more.  Can’t wait to start playing with it!

The afternoon started with David Collins and Mike Niederquell having an informal discussion about the sound design of Hohokum. Super awesome that they did a live demo during their talk. Not only are there not enough live demos at GDC, but watching Mike play through some of the levels made me really want to play the game again. It was cool to watch such a fun, light informal talk and also bask in the joy that is Hohokum. Seriously, if you haven’t experienced it (I would say “play,” but it’s less of a game and more of a audiovisual experience) you should definitely seek it out and give it a go!

A perfect cap on GDC was Dwight Okahara and Herschell Bailey from Insomniac giving a glimpse into the open world sound design of Sunset Overdrive. Key takeaways here were that the audio team helped drive and sculpt the irreverent style of the game by implementing offbeat audio into early “gritty” concepts which brought the rest of the team around to the more fun style we now know and love as Sunset City. They showed off some of their fun tech like contextual storefront dialogue and the horde crowd/walla system and it was fun and refreshing to see such a talented team facing the same frustrations my own team does with streaming, lack of programming resources and other annoyances that plague our daily audiocentric lives.

So that was the talks that I made it to. Granted for every audio talk I went to, there was another at the same time. I missed tons of great talks from Matthew Marteinsson of Klei talking about Early Access to  the PopCap team blowing minds with their work with Wwise and 5mb of memory to create Peggle Blast on iOS to Jon Moldover, Brian Schmidt and others talking about turning music games into instruments and more of an interactive experience.

One final note which I’ve said so many times this past week and hope to never stop repeating: the game audio community is something truly special and wonderful. Hanging out with and meeting so many inspiring men and women and being able to openly share our passion is such a fortunate thing. Of all the people I met, hung out with, joked with, talked shop with, etc., there wasn’t an ounce of ego anywhere. Everyone in the community seems dedicated to each other and hellbent on push our entire industry forward together and I can’t express how lucky I feel to be just a small part of that experience.

Recently I was working on a project in which a country road had a small drainage ditch to the side of it with flowing water in it. I looked at it once, and instantly thought, “I need to add a sound for that!”

Two weeks later, I was taking a hike through Cougar Mountain Regional Park (didn’t see any cougars– feline or otherwise), when I came across a very similar scenario in real life: a small stream of water flowing downhill. I stopped, looked, and listened, but to my surprise I heard no water trickling or babbling sounds emanating from this little stream.

If I went back and removed the sound from my project, someone could walk through the world, see that ditch, and wonder, “why the hell isn’t there a water flowing sound coming from that water?” The simple point here is that our perception of sound often differs from the reality of sound, and in games (or any form of media for that matter) we need to carefully weigh this when crafting an aural landscape. If a user is expecting a sound and it’s not there, it makes a negative impression. Not necessarily because the overall sound design is bad, but rather s/he notices a sound is missing. We have broken the wall of immersion. In the real world, slow moving water needs speed, but it also needs an obstruction in its path to cause enough movement to generate an audible sound. In the game world; however, it may just need to exist with the illusion of movement: perhaps it’s just an animated texture, or a shader trick. There doesn’t need to be a rock or an eddy causing a rapid, it’s just there, it’s expected, so let it have sound. Unless of course that goes against the aesthetic you’re trying to develop in the course of your project.

Sound design is all about managing perceptual expectation. We all know how weak gunfire sounds in real life compared to that which we create for games and film. So there is both the need to manage perception in the design of individual sounds as well as on the implementation side of sound design. But how do we choose what aspects in the world should and should not have sound and how those sounds behave? There are two things to consider here: technical and aesthetic.

On the technical side there are decisions to make based on what is available to you. What device(s) are you developing for? How much memory do you have available? Do you have DSP? Is there any sort of scripting or complex behavioral structure at your disposal? How many concurrent sounds can we play? What else may be concurrently going on in the world? Fortunately, as technology evolves, tools and technical specs are both improving so that even mobile games can use Wwise, FMOD, Unreal, etc. to provide the designer with more options, power, creativity, and flexibility to achieve their sonic goals for a project. Handhelds and mobile are losing their “stripped down,” “less powerful” monikers so that the only limitations we may have on our sound design are those we choose to put there. Of course, we’re not to the Mecca of no technical restrictions yet. Even on Playstation 4, I don’t have limitless memory and resources and that’s probably a good thing. Limitations often drive creativity and allow you to see things in a different light. We still need to fit our design into the technology we’re using, it’s just a matter of understanding the limitations of that tech and working through them.

The aesthetic side is more of a gray area. Technical specs are often set in stone, and while you may be able to negotiate for extra resources, you’re still working in an established ballfield. When determining what should have sound and how it should sound, that’s where the creative and artistic part of sound design really kicks in. This is where you get to decide (either by yourself is sometimes with the assistance of a game/creative director or other audio personnel) where you want the audio to take the user and how it should make them feel. There’s no real science in determining what is right or wrong, it’s usually a mix of gut feeling, experience, and inspiration from others that can drive you to the right place creatively.

I do not mean to suggest that technical and aesthetic design decisions are mutually exclusive. On the contrary, in a well designed audio plan, they are intimately entwined, each one informing the other. We generally want to create a believable soundscape within the context of the game world. What that means specifically is part of the beauty and mystery that is our craft. And the key to meaningful sound design is often understanding the differences in perception and reality and ensuring your audio vision for a project matches the sonic landscape you wish to create.

Audiokinetic has released Wwise 2013.1 with many new features, among them PS4 support, ITU BS 1770 compliant loudness meters and HDR audio. We worked with Audiokinetic to develop the HDR feature set over the past year and now that it’s out, I’d like to share some of my best practices that I’ve come up with (so far) in using it:

1). Keep it mellow: The first thing to be aware of is that the Wwise implementation of HDR audio is a relative volume scheme. We initially played with using SPL, similar to DICE’s Frostbite Engine, but abandoned that because a). we learned that even DICE didn’t use real-world SPL values, which sort of negates the whole reasoning behind using real-world values to set volume and b). because not everyone would use HDR and introducing a second volume slider (Volume and SPL) in Wwise just confused and overcomplicated things. So anything you want to be affected by the HDR effect (which may generally include all game sounds except UI, mission critical VO and the like) will live in its own bus with a special HDR effect on it. But this bus should be kept at a reasonable level. Generally around -12 to -18 dB. This will give you headroom in the final mix, and give your loudest sounds the ability to play without clipping. Furthermore when you have lots of very loud sounds plays, a more conservative bus level will allow things to sound cleaner. For individual sound structures, you can start with 0dB as your baseline, bring down sounds that should be quieter in the mix and bump up the louder ones above 0dB so they’ll push the HDR window up when they play.

2). The voice monitor is your best friend – The new voice monitor (shortcut: Ctrl + Shift + H) is a fantastic asset for tuning individual sound levels within the HDR space. Being able to visualize the input and output of all sounds, as well as see what affects the HDR window and how is immeasurably important when it comes to tuning individual sounds within the HDR space or preventing pumping of quiet sounds when a loud sound plays. The voice monitor is a fantastic tool whether or not you’re using HDR, but the ability to see the window behavior really makes it very intuitive as to how the effect works.

3). It’s okay to cheat: Don’t be afraid of a little judicious use of make-up gain to make an important sound punch through without affecting the HDR mix. Make up gain is applied post-HDR effect, so it won’t affect the window movement, but will boost a sound’s level. More importantly, play with the sensitivity slider in the HDR tab to dial in the best curve for your sounds. The HDR window can follow the volume of a sound, but often you only want the initial transient to affect the window and the tail to decay naturally while letting quieter sounds come through. For even more granular control, you can edit the envelope of individual waveforms in the source editor window. As an additional control, you can also reduce the tail threshold for louder sounds. Most of my louder sounds are set to 3 or 6, which means after the first 3 or 6 dB of loudness, the sound is removed from the calculation of the HDR window.

4). Only generate envelopes on the important sounds in the game. This is a simple optimization tip. It takes CPU to constantly analyze the envelope of every sound. I only generate envelopes for the louder sounds in my game (making sure they’re not generated on ambience and incidental effects). It won’t affect the mix, but provides some performance savings.

5). The EBU/ITU-R BS.1770 standard is gold. Keep you game averaging around -23 LUFS/LKFS (based on a minimum of 30 minutes of gameplay). Everytime you play your game, connect Wwise and keep an eye on the Integrated meter in the Loudness Meter. What matters here is the AVERAGE loudness, the longer you capture, the more accurate your measurement. As a rule of thumb, I always keep the loudness meter up and running in my project.

6). Inverse square attenuation make sounds behave naturally – one of the initial “issues” I had once we got HDR working in our game was that using our old attenuation curves (generally an exponential curve over a set distance based on the general loudness of the sound ranging from 15m – 250m) just didn’t work as we needed them to. We wanted attenuation curves to sound natural in a real-world environment, so I created a set of inverse-square curves. The inverse square law states that the volume of a sound is halved by a multiple of every x meters. For example the most common curve we use falls off fifty percent every 4 meters over the span of 80 meters. So at 0m it’s 0 dB, at 4m it’s -6dB, at 8m it’s -12dB, at 16m it’s -18dB, at 32m it’s -24dB, etc. This has the added benefit of limiting the number of attenuation curves needed which is a performance savings. Of course, inverse square curves are not a blanket solution, there will always be times when you want/need something custom, so we still maintain some custom curves.

I’m happy to share the settings I have on my HDR effect, but I feel this will vary based on project, so I’m not sure how useful that would really be for people. Another feature we’ve added is a speaker_type switch controlled by an rtpc which affects the HDR threshold based on the speaker type the user is playing through. The end result is automatic dynamics switching based on speaker type where the better your speaker system, the greater the dynamic range in the mix (similar to what games like Uncharted offer in their audio options menu). In short, there’s a ton of ways to use this great feature, and I’m sure there’s going to be plenty of other tips and tricks people figure out as they start to play around. Enjoy!

During GDC this year and the week after I ended up telling the story of how I got into the industry a few times, so I decided to commit it to the ether for posterity or some false sense of self-worth. I’ve also decided to embarrass myself publicly by digitizing the demo I made way back then in 1998 that got me into game audio.  It is horrible and borders on unlistenable. Well technically you can listen to it, but you wouldn’t want to, and it’s hard to fathom how someone could have heard this monstrosity and then offered me a job.

My story, while it may not have been exactly common 15+ years ago, doesn’t really happen anymore. The short story is that I lied my way into game audio. The longer story is that I was temping at Berkeley Systems, a video game company in Berkeley, CA after graduating college and they liked me enough to keep me on as their shipping guy. I liked it there, but really wanted to be doing something creative, so I started making a lot of noise as such. I was passed up for a production assistant job (thankfully) and ended up talking to their sound designer a couple times because I thought he had such a cool, crazy job. At this point in my life I’d never used a computer program related to sound ever. I knew how to play notes in BASIC and had a cassette 4-track and had done tons of music, tape loops, and other weird experimental stuff ever since I was a kid, but I didn’t know what MIDI was, how to create a sound effect or really much of anything in regards to sound and computers.

Anyway, one day the VP of Product Development called me into his office to tell me they fired their sound designer (apparently he didn’t come into work very often and they’d had to contract out all their sound work as a result). So he wondered what experience I had and if I’d be interested in the job. I couldn’t believe this was happening, so seeing an amazing opportunity, I lied through my teeth, telling him I had tons of experience and had scored some student films, blah blah blah. He asked me to bring in a demo the next day. I ran home that night and banged a couple things out on my sampler (half of which were a couple synthy pad soundscapes I claimed were from a student film I worked on. They weren’t.) and threw another horrible track called “Gall Stone Attack” onto a cassette and gave it to him. The next week he called me into his office and said “It’s nothing you’re ever gonna hear in any of our games, but it shows you know what you’re doing, so you got the job.” I was ecstatic. And because they’d already farmed out their sound work for the next 6 months or more, I locked myself in my office and started teaching myself everything I could about digital audio and sound design. I believe my first experiment in editing digital audio was removing all the guitar solos from Slayer’s Seasons in the Abyss, but that’s a story for another day. Nowadays, kids are coming out of school with degrees in sound design and blowing me away with their skillsets, so this whole thing known as my career could never happen today.

Everything on my demo was recorded with a Roland S-50 12-bit (!) sampler. It had a floppy drive and I had tons of sample disks for everything from pads to horns and strings to sfx. “Gall Stone Attack” also had a Roland R-8mkII drum machine and Casio SK-5 on it (and I think I used the SK-5 on “Silly Torture” as well). Since I had no sequencer or even an audio editor or audio interface for my computer, each track was recorded live onto my Fostex 4-track and mixed down to the cassette below. (I opted to not de-noise these as part of the digitization process, so they could “preserved” in the state in which they were originally heard).

And so without further ado, I present a public shaming: two tracks from my demo reel in early 1998. I cringe when I listen, and laugh a little. My skills have definitely come a long way, but I still can’t believe they listened to this crap and took a gamble on me anyway. I’m eternally grateful and shocked. Be forewarned.  Be gentle.

[soundcloud params="show_comments=false" ]http://soundcloud.com/revdrbradleydmeyer/02_SillyTorture-mp3[/soundcloud]

[soundcloud params="show_comments=false" ]http://soundcloud.com/revdrbradleydmeyer/03_GallStoneAttack-mp3[/soundcloud]

One of Wwise’s few shortcomings is its current lack of support for LFOs. Modulation can be a godsend to make looping static sounds feel way more dynamic and alive. (an example using volume, pitch, and lpf is here). I wanted to outline two different means here you can “cheat” modulation in Wwise using some technical trickery.

1). Modulation RTPC:
This is the simpler method, although somewhat limited in it’s dynamism. Simply create an RTPC linked to a global timer, and once the timer reaches it’s max, it resets itself to 0. I’ve called mine “modulation” with a min of 0 and a max of 100 (units being set in the engine as seconds). I can then draw an rtpc curve for modulation on any sound I want and affect the pitch, volume, lpf, etc. over time (friendly reminder: subtle pitch changes are WAY more appealing than extreme pitch changes).The most important factor here is to remember to have your values at 0 and 100 be identical, so there’s no pop in the loop. The obvious drawback to this solution is that the modulation is uniform with no possibility of change per cycle. However with a 100 second loop, you have a fair bit of time to build a dynamic modulation curve whose looping won’t be easily detected by a user.

2). Using a tremolo effect as an LFO:
This solution comes from Steven Grimley-Taylor who posted about it on the Wwise forums, and is nothing short of a brilliant use of the tools available in Wwise to make an LFO a reality. It also has some limitations, which we’ll discuss in a bit. The basic gist of this concept is to create a white or pink noise sound generator and sidechain it to a tremolo effect. As Steven explains it:

“Create a Sound SFX object with a Tone Generator Source set to White Noise. Then add a tremolo plugin and then a metering plugin which generates an RTPC

The tremolo becomes your LFO control and you can map it anywhere you want. It becomes unstable at faster speeds, but then this is probably not the best solution for Audiorate FM. For normal ‘modulation’ speed LFOs it works a treat.

You can also go into modular synth territory by creating another of these LFO’s and then modulating the frequency of the first LFO with the amplitude of the second.

Oh the LFO audio should be routed to a muted bus, you don’t actually want to hear them, just generate a control RTPC”

I’m currently using a couple of these in my project and it works great, the only drawback is that using the tone generator plus the tremolo per LFO isn’t super cheap (~2 – 3% of CPU), and the more modulators you want to add, the more expensive it gets. But you can drive the parameters of the LFO from other rtpcs, opening up enormous avenues of creativity and evolving sounds.  It’s a really nice way to spice up some bland looping sounds and give them a bit more life.

Sorry, I couldn’t resist being a little snarky as I typed that title out. Every time there’s a new generation of consoles on the horizon, words begin to flow about what “next-gen” means in relation to (insert your discipline here).  For me, there are two interrelated aspects that we can look at to push envelopes further: technological and structural. Technological advances are those made possible by the capabilities of the hardware and how that interacts with software. CD-ROMS meant we could start streaming redbook audio, tons of voiceover, and video. The Xbox’s DSP chip gave us a low pass filter and reverbs built into the system. The PS3’s SPU core architecture gave us an entire core (or more I suppose if you sufficiently bribed your programmers) to do with whatever we wanted: create custom DSP or utilize FMOD or Wwise and go crazy with the delays and reverbs and eqs, etc. The PS4’s 8gb of memory means, given the time to load data into RAM, we have a near limitless reservoir for game resources. Ok, so “near limitless” is probably an over-exaggeration, but we’re talking a 16x increase over the last generation!

By structural, I mean how does the technology create new ways for us to deliver a sonic experience. The sub-industry of mobile and web development have democratized game development significantly, and with them and the rise of Unity as a viable engine has audio middleware solutions like FMOD and Wwise along for the ride as well. Even Wwise, which started out as a PC, 360, PS3 only platform 6 years ago, now has support for iOS, Android, Windows Phone, and direct integration into Unity. With the democratization of tools comes the possibility to use these tools in novel ways. One such example is adaptive mixing. While in console land we’ve been doing this for years (for a great example of this see Rod Bridgett’s discussions of mixing Scarface: The World is Yours for PS2 back in 2006), this is only now being possible across all platforms. And with the potential for the Ouya, Green Throttle, Steam Box, Apple TV and other small Android, iOS and PC-based home consoles in the coming years we should see “next-gen” meaning what can we do to push content to be more impactful no matter the platform.

While I think the structural aspect has far more implications for sound design as a whole, much of what becomes possible is through technology. I want to touch on one specific technology in this post: procedural synthesis and design. Procedural synthesis is nothing new. Guys like Nicolas Fournel and Dylan Menzies have been doing it for years. Audiokinetic have had wind and impact generation tools in Wwise for several years now. Audiogaming’s wind and rain tools are integrated into the new version of FMOD Studio and will be making their way into Wwise soon (not to mention their latest plug-ins for procedural generation of footsteps, vehicle engine models, and fire).  And there have been countless papers and demonstration videos showing off better and better sounding procedural algorithms from the aforementioned to elements like fabric and full physics simulations.

When developing a game, we often take a cross-platform approach because it’s the easiest way to maximize profits for minimal cost: put your product out on every possible platform and you have a multiplier effect on how many people may play/buy your game, ideally with minimal additional effort on your part. Hopefully in the next year, if all these new hardware devices do come out, we’ll be at a point where we have enough processor to utilize procedural synthesis in games across all platforms, and not just minimal use on 360, PS3, and PC. Having these effects not just available, but possible, across all platforms maybe the shot in the arm procedural synthesis needs to finally bridge the gap from “talked about incessantly” to “the here and now.”

These two elements: realtime, dynamic mixing and procedural synthesis, while nothing new, may be the holy grail of audio development for games in the near-future. I am eagerly looking forward to how things shape up over the next few years, to see what others are doing, and to further explore these waters myself.