I finally updated my website’s backend after being on PHP v5.6 (current version is something like 7.4) for far too long. Good news is now you can access the site via https for better security. Now that that is done I hope to post some articles soon about some of the specific sound design and recording we did on Ghost of Tsushima.

For now, here’s a talk we did in December about Sonic Storytelling and exploring some of the ways we use audio to tell our stories. It was a lot of fun to share some of the world with the audio community:

While I’m proud of so much of the audio design in inFamous Second Son, one feature stands out as a testament to never letting go of a good idea. It was a concept, not new or necessarily innovative, that began incubating around 7 years ago. It wasn’t until 2013 that I was able to make the idea work in a title. I thought it’d be fun to trace that feature from its nascent stages through to its full fledged life. To do so, we have to go all the way back in time to a year we called 2007. Ah 2007! There was a palpable hum in the air. The iPhone was introduced by a little upstart company called Apple, Microsoft excitedly released their newest blockbuster (*cough*) Operating System, Vista, and the Nintendo Wii had captured people’s hearts, minds, wrists, and pocketbooks.

I was working at Shaba Games, where we had just finished up the DLC/Gold Edition of Marvel Ultimate Alliance and were looking for a new project. Like many others, we were captivated by the Wii and began working on a concept for a downhill skateboarding game for the platform. Shaba’s other sound designer, Lorien Ferris, and myself began brainstorming ways we could introduce interesting audio to what would ostensibly be a multiplayer racing game. Obviously the skateboard sounds would reign supreme and we came up with an idea of emitters tied to occluder objects such as buildings which would play a quick whoosh as you passed them (an idea I would later harvest for the mobile title, SummitX Snowboarding). Another idea we had was to have music emanating from buildings as you skated by. You’d be going fast and could never go back uphill, so they could be short loops, and once we applied some doppler it would sound awesome!

Unfortunately, while the Wii as a piece of hardware was popular for a slew of years, the software didn’t seem to sell as well, so the project was scrapped before we got very far. BUT after multiple other false starts we were finally given something wholly different and rather exciting: Spider-Man, and what would eventually become Web of Shadows. The goal was straightforward: create a new, unique open world Spider-Man game using the engine from the recently released Spider-man 3. Once again Lorien and I dove into brainstorming cool new features we could implement on the audio front to push the superhero qualities of Spider-man and the real life interactivity of the city. Early on, our storefront music concept was revived. I even added some various loops to embed into some stores simulating dance and jazz clubs and restaurants. Unfortunately we ran into some design problems early on: the storefronts we had in the game didn’t really match the music, they were destructible but we didn’t have a signal to turn off the music when the store was destroyed, and truthfully it just didn’t sound super-convincing to have the sound of filtered talking and clinking dishes and glasses of a restaurant while you’re right outside fighting. You think there’d be screams and hushed whispers. Basically with a tight schedule and a skeleton crew, our storefront music plans would have to wait for another day…

…which came just a year and a half later. We were working on a new superhero title and, with so much of the infrastructure in place now, we spent some time focusing on how to make storefronts believable. We created a multi-stage approach: idle, which would be the default and would play a basic ambient loop. For example some cheesy Italian music emanating from a restaurant. If a fight broke out in the vicinity we would enter a threatened state which would trigger an appropriate one-shot sound effect of screams and maybe instruments falling, dishes breaking, etc. and the music would cease. During high-tension moments (using the same tension meter as our interactive music system) the stores would be silent. Once tension went back down to low, we would slowly ramp up the idle state again until another fight broke out. Perfect plan! Unfortunately the studio ended up shifting gears and we moved from superhero games to music games. The storefront music would lay dormant again…

Fast forward to early 2012. I had just joined Sucker Punch and we were in pre-production on inFamous Second Son. Being back in an open world title, I pretty quickly started to think about my beloved storefront music concept again. Everyone I pitched it to from our creative director to our music team down at Sony HQ loved the concept. So now it was time to design it. The first step was just to get looping sounds emitting from a point in space and figuring out proper attenuation and processing for them. Next it was time to get into the real nitty gritty. I had several challenges to tackle:

A world inside a world

inFamous Second Son takes place in present-day/slightly-future Seattle. It’s not real Seattle, it’s our take on the city, but we still wanted it to be a unique, diverse, funky place, just like real Seattle. We did not want it to be full of grunge music (and that is a story for another day!). I began talking with the environment team to get a sense of the variety of storefronts we would have, and some of what they created helped influence my ideas. Early on, we got an Irish Pub in the game. At which point I thought, man that’d be cool to have it play Irish music during the day and then become a punk club at night. Just like in real life! Then I started to take it further: what if we had traditional Irish music in the earlier times of day, changing to more upbeat, raucous Irish music in the evening and THEN a punk club at night. I was on to something. As we fleshed out the stores my list of music grew and grew. I wanted jazz and Chinese and J-Pop, club, top 40, and why not mariachi music in the Mexican restaurant and thai music, new age music for the yoga studio, and hell even Russian music to put into apartments where the Akulan gangs live? Sure they’re musical stereotypes, but they’re serving the purpose of a low ambient bed, they were never meant to be featured sounds. The result will be filling the city with greater perceived life. I also wanted to reach out to some local bands and get them featured in the game as well. I wanted a lot. So how the hell were we gonna get all this music?

APM to the rescue!

As anyone who’s worked with Sony can attest to, they have some of the most amazingly talented, brilliant people working in their music department. We were very fortunate to have a few of them working closely with us throughout the project. Beyond the game score, we started discussing this source music idea and they carved out some of their budget for a blanket license from APM for stock music. Matt Levine worked directly with APM who would put together playlists for various genres of music we were interested in. He would then send me the lists, which I would review, make notes and approve or ask for more. In the end, we had over 100 tracks in the game spread out over 8 times of day. On the local band side, having been in a band and played with some acts up here I reached out to some friends’ bands and also KEXP, the local college station, and got a list of some potential candidates, several of which made it into the final game. We also started talking to Sir Mix-A-Lot and he really wanted to get some tracks in, too. Now that we had music, we just had to get it playing in game.

Rock Against the Man

As I mentioned, I had earlier rigged up a test playing source music in a test world pretty quickly to help figure out volume, attenuation, and processing. From there, it was on to the challenging part: figuring out how to make it gel in-game. In inFamous Second Son, you play as Delsin Rowe, a rebelistic youth with super powers battling against an evil authoritarian police force, the D.U.P. (Department of Unified Protection, think of the TSA with guns, armor and superpowers). Delsin can clear the DUP out of each district of Seattle as part of the systemic, non-mission open world gameplay. The main theme here is freedom vs. security. The DUP keeps people secure, but Delsin gives them the freedom to do as they wish. To help reinforce this thematically we decided that when the DUP controls a district we’d only hear DUP music. We started with stoic, patriotic sounding cues, but steered the direction more towards syrupy, happy music that provides a wonderfully stark juxtaposition to the menace of the DUP. Once Delsin begins to drive the DUP out of a district, we stop the DUP music from playing in that area and instead let the storefronts come to life with their own individuality. We had a programmer working on the district status rigging, so I asked him to give me a callback signal for when the district status changed. I was then able to use this to determine what district the player was in and whether DUP music should be playing there (it emits from DUP speakers and closed off DUP storefronts), or whether the other storefronts should be allowed to rock in the slowly-becoming-free world. I didn’t feel my initial idea from way back about multiple states work work in this instance. The music acted more as personality for the district than simulating people inside, so I didn’t pursue any kind of multi-state reactive environment. Maybe next time!

At the same time, I wanted some semblance of reactivity and also wanted to ensure the source music didn’t clash with the game score. So I tied the volume of the source music to our tension rtpc (Real-time Parameter Control in Wwise) which is also used for controlling the music intensity. When the player got caught up in combat, the music would fade out, when the combat abated, the source music would slowly ramp back up in volume. As if the owners of the shops were peeking through their windows, and once they saw the DUP dispatched, they cranked up the tunes again. So everything was working great, but now I had dozens and dozens of songs across ten or so genres, how was I going to make it all fit in a shippable state?

Making it fit

Beyond the goal and using source music to bring more life to our fictitious Seattle, I also wanted breadth and variation within the music so you wouldn’t hear the same cue EVERY time you passed a storefront. With a blanket license from APM plus around 20 local musician tracks the content was near limitless. Our soundbank budget, unfortunately, was not. However, every time we change the time of day in the game, we do a load to bring in our new skybox and other time-specific content. In fact, I was already loading all of my ambient sounds with these time of day loads. I devised a scheme to load certain music which could play at any time of day in our core ambient bank, which is always loaded. This ended up being the DUP music and our local acts. For the rest of the store fronts, I would load in 3-5 cues per TOD per genre. This way we have some variation during each time of day, as well as completely new tracks for most storefronts for each time of day change. For the local music, we had all 20 tracks in a random playlist emanating from Sonic Boom Records (a real Seattle record store), Sir Mix-A-Lot played from some of our neon-drainable low-rider hatchbacks (we HAD to have My Hooptie for that!), and the aforementioned Irish punk club featured 3 bands each rotating through a set of 5 songs each. You could theoretically stand by the Irish pub at night and enjoy a whole night of music (if it wasn’t so much fun to run around and use your powers instead!)

My budget for the TOD banks was 7mb, of which I used 2-3mb for source music at VERY low bitrates. We processed them heavily with severe low pass filters and reverb, so we really didn’t need a lot of high end, and the lower encoded bitrates (24kbps OGG) aided in making the tracks sound like they were coming out of crappy speakers inside the storefronts. Most of the cues were edited to around 60-90 seconds since most people wouldn’t really be standing around listening to the music, and we wanted more quantity of tracks than longer songs for this reason.

Here’s a video showing off just a few of the myriad storefronts we added music to. If you have a copy of Second Son, I highly suggest pushing the DUP out of some districts and running around to see how the source music aids in filling in the world without stepping on the score or any critical gameplay. It’s a subtle effect no one would likely notice, but subtlety is often the key to effective sound design.

Unity is an awesome engine for quickly iterating and building game content. The audio features have definitely improved over the years, but it’s still rather limited in many ways.  Randomization of pitch, volume, lpf, or even sounds can only be done with a little bit of scripting savvy.  Someday, either Unity will fix this or I’ll publish my scripts for these common practices on the asset store .   One of the features we added at Free Game Games which I’m most proud of was a psuedo-occlusion scheme utilizing trigger boxes and an enum (and enumerated static variable) to attenuate and apply a low pass filter on certain sounds. This feature was used prominently throughout Free Range Games’ canceled skateboarding game, as well as Freefall Tournament, which is playable on Kongregate.com.  It was one of the more advanced features we added to Unity on the audio side.  I relied heavily on the scripting wizardry of Jeff Wood, a fantastic designer whom I worked with both at Free Range Games and Shaba Games before that, to handle most of the technical scripting. Our solution was not necessarily the best method to occlude sounds, but it was functional, so I’d like to outline the system so that people can ideally glean some information from it and possibly improve upon it themselves.

The core script components of our system were an AudioOcclusionTrigger and an AudioOcclusionObject.  All objects the we wanted to occlude would have an AudioOcclusionObject script attached to it.  To trigger the occlusion we created trigger boxes and attached an AudioOcclusionTrigger to that trigger box.  Since we may want an object to be occluded in multiple boxes, we created an enum containing a list of occlusion “categories.”  This list was rather arbitrary and dependent on the level design.  I believe we had things like “Hallway,” “ExtAmbience”,  and “IntAmbience” So for example in our skateboarding game, we had a warehouse level in which two cavernous rooms were connected by a small hallway. Each room had an ambient emitter which played a looping ambient drone and  occasional one shots and a PA loudspeaker which was pumping out our licensed music soundtrack. The hallway had a trigger box around it with an AudioOcclusionTrigger script labeled as “hallway.” The ambience and music emitters in each large room where tagged with the hallway enum with their respective AudioOcclusionObject scripts, and whenever the player entered that trigger, all the sounds which contained an AudioOcclusionObject script with the category set to “hallway” would  attenuate and get filtered over time.  And when the player exited that box, the reverse wold happen (the objects volume and lpf would be restored to where they were prior to occlusion).

Here is a short video demonstrating the effect on the music and ambience:

We did a lot of safety checks to make sure the sounds aren’t already occluded before attempting attenuation, creating filters if they don’t exist, etc. but it’s still a far from perfect solution.  It really only works well with fairly simple geometry and the more occlusion categories you add, the crazier it can get to keep track of your objects and make sure everything is set up properly.

But there you go.  If you’d like to download the scripts and check it out yourself, they’re available here.