PRINCIPLES OF GAME AUDIO

AND SOUND DESIGN

Principles of Game Audio and Sound Design is a comprehensive introduction to the art of sound for games and interactive media using Unity. This accessible guide encompasses both the conceptual challenges of the artform as well as the technical and creative aspects, such as sound design, spatial audio, scripting, implementation and mixing.

Beginning with basic techniques, including linear and interactive sound design, before moving on to advanced techniques, such as procedural audio, Principles of Game Audio and Sound Design is supplemented by a host of digital resources, including a library of ready-to-use, adaptable scripts. This thorough introduction provides the reader with the skills and tools to combat the potential challenges of game audio independently.

Principles of Game Audio and Sound Design is the perfect primer for beginner- to intermediate-level readers with a basic understanding of audio production and Unity who want to learn how to gain a foothold in the exciting world of game and interactive audio.

Jean-Luc Sinclair has been a pioneer in the field of game audio since the mid-1990s. He has worked with visionaries such as Trent Reznor and id Software and has been an active producer and sound designer in New York since the early 2000s. He is currently a professor at Berklee College of Music in Boston and at New York University, where he has designed several classes on the topic of game audio, sound design and software synthesis.

PRINCIPLES OF GAME AUDIO

AND SOUND DESIGN

Sound Design and Audio

Implementation for Interactive

and Immersive Media

Jean-Luc Sinclair

First published 2020

by Routledge

52 Vanderbilt Avenue, New York, NY 10017

and by Routledge

2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN

Routledge is an imprint of the Taylor & Francis Group, an informa business

© 2020 Taylor & Francis

The right of Jean-Luc Sinclair to be identified as author of this work has been asserted by him in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.

All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers.

Trademark notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Library of Congress Cataloging-in-Publication Data Names: Sinclair, Jean-Luc, author.

Title: Principles of game audio and sound design : sound design and audio implementation for interactive and immersive media / Jean-Luc Sinclair.

Description: New York, NY : Routledge, 2020. | Includes index.

Identifiers: LCCN 2019056514 (print) | LCCN 2019056515 (ebook) |

ISBN 9781138738966 (hardback) | ISBN 9781138738973 (paperback) |

ISBN 9781315184432 (ebook)

Subjects: LCSH: Computer games—Programming. | Sound—Recording and reproducing—Digital techniques. | Computer sound processing. | Video games—Sound effects.

Classification: LCC QA76.76.C672 S556 2020 (print) | LCC QA76.76.C672

(ebook) | DDC 794.8/1525—dc23

LC record available at https://lccn.loc.gov/2019056514

LC ebook record available at https://lccn.loc.gov/2019056515

ISBN: 978-1-138-73896-6 (hbk)

ISBN: 978-1-138-73897-3 (pbk)

ISBN: 978-1-315-18443-2 (ebk)

Typeset in Classical Garamond

by Apex CoVantage, LLC

Visit the companion website: www.routledge.com/cw/sinclair

BRIEF CONTENTS

1 Introduction

1

2 The Role of Audio in Interactive and Immersive Environments

7

3 The Game Engine Paradigm

24

4 The Audio Engine and Spatial Audio

43

5 Sound Design – The Art of Effectively Communicating

With Sound

70

6 Practical Sound Design

115

7 Coding for Game Audio

147

8 Implementing Audio: Common Scenarios

173

9 Environmental Modeling

214

10 Procedural Audio: Beyond Samples

238

11 Adaptive Mixing

251

12 Audio Data Reduction

276

Index

287

DETAILED CONTENTS

1 Introduction

1

1 The Genesis of Audio in Games 1

2 From Sample Playback to Procedural Audio 3

3 How to Use This Book 5

What This Book Is 5

What This Book Isn’t 6

2 The Role of Audio in Interactive and Immersive Environments

7

1 Inform, Entertain, Immerse 7

1 Inform: How, What 8

a Geometry/Environment: Spatial Awareness 9

b Distance 10

c Location 10

d User Feedback and Game Mechanics 11

2 Entertain 12

a Sound Design 12

b Music and the Mix 13

3 Defining Immersion 14

2 Challenges of Game Audio 17

1 Implementation 17

2 Repetition and Fatigue Avoidance 18

3 Interactive Elements and Prototyping 19

4 Physics 20

5 Environmental Sound Design and Modeling 21

6 Mixing 21

7 Asset Management and Organization 22

3 The Game Engine Paradigm

24

1 What Is a Game Engine 24

The Unity3D Project Structure 25

1 Level Basics 101 26

a 2D, 3D and Cartesian Coordinates 26

b World Geometry 27

c Lighting 28

d Character Controllers 28

e Cameras 29

2 Elements of a Level 29

a Everything Is an Object 30

viii

DETAILED CONTENTS

b Transform 30

c Sprites 30

d Meshes 30

e Models 30

f Textures 31

g Shaders 31

h Materials 31

i Terrain 31

j Skyboxes 32

k Particles Systems 32

l Colliders 32

m Triggers/Trigger Zones 33

n Lighting 33

o Audio 34

p Prefabs 34

2 Sub Systems 35

1 Animation 35

2 Input 37

3 Physics 38

Rigidbodies and Collision Detection 38

Physics Materials 38

Triggers 39

Raycasting 39

4 Audio 40

5 Linear Animation 41

6 Additional Sub Systems 42

4 The Audio Engine and Spatial Audio

43

1 Listeners, Audio Clips and Audio Sources 43

1 The Audio Listener 43

Audio Clips 44

Audio Sources 45

2 Audio Source Parameters 46

3 Attenuation Shapes and Distance 47

a Spherical Spreading 48

b Sound Cones – Directional Audio Sources 50

c Square/Cube 50

d Volumetric Sound Sources 51

e 2D, 3D or 2.5D Audio? 51

4 Features of Unity’s Audio Engine 52

a Audio Filters 52

b Audio Effects 52

c Audio Mixers 53

2 Audio Localization and Distance Cues 53

1 Distance Cues 54

a Loudness 54

b Dry to Reflected Sound Ratio 55

DETAILED CONTENTS

ix

c Low Pass Filtering With Distance 55

d Spatial Width 56

2 Localization Cues 56

a Localization on the Horizontal Plane 56

b Localization on the Vertical Plane 58

3 Implementing 3D Audio 58

a Object-based Audio and Binaural Renderings 58

b Working With HRTFs 61

c Multichannel Audio and Ambisonics 62

4 Optimizing Sound Design for Spatialization 68

a Putting It all Together 68

b Working With 2D and Multichannel Audio 68

c Working With Ambisonics 68

d Working With Object-Based Audio 68

5 Sound Design – The Art of Effectively Communicating

With Sound

70

1 The Art of Sound Design 70

1 A Brief History of Sound Design 70

2 Sound Design – Basic Considerations 72

a Effective Sound Design 72

b Sound Design Guidelines 74

3 Getting the Right Tools 76

a Equalization 77

b Dynamic Range 77

c Reverberation 78

d Harmonic Processors 78

e Metering Tools 78

f Utilities 80

4 Microphones 80

a Microphone Choice: Dynamic vs. Condensers 80

b Mic Placement 82

5 Sound Design – Before You Start 82

a Always Use High Quality Material 83

b Don’t Get Too Attached 84

c Build and Learn 84

d Listen for the Expected and the Unexpected 84

e Layers 85

f Be Organized 85

g Communicate 86

h Experiment, Experiment, Experiment 86

2 Basic Techniques 86

1 Layering/Mixing 86

2 Pitch Shifting 87

a Playback Speed Modulation 87

b Granular Synthesis 88

c Fast Fourier Transform-Based Algorithms 89

x

DETAILED CONTENTS

3 Distortion 89

a Saturation 90

b Overdrive 91

c Distortion 91

d Bit Crushing 92

4 Compression 92

a Blending Through Bus Compression 94

b Transient Control 94

c Inflation 95

5 Equalization/Filtering 95

a Equalization for Sound Design 95

b Resonance Simulation 96

6 Harmonic Generators/Aural Exciters 97

7 Granular Synthesis and Granulation of Sampled Sounds 97

a Granular Synthesis Terminology 98

b Sound Design Applications of Granular Synthesis 99

8 DSP Classics 100

a Ring Modulation/Amplitude Modulation 100

b Comb Filtering/Resonators 101

9 Reverberation 102

a Indoors vs. Open Air 102

b Reverb Parameters 105

c Reverberation for Environmental Modeling 106

d Reverberation as a Dramatic Tool 107

10 Convolution 107

a Optimization 109

b Speaker and Electronic Circuit Emulation 109

c Filtering/Very Small Space Emulation 110

d Hybrid Tones 110

11 Time-Based Modulation FX 110

a Chorus 110

b Flanger 111

c Phasers 112

d Tremolo 112

12 Foley Recording 113

6 Practical Sound Design

115

1 Setting Up a Sound Design Session and Signal Flow 115

1 Signal Flow 116

a Input 116

b Inserts 116

c Pre-Fader Send 117

d Volume Fader 117

e Metering: Pre-Fader vs. Post Fader 117

f Post-Fader Send 118

g Output 118

DETAILED CONTENTS

xi

2 Working With Video 118

a Know Your Frame Rate 118

3 Clipping Is Easy – Mind the Signal Path 119

Use the Dynamic Range 120

4 Setting Up a Basic Session for Linear Mixes and Cut Scenes 122

a Music, Dialog and Sound Effects 122

b Inserts vs. Effects Loops for Reverberation 122

c Setting Up the Mix Session 123

d Master Output and Sub Maste 124

e Submixes and Effects Loops 124

f Further Enhancements 125

2 Practical Sound Design and Prototyping 126

1 Guns 126

a One Shot vs. Loops 126

b General Considerations 127

c Designing a Gunshot 128

2 Prototyping Vehicles 132

a Specifications 132

b Selecting Your Material 133

c Processing and Preparing Your Material 133

d Building a Prototype 134

3 Creature Sounds 136

a Primary vs. Secondary Sounds 137

b Emotional Span 137

c Working With Vocal Recordings 138

d Working With Animal Samples 141

e Working With Non-Human or Animal Samples 143

4 An Adaptive Crowd Engine Prototype in MaxMSP 143

7 Coding for Game Audio

147

1 Why Learn to Code? 147

1 Syntax and Logic 148

2 Algorithms 148

3 Basic Object-Oriented Programming Concepts 149

a Procedural vs. Object-Oriented 149

b Encapsulation and Inheritance 150

2 An Intro to C#: Syntax and Basics 151

1 Our First Script 151

2 Variables, Constants, Data Types Operators, Arrays

and Lists 154

a Data Types 154

b Variables 154

c Arrays 155

d Lists 157

e Access Modifiers 158

3 Accessing a Function From Another Class 159

xii

DETAILED CONTENTS

3 Playing Audio in Unity 160

1 Our First Audio Script 160

2 Play() vs. PlayOneShot() 163

3 Using Triggers 164

4 Sample Randomization 166

5 Detecting Keyboard Events 167

6 Audio-Specific Issues 168

a Timing – Frame Rate vs. Absolute Time 168

b Linear vs. Logarithmic Amplitude 170

8 Implementing Audio: Common Scenarios

173

1 Before You Start: Preparing Your Assets 173

2 Ambiences and Loops 174

1 Creating Ambiences and Loops 175

a Seamless Loops 175

b Creating a Simple Loop – Looping Techniques 176

c Creating Variations 178

2 Implementing Our Loops in a Unity Level 178

a Challenges 178

b Spatial Distribution 180

c Working With the Time Property to Avoid Phasing

Issues 181

3 Random Emitters 182

a A Simple Random Emitter Algorithm 183

b Coroutines 183

4 Ambiences, Putting It All Together 188

5 Sample Concatenation 189

a Creating Variations With Footsteps Samples 189

b Case 1: Swapping Audio Clips 190

c Case 2: Using PlayScheduled () 192

6 Collisions 193

a Detecting Collision 193

b Velocity-based Sample Selection 195

7 Raycasting and Smart Audio Sources 197

a Implementing Occlusion With Raycasting 197

b Avoiding the Pebble Effect 199

8 Animation Events 201

9 Audio Fades 204

10 Distance Crossfades 206

11 Working With Prefabs 210

a Creating a Smart Intermittent Emitter Prefab With

Occlusion 210

b Instantiating a Prefab From Scripting 210

c Destroying an Object Instantiated From a Prefab 211

d Instantiating Audio Emitters at Random Locations

in 3D 212

DETAILED CONTENTS

xiii

9 Environmental Modeling

214

1 What Is Environmental Modeling? 214

1 Reverberation 215

a Pre-Computed vs. Real Time Computation 216

b Absorption Coefficients 216

c Environmental Modeling With Reverberation

in Unity 216

d Unity’s Reverberation Parameters 217

2 Best Practices for Environmental Modeling 219

a Late vs. Early Reflections 219

b Reflections Level 219

c Density and Diffusion 220

d High Frequencies vs. Low Frequencies 220

3 Reverb Zones, Effects Loops and Audio Reverb Filters 221

a Reverb Zones 221

b Adding Reverb as an Effect Loop Using the Mixer 223

c Audio Reverb Filters 224

2 Distance Modeling 224

1 Filtering as a Product of Distance 224

a Adding a Low Pass Filter That Will Modulate its Cutoff

Frequency Based on Distance 224

b Width Perception as Product of Distance 225

c Dry to Wet Ratio as a Product of Distance 227

d Distance Simulation: Putting It All Together 229

3 Additional Factors 230

1 Occlusion, Obstruction, Exclusion 230

a Occlusion 231

b Obstruction 231

c Exclusion 232

2 Distance Crossfades 233

3 Doppler Effect 234

10 Procedural Audio: Beyond Samples

238

1 Introduction, Benefits and Drawbacks 238

1 What Is Procedural Audio? 239

a Procedural Audio, Pros and Cons 239

b Approaches to Procedural Audio 241

2 Practical Procedural Audio: A Wind Machine and a Sword

Collision Model 242

1 A Wind Machine in MaxMSP With Subtractive

Synthesis 242

Making the Model Flexible 245

2 A Sword Maker in MaxMSP With Linear Modal

Synthesis 246

Spectral Analysis 248

Modeling the Impulse 249

xiv

DETAILED CONTENTS

Modeling the Resonances 250

Making the Model Flexible 250

11 Adaptive Mixing

251

1 What’s in a Mix? Inform and Entertain (Again) 251

1 Mix Considerations 251

2 Music, Dialogue and Sound Effects 253

3 Planning and Pre-Production 254

a SubMixing 254

b Routing 255

c Dynamic Range 256

d Passive vs. Active Mix Events 258

2 The Unity Audio Mixer 259

1 Adding Groups to the Unity Mixer 259

2 The Audio Group Inspector 260

3 Working With Views and Colors in the Unity Mixer 261

Creating Views in Unity 262

4 Adding Effects to Groups in Unity 262

5 Inserts vs. Effect Loops 263

6 Setting Up an Effect Loop for Reverberation in Unity Using

Send and Receive 264

Note on Adjusting Levels During Gameplay 265

7 Ducking in Unity 266

Setting Up a Ducking Compressor in Unity 266

3 Snapshots, Automation and Game States 266

1 Working With Snapshots 267

2 Recalling Snapshots via Scripting 268

3 Editing Mixer and Plugin Parameters via Scripting 270

4 Exposing a Parameter: Controlling a Volume Slider 270

4 Good Practices 271

Mix Levels 273

12 Audio Data Reduction

276

1 Digital Audio: A Quick Review 276

1 Pulse Code Modulation 276

2 File Size Calculation 277

2 Data Reduction Strategies 278

1 Speech vs. Generic Audio 279

2 Bit Rates 279

3 Perceptual Coding 280

The Trade-Off 280

4 Common File Formats 280

a MP3 280

b Advanced Audio Coding 281

c Ogg Vorbis 281

d AC-3 Dolby Digital 282

e Adaptive Differential Pulse Code Modulation 282

DETAILED CONTENTS

xv

3 Data Reduction Good Practices 282

4 Data Reduction Options in Unity 283

1 File Options 283

2 Load Type 284

3 Compression Formats Options 285

Sample Rate Setting 286

Index

287

1 INTRODUCTION

Interactive and Game Audio

‘Simplicity is the ultimate sophistication’.

– Leonardo Da Vinci

1. The Genesis of Audio in Games

Video games are a relatively new art form, one that was borne out of the boredom and curiosity of computer scientists, advances in technology and the human need for new entertainment. It is generally agreed upon that the first commercially released, mass produced video game was an arcade game called Computer Space, in 1971, by Nutting Associates. The origins of video games, however, can be traced to the Massachusetts’ Institute of Technology in the United States, where, in 1962, Steve Russel developed Spacewar! on a DEC PDP-1 computer. But it was in 1972 that the iconic game Pong was released. Pong was perhaps the tipping point, the game that took video games out of the realm of computer scientists, science fiction fans and software engineers and brought it out to the general public, introducing the art form to our culture at large. The game was not about computers or spaceships and as such didn’t necessarily and specifically appeal to the science and computer nerds amongst us, at least not specifically. It was, of all things, about sports. Table tennis, a game most people could relate to and probably enjoy or have enjoyed playing at some point. There perhaps was the genius behind it all, when Nolan Bushnell, who co-founded the mythical gaming company Atari, asked programmer and game developer Allan Acorn to create a table tennis game as an exercise. Although extremely primitive by today’s standards – the game was black and white, the graphics were entirely made up of squares and rectangles and the gameplay was extremely simple –

still, it was fun to watch for onlookers and the game demanded attention wherever it was found. Pong’s contribution to the video game industry and our culture in general cannot be understated.

In many ways, Pong hit all the marks a successful game ought to. It was easy to learn but hard to master, could be played alone or with a friend and was just the right amount of difficult (the ball speed would slowly increase as the play continued on then reset at the next play). In some ways, the soundtrack was perhaps the crudest aspect of the game. There was no

2

INTRODUCTION

music, a simple, musical ping to let you know you had hit the ball, a similar sound but slightly lower in pitch when the ball hit the walls and a slightly noisier sound, more akin to a buzzer, when you failed to hit the ball at all.

Yet, this simple audio implementation, realized by someone with no audio training, still resonates with us to this day and was the opening shot heard around the world for game audio. Indeed, Allan Acorn may not have studied modern sound design, but his instincts for game development extended to audio as well. The soundtrack was definitely primitive, but it reinforced and possibly even enhanced the very basic narrative of the game and is still with us today.

To say that technology and games have come a long way since then would be both an understatement and commonplace. Today’s games bear little resemblance to Pong. The level of sophistication of technology used by modern game developers could not have been foreseen by most Pong gamers as they eagerly dropped their quarters in the arcade machine.

1972 also marked what’s commonly referred to as the first generation of home gaming consoles, with the release of a few devices meant for the general public. One of the most successful of these was the Magnavox Odyssey. It had no audio capabilities whatsoever, and although it enjoyed some success, its technology was a bit crude, even for its time. The games came with overlays that the gamer had to place on their TV screen to make up for the lack of graphic processing power, and with hindsight, the Odyssey felt a bit more like a transition into interactive electronic home entertainment systems than the first genuine video gaming console. It wasn’t until the next generation of home gaming hardware and the advent of consoles such as the Atari 2600, introduced in 1977, that the technology behind home entertainment systems became mature enough for mass consumption and started to go mainstream and, finally, included sound.

The Atari 2600 was a huge commercial success. It made Atari an extremely successful company and changed the way we as a culture thought of video games. Still, it suffered from some serious technical limitations, which made it difficult to translate the hit coin-operated games of the times such as Pac Man or even Space Invaders into compelling console games. Still, these did not stop Atari from becoming one of the fastest growing companies in the history of the US. When it came to sound, the Atari 2600 had a polyphony of two voices, which was usually not quite enough for all the sounds required by the games, especially if the soundtrack also included music.

Besides the limited polyphony, the sound synthesis capabilities of the 2600

were also quite primitive. The two voice polyphony was created by two onboard audio chips that could only produce a very narrow array of tones, pitches and amplitude levels. No audio playback capabilities and limited synthesis technology meant that the expectation of realistic sound was off the table for developers back then.

It’s also sometimes easy to forget that nowadays, when major game studios employ thousands of designers, coders and sound designers, game development

INTRODUCTION

3

Figure 1.1

in the early days of the industry was a very personal matter, often just one person handling every aspect of the game design, from game logic to graphics and, of course, music and sound design. Sounds in early video games were not designed by sound designers, nor was the music written by trained composers.

Perhaps it is the combination of all these factors, technical limitations and limited expertise in sound and music, combined with a new and untested artform pioneered by visionaries and trailblazers, that created the aesthetics that we enjoy today when playing the latest blockbusters.

2. From Sample Playback to Procedural Audio

Technology evolved quickly after the Atari 2600. As the graphics and gameplay improved with each generation of new hardware, audio sample playback technology started to find its way into video games in arcades at first and in-home entertainment systems later on. Although the first attempts to implement sample playback in games were not always very satisfying or even convincing, due to the initial limitations of the technology such as low sample rates (as low as 11Khz), bit depth (as low as 8 bits) and heavily compressed

4

INTRODUCTION

audio formats at low bit rates, eventually, as the technology improved so did the fidelity of the audio samples we could include and package in our games.

And so, eventually, along with audio playback technology and the ability to use recorded sound effects in games, games soundtracks started to improve in terms of fidelity, impact and realism. It also started to attract a new generation of sound designers, often coming from linear media and curious or downright passionate about gaming. Their expertise in terms of audio production also helped bring game soundtracks out of the hands of programmers and into those of dedicated professionals. Although game audio still suffered from the stigma of the early days of low fidelity and overly simplistic soundtracks, over time these faded, and video game studios started to call upon the talents of established composers and sound designers to improve the production values of their work further still. With better technology came more sophisticated games, and the gaming industry started to move away from arcade games toward games with complex story lines and narratives. These, in turn, provided sound designers and composers with more challenging canvases upon which to create and, of course, also provided more challenges for them to overcome. More complex games required more sounds and more music, but they also demanded better sounds and music, and the expectations of the consumers in terms of production values started to rival those of Hol-lywood blockbusters. This, however, meant much more than to simply create more and better sounds. Issues in gaming, which had been overlooked so far, became much more obvious and created new problems altogether. It was not quite enough to create great sounds, but the mix and music had to be great while at the same time adapt and reflect the gameplay. This demanded the creation of new tools and techniques.

Over the years, however – with increasing levels of interactivity and complexity in gameplay, sample playback’s dominance in the world of game audio and the inherent relative rigidity that comes with audio recordings – signs that other solutions were needed in order for our soundtracks to respond to and keep up with the increasingly complex levels of interaction available in games started to appear. This became more obvious when real-world physics were introduced in gaming. With the introduction of physics in games, objects could now respond to gravity, get picked up and thrown around, bounce, scrape and behave in any number of unpredictable manners. The first major release to introduce ragdoll physics is generally agreed to be Trespassers: Jurassic Park, a game published in 1998 by Electronic Arts. Although game developers usually found ways to stretch the current technologies to provide acceptable solutions, it was impossible to truly predict every potential situation, let alone create and store audio files that would cover them. Another crack in the façade of the audio playback paradigm appeared more recently, with the advent of virtual and augmented reality technologies. The heightened level of expectations of interaction and realism brought on these new technologies means that new tools had to be developed still, especially in the areas of environmental modeling and procedural audio.

INTRODUCTION

5

Procedural audio is the art and science of generating sound effects based on mathematical models rather than audio samples. In some ways it is a return to the days of onboard sound chips that generated sound effects from primitive synthesis chips in real time. Generating sounds procedurally holds the promise of sound effects that can adapt to any situation in the game, no matter what.

Procedural audio is still a relatively nascent technology, but there is little doubt that the level of expertise and fluency in audio technologies significantly increases with each new technical advance and will keep doing so. As a result, we can expect to see a fragmentation in the audio departments of larger game development studios, labor being divided in terms of expertise, perhaps along a similar path to the one seen in graphic departments. Sound design and the ability to create compelling sounds using samples are going to remain a crucial aspect of how we generate sounds, but in addition we can expect to see increased specialization in several other areas, such as:

• Spatial audio: the ability to create and implement sound in 360 degrees around the listener.

• Procedural sound synthesis: designing audio models via scripting or programming that can accurately recreate a specific sound.

• Virtual reality and augmented reality audio specialists: working with these technologies increasingly requires a specific set of skills specific to these mediums.

• Audio programming and implementation: how to make sure the sound designed by the audio team is triggered and used properly by the game engine.

• Technical sound design: the ability to connect the sound design team to the programming team by designing specialized tools and optimizing the overall workflow of the audio pipeline.

Each of these topics could easily justify a few books in their own rights, and indeed there are lots of great tomes out there on each specific topic. As we progress through this book, we will attempt to demystify each of these areas and give the reader not only an overview of the challenges they pose but also solutions and starting points to tackle these issues.

3. How to Use This Book

What This Book Is

This book is about the soundtrack of video games – focusing on sound effects rather than music – and about interactive audio in general. The days when a single person, no matter how talented or gifted, could write a blockbuster video game from start to finish on their own are long gone, and the level of technical expertise required in every aspect of game development continues to rise with no end in sight. Today an audio developer, regardless of their place

6

INTRODUCTION

in the audio team, needs to be fluent with a number of software packages, from multiple digital audio workstations to increasingly more sophisticated audio processors, sound design techniques, adaptive mixing techniques, spatial audio, coding and procedural audio techniques.

Over the course of the next few chapters we will examine the purposes served by a game audio soundtrack; the various components that make up a game engine; how to approach sound design and the basics of scripting, of audio implementation, of adaptive mixing, of data reduction and of procedural audio. We will use Unity as our game engine, but a lot of these concepts will apply to your work in other game engines and in some cases to linear media as well. By the end of this book, the reader will have obtained a solid understanding of the techniques and solutions used to address common issues in game audio and should have a strong foundation from which to approach most situations. While we tried to keep the book software agnostic, Unity will be used to demonstrate a lot of the issues dealing with implementation and scripting. For some of the chapters in this book you will find material available on the companion website. These examples are meant to complement and enhance your experience with the book and provide you with additional perspective and material. When it comes to the chapters dealing with coding, we have provided several Unity projects, each containing the scripting examples covered in the book, as well as additional examples. These scripts and projects are intended as starting points, meant to be customized to fit your desired outcome.

What This Book Isn’t

This is not a book intended to teach the reader Unity. There are many fantastic books and resources on the topic, and while you do not need to be an expert with Unity to get the most out of this book, it is strongly encouraged to spend some time getting acquainted with the interface and terminology and to run through a few of the online tutorials that can be found on the Unity website.

No prior knowledge of computer science or scripting is required; Chapters seven and eight will introduce the reader to C#, as well as audio-specific issues that deal with audio coding.

If you are reading this, you probably have a passion for gaming and sound.

Use that passion and energy, and remember that, once they are learned and understood, rules can be bent and broken. We are storytellers, artists and sound enthusiasts. It is that passion and enthusiasm that for several decades now has fueled the many advances in technology that make today’s fantastic games possible and that will create those of tomorrow.

2 THE ROLE OF AUDIO IN

INTERACTIVE AND IMMERSIVE

ENVIRONMENTS

Learning Objectives

The purpose of this chapter is to outline the major functions performed by the soundtrack of a video game, as well as to layout the main challenges facing the modern game audio developer.

We shall see that audio plays a multi-dimensional role, covering and supporting almost every aspect of a game or VR environment, from the obvious, graphics animation, to the less obvious, such as narrative, Artifcial Intelligence and game mechanics, to name but some. All and all, the soundtrack acts as a cohesive layer that binds the various components of a game together by providing us with a consistent and hopefully exciting sensory experience that deals with every sub system of a game engine.

1. Inform, Entertain, Immerse

What is the purpose of audio in games? What makes a player turn up the volume in a game instead of streaming their favorite music playlist?

Games have come a long way since the days of the Atari 2600 and its embryonic soundtracks, the blips and noises still in our collective memory today. Newer, better technologies have come online, giving sound designers new tools and more resources with which to create the soundtracks of future games. Yet, even with the recent technological advances, crafting a compelling soundtrack remains a tricky affair at best, reminding us that technology isn’t everything, and that, at its core, the issues facing the modern sound designer have at least as much to do with the narrative we strive so hard to craft as with the tools at our disposal. So perhaps we should begin our investigation not so much by looking at the tools and techniques used by professionals but by understanding the aims and challenges gaming confronts us with, and how to best tackle them.

Understanding these challenges independently from the technology involved will allow us to ultimately get the best out of the tools available to us, whatever those may be, whether we are working on a AAA game for the latest generation of dedicated hardware or a much more humble mobile app.

8

THE ROLE OF AUDIO

If we had to sum up the purpose of sound in games and interactive media we could, perhaps, do it with these three words: inform, entertain, immerse. The role of the sound designer and audio engineer in interactive media is to pursue and attain these goals, establishing a dialogue between the player and the game, providing them with essential information and data, that will help them navigate the game. Perhaps a simple way to think about how each event fits within the overall architecture of our soundtracks is through this simple equation:

Data + Context = Information

It is easy to understand the entertain portion of our motto. The soundtrack (a term that refers to music, dialog and SFX) of a AAA game today should be able to compete with a high-end TV or film experience. We expect the sound design to be exciting, larger than life and original. That is a challenge in itself, of course. Additionally, however, in order to create a fully encompassing gaming experience, it is also important that we provide useful feedback to the player as to what is happening in the game both in terms of mechanics and situational awareness. Using the soundtrack to provide gamers with information that will help them play better and establish a dialog with the game is a very powerful way to maximize the impact of the overall experience. Indeed, as we shall see, even a simple, mobile arcade game type can be significantly improved by a detailed and thoughtful soundtrack, and the user’s experience vastly heightened as a result. Effective aural communication will also certainly greatly contribute to and enhance the sense of immersion that so many game developers aspire to achieve.

In a visually driven media world we tend to underestimate – or perhaps take for granted – how much information can be conveyed with sound. Yet constantly in our daily lives we are analyzing hundreds of aural stimuli throughout the day that provide us with information on our surroundings, the movement of others, alert us to danger or the call of a loved one and much, much more.

In effect, we experience immersion on a daily basis; we simply call it reality, and although gaming is a fundamentally different experience, we can draw upon these cues from the real world to better understand how to provide the user with information and how to, hopefully, achieve immersion.

Let us take a closer look at all three of these concepts, inform, entertain and immerse, first in this chapter, then in more detail throughout the rest of this book as we examine strategies to develop and implement audio assets for a number of practical situations.

1. Inform: How, What

In a 3D or VR environment sound can and must play an important role when it comes to conveying information about the immediate surroundings of the user.

Keeping in mind that the visual window available to the player usually covers between 90–120 degrees out of 360 at any given time, sound quickly becomes indispensable when it comes to conveying information about the remaining

THE ROLE OF AUDIO

9

portion of the environment. It should also be noted that, while the visual field of humans is about 120 degrees, most of that is actually peripheral vision; our actual field of focus is much narrower. The various cues that our brain uses to interpret these stimuli into a distance, direction and dimension, will be examined in more detail in a future chapter, but already we can take a preliminary look at some of the most important elements we can extract from these aural stimuli and what they mean to the interactive and immersive content developer.

a. Geometry/Environment: Spatial Awareness

In a game engine, the term geometry refers to the main architectural elements of the level, such as the walls, stairs, large structures and so on. It shouldn’t be surprising that sound is a great way to convey information about a number of these elements. Often, in gaming environments, the role of the sound designer extends beyond that of creating, selecting and implementing sounds. Creating a convincing environment for sound to propagate in is often another side of the audio creation process, known as environmental modeling. A well-designed environment will not only reinforce the power of the visuals but is also a great way to inform the user about the game and provide a good backdrop for our sounds to live in.

Figure 2.1

Some of the more obvious aspects of how sound can translate into information are:

• Is the environment indoors or outdoors?

• If indoors, what is the order of the size of the room we find ourselves in?

10

THE ROLE OF AUDIO

• If outdoors, are there any large structures, natural or man-made, around?

• Do we have a clear line of sight with the sound we are hearing, or are we partially or fully cut off from the source of the sound? We can isolate three separate scenarios:

1. We are fully cut off from the audio source. The sound is happening in an adjacent room or outside. This is known as occlusion. There is no path for the direct or reflected sound to get to the listener.

2. The path between the audio source and the player is partially obstructed, as in a small wall or architectural feature (such as a column for instance) blocking our line of sight. In this case the direct audio path is blocked, but the reflected audio path is clear: that is known as obstruction.

3. The direct path is clear, but the reflected sound path isn’t, blocking the reverberated sound: this is known as exclusion.

Each of these situations can be addressed and simulated in a soundtrack and provide the user with not just an extremely immersive experience but also valuable information to help them navigate their environment and the game itself.

b. Distance

We have for a long time understood that the perception of distance was based primarily on the amount of dry vs. reflected sound that reaches our ears and that therefore reverberation played a very important role in the perception of distance.

Energy from reverberant signals decays more slowly over distance than dry signals, and the further away from the listener the sound is, the more reverb is heard.

Additionally, air absorption is another factor that aids us in perceiving distance. Several meteorological factors contribute to air absorption; the most important ones are temperature, humidity and distance. The result is the noticeable loss of high frequency content, an overall low pass filtering effect.

Most game engines, Unity being one of them, provide us with a great number of tools to work with and effectively simulate distance. It does seem, however, that, either due to a lack of knowledge or due to carelessness, a lot of game developers choose to ignore some of the tools at their disposal and rely solely on volume fades. The result is often disappointing and less-than-convincing, making it difficult for the user to rely on the audio cues alone to accurately gauge distance.

c. Location

The perception of the location of a sound in terms of direction in 360 degrees is a little more complex a mechanism, as it in fact relies on multiple mechanisms. The most important are:

THE ROLE OF AUDIO

11

• Interaural time difference : the time difference it takes for sound to reach both the left and right ears.

• Interaural intensity difference: the difference in amplitude between the signal picked up by the left and the right ear.

• The precedence effect: in a closed space, the precedence effect can also help us determine the direction of the initial sound source. It was demonstrated by Dr Helmut Haas in 1949 that humans, when confronted to under certain circumstances, will determine the location of a sound based on the first arriving wave.

As outlined with these principles, our ability to discern the direction a sound comes from is dependent on minute differences in time of arrival and relative intensities of signals to both ears. While some of these phenomena are more relevant with certain frequencies than others (we almost universally have an easier time locating sounds with high frequency content, for instance), it is almost impossible to determine the location of a continuous tone, such as a sine wave playing in a room (Cook ’99). A good game audio developer will be able to use these phenomena to their advantage.

The process currently used to recreate these cues on headphones is a technology called Head Related Transfer Functions, which we shall discuss in Chapter four.

Another somewhat complimentary technology when it comes to spatial audio is ambisonic recording. While not used to actually recreate the main cues of human spatial hearing, it is a great way to compliment these cues by recording a 360-degree image of the space itself. The Unity game engine supports this technology, which their website describes as an ‘audio skybox’.

Ambisonic and their place in our sonic ecosystem will also be discussed further in upcoming chapters.

d. User Feedback and Game Mechanics

This might be less obvious than some of the previous concepts discussed up until now, as in some ways, when successfully implemented, some of the features about to be discussed might not – and perhaps should not – be noticed by the casual player (much to the dismay of many a sound designer!).

On a basic level, audio based user feedback is easily understood by anyone who ever had to use a microwave oven, digital camera or any of the myriad consumer electronics goods that surround us in our daily lives. It is the Chime Vs. Buzzer Principle that has governed the sound design conventions of consumer electronics good for decades – and TV quiz shows for that matter.

The simplest kind of feedback one can provide through sound is whether an action was completed successfully or not. The Chime Vs. Buzzer Principle is actually deceptively simple, as it contains in its root some of the most important rules of sound design as it relates to user feedback:

12

THE ROLE OF AUDIO

The chime almost universally symbolizes successful completion of an action, or positive feedback. It is a pleasant, musical sound that we associate with immediate action and positive sentiments. The buzzer, of course, is noisy, unpleasant to the ear and associated with negative feedback and negative sentiments. Both these sounds have the benefit of being easy to hear, even at mod-erate levels in a somewhat crowded or noisy environment, although the chime appears to achieve similar results while remaining pleasant to the listener.

These qualities, being easy to hear in a noisy environment, easy to understand when heard (also known as legibility), make them prime examples of the specific demands that user feedback sound design requires.

Sound can provide much more complex and subtle feedback as well. Adding a low tone to the mix when entering a room can induce an almost sub-liminal sense of unease in the player; a sound can inform us of the material that something is made of even though it might not be clear visually. There are many variations of the Chime Vs. Buzzer Principle in gaming. Contact sounds, the sound the game makes if you hit a target, for instance, are one such great example, but there are far too many for us to list here. As you can see, there are many ways to use the Chime Vs. Buzzer Principle in your games, and coming up with creative ways to take advantage of our innate understanding of this principle provides the game developer with endless opportunities for great sound design.

Additionally, the mix itself is an effective way to provide information to the player. By altering the mix – for instance the balance between music, dialog and FX – or even by changing the relative balance between sound effects, the game can attract the attention of the player and focus it on a specific element or, in turn, distract the attention of the player.

2. Entertain

The focus of this book being on sound design and not composition, we will think of music in relation to the sound design and overall narrative and emotional functions it supports.

a. Sound Design

We all know how much less scary or intense even the most action-packed shots look when watched with the sound off. If you haven’t tried, do so. Find any scary scene from a game or movie, and watch it with the sound all the way down. Sound allows the story-teller to craft and compliment a compelling environment that magnifies the emotional impact of the scene or game, increasing the amount of active participation of the gamer. An effective combination of music and sound design, where both work together, plays a critical role in the overall success of the project, film or game.

Sound design for film and games remains still today, to an extent, a bit of a nebulous black art – or is often perceived as such – and one that can truly be

THE ROLE OF AUDIO

13

learned only through a long and arduous apprenticeship. It is true that there is no substitute for experience and taste, both acquired through practice, but the vast amount of resources available to the student today makes it a much more accessible craft to acquire. This book will certainly attempt to demystify the art of sound design and unveil to students some of the most important techniques used by top notch sound designers, but experimentation by the student is paramount.

As previously discussed, sound supports every aspect of a video game – or should anyway. If we think of sound as simply ‘added on’ to complete the world presented by the visuals, we could assume that the role of sound design is simply to resolve the cognitive dissonance that would arise when the visuals are not complemented by sound.

Of course, sound does also serve the basic function of completing the visuals and therefore, especially within VR environment, allows for immersion to begin to take hold, but it also supports every other aspect of a game, from narrative to texturing, animation to game mechanics. A seasoned sound designer will look for or create a sound that will not simply complete the visual elements but also serve these other functions in the most meaningful and appropriate manner.

b. Music and the Mix

While this book does not focus on music composition and production, it would be a mistake to consider sound design and music in isolation from each other. The soundtrack of any game (or movie) should be considered as a whole, made up of music, dialog, sound effects and sometimes narration.

At any given time, one of these elements should be the predominant one in the mix, based on how the story unfolds. A dynamic mix is a great way to keep the player’s attention and create a truly entertaining experience. Certain scenes, such as action scenes, tend to be dominated by music, whose role is to heighten the visuals and underline the emotional aspect of the scene. A good composer’s work will therefore add to the overall excitement and success of the moment. Other scenes might be dominated by sound effects, focusing our attention on an object or an environment. Often, it is the dialog that dominates, since it conveys most of the story and narrative. An experienced mixer and director can change the focus of the mix several times in a scene to carefully craft a compelling experience. Please see the companion website for some examples of films and games that will illustrate these points further.

Music for games can easily command a book in itself, and there are many out there. Music in media is used to frame the emotional perspective of a given scene or level. It tells us how to feel and whom to feel for in the unfolding story. I was lucky enough to study with Morton Subotnick, the great composer and pioneer of electronic music. During one of his lectures, he played the opening scene to the movie The Shining by Stanley Kubrick. However, he kept changing the music playing with the scene. This was his way to illustrate some

14

THE ROLE OF AUDIO

of the obvious or subtle ways in which music can influence our emotional perception of the scene. During that exercise it became obvious to us that music could not only influence the perceived narrative by being sad or upbeat or by changing styles from rock to classical but that, if we are not careful, music also has the power to obliterate the narrative altogether. Additionally, music has the power to direct our attention to one element or character in the frame.

Inevitably, a solo instrument links us emotionally to one of the characters, while an orchestral approach tends to take the focus away from individuals and shifts it toward the overall narrative.

Although we were all trained musicians and graduate students, Subotnick was able to show us that music was even more powerful than we had thought previously.

The combination of music and sound can not only be an extremely powerful one, but it can play a crucial role in providing the gamer with useful feedback in a way that neither of these media can accomplish on their own, and therefore communication between the composer and sound design team is crucial to achieve the best results and create a result greater than the sum of its parts.

3. Defning Immersion

Entire books have been dedicated to the topic of immersion – or presence – as psychologists have referred to it for several decades. Our goal here is not an exhaustive study of the phenomenon but rather to gain an understanding of it in the context of game audio and virtual reality.

We can classify virtual reality and augmented reality systems into three categories:

• Non-immersive systems: typically, simple Augmented Reality systems that affect one sensory input. Playing a 3D game on a laptop is a common example. This is the type of system most people are familiar with.

• Semi-immersive systems: typically allows the users to experience a 3D

world while remaining connected to the real world. A flight simulator game played on a multiscreen system with realistic hardware, such as a flight yoke, would be a good example of such a system.

• Fully immersive systems: affect all or most sensory inputs and attempt to completely cut off the user from their surroundings through the use of head mounted displays, headphones, and additional systems such as gaming treadmills, which allow the user to walk or even run though a virtual environment.

An early definition of presence based on the work of Minski, 1980 would be: The sense an individual experiences of being physically located in an environment different from their actual environment, while also not realizing the role technology is playing in making this happen

THE ROLE OF AUDIO

15

We in the gaming world tend to think of presence or immersion as a rather novel topic, one that came about with games and virtual reality. Truly, however, the concept has been part of conventional medias such as literature for hundreds of years. Narrative immersion happens when a player or reader is so invested in the plot that they momentarily forget about their surroundings.

There is no doubt, however, that games and virtual reality have given us a new perceived dimension in the immersive experience, that is, the possibility to act in an environment, not simply having the sensation of being there. So, what are the elements that scientists have been able to identify as most likely to create immersion?

The research of psychologist Werner Wirth suggests that successful immersion requires three steps:

1. Players begin to create a representation in their minds of the space or world the game is offering.

2. Players begin to think of the media space or game world as their main reference (aka primary ego reference).

3. Players are able to obtain useful information from the environment.

Characteristics that create immersion tend to fall in two categories: 1. Characteristics that create a rich mental model of the game environment.

2. Characteristics that create consistency amongst the various elements of the environment.

Clearly, sound can play a significant role in all these areas. We can establish a rich mental model of an environment through sound by not only ‘scoring’

the visuals with sound but by also by adding non-diegetic elements to our soundtrack. For instance, a pastoral outdoor scene can be made more immersive by adding the sounds of birds in various appropriate locations, preferably randomized around the player, such as trees, bushes etc. Some elements can be a lot more subtle, such as the sound of wood creaking, layered every once in a while, with footsteps over a wooden surface, for instance. While the player may not be consciously cognizant of such an event, there is no doubt that these details will greatly enhance the mental model of the environment and therefore contribute to creating immersion.

Consistency, this seemingly obvious concept, can be trickier to implement when it comes to creature sounds or interactive objects such as vehicles. The sound an enemy makes while it is being hurt in battle should be different than the sound that same creature might make when trying to intimidate its enemies, but it should still be consistent overall with the expectations of the player based on the visuals and, in this case, the anatomy of the creature and the animation or action. Consistency is also important when it comes to sound propagation in the virtual environment, and, as was seen earlier in this

16

THE ROLE OF AUDIO

chapter, gaming extends the role of the sound designer to modeling sound propagation and the audio environment in which the sounds will live.

Inconsistencies in sound propagation will only contribute to confusing the player and cause them to eventually discard any audio cue and rely entirely on visual cues.

Indeed, when the human brain receives conflicting information between audio and visual channels, the brain will inevitably default to the visual channel. This is a phenomenon known as the Colavita visual dominance effect.

As sound designers, it is therefore crucial that we be consistent in our work. This is not only because we can as easily contribute and even enhance immersion as we can destroy it, but beyond immersion, if our work is confusing to the player, we take the risk of having the user discard audio cues altogether.

It is clear that sensory rich environments are much better at achieving immersion. The richness of a given environment maybe given as:

• Multiple channels of sensory information.

• Exhaustiveness of sensory information.

• Cognitively challenging environments.

• Possessing a strong narrative element.

Additionally, while immersion can be a rather tricky thing to achieve, it is rather easy to break. In order to maintain immersion, research suggests that these elements are crucial:

• Lack of incongruous audio/visual cues.

• Consistent behavior from objects in the game world.

• Continuous presentation of the game world – avoid commercials, level reset after a loss.

• The ability to interact with objects in the game world.

While some of these points may be relatively obvious, such as the lack of presence of incongruous elements (such as in-game ads, bugs in the game, the wrong sound being triggered), some may be less so. The third point presented in this list, ‘continuous presentation of the game world’, is well illustrated by the game Inside by Playdead studios. Inside is the follow-up to the acclaimed game Limbo, and Inside’s developers took a very unique approach to the music mechanics in the game. The Playdead team was trying to prevent the music from restarting every time the player respawned after being killed in the game. Something as seemingly unimportant as this turns out to have a major effect on the player. By not having the music restarted with every spawn, the action in the game feels a lot smoother, and the developers have removed yet one more element that may be guilty of reminding the player they are in a game, therefore making the experience more immersive. Indeed, the game is extremely successful at creating a sense of immersion.

THE ROLE OF AUDIO

17

It is important to note than the willingness to be emotionally involved is also an important, perhaps crucial, factor to achieving immersion. This is something that developers have no control over and that pre-supposes the desire of the user to be immersed. This is sometimes referred to as the ‘Fan Gene’. As a result, two users may have wildly differing experiences when it comes to the same experience, based, partially, on their willingness to ‘be immersed’.

2. Challenges of Game Audio

In spite of the improvements that each new generation of hardware brings with every anticipated release, developers are forced to come to one ineluc-table conclusion: no matter how new, exciting, revolutionary, even, each new generation of tools is, we are almost always at some point contending with finite resources. It could be said that developers working on mobile gaming today are facing similar challenges as their peers did when developing games on the first generation of gaming consoles. In that regard, the range of technologies available to us today requires the modern developer to deal with a massive range of hardware and capabilities, demanding a level of expertise that is constantly evolving and increasing.

1. Implementation

It is impossible to understate the importance and impact of implementation on the final outcome, although what implementation actually consists of, the process and its purpose often remain a somewhat nebulous affair.

In simplistic terms, implementation consists of making sure that the proper sounds are played at the right time, at the right sound level and distance and that they are processed in the way the sound designer intended. Implementation can make or break a soundtrack and, if poorly realized, can ruin the efforts of even the best sound designers. On the other hand, clever use of resources and smart coding can work their magic and enhance the efforts of the sound designers and contribute to creating a greater sense of immersion.

Implementation can be a somewhat technical process, and although some tools are available that can partially take out the need for scripting, some programming knowledge is definitely a plus in any circumstance and required in most. One of the most successful third-party implementation tools is Audiokinetic’s Wwise, out of Montreal, Canada, which integrates seamlessly with most of the leading game engines such as Unity, Unreal and Lumberyard. The Unreal engine has a number of tools useful for audio implementation. The visual scripting language Blue Print developed by Epic is a very powerful tool for all-purpose implementation with very powerful audio features. As a sound designer or audio developer, learning early on what the technical limitations of a game, system or environment are is a crucial part of the process.

18

THE ROLE OF AUDIO

Because the focus of this book is to work with Unity and with as little reliance on other software as possible, we will look at these concepts and implementation using C# only, although they should be easy to translate into other environments.

2. Repetition and Fatigue Avoidance

We have already seen in Chapter one that the first generations of gaming hardware did not rely on stored PCM data for audio playback as is mostly the case today but instead used on-board audio chips to synthesize sounds in real time.

Their concerns when it came to sound therefore had more to do with number of available voices than trying to squeeze as many samples as possible on a disc or download. Remember that the Atari 2600 had a polyphony of two voices.

The 1980s saw the rise and then dominance of PCM audio as the main building blocks of game soundtracks. Audio samples afforded a level of realism that was unheard of until then, even at the low resolutions early hardware could (barely) handle. Along with increased realism, however, came another host of issues, some of which we are still confronted with today.

Early video game systems had very limited available RAM, as a result of which games could ship only with a small amount of samples. Often these samples were heavily compressed (both in terms of dynamic range and data reduction), which severely reduced the fidelity of the recording or sound, making them hard to listen to, especially over time. In addition, since so few samples could be included, they were played frequently and had to be used for more than one purpose. In order to deal with listener fatigue, game developers early on developed techniques that are still relevant and in use today, the most common being randomization.

The use of random and semi-random techniques in sound and music, also known as Stochastic techniques, had been pioneered by avant-garde composers such as John Cage and Iannis Xenakis in the 1950s and 1960s. These techniques, directly or indirectly, have proved to be extremely helpful for game developers.

The use of random behaviors is a widespread practice in the gaming industry, which can be applied to many aspects of sound.

Randomization can be applied to but is not limited to: 1. Pitch

2. Amplitude

3. Sample Selection

4. Sample concatenation – the playback of samples sequentially 5. Interval between sample playback

6. Location of sound source

7. Synthesis parameters of procedurally generated assets (Working examples of each of the techniques listed in the following and more are provided in the scripting portion of the book.)

THE ROLE OF AUDIO

19

The most common of these techniques is the randomization of pitch and amplitude, often built into game engines, such as Unreal, in which it’s been implemented as a built-in feature for iterations. Pitch and amplitude randomization might be a good start, but it’s often no longer enough to combat listener fatigue. Nowadays developers rely on more sophisticated techniques, often combining the randomization of several parameters. These more advanced, combinatorial techniques are sometimes referred to as procedural, a term in this case used rather loosely. In this book, we will tend to favor the stricter definition of the term procedural, that is, real time creation of audio assets, as opposed the real time manipulation of existing audio assets. The difference between procedural asset creation and advanced stochastic techniques are sometimes blurry, however. These more advanced random or stochastic techniques are certainly very important, and their usefulness should not be underestimated.

3. Interactive Elements and Prototyping

One of the challenges that even very accomplished sound designers coming from linear media tend to struggle with the most initially when working in gaming is the interactive elements, such as vehicles, machines, weapons and other devices the user may interact with. Interactivity makes it difficult to predict the behavior of a game object and therefore cannot be approached in a traditional linear fashion. How can one design sounds for a vehicle not knowing in advance how the user will interact with it?

Simple things such as acceleration, braking sounds and the sound of tires skidding when the vehicle moves at high speed are all of a sudden part of a new equation.

The answer when addressing these issues is often prototyping. Prototyping consists of building an interactive audio model of the object, often in a visual environment such as Cycling74’s MAXMSP, Native Instrument’s Reaktor or Pure Data by Miller Puckette, to recreate the intended behavior of the object and test in advance all possible scenarios to make sure that our sound design is on point and, just as importantly, that the sounds behave appropriately. For instance, in order to recreate the sense of a vehicle accelerating, the engine loop currently playing back might get pitched up; inversely, when the user is slamming on the breaks the sample will get pitched down, and eventually, in more complex simulation, another sample at lower RPM might get triggered if the speed drops below a certain point and vice versa.

Working with interactive elements does imply that sounds must be ‘animated’ by being pitched up, down, looped and processed in accordance with the circumstances. This adds another layer of complexity to the work of the sound designer: they are not only responsible for the sound design but also for the proper processing and triggering of these sounds. The role of the sound designer therefore extends to determining the range of the proper parameters

20

THE ROLE OF AUDIO

for these actions, as well as the circumstances or threshold for which certain sounds must be triggered. The sound of tires skidding would certainly sound awkward if triggered at very low speeds, for instance. Often, these more technical aspects are finely tuned in the final stages of the game, ideally with the programming or implementation team, to make sure their implementation is faithful to your prototype. In some cases, you might be expected to be fluent both as a sound designer and audio programmer, which is why having some scripting knowledge is a major advantage. Even in situations where you are not directly involved in the implementation, being able to interact with a programmer in a way they can clearly comprehend, with some knowledge of programming, is in itself a very valuable skill.

4. Physics

The introduction and development of increasingly more complex physics engines in games introduced a level of realism and immersion that was a small revolution for gamers. The ability to interact and have game objects behave like ‘real-world’ objects was a thrilling prospect. Trespasser: Jurassic Park, released in 1998 by Electronic Arts, is widely acknowledged as the first game to introduce ragdoll physics, crossing another threshold toward full immersion. The case could be made that subsequent games such as Half Life 2, published in 2004 by Valve Corporation, by introducing the gravity gun and allowing players to pick up and move objects in the game, truly heralded the era of realistic physics in video games.

Of course, physics engines introduced a new set of challenges for sound designers and audio programmers. Objects could now behave in ways that were totally unpredictable. A simple barrel with physics turned on could now be tipped over, dragged, bounce or roll at ranges of velocities, each requiring their own sound, against any number of potential materials, such as concrete, metal, wood etc.

The introduction of physics in game engines perhaps demonstrated the limitations of the sample-based paradigm in video game soundtracks. It would be impossible to create, select and store enough samples to perfectly cover each possible situation in the barrel example. Some recent work we shall discuss in the procedural audio chapter shows some real promise for real-time generation of audio assets. Using physical modeling techniques we can model the behavior of the barrel and generate the appropriate sound, in real time, based on parameters passed onto us by the game engine.

For the time being, however, that is, until more of these technologies are implemented in production environments and game engines, we rely on a combination of parameter randomization and sample selection based on data gathered from the game engine at the time of the event. Such data often include the velocity of the collision and the material against which the collision occurred. This permits satisfactory, even realistic simulation of most scenarios with a limited number of samples.

THE ROLE OF AUDIO

21

5. Environmental Sound Design and Modeling

In creating the soundtrack for a large 3D game or environment, one should consider the resulting output as a cohesive whole instead of a collection of sounds playing somewhat randomly on top of each other. This kind of fore-sight and holistic approach to sound design allows for much more engaging and believable environments and a much easier mix overall. The soundtrack of a game is a complex environment, composed of many layers playing on top of each and changing based on complex parameters determined by the gameplay.

In a classic first-person shooter game, the following groups or layers of sounds could be playing at any single time over each other:

• Room tones: drones, hums.

• Environmental sounds: street sounds, weather.

• Dialog and chatter.

• Foley: footsteps, movement sounds.

• Non-player characters: AI, creatures, enemies.

• Weapons: small arms fire, explosions.

• Machinery: vehicles, complex interactive elements.

• Music.

This list does gives us a sense of the challenge that organizing, designing, prioritizing and playing back all these sounds together and keeping the mix from getting cluttered represents.

In essence, we are creating a soundscape. We shall define soundscape as a sound collage that is intended to recreate a place and an environment and provide the player with an overall sonic context.

In addition to having the task of creating a cohesive, complex and responsive sonic environment, it is just as important that the environment itself, within which these sounds are going to be heard, be just as believable. This discipline is known as environmental modeling and relies on tools such as reverberation and filtering to model sound propagation. Environmental modeling is a discipline pioneered by sound designers and film editors such as Walter Murch that aims at recreating the sonic properties of an acoustical space – be it indoors or outdoors – and provides our sounds a believable space to live in. The human ear is keenly very sensitive to the reverberant properties of most spaces, even more so to the lack of reverberation. Often the addition of a subtle reverberation to simulate the acoustic properties of a place will go a long way in creating a satisfying experience but in itself may not be enough.

Environmental modeling is discussed in further detail in this book.

6. Mixing

The mix often remains the Achilles’ heels of many games. Mixing for linear media is a complex and difficult process usually acquired with experience.

Mixing for games and interactive media does introduce the added complexity

22

THE ROLE OF AUDIO

of unpredictability, as it isn’t always possible to anticipate what to expect sonically in an interactive environment where events may unfold in many potential ways. We must teach the engine to deal with all potential situations using a carefully thought-out routing and rules of architecture for the game to follow.

In most situations the game has no or little awareness of its own audio output.

Our challenge is, as it is so often in game audio, twofold: ensure a clean, crisp and dynamic mix while making sure that, no matter what, critical audio such as dialog is heard clearly under any circumstances and is given priority.

Discussing the various components of a good mix is beyond the scope of this chapter and shall be addressed in detail in Chapter twelve.

7. Asset Management and Organization

A modern game or VR simulation requires a massive number of audio assets.

These can easily range in the thousands, possibly tens of thousands for a AAA game. Managing these quickly becomes a challenge in itself. Game engines, even third-party software such as Wwise, should be thought of as integration and creative tools rather than asset creation tools. The line between the two is not always an absolute one, but as a rule you should only import in the game engine and work with polished assets ready to be plugged in as quickly and painlessly as possible. While you can fix some issues during the implementation process, such as amplitude or pitch adjustments, you should avoid consistently relying on adjusting assets in the game engine for matters that could have been taken care of sooner. This tends to cost time and create unnecessarily complex projects. It is much more time-efficient to make sure all assets are exported and processed correctly prior to importing them.

An asset delivery checklist, usually in the form of a spreadsheet, is a must. It should contain information about the following, but this list is not exhaustive:

• Version control: you will often be dealing with multiple versions of a sound, level, game build etc. due to fixes or changes. Making sure you are working with and delivering the latest or correct file is obviously imperative.

• Deadlines: often the work of the sound design team is split up into multiple deadlines for various assets types in order to layer and optimize the audio integration and implementation process. Keeping track of and managing multiple deadlines is a highly prized and useful organizational skill.

• Consistency and format: making sure that all the files you will be delivering are at the proper format, sample rate, number of channels and at consistent sound levels across variations, especially for sounds that are related (such as footsteps for instance), quickly becomes challenging and an area where it is easy to make mistakes.

• Naming convention: dealing with a massive number of assets requires a naming convention that can easily be followed and understood by all

THE ROLE OF AUDIO

23

the team members. The naming convention should be both descriptive and as short as possible:

Hero_Fstps_Walk_Wood_01.wav

Hero_Fstps_Walk_Metal_02.wav

Hero_Fstps_Run_Stone_09.wav

Deciding on a naming convention is something that should be carefully considered in the preproduction stages of the game, as it will be very inconvenient to change it halfway through and could cause disruptions in the production process. Keep in mind that audio files are usually linked to the engine by name.

Conclusion

The functions performed by the soundtrack of a video game are complex and wide ranging, from entertaining to providing user feedback. The goal of an audio developer and creator is to create a rich immersive environment while dealing with the challenges common to all audio media – such as sound design, mixing and supporting the narrative – but with the added complexities brought on by interactive media and specific demands of gaming. Identifying those challenges, establishing clear design goals and familiarity with the technology you are working with are all important aspects of being successful in your execution. Our work as sound designers is often used to support almost every aspect of the gameplay, and therefore the need for audio is felt throughout most stages of the game creation process.

3 THE GAME ENGINE PARADIGM

Learning Objectives

When sound designers and composers get into gaming, one of the most nebulous concepts initially is the game engine and its inner workings. In this chapter, using Unity as our model, we will attempt to demystify the modern game engine, take a look at the various components and sub systems that make up a modern game engine and understand what they each do and are responsible for. In addition, we will look at the various elements that comprise a typical level in a 2D or 3D video game, as well as the implications on the sound design and implementation. This chapter is not intended to be a specifc description of the inner workings of a specifc game engine but rather a discussion of the various parts and sub systems that comprise one, using Unity as our teaching tool. Readers are encouraged to spend time getting acquainted with Unity (or any other engine of their choice) on their own to develop those skills.

1. What Is a Game Engine

Early video games such as arcades or games developed for the first generation of gaming consoles, such as the Atari 2600, often used to be the work of a single person. Often that person was a programmer, who also moonlighted as a graphic artist, animator, sound designer, game designer, composer and whatever other tasks were required for the game. While that may have been okay for a single, very talented individual to manage then, game engine technology quickly evolved past the point where a single person could claim to be able to efficiently take care of all the various aspects of game production.

The question at the heart of this chapter is: what is a game engine?

It may be easier to think of a game engine as a collection of dedicated sub systems interacting with each other rather than a monolithic entity. Some sub systems take care of rendering graphics; others select and manage animations; others deal with networking, physics and, of course, sound. Sometimes some of these sub systems can be enhanced or replaced by more advanced and capable third-party software. But before delving deeper into these matters, let’s make sure we understand how a Unity project is structured.

THE GAME ENGINE PARADIGM

25

The Unity3D Project Structure

Creating a new project in Unity can be done via the Unity Hub application. It is highly recommended to work with the Unity hub as it gives the user a way to keep track of new projects but also of multiple versions of the Unity engine.

It is possible to have multiple versions of Unity installed on your computer simultaneously, and, in some cases, you will need to, as new versions of the software may not be backwards compatible. In other words, a project created in a version of Unity may not run in a newer version of the engine, and you may need to work with multiple versions of Unity if you are working on several projects at once or if you are trying to open older projects.

Note: when opening a project made with an older version of the engine with a newer version of Unity you will be asked if you made a backup before continuing.

Unity will also ask if you wish to upgrade the project. Please note than when Unity upgrades a project, it might no longer compile and require you to fix/update the project. Reopening that same project with an older version of the software will NOT fix these issues, so do be careful when working with different versions of the engine, and do make sure your projects are backed up before upgrading them.

In order to create a new project, click the New button in the Unity Hub software.

Figure 3.1

You will then be asked to name your project, select a location and choose the type of project you wish to create: 2D, 3D or some of the other options available. Click create when done.

When you create a new Unity project, the software will create several new folders on your hard drive with a predetermined structure.

26

THE GAME ENGINE PARADIGM

Figure 3.2

Of all the folders Unity created for your project, the assets folder is the one we will focus on the most, as every asset imported or created in Unity will show up in this folder. Since you can expect a large number of files of various types to be located in the folder, organization and naming conventions are key.

Note: the project structure on your hard drive is reflected in the Unity editor. Whenever you import an asset in Unity, a local copy in the project folder is created, and it is that copy that will be referenced by the game from now on. The same is true when moving assets between folders in the Unity Editor. You should always use the Unity editor to move and import assets and not directly move or remove files from a Unity project via the Finder but always using the Unity editor. Failing to do so may result in the project getting corrupted, behaving unpredictably or simply force-quitting without warning.

Unity scenes vs. projects: there may be some confusion between a Unity scene and a Unity project. A Unity project consists of all the files and assets within the folder with your project’s name that were created when you clicked the create button. When opening a Unity project, this is the folder you should select when opening the project from the Unity Hub or Editor. A Unity scene is what we most commonly think of as a level. That is a playable environment, either 2D or 3D. But scenes can be used for menus, splash screens etc.

1. Level Basics 101

a. 2D, 3D and Cartesian Coordinates

When creating a game level or Unity scene, the first question is whether to create a 2D or 3D level. This of course will depend on the desired gameplay, although the lines between 2D and 3D can be somewhat blurry these days.

For instance, some games will make use of 3D assets, but the camera will be located above the level, in a bird’s eye view setting also known as ortho-graphic, giving the gameplay a 2D feel. These types of games are sometimes known as 2.5D but are in fact 3D levels. The opposite can also be true, where

THE GAME ENGINE PARADIGM

27

we have seen 2D gameplay with 3D graphics. In both these cases, you would need to create a 3D level in order to manage the 3D graphics.

Both 2 and 3D levels are organized around a Cartesian coordinate system: Figure 3.3

A 2D level will only have the X and Y axes.

Coordinates are a very important part of working with Unity, as all object placement in the level will be done using coordinates. Do take some time to become familiar and comfortable with the Cartesian coordinate system.

Note: Unity defaults to 1 unit = 1 meter.

Coordinates can be world coordinates or local coordinates. World coordinates are the way we locate game objects in the level; each X, Y and Z coordinates to a unique location in the level. Local coordinates are referring to each individual object. That means that, in addition to the world coordinate system, every object has its own system of coordinates. Local coordinates are useful when it comes to object manipulation, transformation and creating parent/

child hierarches with other game objects.

A game level is a fully functioning world, usually where the game takes place, either 2D or 3D, that is comprised of at least these few items:

b. World Geometry

World geometry usually refers to the static architectural elements, such as walls, floors etc. More complex objects, such as furniture or vehicles are generally not considered geometry, and unlike world geometry, which is usually created

28

THE GAME ENGINE PARADIGM

in the game engine itself, more complex objects, 2D and 3D models are usually imported into the game engine and created in a third-party graphic software.

c. Lighting

At least one light will be necessary in order for the level not to be completely dark. There are many types of lights available to the level designer, which we will look at in more detail later on in this chapter. When creating a new level, Unity provides a default light.

d. Character Controllers

A character controller is the interface between the player and the game. It allows the player to look, move around the level and interact with the environment. There are several types of character controllers: player controllers – which are meant to be controlled by human beings – and NPCs, non-player controllers, meant to control AI characters in the game without human input. Often the character controller is tied to a graphical representation of your character or avatar in the game.

Player characters also fall into two categories: first- and third-person controllers. With a third-person character, the player can see their avatar’s body on the screen, whereas with a first-person controller the player will usually only see through the eyes of their character and may not be able to see their own avatar at all. In fact, with the default first-person character controller in Unity, the player’s avatar is simplified down to a capsule. This simplifies computation while still giving the game engine a good way to be aware of the character’s dimension and scale.

Figure 3.4a First-person controller

THE GAME ENGINE PARADIGM

29

Figure 3.4b Third-person controller

e. Cameras

The camera is the visual perspective through which the level is rendered. The camera’s placement usually depends on the type of character controller used in the game and the game itself. A first-person controller will usually have the camera attached to the avatar of the main character, usually at or near head level. With a third-person controller the camera will usually be placed above and behind the avatar, sometimes known as a ‘behind the shoulder’

camera.

The camera can also be placed fully above the level, known as top-down or isometric placement. This is a bit more common in 2D games such as platformer games or in strategy games.

These four elements, geometry, lights, a character controller and a camera, may be indispensable in order to create a basic level, but it will be a rather boring one. A few additional elements are required to make this a somewhat interesting and compelling level.

2. Elements of a Level

The following section is an introduction to some of the most commonly found objects in game levels, whether in Unity or other game engines, but it is by no means an extensive list of all Unity objects. Some of these objects may have other names in other game engines but are common across most engines.

30

THE GAME ENGINE PARADIGM

a. Everything Is an Object

Before going further in our study of game engines, it is important to understand that everything that appears in a level is considered by Unity as an object.

There are many different types of objects, of course; some are invisible to the player, some are actual objects in the level.

Generally speaking, an object’s behavior is determined by one or multiple scripts. If an object is visible in the level (not all objects are visible), its representation in the level is known as mesh. Objects can be made invisible in the level by disabling their mesh component.

b. Transform

Every game object in a scene has a transform component. The transform component determines the position, rotation and scale of an object. We can use this component to move an object on the screen by updating its position with every frame and do the same thing for its rotation and scale.

Figure 3.5

c. Sprites

In 2D games, models tend to be made of 2D images, often sequenced in a particular order to create the illusion of animation, such as a character walking. The sprites are then moved along the X and Y axis based on user input or gameplay.

d. Meshes

Meshes are 3D objects made of interconnected polygons, triangular shapes used to create more complex shapes. The reason polygons have become popular has to do with their efficiency in terms of rendering. For reasons of efficiency also, polygons are usually only rendered from one side, the visible side to the player.

Often the other side of the polygon, the hidden side, is simply not rendered, and moving the camera behind a model may result in the model disappearing altogether, which is perfectly normal behavior. Strictly speaking, the mesh is only the vertex data, representing the shape and dimension of the object.

e. Models

While the world geometry, walls, floors and ceilings are usually created within the game engine itself, game engines are not well-suited for the generation of more detailed objects, such as furniture, vehicles and other weapons you will find in a game. Those objects, or models, are usually imported and created in other software packages.

THE GAME ENGINE PARADIGM

31

Models are usually comprised of a mesh but also textures, materials, animations and more depending on the desired appearance and functionality. When referring to a model, we usually mean all of these, not just the mesh. Models may be imported from 2D and 3D modeling software or as packages from the asset store.

f. Textures

Textures are 2D images that get applied to 3D objects in order to give them detail and realism. When creating geometry in Unity, such as a wall for instance, they are created with a default solid white color. By applying textures, we can make that wall look like a brick wall, a wooden fence or any other material.

Figure 3.6 shows an untextured wall next to a textured one for contrast.

Figure 3.6

g. Shaders

Shaders determine the properties of how the model will respond to light, its color, how matte or reflective it is, which textures to apply and many other properties.

h. Materials

Materials are a way for Unity to combine shaders and textures, providing a convenient way to describe the physical appearance of an object and giving the designer one more level of control over the process. Materials are applied to an object, and from the material shaders and textures are applied.

i. Terrain

Terrains are generally used to recreate outdoor landscapes, such as hills or sand dunes, where the ground shape is highly irregular and could not realistically be

32

THE GAME ENGINE PARADIGM

simulated using primitive geometric shapes. Often terrains start as a flat mesh that is sculpted by the level designer into the desired shape.

j. Skyboxes

Skyboxes are used to create background images for levels that extend or give the illusion to extend beyond the level itself, often, as the name implies, for the purpose of rendering skies. This is done by enveloping the level in a box or sphere and projecting an image upon it.

k. Particles Systems

Most game engines include particle systems. These are used to model smoke, fire, fog, sparks etc. Particle systems can grow into rather complex and computationally intensive systems.

l. Colliders

Collision detection is at the core of gaming and has been since the early days of Pong. In order for the game engine to register collisions and to prevent other objects in the game from going through each other, a collider component is added. Colliders tell the game engine what the shape and dimensions of an object are, as far as collision are concerned. Rather than computing collision on a polygon per polygon basis using the exact same shape as the object’s mesh, which is computationally expensive, colliders are usually invisible and made of simple shapes, known as primitives, in order to maintain efficiency while still maintaining accurate results. For instance, a first-person controller is abstracted down to a capsule collider matching the height and width of a character in the game, or a stool might be simplified down to a cube collider.

Figure 3.7

Note: The green outline shows a box collider. Even though the object would be invisible in the game engine, because its mesh renderer is turned of, it would still be an obstacle for any player.

THE GAME ENGINE PARADIGM

33

m. Triggers/Trigger Zones

A trigger or trigger zone is a 2D or 3D area in the level that is monitored for collisions but, unlike a collider, will not block an actor from entering it. Triggers are a staple of video games. They can be used to play a sound when a player enters a particular area or trigger an alarm sound, start a cinematic sequence, turn on or off a light etc. Trigger zones can keep track of whether a collider is a entering an area, remaining in an area or exiting an area. In Unity a trigger component is actually a collider component whose is trigger is checked, so, like colliders, triggers are usually made of simple geometric shapes such as squares, cubes or spheres.

Triggers and colliders are discussed in more depth in the rest of this book.

Figure 3.8

n. Lighting

Lighting is a very complex topic, one that can make all the difference when it comes to good level design and that takes time and experience to master.

For our purposes as audio designers, however, our understanding of the topic needn’t be an in-depth one but rather a functional one. The following is a short description of the most common types of lights found in game engines.

Note: in Unity lights are added as components to existing objects rather than being considered objects themselves.

Point lights: point lights emit lights in every direction and are very common for indoor lighting. They are similar to the household lightbulb.

Spotlights: light is emitted as a cone from the origin point outwards and can be aimed at a specific location while keeping other areas dark.

Area lights: area lights define a rectangular area where light is distributed evenly across.

Ambient lights: ambient lights are lights that don’t appear to have a point of origin but illuminate a large area.

Directional lights: often used to recreate daylight illumination, while directional lights can be aimed, they will illuminate an entire level. For that reason, they

34

THE GAME ENGINE PARADIGM

are often used in lieu of sunlight. At the time of this writing, when creating a new scene in Unity, a directional light is added to every scene.

o. Audio

Unity, like a number of audio engines, relies on a structure for its audio engine based around three main object types and additional processors. The main three object types are:

• Audio sources: the audio is played in the level through an audio source, which acts as a virtual speaker and allows the audio or level designer to specify settings such as volume, pitch and additional properties based on the game engine.

• Audio clips: audio clips are the audio data itself, in compressed format, such as ogg vorbis or ADPCM or uncompressed PCM audio. Audio clips are played back through an audio source. Most game engines use audio sources as an abstraction layer rather than directly playing back the audio data (without going through an audio source). This gives us a welcome additional level of control over the audio data, such as control of pitch, amplitude and more depending on the game engine.

• Listeners: the listener is to the audio what the camera is to the visuals; it represents the auditory perspective through which the sound will be rendered. Unless you are doing multiplayer levels, there usually should be only one audio listener per scene, often but not always attached to the camera.

Listeners and audio sources are usually added as components, while audio clips are loaded into existing audio sources. As we shall see shortly, Unity also provides developers with a number of additional processors, such as mixers and processing units.

p. Prefabs

Game objects in Unity can quickly become quite complex, with multiple components, specific property values and child objects. Unity has a system that allows us to store and easily instantiate all components and settings of a game object known as prefabs . Prefabs are a convenient way to store these complex game objects and instantiate them easily and at will.

A Prefab can be instantiated as many times as desired, and any changes made to the Prefab will propagate to all instances of the prefab in a scene, although it is possible and easy to make changes to a single instance without affecting the others as well. The process of changing the settings on one instance of a prefab is known as overriding.

Prefabs are very useful for instantiating objects at runtime, which can apply to audio sources, as a way to generate sounds at various locations in a scene for instance.

When adding sound to a prefab, it is much more time effective to edit the original prefab, located in the assets folder, rather than editing individual instances separately.

THE GAME ENGINE PARADIGM

35

2. Sub Systems

At the start of this chapter we stated that a game engine is a collection of sub systems. Now we can take a closer look at some of the individual systems that make up a modern game engine and that, as sound designers, we find ourselves having to support through our work.

1. Animation

Most game engines include an in-depth animation system, and Unity is no exception. Unity’s animation system is also sometimes called Mecanim. Animations, whether 2D or 3D, are used very commonly in game engines. 3D characters rely on a number of animation loops for motion, called animation clips in Unity, such as walking, running, standing or crouching, selected by the game engine based on the context for AI characters or by the player’s actions for player characters.

Figure 3.9

Animation clips contain information such as position, rotation, scale or movement tied to a timeline and are the foundation of animation sub systems.

Animation clips can be created in Unity or imported from a third-party software package. These clips are organized in a graphical structure known as an animation controller. It is the task of the animation controller to determine which animation clip the engine should be playing and which to use next.

Animations can also be blended together.

36

THE GAME ENGINE PARADIGM

Figure 3.10

Animation controllers are used for simple tasks such as a sliding door or very complex ones such as a humanoid character. Since humanoid characters are quite a bit more complex, Unity has a dedicated sub system known as Avatar for mapping and editing animations to humanoid characters. Animation clips are organized graphically as a flowchart in the animation controller and use a state machine, which holds the animation clips and the logic used to select the proper clip, transition and sequence.

These elements can be added to a game object via the animation component, which holds a reference to an animation controller, possibly an Avatar, and in turn the animation controller holds references to animation clips.

Audio may be attached to animation clips via the use of animation events.

animation events can call a function located in a script – which in turn can trigger the appropriate sound – and are added to specific frames via a timeline.

For instance, in the case of a walking animation we would add an animation event each time the character’s feet touch the ground, calling a function that would trigger the appropriate sound effect.

Figure 3.11

THE GAME ENGINE PARADIGM

37

Learning about the animation system of the game engine you are working with is important in order to know where to attach sounds or scripts and how to do so. You will find that while different game engines offer different features and implementation, the heart of its animation system will usually be supported by animation clips triggered by a state machine.

2. Input

Input in Unity is usually entered using the keyboard, gamepad, mouse and other controllers such as VR controllers. Since it is difficult to know in advance what the player will be working with, it is recommended to use Unity’s input manager rather than tying actions to specific key commands for optimal compatibility.

The input manager can be accessed in the setting manager located under the edit menu: edit->project settings. Select the input tab on the right-hand side: Figure 3.12

Unity uses a system of Axes to map movement. The vertical axis is typically mapped to the S and W keys and the horizontal axes to the A and D keys.

There are also three Fire modes, Fire 1, 2 and 3.

The positive horizontal axis is mapped to the D key – or right The negative horizontal axis is mapped to the A key – or left The positive vertical axis is mapped to the W key – or up The positive vertical axis is mapped to the S key – or down

38

THE GAME ENGINE PARADIGM

Fire 1 is mapped to the control key – or left mouse button Fire 2 is mapped to the option key – or right mouse button Fire 3 is mapped to the command key – or middle mouse button These are the default mappings, and they can be customized from the input manager to fit every situation. It is recommended to refer to the Unity manual for a complete listing and description of the options available to the developer from the input manager.

The input manager is a great way to standardize the control over multiple platforms and input devices. It is recommended to work with the input manager when sounds must be triggered in response to events in the game rather than attaching them directly to keystrokes. This will ensure the sounds will always be triggered regardless of the controller the user is playing with.

3. Physics

A modern game engine has to have a comprehensive physics engine in order to be able to recreate the expected level of interaction and realism of modern games.

The most common iteration of physics in games is collision detection, without which most games simply would be impossible to make.

Rigidbodies and Collision Detection

Rigidbodies are required to be added to objects to enable Unity’s physics engine. Rigidbodies are added as a component and will make objects they are applied to respond to gravity. In order for collision to be detected, another component, a collider must be added, as previously mentioned. Colliders usually approximate the shape of the object they are applied to in order to maximize performance. When colliders are added to a game object without a rigidbody component, they are known as static colliders and are used for the level geometry, such as walls. These can interact with other colliders but will not be moved or displaced in response to a collision. When a collider is added to an object with a rigidbody component it is known as a dynamic collider.

Rigidbodies have properties that can be adjusted by the user in order to adjust the behavior of the game object they are added to. A complete listing can be found in the Unity documentation. These properties allow us to adjust mass, air resistance and the method for collision detection. The property isKinematic allows us to turn off an object’s physics properties altogether when set to true. When an object is governed by physics it shouldn’t be moved by updating its transform properties but rather by applying forces to it.

Physics Materials

In order for colliders to mimic the property of their surface materials, physics materials can be added to game objects. The properties of a physics material include detailed control over bounciness and friction in order to create various surface types, such as plastic, stone, ice etc.

THE GAME ENGINE PARADIGM

39

Triggers

Triggers have already been discussed earlier in this chapter, but they are part of the physics engine in Unity, depending on how their isTrigger property is set. When false, triggers will be used to detect collisions between game objects with collider components, and the object they are applied to will behave as a solid one.

Collision detection is a complex and fascinating topic, much greater in scope than this chapter. The reader is encouraged to read further about it in the Unity online documentation.

Raycasting

Raycasting is a very powerful technique used in gaming that consists of drawing a ray or a line in a specific direction and length and seeing if it hits any colliders. The ray is invisible and is a very useful way to detect any object in the path of a projectile, but that also has many applications in the world of sound.

Raycasting can be used, as we shall see later in the book, to detect obstacles located between the listener and an audio source, allowing us to process the audio accordingly and model our environment more accurately, among so many other applications for this tool.

The sphere is raycasting to the camera, where the listener is located. The wall will be detected by the ray, and the information can be used to trigger a low pass filter to simulate partial occlusion.

Figure 3.13

40

THE GAME ENGINE PARADIGM

The physics portion of Unity is both vast and somewhat intuitive, but it certainly takes some practice to feel really comfortable with it. Dynamic rigidbodies especially can present difficult challenges to the sound design and implementation team as their behavior can be both complex and unpredictable. For that reason, it’s important to understand the basics of the implementation of physics objects in the game engine you are working with since it will help you a great deal in understanding the behavior of these objects and coming up with solutions to address them.

4. Audio

The Unity audio engine is powerful and provides game developers with a wide range of tools with which to create our sound worlds. Unity features 3D spatialization capabilities, a number of audio filters, which are audio processors such as low pass filters and echoes, as well as mixers, reverberation and more.

These effects are covered in more detail in Chapter four.

The Unity audio settings is where the global settings for the audio engine are found, under the edit menu: edit->project settings->audio Figure 3.14

The following parameters are defined under audio settings:

• Global volume: will act as a last gain stage and affect the volume of all the sounds in the project.

• Volume rolloff scale: controls the curve of all the logarithmic based audio source. One is intended to simulate real-world conditions while values over 1 make the audio sources attenuate faster. A value under 1

will have the opposite effect.

• Doppler factor: controls the overall doppler heard in the game, affecting how obvious or subtle it will appear. This will affect all audio files playing in the game. A value of zero disables it altogether, and 1 is the default value.

• Default speaker mode: this controls the number of audio channels or speaker configuration intended for the game to be played on, from mono to 7.1. The default is 2 for stereo. The speaker mode can be changed during the game using script.

THE GAME ENGINE PARADIGM

41

• System sample rate: the default is 0, which translates as using the sample rate of the system you are running. Depending on the platform you may or may not be able to change the sample rate, and this is intended as a reference.

• DSP buffer size: sets the size of the DSP buffer. There is an inherent tradeoff between latency and performance. In the digital audio world latency is the time difference between the moment an audio signal enters a digital audio system and the moment it leaves the audio converters. The option best latency will minimize the audio latency but at the expense of performance; good latency is intended as a balance between the two, and best performance will favor performance over latency.

• Max virtual voices: a virtual audio source is one that has been bypassed but not stopped. It is still running in the background. Audio voices are made virtual when the number of audio sources in the scene exceeds the max number of available voices, by default set to 32. When that number is exceeded, audio voices deemed less important or audible in the scene will be made virtual. This field controls the number of virtual audio voices that Unity can manage.

• Max real voices: number of audio voices Unity can play at one time.

The default is 32. When that number is exceeded Unity will turn the softest voice virtual.

• Spatializer plugin: Unity allows the user to use third-party plugins for audio spatialization. Once an audio spatializer package has been installed, you can select it here.

• Ambisonic decoder plugin: Unity supports the playback of ambisonic files. This field allows you to choose a third party plugin for the rendering of the ambisonic file to binaural.

• Disable Unity audio: when checked, Unity will turn off the audio in standalone builds. The audio will still play in the editor, however.

• Virtualize effects: when checked Unity will dynamically disable spatialization and audio effects on audio sources that have been virtualized or disabled.

The Unity audio engine supports multiple file formats, such as AIF, WAV, OGG and MP3. Mixers provide us with a convenient way to organize and structure our mixes, and the built-in audio effects are flexible enough to allow us to deal with most situations. The audio implementation does lack a few features available in other game engines, such as randomization of volume and pitch for audio sources or directional audio sources, but most of these features can be easily implemented with some scripting knowledge.

5. Linear Animation

Unity, like a lot of modern game engines, also often features a linear sequenc-ing tool for cut scenes and linear animations. In Unity the timeline window

42

THE GAME ENGINE PARADIGM

can be used to create cinematic sequences using tracks upon which to position clips that are attached to objects. The timeline window allows us to create tracks upon which multiple clips can be layered and sequenced, much more along the lines of a traditional audio or video editing software.

Figure 3.15

This is a better solution than the animation window when it comes to creating more complex linear animation sequences involving multiple objects. Audio clips can also be used to score the sequences.

6. Additional Sub Systems

Along with the systems outlined earlier, a modern game engine such as Unity contains additional functionality for handling other areas of gameplay that could impact the job of the audio team. Networking is a big part of modern gaming, and it us usually handled by a dedicated section of the engine. Multiplayer games usually bring with them the issues of multiple listeners and sound prioritization and propagation.

Conclusion

A game engine is a complex ecosystem comprised of multiple sub systems working together to support the gameplay. Understanding how they coexist and function is a valuable skill as sound often supports many, if not all, of these sub systems, and understanding the possibilities and limitations of these systems will help the audio team make more informed decisions and utilize the available technologies to their full extent. Although the job of the audio team does not usually extend to level and game design, the student is encouraged to learn about the basics of how to put together a simple arcade style game, from start to finish. There are lots of tutorials available directly from the Unity website that will give the reader a better sense of how these various components interact and gain a deeper understanding of how a game engine actually operates.

4 THE AUDIO ENGINE AND

SPATIAL AUDIO

Learning Objectives

In the previous chapter we looked at the various components and sub systems that make up a game engine. in this chapter we shall focus our attention on the audio system with an-in depth look at its various components, from listeners to audio sources, from reverberation to spatial audio implementation. By the end of this chapter the student will have gained a solid understanding of the various audio components and capabilities of the Unity engine and of similar game engines. We will also take a close look at the mechanisms and technologies behind spatial audio and how to start to best apply them in a game context.

1. Listeners, Audio Clips and Audio Sources

While a number technologies are available to implement audio in game engines, most have settled on a basic layout, with various degrees of feature implementation. This model revolves around three basic audio objects.

Although the terminology used in this chapter focuses on Unity’s implementation, other game engines tend to build upon a similar architecture. The three objects at the core of the audio engine are listeners, audio clips and audio sources. Let’s take a close look at them.

1. The Audio Listener

The audio listener allows us to hear the audio in the game and represents the auditory perspective rendered in the game when playing back spatial audio. Although there can be situations where multiple listeners are required, usually in multiplayer games, in single player situations there should only be one listener per level. Without an audio listener, no audio will be heard. Audio listeners are added to a game object, often the camera, as a component.

44

THE AUDIO ENGINE AND SPATIAL AUDIO

Audio Clips

Audio clips hold the actual audio data used in the game. In order to play back an audio clip, it must be added to an audio source (discussed next). Unity supports the following audio formats:

• aif files.

• wav files.

• mp3 files.

• ogg vorbis files.

Mono, stereo and multichannel audio (up to eight channels) are supported by Unity. First order ambisonic files are also supported. When an audio file is imported in Unity, a copy of the audio is created locally and a metadata file is generated with the same name as the audio file and a .meta extension. The meta file holds information about the file such as format, quality (if applicable), whether the file is meant to be streamed, its spatial setting (2D vs. 3D) and its loop properties.

Figure 4.1

THE AUDIO ENGINE AND SPATIAL AUDIO

45

Audio Sources

Audio sources are the virtual speakers through which audio clips are played back from within the game. Audio sources play the audio data contained in audio clips and give the sound designer additional control over the sound, acting as an additional layer. This is where we specify if we want the audio file to loop, to be directional or 2D (whether the sound pans as we move around it or plays from a single perspective) and many more settings, each described in more detail later.

Note: audio sources can be added as a component to an existing object, but for the sake of organization I would recommend adding them to an object dedicated to hosting the audio source as a component. With a careful naming convention, this will allow the designer to quickly identify and locate the audio sources in a given level by looking through the hierarchy window in Unity. Ultimately, though, every designer’s workflow is different, and this is merely a suggestion.

Audio sources are rather complex objects and it is worth spending some time familiarizing yourself with the various parameters they give the game audio designer control over.

Figure 4.2

46

THE AUDIO ENGINE AND SPATIAL AUDIO

2. Audio Source Parameters

Audio clip: use this field to select the audio clip to be played by that audio source. You must import the audio assets first for them to show up as an option in this field.

Output: use this field to select a mixer’s group input – or submix – to route the audio source to. If none is selected the audio output will default to the master fader.

Mute: when checked will mute the output of the audio source.

Bypass effects: when checked, the audio output of that audio source will not be routed through any effects that were applied. This is a quick way to listen to the sound completely unprocessed.

Bypass listener effects: when checked, global effects applied to the listener will not be applied to this audio source. (irrelevant if the audio source is routed through a mixer group).

Bypass reverb zone: when checked, audio from that audio source will not be routed to any reverb zone applied.

PlayOnAwake: when checked, the audio source will start playing as soon as the level starts running.

Loop: when checked, the audio will keep on looping.

Volume: amplitude of the audio source on a linear scale from 0, no audio, to 1, full audio, at a distance of one unit.

Pitch: pitch of the audio source, from –3 to 3. 1 represents actual pitch, 0.5 an octave down and 2 an octave up from the original pitch. Negative values will play the sound backwards.

Stereo pan: allows the panning of files in the stereo field. –1 = left, 0 =

center and 1 = right. Disabled if the spatial blend is set to 3D.

Spatial blend: determines if the audio source will be spatialized in 2D, 3D

or a combination of both. With a setting of zero, the sound will not be spatialized in 3 dimensions or appear to come from a specific place in the level. The sound will not decay with distance but can still be panned left-right using the stereo pan slider. Furthermore, the sound will not pan relative to the position of the listener and will appear to be static. This setting is usually appropriate for voiceover, music and UI sounds.

By setting this parameter to 1 the audio will playback in full 3D and will be localized using the 3D engine. The position of the sound will appear to change relative to the position of the listener and will not be heard when the player is outside the audio source’s maximum range. Use this setting when you want your audio sources to have a clear sense of location in your level.

Priority: used to determine the importance of each audio source relative to each other. This setting comes in handy if Unity runs out of available audio voices and is therefore forced to mute some. A setting of 0 gives the audio source the highest priority and 256 the least. The

THE AUDIO ENGINE AND SPATIAL AUDIO

47

Unity manual suggests 0 for music so that music tracks do not get interrupted, while sounds that may not be crucial to the gameplay or the level should be assigned a lower setting.

Reverb zone mix: this parameter determines how much of the audio source’s signal will be routed through a reverb zone, if one is present.

This acts as the dry/wet control found in traditional reverb unit, allowing you to adjust how much reverb to apply to each audio source.

Doppler level: controls the amount of perceived change in pitch when an audio source is in motion. Use this parameter to scale how much pitch shift will be applied to the audio source when in motion by the engine.

Spread: controls the perceived width in degrees of a sound source in the audio field. Generally speaking, as the distance between a sound and the listener decreases, the perceived width of a sound increases. This parameter can be changed relative to distance to increase realism using a curve in the 3D sound settings portion of an audio source.

Volume roll off: This setting controls how a 3D sound source will decay with distance. Three volume roll off modes are available, logarithmic, linear and custom. Logarithmic tends to sound the most natural and is the most intuitive as it mimics how sound decays with distance in the real world. Linear tends to sound a little less natural, and the sound levels may appear to change drastically with little relation to the actual change in distance between the listener and source. Custom will allow the game designer to control the change in amplitude over distance using a curve for more precise control.

Note: always make sure the bottom right portion of the curve reaches zero, otherwise even a 3D sound will be heard throughout an entire level regardless of distance.

Minimum distance: the distance from the sound at which the sound will play at full volume.

Maximum distance: the distance from the sound at which the sound will start to be heard. Beyond that distance no sound will be heard.

3. Attenuation Shapes and Distance

When working with 3D levels, the way sounds are represented in the world, how far they can be heard, how they sound up close or from a distance and whether they pan as the listener moves about or not, are crucial aspects of building something believable. It is worth spending a little more time specifically discussing the different ways that audio sources can be set up – in Unity and beyond – and how to adjust a given source to obtain the best results.

Note: the following applies to 3D audio sources. 2D audio sources will be played back as mono or stereo files, at the same volume regardless of the position of the listener.

48

THE AUDIO ENGINE AND SPATIAL AUDIO

a. Spherical Spreading

Spherical spreading over distance is probably the most common attenuation shape in games. In this configuration, the audio source will spread outwards as a sphere, and be heard from all directions, based on a minimum and maximum distance parameter.

Outside area: no sound is heard

Audio Source

Inner Radius:

sound plays at full volume

within this area

Outer Radius:

sound fades in as you enter this

area and increases in volume as

you get closer to the inner radius

Figure 4.3

The maximum distance, expressed in game units, specifies how far from the source or the object it is attached to the audio will be heard. At any point beyond the maximum distance the audio will not be heard and will start fading in once the listener enters the maximum distance or radius. As you get closer to the audio source, the sound will get louder until you reach the minimum distance, at which point the audio will play at full volume. Between the two distances, how the volume fades out, or in, is specified by the fall-off curve, which can be either linear, logarithmic or custom:

• For more natural-sounding results, it is recommended to start with a logarithmic fall-off curve and adjust as needed.

• Linear is not recommended when looking for realistic, smooth sounding results but can be useful when a non-realistic behavior is desired.

• Custom is very useful when a specific behavior is required. This allows the game designer to draw a curve that represents the behavior of sound over distance and does not have to be rooted in real-world behavior. A sound could get louder as you get further away from it for instance.

While some game engines allow for a more flexible implementation, unfor-tunately, at the time of this writing Unity only implements audio sources as spheres. This can create issues when trying to cover all of a room, which

THE AUDIO ENGINE AND SPATIAL AUDIO

49

doesn’t happen to be circular in shape, which of course is most rooms. This leaves us with two options, leaving the corners of the room uncovered or increasing the radius of the shape to the so that it encompasses the entire space, but the sound will also spill over into the next room or outside area.

Figure 4.4

Figure 4.5

50

THE AUDIO ENGINE AND SPATIAL AUDIO

Although Unity does not allow one to natively alter the shape by which the audio spreads out into the level, other game engines and audio middleware allow the designer to alter the shape of the audio source. Other game engines and audio middleware, however, do, and other shapes are available.

b. Sound Cones – Directional Audio Sources

Sound cones allow the game designer to specify an angle at which the sound will be heard at full volume, a wider angle where the sound level will begin to drop and an outside angle where sound might drop off completely or be severely attenuated. This allows us to create directional audio sources and can help solve some of the issues associated with covering a square or rectangular area with spherical audio sources.

Sounds cones are particularly useful when we are trying to draw the player to a certain area, making it clearer to the player as to the actual location of the audio source.

Sound cones are very useful and can be recreated using a little scripting knowledge by calculating the angle between the listener and the sound source and scaling the volume accordingly.

Audio Source

Inner Radius:

sound playing at full volume

Outer Radius:

sound playing at lower volume

Distance From Source

Outer Cone

Inner Cone

Figure 4.6

c. Square/Cube

As the name implies, this type of audio source will radiate within a square or cube shape, making it easier to cover indoor levels. There again we find a minimum and maximum distance.

THE AUDIO ENGINE AND SPATIAL AUDIO

51

Inner Square:

sound plays at full volume

in this area

Outer Square:

sound fades

Figure 4.7

d. Volumetric Sound Sources

Volumetric is a somewhat generic term for audio sources that evenly cover a surface area – or volume – instead of emanating from a single point source.

Some game engines allow the game designer to create very complex shapes for volumetric audio sources, while some stay within the primitive geometric shapes discussed earlier. Either way these shapes are useful for any situation where the audio needs to blanket a whole area, rather than coming from a single point in space, such as a large body of water or a massive engine block.

Volumetric sound sources can be difficult to model using Unity’s built in tools, but a combination of a large value for the spread parameter with the right value for the spatial blend may help.

e. 2D, 3D or 2.5D Audio?

Most sound designers when they start working in the gaming industry understand the need to have both non-localized 2D sounds, such as in-game announcements, that will be heard evenly across the level no matter where the players are, as well as the need to have 3D localized audio files, such as a sound informing us as to the location of a pickup for instance, only audible when close to these objects and having a clear source of origin. Why, however, Unity gives the designer the option to smoothly go from 2D to 3D may not be obvious.

The answer lies in a multitude of possible scenarios, but one of the most common ones is the distance crossfade. Distance crossfades are useful when the spatial behavior of a sound changes relative to distance. Some sounds that can be heard from great distances will switch from behaving as 3D sound sources, clearly localizable audio events, to 2D audio sources when heard up close. A good example would be driving or flying toward a thunderstorm.

From miles away, it will appear to come from a particular direction, but when

52

THE AUDIO ENGINE AND SPATIAL AUDIO

in the storm, sound is now coming from every direction and is no longer localizable. In many cases, however, it is worth noting that different samples will need to be used for the far away sound and the close-up sound for added realism. In our case, a distant thunderstorm will sound very different from the sound of the same storm when ‘in’ it.

Another situation where you might want to have a sound neither fully 2D

nor 3D is you want a particular audio source to be audible from anywhere in a large map but only become localizable as you get closer to it. In such a case, you might want to set the audio source to a spatial blend value of 0.8. The sound will be mostly 3D, but since it isn’t set to a full value of 1, it will still be heard across the entire level.

4. Features of Unity’s Audio Engine

Unity’s audio engine also provides us with additional features for audio processing, audio filters, and audio effects and the ability to create multiple audio mixers for flexible routing. A lot of these features will be explained in more detail in further chapters, such as the adaptive mixing chapter.

a. Audio Filters

Audio filters may be applied to an audio source or listener as components, and one should be mindful of the order in which they are added as the signal will be processed in that order. It is always possible to re-arrange components, however, by clicking on the component’s gear icon at the top right of the component and selecting either the move up or move down option.

An audio filter applied to the listener will be heard on every audio source in the level.

Unity provides the following effects as audio filters:

• Audio Low Pass Filter.

• Audio High Pass Filter.

• Audio Echo.

• Audio Distortion Filter.

• Audio Reverb Filter.

• Audio Chorus Filter.

b. Audio Efects

Audio effects are applied to the output of an audio mixer group as individual components, and, as was the case for audio filters, the order of the components is important, and the signal will be processed in the order through which the components are ordered.

Audio effects are discussed in more detail in the adaptive mixing chapter, but their list includes:

THE AUDIO ENGINE AND SPATIAL AUDIO

53

• Audio Low Pass Effect.

• Audio High Pass Effect.

• Audio Echo Effect.

• Audio Flange Effect.

• Audio Distortion Effect.

• Audio Normalize Effect.

• Audio Parametric Equalizer Effect

• Audio Pitch Shifter Effect.

• Audio Chorus Effect.

• Audio Compressor Effect.

• Audio SFX Reverb Effect.

• Audio Low Pass Simple Effect.

• Audio High Pass Simple Effect.

c. Audio Mixers

Unity also features the ability to instantiate audio mixers, which allows us to create complex audio routing paths and processing techniques and add effects to our audio for mixing and mastering purposes.

When you create an audio source, you have the option to route its audio through a mixer by selecting an available group using the output slot (more on that in the adaptive mixing chapter).

Groups can be added to the mixer to provide additional mixer inputs.

Groups can be routed to any other audio mixer present in the scene, allowing you to create very intricate mixing structures.

Please refer to the adaptive mixing chapter for an in-depth discussion of audio mixers in Unity.

2. Audio Localization and Distance Cues

In order for us to understand the game engine’s implementation of 3D audio it is useful to first understand the way human beings relate to distance and direction. How do we estimate distance and tell the origin of a sound in a complex 360-degree real-world environment? A thorough study of distance and localization cues is well beyond the scope and ambitions of this book, however, it is extremely helpful to understand the fundamental concepts involved with the perception of both in order to take advantage of the current and future spatial audio technologies, especially as these technologies have implications for both game and sound design.

When discussing 360 audio, it is common to express the position of sound sources in terms of Azimuth, the angle between the sound source and the listener on the median plane; elevation, the angle between the listener and the audio source on the horizontal plane and of course distance, which we will begin our examination with.

54

THE AUDIO ENGINE AND SPATIAL AUDIO

Figure 4.8

1. Distance Cues

In order to evaluate the distance from an object in the real world, humans rely on several cues. These cues, in turn, when recreated or approximated virtually, will give the listener the same sense of distance we would experience in our everyday life, allowing us as sound designers to create the desired effect. The main distance cues are:

• Loudness or perceived amplitude.

• Dry to reflected sound ratio.

• Timbre.

• Width (the perceived size of a sound in the audio field).

a. Loudness

Although loudness may seem like the most obvious cue as to the distance of a sound, it does not on its own tell the whole story. In fact, simply turning down the volume of an audio source and nothing else will not necessarily make it seem further away; in most cases it will only make it softer. The ability of human beings to perceive distance is fundamentally and heavily dependent on environmental cues and, to a lesser degree, some familiarity with the sound itself. Familiarity with the sound will help our brain identify the cues for distance as such rather than mistaking them as being part of the sound.

Physics students learning to understand sound are often pointed to the inverse square law as to understand how sound pressure levels change with distance. The inverse square law, however, is based on the assumption that waves spread outwards in all directions and ignores any significant environmental factors. In such conditions an omnidirectional sound source will decay by 6dB

THE AUDIO ENGINE AND SPATIAL AUDIO

55

for every doubling of distance. This is not a very realistic scenario, however, as most sounds occur within a real-world setting, within a given environment where reflections are inevitable. Furthermore, the pattern in which the sound spreads is also a significant factor in how sound decays with distance. Most audio sources are not truly omnidirectional and will exhibit some directionality, which may vary with frequency. If the audio source is directional instead of omnidirectional, that drop changes from 6dB per doubling of distance to about 3dB (Roginska, 2017).

Loudness is only a part of the equation that enables humans to appreciate distance. Loudness alone is most effective when the listener is very close to the sound source and environmental factors such as reflections are negligible.

Research also suggests that when loudness is the main factor under consideration, human perception does not necessarily agree with the inverse square law, as for most people a doubling of distance is associated with a doubling of amplitude, which is closer to 10dB (Stevens & Guirao, 1962; Begault, 1991).

b. Dry to Refected Sound Ratio

Another key factor in the perception of distance under non-anechoic conditions, that is to say in any reflective environment or real-world conditions, is the ratio of direct sound to reflected sound. The ratio is a function of the distance between the audio source and the listener and provides us with important cues when it comes to distance. The ratio of reverberated to direct signal or R/D ratio is often used as a way of creating distance within mixes using reverberation, and most game engines will implement some technology to emulate this phenomenon. The exact science behind how to calculate the ratio of reflected to direct sound is quite complex, but it is not necessary to be scientifically accurate when doing sound design, with the exception, perhaps, of doing sound design for simulations. As we get further away from the sound the ratio of reflected sound present should increase and decrease as we get closer to it.

It is also worth mentioning that once we go past a given distance, the sound will attain an event horizon point where it simply doesn’t get much softer in spite of an increase in distance between the sound and the listener. This point, sometimes referred to as critical distance or reverberation radius, happens when the sound heard is mostly made up of reflections and the dry signal’s contributions to the sound become insignificant in comparison.

c. Low Pass Filtering With Distance

In the real world, high frequencies get attenuated with distance due to air absorption and atmospheric conditions. The amount of filtering over distance will vary with atmospheric conditions, and a loss of high frequency might also be due to the shorter wavelength of these frequencies and their inherent directionality. There, also, our purpose is not the scientific simulation of such a phenomenon but rather to take advantage of this phenomenon to better simulate distance in our games.

56

THE AUDIO ENGINE AND SPATIAL AUDIO

d. Spatial Width

Environmental factors, especially reflections, may also account for other less obvious phenomena that are somewhat subtle but when combined with other factors will create a convincing overall effect. One such factor is the perceived width of a sound over distance. Generally speaking, as we get closer to a sound, the dry signal will occupy more space in the sound field of the listener and become smaller as we get farther away. This effect might be mitigated when the wet signal is mixed in with the dry signal, however. This is relatively easy to implement in most game engines, certainly in Unity as we are able to change the spread property of a sound source, as well as its 2D vs.3D properties. Such details can indeed add a great level of realism to the gameplay. In spite of the mitigating effect of the wet signal, generally speaking, the overall width of a sound will increase as we get closer to it. Most game engines, Unity included, will default to a very narrow width or spread factor for 3D sound sources. This setting sounds artificial for most audio sources and makes for a very drastic pan effect as the listener changes its position in relation to the sound. Experimenting with the spread property of a sound will generally yield very positive results.

Another such factor has to deal with the blurring of amplitude modulation of sounds as they get further away. This can be explained by the increased contribution of the reverberant signal with distance. Reflections and reverberation in particular naturally have a ‘smoothing’ effect on the sound they are applied to, something familiar to most audio engineers. A similar effect happens in the real world.

2. Localization Cues

In order to localize sounds in a full 360 degrees, humans rely on a different set of cues than we do for distance. The process is a bit more complex, as we rely on different cues for localization on the horizontal plane than we do on the vertical plane, and although spatial audio technology is not entirely new – ambisonic recordings were first developed in 1971 for instance – only recently has the technology both matured and demanded wider and better implementation.

Additionally, the localization process is a learned one. The way humans localize sounds is entirely personal and unique to each individual, based on their unique dimension and morphology, which does make finding a universally satisfying solution difficult.

a. Localization on the Horizontal Plane

When considering spatial audio on the horizontal plane, the main cues tend to fall into two categories: interaural time difference – the time difference it takes for the sound to reach both ears – and interaural intensity difference, also sometimes referred to as interaural level difference, which represents the difference in intensity between the left and right ear based on the location of the audio source around us. Broadly speaking, it is accepted that the interaural

THE AUDIO ENGINE AND SPATIAL AUDIO

57

intensity difference is relied upon for the localization of high frequency content, roughly above 2Khz, while the interaural time difference is more useful when trying to localize low frequencies. At high frequencies a phenomenon known as head shadowing occurs, where the size of an average human head will act as an obstacle to sounds with short wavelengths, blocking high frequencies. As a result, the difference in the sound at both ears isn’t just a matter of amplitude, but the frequency content between each ear will also be different.

At low frequencies that phenomenon is mitigated by the longer wavelengths of the sounds, allowing them to refract around the listener’s head. For low frequencies the time difference of arrival at both ears is a more important factor.

Figure 4.9

There are limitations to relying solely on IIDs and ITDs, however. In certain situations, some confusion may remain without reliance on additional factors.

For instance, a sound placed directly in front of or in back of the listener at the same distance will yield similar results for both interaural time difference and interaural intensity differences and will be hard to differentiate. In the real world, these ambiguities are resolved by relying on other cues, environmental, such as reflections, filtering due to the outer ear and even visual cues.

Figure 4.10

58

THE AUDIO ENGINE AND SPATIAL AUDIO

b. Localization on the Vertical Plane

Neither IID and/or ITD are very effective cues for localizations on the vertical plane, as a sound located directly above or below the listener may yield the same data for both. Research suggests that the pinna – or outer ear – provides the most important cues for the localization of sounds on the vertical plane.

This highlights the importance of the filtering that the outer ear and upper body structure perform in the localization process, although here again environmental factors, especially reflection and refraction, are useful to help with disambiguation.

3. Implementing 3D Audio

3D audio technologies tend to fall in two main categories, object-based and channel - based. Object-based audio is usually mono audio, rendered in real time via a decoder, and it relies on metadata for the positioning of each object in a 3D field. Object-based technology is often scalable, that is, the system will attempt to place a sound in 3D space regardless of whether the user is playing the game on headphones or on a full-featured 7.1 home stereo system, although the level of realism may change with hardware.

Channel-based audio, however, tends to be a bit more rigid, with a fixed audio channel count mapped to a specific speaker configuration. Unlike object-based audio, channel-based systems, such as 5.1 audio formats for broadcasting, tend to not do very well when translated to other configurations, such as going from 5.1 to stereo.

In the past few years, we have seen a number of promising object-based-audio technologies making their way into home theaters such as Dolby Atmos and DTS:X. When it comes to gaming, however, most engines implement 3D localization via head related transfer functions or HRTFs for short. When it comes to channel-based technology, ambisonics have become a popular way of working with channel-based 3D audio in games and 360 video.

a. Object-based Audio and Binaural Renderings

The most common way to render 3D audio in real time in game engines relies on HRTFs and binaural renderings. A binaural recording or rendering attempts to emulate the way we perceive sounds as human beings by recording IID and ITD

cues. This is done by recording audio with microphones usually placed inside a dummy human head, allowing the engineer to record the natural filtering that occurs when listening to sound in the real world by capturing both interaural time differences and interaural intensity differences. Some dummy heads can also be fitted with silicone pinnae, which further records the filtering of the outer ear, which, as we now know, is very important for localization on the vertical

THE AUDIO ENGINE AND SPATIAL AUDIO

59

plane, as well as disambiguation in certain special cases, such as front and back ambiguity.

Head related transfer function technology attempts to recreate the ITD

and IID when the sound is played back by ‘injecting’ these cues into the signal, via a process usually involving convolution, for binaural rendering.

In order to do so, the cues for localization are first recorded in an anechoic chamber in order to minimize environmental factors, by using a pair of microphones placed inside a dummy’s head. The dummy’s head is sometimes mounted on top of a human torso to further increase realism. A full bandwidth audio source such as noise is then played at various positions around the listener. The dummy, with microphones located in its ears, is rotated from 0 to 360 degrees in small increments in order to record the IID

and ITD cues around the listener. Other methods and material may be used to accurately collect this data. This recording allows for the capture of IID

and ITD at full 360 degrees and if implemented can provide cues for elevation as well.

Figure 4.11

Once they have been recorded, the cues are turned into impulse responses that can then be applied to a mono source that needs to be localized in 3D via convolution.

60

THE AUDIO ENGINE AND SPATIAL AUDIO

Left ear

Right channel

Real time

Right ear

Amplitude

convolution

Left channel

Time

Signal to be

localized in 3D

Figure 4.12

HRTFs remain the most effective method to recreate 3D audio on headphones, but they do have some limitations. The main issue with HRTFs is that, localization being a learned process, it is unique to each individual.

The one size fits all approach of using an idealized dummy’s head to capture IIDs and ITDs simply doesn’t work well for everyone. Interaural differences are indeed different for everyone, and the cues recorded with one dummy may or may not approach yours. If they do, then the desired effect of being able to localize sound in 3D over headphones works quite well. If they do not, however, the audio source may appear to come from a different place than intended, or, worse, a phenomenon known as ‘inside the head locatedness’ may occur, in which the listener, unable to properly resolve the cues presented to them, will have the sensation that the sound is coming from inside their own head. It is interesting to note that research has shown that after prolonged exposure to any given set of HRTFs, even if the set initially did not match the listener’s morphology, localization accuracy will improve over time. Additionally, HRTF technology does suffer from a few additional challenges. Mono sources are best when working with HRTFs, and while some audio engines such as Unity do allow the spatialization of stereo sources, some will only process mono audio sources. While this limitation may be disappointing to sound designers initially, stereo sources, in order to be processed through HRTFs, would have to be split into two mono channels, each then rendered on both the left and the right headphone with the appropriate interaural differences, then summed together. The results are usually disappointing due to phasing issues. Another issue when it comes to HRTFs is the artifacts of the convolution process itself, which may somewhat degrade the quality of the sound. This loss of fidelity and potential artifacts might be most audible when dealing with moving audio sources. This may be most noticeable

THE AUDIO ENGINE AND SPATIAL AUDIO

61

for moving audio sources, which may in some cases add a slightly unpleasant zipping sound to the audio.

Lastly, HRTFs work best on headphones, and, when translated to stereo speakers, the effect is usually far less convincing, due in no small part to the cross talk between the left and right speaker, which is of course not present on headphones. Crosstalk greatly diminishes the efficacy of HRTFs, although some technologies have attempted to improve the quality and impact of HRTFs and binaural rendering on speakers.

In recent years we have seen a burst in research associated with optimizing HRTF technology. The ideal solution would be to record individualized HRTFs, which remains quite impractical for the average consumer.

The process is quite time consuming and expensive and requires access to an anechoic chamber. It is also quite uncomfortable as the subject needs to remain absolutely immobile for the entire duration of the process. Although fully individualized HTRFs remain impractical for the time being, developers continue to find ways to improve the consumer’s experience. This could mean offering more than one set of HRTF measurements to choose from, creating a test level to calibrate the HRTFs to the individual and calculating an offset or a combination of the previous elements.

In spite of these disadvantages, HRTFs remains one of the most practical solutions for delivering 3D audio on headphones, and provides the most flexibility in implementation, as most game engines natively support it, and there are a number of third-party plugins available, often for free.

Binaural rendering has also been shown to improve the intelligibility of speech for video conferencing applications by taking advantage of the effect of spatial unmasking. By placing sounds in their own individual location, all sounds, not just speech, become easier to hear and understand, improving the clarity of any mix.

b. Working With HRTFs

Whether a sound should be playing as 3D or 2D ideally should be known by the design stage, prior to implementation in the game. This will allow the sound designer to make the best decisions to optimize each sound. For any sound that requires 3D localization, HRTFs remain one of the best options.

However, not all audio files will react well to HRTF processing, and in some cases the effect might be underwhelming or simply ineffective. In order to get the best results, we should keep the following in mind: 1. HRTFs work best on mono signals. When doing sound design for 3D sounds, work in mono early in the process. This will prevent any

62

THE AUDIO ENGINE AND SPATIAL AUDIO

disappointing results down the line. Most DAWs include a utility plug in that will fold sounds to mono. It might be a good idea to put one on your master bus.

2. HRTFs are most effective when applied to audio content with a broad frequency spectrum. High frequencies are important for proper spatialization. Even with custom HRTFs, sounds with no high frequency content will not localize well.

3. When it comes to localization, transients do matter. Sounds lacking transients will not be as easy to localize as sounds with a certain amount of snappiness. For sounds that provide important locational information to the player, do keep that in mind. If the sound doesn’t have much in the way of transients, consider layering it with a sound source that will provide some.

c. Multichannel Audio and Ambisonics

a. Stereo

Although video games are highly interactive environments and object-based audio is a great tool to address these dynamic needs, channel-based audio, while not as flexible in some regards as object-based, still has a place in game audio and is very well suited to gaming in several regards.

Multichannel audio is more computationally efficient than object-based audio, and not every sound in a game needs to be 3D. Some of these 2D

sounds lend themselves well to stereo audio. Examples of 2D sound candidates in 3D environments include wind and some ambiences, announcers, user interface sounds and music, amongst others. A lot of these might even sound awkward if 3D localized. Wind, for instance, in real life, does not appear to emanate from a single direction in space, nor does it pan around as the listener moves about in a level. For 2D sounds, such as wind or in-game announcement, stereo files are usually well-suited and can still be adjusted at run time in the stereo field from within Unity using the stereo pan slider on the audio source the file is associated with.

b. Surround Channel-Based Formats: 5.1

The 5.1 standard comes to us from the world of movies and broadcast where it was adopted as a standard configuration for surround sound. The technology calls for five full spectrum speakers located around the listener and a subwoofer. The ‘5’ stands for the full speakers and the ‘.1’ for the sub. This type of notation is common, and you will find stereo configurations described as 2.0.

THE AUDIO ENGINE AND SPATIAL AUDIO

63

Figure 4.13

The main applications for 5.1 systems in games are monitoring the audio output of a video game and the scoring of cinematic scenes in surround. Most gamers, however, tend to rely on headphones rather than speakers for monitoring, but 5.1 can still be a great way for the sound designer to retain more control over the mix while working with linear cutscenes as well as making them sound much more cinematic. Video games mix their audio outputs in real time and do so in a way that is driven by the gameplay. Events in the game are panned around the listener based on their location in the game, which can sometimes be a bit disconcerting or dizzying if a lot of events are triggered at once all around the listener. Working with 5.1 audio for cutscenes puts the sound designer or mix engineer back in control, allowing them to place sounds exactly where they want them to appear, rather than leaving that decision to the game engine.

The viewer’s expectations change quite drastically when switching from gameplay to non-interactive (linear) cutscenes. This is a particularly useful thing to be aware of as a game designer, and it gives us the opportunity, when working with 5.1 surround sound, to make our games more cinematic sounding by using some of the same conventions in our mix than movie mixers may use. These conventions in movies were born out of concerns for story-telling,

64

THE AUDIO ENGINE AND SPATIAL AUDIO

intelligibility and the best way to use additional speakers when compared to a traditional stereo configuration.

In broadcast and film, sounds are mixed around the listener in surround systems based on a somewhat rigid convention depending on the category they fall into, such as music, dialog and sound effects. An in-depth study of surround sound mixing is far beyond the scope of this book, but we can list a few guidelines for starting points, which may help clarify what sounds go where, generally speaking. Do keep in mind that the following are just guidelines, meant to be followed but also broken based on the context and narrative needs.

FRONT LEFT AND RIGHT SPEAKERS

The front left and right speakers are reserved for the music and most of the sound effects. Some sound effects may be panned behind the listener, in the rear left-right speakers, but too much going on behind them will become distracting over time, as the focus remains the screen in front of the player.

Dialog is rarely sent to these speakers, which makes this stereo axis a lot less crowded than classic stereo mixes.

CENTER SPEAKER

The center speaker is usually reserved for the dialog and little else. By having dialog on a separate speaker, we improve intelligibility and delivery, as well as free up a lot of space on the left and right front speakers for music and sound effects. By keeping the dialog mostly in the center, it makes it easier to hear regardless of the viewer’s position in the listening space.

REAR LEFT AND RIGHT SPEAKERS

These are usually the least busy; that is where the least signal or information is sent to, save the subwoofer. They are a great way to create immersion, however, and ambiences, room tones and reverbs are often found in these speakers. If the perspective warrants it, other sounds will make their way there as well, such as bullets ricochets, impacts etc.

SUBWOOFER

Also referred to as LFE, for low frequency effects, the subwoofer is a channel dedicated to low frequencies. Low frequencies give us a sense of weight, and sending a sound to the LFE is a great way to add impact to it. It should be noted that you should not send sounds only to the subwoofer but rather use it to augment the impact of certain sounds. Subwoofers, being optimized for low frequencies, are usually able to recreate frequencies much lower than the traditional bookshelf type speakers, but their frequency response is in turn much more limited, rarely going above 150Hz. Additionally, the subwoofer channel often gets cut out altogether when a surround mix is played through a different speaker configuration, so any information sent only to the LFE will be lost.

THE AUDIO ENGINE AND SPATIAL AUDIO

65

Some mix engines or third-party audio middleware software will give the sound designer the ability to force certain sounds from the game engine to specific channels in a 5.1 configuration. It is recommended to keep the center channel for dialog and avoid routing music and SFX to the center speaker. The reason is that, having an additional speaker in the center, in front of the listener, may create a heavier-than-usual center image, since in stereo we are used to relying on the left and right speakers to create a center image. Relying on both the left and right speakers and the center speaker will make for a very strong center image. This may make the mix feel somewhat heavy in the front and overall unbalanced. Additionally, it will make the dialog easier to mix and hear if no other sounds or few sounds are sent to the center speaker.

Although more costly for the consumer due to the additional required hardware (speakers, multichannel capable sound card and amplifier), 5.1

audio does present some real benefits over stereo or quad type configurations and provides the game designer an alternative to headphones, especially when it comes to immersive audio.

Ambisonics

Although it was a technology studied primarily in academic circles for the longest time, support for ambisonics has become standard in game engines and audio middleware since the advent of virtual reality and 360 video. This technology offers an interesting alternative to stereo audio or HRTF/object-based audio. Ambisonic technology is the brain child of British engineer Michael Gerzon who developed a method to record and play back audio in full 360-degree surround. The technology is also speaker-independent and can easily be scaled to various configurations. Ambisonic recordings can be played on headphones via binaural rendering and on any multi-speaker configuration such as quad, 5.1 or 7.1. Ambisonic recordings can also be played on stereo speakers, but their impact greatly suffers as stereo speakers are not a favored way of delivering binaural renderings due to issues with crosstalk.

Note: when playing back ambisonic recording on headphones, you may expect the same side effects as you would with HRTFs, such as mixed results in accuracy.

Ambisonics is often described as a full sphere, surround format. Full sphere meaning the technology records signals in surround both on the horizontal plane, around the listener but also vertically above and below the listener.

This happens by recording multiple channels simultaneously, usually with a dedicated ambisonic microphone whose capsules are arranged in a tetrahe-dral formation. The accuracy of the recording and of the positioning of the elements in the 360-degree sphere around the microphone depends on the order of the recording. First - order ambisonic recordings rely on four channels to capture a full 360-degree sonic image. Second - order ambisonics use nine channels, and third - order ambisonic recordings rely on 16 channels, all the way to sixth-order, which uses 49 channels. The increase of complexity from

66

THE AUDIO ENGINE AND SPATIAL AUDIO

first- to second-order does yield additional spatial accuracy; however, recording, processing and implementation becomes complex due to the increased number of channels, and first-order ambisonics are the preferred format for games, Virtual Reality and 360 video.

Figure 4.14

Because of their ability to rapidly capture audio in full 360 degrees, ambisonics are a good option when it comes to efficiently recording complex ambiences and audio environments. By using a first order ambisonic microphone and a multitrack recorder, one can record a detailed picture of an audio environment in 360, with minimal hardware and software requirements. Ambisonics may also be synthesized in a DAW by using mono sources localized in 3D

around a central perspective and rendered or encoded into an ambisonics file.

Ambisonics recordings do not fall under the object-based category, nor are they entirely similar to some of the traditional, channel-based audio delivery system such as 5.1 Dolby Digital. As mentioned previously, ambisonics recordings do not require a specific speaker configuration, unlike 5.1 Dolby Digital or 7.1

surround systems, which rely on a rigid speaker structure. The ability of first-order ambisonic recordings to capture a full 360-degree environment with only four audio channels and the ability to project that recording on a multitude of speaker configurations is indeed one of the main appeals of the technology.

In fact, for certain applications ambisonics present some definite advantages over object-based audio. Recording or synthesizing complex ambiences that can then be rendered to one multichannel audio file is more computationally efficient than requiring the use of multiple audio sources, each localized in 360, rendered at run time. In most cases it is also faster to drop an ambisonics file in your game engine of choice than it would be to create and implement multiple audio sources to create a 360 ambience. Decoding an ambisonics

THE AUDIO ENGINE AND SPATIAL AUDIO

67

recording is a fairly efficient computational operation, and the load on the audio engine can be decreased by using an ambisonics recording over the use of several audio sources, each requiring to be localized in real time using HRTFs, for instance.

The most common format for ambisonics is known as the B format. It is comprised of four channels, labelled W, X, Y and Z. The W channel is a omni directional recording, the X represents the front-back axis, Y represents the left-right asset and Z the up and down axis. A raw recording done via an ambisonic microphone is often referred to as A format. A format files need to be decoded, usually in a B format. There are two B formats types, AmbiX and Fuma, which are similar but not interchangeable. An A format file can be turned into a B format file using a software decoder, not unlike a Mid/Side recording. Once it has been turned into the appropriate B format file (check with the documentation of the software you are using to find out which B format to use), the file is ready for use.

In Unity, ambisonics recordings must be played through a third-party plug in, such as Facebook’s oculus tools, which also includes additional tools as well such as a powerful spatializer.

Ambisonics are very efficient and offer the ability to capture a 360-degree audio sphere around a single, static point. That sphere can then be manipulated, usually via rotation, to match the viewer’s perspective and current point of view, dynamically adjusting to changes in the game or video. The computational savings of using ambisonics can be significant over dedicated mono sources that each would require to be spatialized in 3D and is a very good alternative when dealing with complex ambiences that would otherwise require many audio sources, each localized in 3D individually. There are limitations to what this technology offers, and these should also be noted in order to make the best possible choice for a given project.

Ambisonics recordings or files have limited interactivity. They do allow the user to rotate the recording to match the viewer’s perspective, but once recorded or encoded the spatial relationship between the events is set and cannot be changed.

Although this is somewhat subjective, it is also generally agreed that object-based audio is usually more precise than first-order ambisonics, and when more accurate positioning is required, object-based solutions might be better.

The audio in ambisonics recording is forever at ‘arm’s length’; no matter how far the listener walks toward the direction of an audio source in the 3D

world, they will never reach the actual position of that audio source. That makes ambisonics inappropriate for foreground elements that the player may be able walk up to, for which object-based audio is still the best solution.

In spite of these limitations, as pointed out earlier in the chapter, ambisonics remain a good option for working with complex, surround sound ambiences, with elevation information, while remaining a relatively inexpensive solution computationally.

68

THE AUDIO ENGINE AND SPATIAL AUDIO

4. Optimizing Sound Design for Spatialization

a. Putting It all Together

A hierarchy seems to naturally emerge when it comes to using and combining the technologies we just looked at in order to create an immersive environment.

Ambisonics provide us with a very efficient way of capturing or rendering a full sphere environment, well suited for backgrounds, ambiences and other non-primary audio sources.

Stereo files are well suited for 2D, non-directional audio, from environmental sounds to in-game announcements, music and dialog.

Object-based audio, using HRTFs, is usually best for foreground 3D

sounds and audio emitters.

By combining these technologies we can create a fully immersive audio environment, which will complement and possibly augment and elevate the visuals.

b. Working With 2D and Multichannel Audio

Stereo audio is a good option for a number of situations, in 2D or 3D environments. In a 3D world coordinate, stereo files are useful for:

• Ambiences, such as wind, rain, outdoors and city sounds.

• Music.

• UI (User Interface) sounds.

• In-game announcements, critical dialog, narration.

• Player’s own sounds, such as footsteps, Foley, breathing etc.

c. Working With Ambisonics

Ambisonics can fit within a hierarchy of sounds within a 3D environment as they are a very efficient way to encode complex 3D data at minimal computational costs. By encoding non critical sounds in 360 degrees on four channels we can save the number of 3D audio sources. Ambisonic files are useful for:

• Surround ambiences.

• Complex room tones.

• Synthesizing complex environments and rendering them to a single file.

d. Working With Object-Based Audio

Object-based audio, sounds that need to be localized in 2 or 3D by the player, is well suited for:

• Any sound that may give crucial information to the player.

• 3D emitters, such as birds in an outdoors environment, water dripping in a cave.

• Enemy weapon sounds, vehicles, AI characters, other players.

THE AUDIO ENGINE AND SPATIAL AUDIO

69

When combining these formats for our purposes, a hierarchy naturally emerges: 3D, object based – emitters, in world sounds

Stereo files – weather – announcements etc.

Ambisonic bed – ambiences

3D Object-based: world sounds, emitters

Stereo / Multichannel: weather, announcements

Ambisonic Bed: 360-ambiences

Figure 4.15

Conclusion

The audio engine is particularly complex sub system of the game engine, and regardless of the engine you are working with, as a sound designer and game audio designer it is important that you learn the features of the game engine you are working with in order to get the most out of it. Most audio engines rely on a listener – source – audio clip model, similar to Unity’s.

From this point on, every engine will tend to differ and offer its own set of features. Understanding spatial audio technology is also important to every sound designer, and spending time experimenting with this technology is highly recommended.

5 SOUND DESIGN – THE ART OF

EFFECTIVELY COMMUNICATING

WITH SOUND

Learning Objectives

In this chapter we look at the craft of sound design and attempt to demystify it.

We will ask what is efective sound design, how to properly select samples and tools for this trade and how to use them in common and less common ways to achieve the desired results.

By the end of this chapter we expect the reader to have a solid foundation on the topic and to be armed with enough knowledge to use a variety of tools and techniques. Whether you are a novice or have some experience with the subject, there is science behind what we do, how the tools are created and how we use them, but sound design is frst and foremost an artform and ultimately should be treated as such.

1. The Art of Sound Design

1. A Brief History of Sound Design

As we saw in Chapter one, video games are a relatively new medium, but sound design isn’t. It takes its roots in theatre and was used to augment the impact of dramatic scenes and help create immersion, before that term was even articulated. Early sound designers had to be crafty and create unique contraptions to create all types of needed sounds. Some devices became popular and even standardized, such as the aeoliphone or wind machine. The aeoliphone consisted of a rotary device, a wooden cylinder outfitted with wooden slats that the operator would use by using a crank to spin the slats against a rough canvas. The aeoliphone was used in both orchestral and theatrical settings, and by varying the speed at which the operator would crank the device, various wind types and intensities were possible.

This type of ‘contraption-based sound design’ was in use until and through most of the 20th century, certainly through the golden age of radio and early cartoons and movies. For a very long time indeed, this was the only way to create sounds from a stage or recording studio. (Keep in mind that it wasn’t

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

71

until the 1960s and 1970s that recording equipment became portable, cheap enough and reliable enough to allow audio engineers to record sound on location.)

One of the pioneers and master of these techniques applied to visual media was Jimmy MacDonald, the original head of the Disney sound effect department. MacDonald was also a voice actor, most notably the voice of Mickey Mouse. Since recording equipment was expensive, very bulky and therefore could not be moved out of the studio to record a sound, Mac Donald and his colleagues invented a multitude of devices and contraptions to create his sound world. These contraptions were then performed to picture in real time by the sound artist, which required both practice and expertise.

Disney’s approach was contrasted by the Warner Brothers team on their

“Looney Tunes” and “Merry Melodies” cartoons, as early as 1936. Sound designer Tregoweth Brown and composer Carl Stalling worked together to create a unique sound world that blended musical cues to highlight the action on the screen, such as timpani hits for collisions or pizzicato strings for tip toeing, together with recorded sounds extracted from the growing Warner Brother audio library. In that regard, Brown’s work isn’t dissimilar the work of music concrete pioneers such as Pierre Schaeffer in Paris, who was using pre-recorded sounds to create soundscapes, and Brown was truly a pioneer of sound design. Brown’s genius was to re-contextualize sounds, such as the sound of a car’s tire skidding played against a character making an abrupt stop.

His work opened the door to luminaries such as Ben Burtt, the man behind the sound universe of Star Wars.

Ben Burtt’s work is perhaps the most influential of any sound designer to date. While the vast majority of his work was done for movies, most notably for the Star Wars film franchise, a lot of his sounds are also found in video games and have influenced almost every sound designer since. Burtt’s genius comes from his ability to blend sounds together, often from relatively common sources, in such a way that when played together to the visual they form a new quantity that somehow seamlessly appears to complement and enhance the visuals. Whether it is the sound of a light saber or a Tie fighter, Burtt’s work has become part of our culture at large and far transcends sound design

A discussion of sound design pioneers would be incomplete without mentioning Doug Grindstaff, whose work on the original TV show Star Trek between 1966 and 1969 has also become iconic but perhaps slightly overlooked. Grindstaff ’s work defined our expectations of what sliding doors, teleportation devices, phasors and many other futuristic objects ought to sound like. Grindstaff was also a master of re-purposing sounds. The ship’s engine sound was created with an air conditioner, and he made sure that each place in the ship had its own sound. The engineering section had a different tonality than the flight deck, which was something relatively new at the time. It allowed the viewer to associate a particular place with a tone, and an avid viewer of the show could tell where the action was taking place without

72

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

needing to look at the picture. In that regard, Grindstaff ’s work was visionary and helped further expectations on the role of sound design in visual media.

2. Sound Design – Basic Considerations

In his essay Dense Clarity – Clear Density, sound designer and film editor Walter Murch pointed out that, over time, the soundtracks of movies have continued to increase in complexity and sophistication, from early movies requiring under 20 sounds for the entire soundtrack, to modern movies now requiring many thousands.

One of the most common mistakes sound designers tend to make when they start out is to attempt to make things sound a little too realistic and, ultimately, not quite interesting enough. The pursuit of realism, albeit a worthy one, is ultimately optional and sometimes futile, as ultimately underwhelming in most situations. This is true of both film, games and VR experiences. We might of course want the user experience to ‘feel’ real, but in order to achieve that we may have to take liberties with the real world. We are story tellers; serving the story, not reality, ought to be our primary concern. While this chapter focuses on gaming, most of the concepts here can also be applied to other visual media.

a. Efective Sound Design

Perhaps a good place to start a practical discussion of sound design is to attempt to answer the question: what is effective sound design? As the title of this chapter states, sound design is about effective communication through sound, for a given medium. Every medium and genre tends to have its own conventions, but there are a lot of commonalities across all.

Sound is a highly difficult thing to describe. It cannot be seen, easily measured or quantified. But certain adjectives or words resonate. Interesting sound design should have depth and texture. Even a seemingly simple sound may have several layers upon closer inspection and can sonically be detailed, creating a rich spectrum even if meant to be subtle. This approach to sound design is an ‘active’ one, where the sound designer seeks not only to match the visuals but enhance them, becoming a contributor to the overall narrative.

I often like to contrast the full and always exciting and dazzling sound world of Star Wars to the brilliant, wonderfully understated sound world of the movie No Country for Old Men by the Coen brothers. While Star Wars and the stunning world created by sound designer Ben Burtt is simply breathtak-ing, the universe of No Country for Old Men is sparse yet extremely detailed and deliberate and certainly has a lot of texture. It creates tension by letting the listener focus on the sound of a candy wrapper slowly expand after it is discarded and immerses us in its world by subtle mix moves that seamlessly take us in and out of objective and subjective space.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

73

These two very different approaches to sound design perhaps explain why it is so difficult to teach sound design in a systematic manner, since context and intention are so important to our craft. There are, however, certain concepts and techniques we can rely on when dealing with common sound design problems. Please note that the following is intended as a guideline, and that, each situation being different, we must ultimately rely on the conventions of the genre, our ears and taste.

When considering what makes effective and interesting sound design, here are a few points to consider:

1. Effective sound design is exciting, often bigger than life.

2. Effective sound design is congruent with the visuals. A great sound is useless if when put to visuals it doesn’t complement them well. Some sounds on the other hand will only come to life when played against the right visuals.

3. Effective sound design is always clearly legible, that is, its purpose or meaning in the game, scene or overall context should be clearly understood and unambiguous. Ambiguity arises when a sound could be attributed to more than one object in the scene or when the gamer or viewer is confused as to why the sound was played at all.

4. Effective sound design is stylistically appropriate. While a given sound may be extremely effective in the context of a medieval game, it may not work at all in the context of a science fiction piece. Another way to look at this is that a sound that may be extremely effective in the context of a retro 8bit game, would probably appear completely wrong, possibly comical, in a modern first-person shooter game. We must adhere to the conventions of the medium and genre, unless there’s a very good reason to break these.

5. Effective sound design provides information to the user. This can mean information about the object itself, such as its weight and texture, as well as the purpose of the object in the context of the game.

Is it a positive development? Does this object or character constitute a threat to me?

6. Complete silence should be avoided. Inexperienced sound designers may sometimes try to use silence as a dramatic device by turning off all sound effects and music in a scene. However, by inserting silence into a scene, the attention of the viewer/player will be diverted to the sounds in their immediate environment, turning their attention away from the game, and the desired impact is not achieved. From the sound of an air conditioner unit in the background to a car passing by, their attention might start to turn to their surroundings, effectively breaking immersion.

7. Always break complex sounds up into separate layers, each layer serving its own purpose. A gun for instance may be broken down into a very sharp snappy transient, a detonation sound and a sub layer,

74

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

altogether creating a rich full spectrum and impactful sound. Breaking sounds into layers is especially useful in game audio where variations of the same sounds are often required to break up monotony. This allows the sound designer to create multiple variations by swapping one or two layers at a time, therefore creating endless variations without sacrificing continuity and consistency.

Figure 5.1 Gunshot separated into three layers here 8. Avoid repeating samples. Certainly not back to back. Do not select or allow the game engine to select the exact same sound twice in a row or more. Hearing the same footstep sample four times in a row will sound artificial and synthetic. Your ear is quite sensitive to this sort of duplication and it immediately sounds artificial and breaks immersion.

b. Sound Design Guidelines

In addition to these guidelines, several general principles can be outlined that may help budding sound designers

The Size of an Object Can Often Be Related to the Pitch of the Sound The same sample played at different pitches will imply different sizes for the object that creates the sound. The high-pitched version of the sound will imply a smaller size, while lower-pitched versions, a larger size.

A car engine loop, if pitch shifted an octave, will tend to imply a much smaller object, such as a toy or RC model. Likewise, if pitch shifted down, it will imply a truck or boat.

The Mass or Weight of a Sound Is Often a Product of How Much Bottom End Is Present in the Sound

By adding bottom end, either via an equalizer or using a sub harmonic synthesizer, we can make objects feel heavier, increasing their perceived mass.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

75

Likewise, cutting the bottom end of a sound makes it feel lighter. This is often used in footsteps, for instance, where a smaller character’s footsteps may be high pass filtered in order to better match the size/weight of the character on the screen and make them appear lighter. Remember, however, that in order for an equalizer to be effective, there already has to be some energy in the frequency band you are trying to boost or cut. If there is no information there and you are trying to add weight to a sound, then rather than using an equalizer, use a subharmonic synthesizer plugin.

Transients Are Crucial to Sharp Detonation, Impacts and Percussive Sounds

Transients, sharp spikes in amplitude usually associated with the onset of percussive sounds, are what give these sounds their snappy and sudden quality.

Preserve them. Be careful not to over-compress for instance. By limiting the dynamic range of a sound it is easy to lower the amplitude spikes of the transients relative to the rest of the sound. Transients ultimately require dynamic range. For a snappy and impactful gun, make sure that the attack portion of the sound isn’t reduced to the point where you no longer can tell where the transient ends and where the rest of the waveform begin.

Figure 5.2

Softer Does Not Mean Further Away

Distance will be discussed further in the environment modeling section, but remember that distance is a product of several factors: Amplitude, certainly but also: wet to reverberant signal ratio, pre-delay time in the reverb, high pass

76

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

and low pass filtering and even the blurring of amplitude modulation. Without these other cues, lowering the amplitude of a sound will not make the sound appear farther away, only softer.

The Law of Two and a Half

Articulated by legendary sound designer and film editor Walter Murch when dealing with footstep sounds for his work on the film THX1138, this law can be applied to other contexts as well. The Law of Two and a Half states that our brain can keep track of up to two people’s footsteps at once, but once a third person’s footsteps are added to the mix, the footsteps are no longer evaluated individually but rather as a group of footsteps, a single event, at which point sync matters a lot less, or any sync point is as good as any. Walter Murch goes beyond footsteps, and he extrapolated his concept to other sounds. When the mind is presented with three or more similar events happening at once, it stops treating them as simultaneous individual events and rather treats them as a group. In fact, when we attempt to sync up frame by frame three or more character’s footsteps in a scene, the effect achieved will just be confusing and clutter the mix and ironically feel less realistic.

The Audio-visual Contract

There is a magic to seeing and hearing sound syncing up together on a screen.

The effect is different from either of these senses being stimulated independently. Michel Chion argued that when images and sounds are played in sync, the viewer’s experience transcends both sound and picture to create a new, trans-sensory experience. This phenomenon, known as multi - modal integration, allows us great latitude as sound designers and is a staple of the sound design experience. In a way, our brain, ears and eyes want to agree, and your senses fuse into a new one, where we as sound designers can insert our creative vision and sounds. There are limits to the power of this contract between senses, however. Synchronization between the audio and visual appears critical, as well as a basic level of congruency between the visuals and sound.

Do keep this in mind when doing sound design. You brain wants to believe.

Breaking those rules, however, will break the illusion, and your brain simply will discard the audio in favor of the visuals.

3. Getting the Right Tools

It is easy to get distracted by the overabundance of plugins available on the market and get overwhelmed by the sheer volume of it all. Getting the right tools is central to being successful as a sound designer. Rather than gathering as many processors as possible and never fully learning any of them, I recommend the opposite approach: learn a few key plugins and processors very well;

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

77

get to know them in depth. These are the plugins I would recommend getting the most intimate with.

a. Equalization

A clean, full featured transparent and ‘surgical’ equalizer, capable of precise, narrow, deep boosts and cuts. This will be very helpful when trying to clean up a sound by targeting individual frequencies or perform massive boosts and cuts in order to shape the sound into its final iteration. Advances in technology have made it possible to create equalizers, which can truly be used as sound shapers. Get one.

Conversely, get an equalizer that might not have the same capabilities as your surgical equalizer but that will add a little gloss or personality to your sound, such as a replica of any classic hardware Neve or Pultec models. You will use these for very different reasons such as for ‘broader strokes’, when a sound might need a little help standing out in a mix for instance or simply to be made a little more interesting.

b. Dynamic Range

A very precise and somewhat transparent compressor. Compression, past a point anyhow, is never really transparent, but you will want something that allows you to control the dynamic range of a sound without imparting too much of its own sound on it. Look for a compressor with independent attack and release time, knee and ratio controls. Control over attack and release time will help you manage transients and shape the overall dynamic range of a sound with greater accuracy.

As with the equalizer, do also get a more classic sounding compressor that might not have all the controls and flexibility of the one mentioned earlier but that will also impart to your sound a little more personality. There are many classic compressor software emulations available to the sound designer today. Some of the most commonly used ones include the UREI 1176 limiting amplifier or the mythical Universal Audio LA-2A leveling amplifier. These tend to sound quite musical and can be used to impart to the sounds a bit more personality, making them more interesting to listen to.

At times, especially with gunshots and explosions, you will want to make a sample as loud as possible. In such cases, a loudness maximizer will be extremely helpful. Loudness maximizers are also often used in music mastering. In music as in sound design, they allow us to increase the perceived loudness of a sound by raising the audio levels of the softer portions of a sound relative to its peaks. Loudness maximizers also allow us to make sure that no audio signals get past a certain level, which is in itself a very useful tool when trying to prevent signals from clipping or to remain within certain guidelines for broadcasting standards. Do be mindful of transients, as loudness maximizers tend to be rough on them and destroy them.

78

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

c. Reverberation

A good, realistic sounding convolution reverb to simulate real-world spaces. A lot of DAWs these days come with good reverb units. Convolution-based reverb plugins are usually the best at emulating real spaces. That is because convolution reverbs use actual recordings of indoor and outdoor spaces, which based on the quality of the recording and of the plugin can sound quite spectacular when applied in a mix. Convolution reverb plugins can be used for much more than the simulation of real-world environments and can be great for modeling classic reverbs, such as plates, spring reverbs or other classic gear but will probably be your go-to to add a convincing ambiences and spaces to your sounds.

You will also need a good procedural, ‘creative’ reverb plugin that can be used for realistic applications and also non-realistic effects, such as infinite reverb times, freezing effects, or adding a little shine to a sound. Some reverb processors will also allow you to pitch shift the reverb itself independently of the dry signal for added sparkle to your sounds, an effect you can always recreate by pitch shifting the reverb at a later stage with a separate pitch shifter. A procedural reverb is one where the reverberation is computed using algorithms that create the reflections from scratch, and they tend to give the sound designer more control than some convolution-based plugins. While non-convolution-based reverbs can be used to simulate real spaces as well, they are great as part of your sound design toolkit and are sound design tools in their own rights.

Reverb can be further processed to give us more exciting sounding results –

something that is often overlooked. Following a reverb plugin with chorus will often make the reverb wider; adding a flanger after a reverb plugin will make a somewhat dull and boring reverb more interesting to the ear by giving it movement and making it more dynamic. Reverb can also be ‘printed’

separately, that is rendered to an audio file and processed further (reversed, granularized etc.). The possibilities are endless and exciting.

d. Harmonic Processors

Harmonic processors are a generic term for distortion/saturation plugins. Distortion is an extremely helpful tool for any sound designer. Ideally you are looking for a plugin or plugins that can go from mild saturation to extreme distortion and are flexible enough to fit a wide range of situations. There are many different flavors of distortion available to the sound designer, most very useful, from saturators to bit crushers, so you will likely end up relying on a few plugins for distortion, but, as always, focus on a few choice plugins and learn them well.

e. Metering Tools

A LUFS-based loudness meter. LUFS meters have become the standard way of measuring loudness, and with good reason. They are much more accurate than previous RMS or VU meters and allow you to track the evolution of loudness of a sound or a mix over time with great accuracy. At some point after a

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

79

few hours of work, your ears will become less accurate and you might have a harder time keeping track of the perceived loudness of your audio assets.

This can be a critical issue, especially in gaming where multiple variations of a sound are often expected to be delivered. If a bit of stitched dialog sounds louder than the rest of the files it is meant to be triggered with, you will end up having to fix it at a later time, where it might not be as convenient to do so.

Although video games yet have to be as strictly standardized as broadcast in terms of expected loudness (broadcasting standards such as ITU-R BT1770

are more stringent), a good LUFS meter will also help you monitor the consistency of your mix, which does make it rather indispensable.

A good spectrum analyzer software. Rather than display the amplitude of the signal over time, which all DAWs and audio editors do by default, spectrum analyzers display the energy present in the spectrum over the full frequency range of the sample. In other words, they display the frequency content and change over time of a sound. This is an exceedingly helpful tool when trying to analyze or understand how a sound works. Some will allow you to only audi-tion a portion of the spectrum, very helpful if you are trying to focus on one aspect of the sound and want to isolate it from the rest of the audio. A good spectrum analyzer will make it easy to see with precision the frequency starting and ending point of filter sweeps; the behavior, intensity and trajectory of individual partials, and some will even allow you to modify, for instance, transpose selected partials while leaving the rest of the sound untouched. Whenever you wish to find out more about a sound, inspect its spectrum.

Figure 5.3

80

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

f. Utilities

A good batch processor. When working on games, you will inevitably end up working on large batches of sounds that need to be processed similarly. A good batch processor will be a massive time saver and ultimately help you make the most out of your time. Batch processors can perform functions such as conversion to a different format; applying a plug in, such as a high pass filter to clean up a number of audio files at once etc. Batch processing is also a useful tool when working on matching loudness levels across multiple audio files by applying a loudness normalization process. Batch processing can also be used to ensure clean assets are delivered by getting rid of silence on either end of the audio file or by applying micro fades at the beginning and end of the file to get rid of any pops and clicks.

The plugins listed earlier are certainly not the only ones you will need and add to your workflow. A multiband compressor, noise remover, delays and others will find their way into your list.

4. Microphones

There is no better way to create original content than by starting with recording your own sounds for use in your projects. Every sound designer should include in their setup a quick way to record audio easily in the studio, by having a microphone always setup to record. Equally important is being able to record sound on location, outside the studio. In both cases, the recording itself should be thought of as part of the creative process, and the decisions you are making at that stage, whether consciously or not, will impact the final result and how you may be able to use the sound. The following is not intended as an in-depth look at microphones and microphone techniques but rather to point out a few key aspects of any recording, especially in the context of sound effects recordings. The student is highly encouraged to study some basic microphone techniques and classic microphones.

a. Microphone Choice: Dynamic vs. Condensers

When in the studio, you are hopefully dealing with a quiet environment that will allow you a lot of freedom on how to approach the recording. Regardless of where the recording takes place, always consider the space you are recording in when choosing a microphone. In a noisy environment you may want to default to a good dynamic microphone. Dynamic microphones tend to pick up fewer details and less high-end than condenser microphones, which means that in a noisy environment, where street sounds might sneak in for instance, they might not pick up the sounds of the outside nearly as much as a condenser microphone would. Of course, they will also not give you as detailed a recording as a condenser, and for that reason condenser microphones are usually favored.

On location sound professionals often use ‘shotgun’ microphones, which are condensers, usually long and thin, with a very narrow pick up pattern, known as a hypercardioid polar pattern. They are very selective and are good

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

81

for recording sounds coming directly from the direction they are pointed to and ignoring all other sounds. They can also be useful in the studio for simple sound effect recordings, but then other types of condensers are usually favored, such as large diaphragm condensers.

Figure 5.4

Large diaphragm condenser microphones are a good go-to for sound effect and voice over recording. They are usually detailed and accurate and are well suited to a wide range of situations.

If you are in a quiet enough environment and are trying to get as much detail as possible on the sound you are trying to record, you may want to experiment with a small diaphragm condenser microphone, which tends to have better transient responses than larger diaphragm microphones and therefore tend to capture more detail.

Lavalier microphones, the small microphones placed on lapels and jackets in order to mic guests on TV talk shows and for public speaking, are usually reserved for live, broadcast speech applications. They can be a great asset to the sound designer, however, because of their small size, which allows them to

82

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

be placed in difficult or impossible to reach places with a regular microphone and therefore capture sounds from unusual perspectives.

Perhaps most importantly, especially as you are starting out and may not have access to a large selection of microphones, is to just record. Do not let lack of high-end equipment get in the way of your work, and use whatever you have at your disposal. Modern recording equipment, even some consumer devices, often provide recordings of good enough quality to work with, even if they may need to be processed a little more than sounds recorded under ideal situations on high-end equipment. So, record, record, record.

b. Mic Placement

Microphone placement is a crucial aspect of the recording, but there, also, do not overthink the technical side, and always focus on the creative aspect.

Ask yourself: how do you wish to use and therefore record the sound you are recording? If you are trying to record a sound up-close, try placing the mic about a foot away and experiment by moving the microphone around the sound source until you get the best results in your headphones.

Perhaps it would help to think of a microphone as a camera. When close up to a subject, you tend to get a lot of detail on that subject but on a relatively small area only. Pulling the camera out will reveal more about the environment around the subject and give us more context but at the expense of the previous level of detail. A microphone works in a similar way. By pulling the microphone away from the source, you will start to hear more of the environment, which may or may not be a good thing. Recording footsteps in a reverberant hallway might make for a great recording but capture too much reverberated sound for the recording to be useful in other situations.

You can also use a microphone to magnify sounds by placing it extremely close to the sound and bring out elements of it rarely heard in it. In some cases, this will also make the sound source appear significantly larger than it is and can be a great way to record unusual sounds. Lavalier microphones, with their small size, are especially useful in creative recordings.

These remarks and suggestions are to be taken as general guidelines, and every situation needs to be assessed individually. Do not place a condenser microphone very close to a very loud audio source hoping to capture more details; the air pressure of loud sounds can be very harmful to the microphone, and always keep safety in mind when recording, especially on location. It is easy to get lost in our sound worlds and forget about the world around us.

5. Sound Design – Before You Start

Working with the right assets is key when sound designing. That means finding the proper raw material and making sure it is suited for our purposes. Audio assets that are noisy or flawed can be difficult to work with and degrade the overall quality and impact of your work. The following are a few guidelines to keep in mind while gathering and preparing your raw assets.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

83

a. Always Use High Quality Material

Always use high quality material in the first place. A mediocre sounding audio file will usually result in mediocre outcome, even after processing. While processing an audio file might improve its quality and render it useable, you will end up spending a lot more time to obtain the desired results than if you had started with a clean file in the first place. Here are a few things to look for:

• Avoid heavily compressed audio file formats such as MP3, which may be acquired from online streaming services, even if it is otherwise the perfect sample. Even when buried in a mix, compressed sounds will stand out and weaken the overall result.

• Work with full bandwidth recordings. Are high frequencies crisp? Is the bottom end clean? Some sound effect libraries include recordings made in the 1960s and even earlier. These will inevitably sound dated and are characterized by a limited frequency response and a lack of crispness. If a frequency band is not present in a recording, an equalizer will not be able to bring it back, and boosting that frequency will only result in nothing at best or the introduction of noise at worst.

• For percussive sounds, make sure transients have been preserved/well recorded. Listen to the recording. Are the transients sharp or snappy?

Have they suffered from previous treatment, such as compression?

When in doubt, import the file in your preferred DAW and inspect the file visually. A healthy transient should look like a clean spike in amplitude, easily picked apart from the rest of the sound.

Figure 5.5

84

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

• Does the sound have baked in reverberation or other effects or recorded ambience that could prevent it from blending well within your mix?

Some recordings are made in environments that can be heard as roomy.

While there are de-reverberation plugins available, they are rarely entirely transparent and will sometimes impact the sound negatively in other ways.

• Is the recording noisy? If some noise is present but initially seems acceptable, that may no longer be true once dynamic range compression is applied, which will tend to bring up the softest parts of a sound and make the noise appear louder relative to the recording post compression. A de-noising stage might help.

b. Don’t Get Too Attached

Don’t get too attached to your material. Sometimes you just have to try another audio file, synth patch or approach altogether to solve a problem.

Every sound designer at some point or another struggles with a particular sound that remains stubbornly elusive. When struggling with a sound, take a step back and try something drastically different, or move on to something else altogether and come back to it later.

c. Build and Learn

You’re going to have to build a consequent sound effect library, usually consisting of purchased or downloaded assets (from online libraries, Foley artists) and your own recordings. Having hundreds of terabytes worth of sounds is absolutely useless if you cannot easily access or locate the sound you need.

There are tasks worth spending time during the sound design process; fum-bling through an unorganized sound library is not one of them. You may want to invest in a sound FX librarian software, which usually allows the user to search by tags and other metadata or simply organize it yourself on a large (and backed up) hard drive or cloud. The best way to learn a sound effect library is to use it, search through it, make notes of what interesting sounds are located where etc. In addition to learning and organizing your library, keep growing it. The best way to do it is to record or process your own sounds. Too much reliance on commercial libraries only tends to make your work rather generic and lacking in personality. Watch tutorials – especially Foley tutorials –

and always be on the lookout for interesting sounds.

d. Listen for the Expected and the Unexpected

Every processor, be it a compressor, equalizer or delay, will tend to affect a sound in more or less subtle and unexpected ways. For instance, a compressor will have a tendency to bring up the softer portions of a recording, which could, if some noise was present but very soft, make the noise a little more

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

85

obvious. Some plug ins will sometimes have a negative side effect on the stereo width of a sound without intending to affect it. Always compare your before and after sound by matching the output levels so that the processed sound isn’t louder or softer than the unprocessed. The loudest one will always tend to sound more appealing at first, which can be very misleading. Then try listening for different things at each comparison pass, by actively tuning your ears and attention.

e. Layers

Don’t try to find a single sample to fit a complex task, such as the roars and grunts of a large creature for instance. Instead try to figure out what are the different layers that could/would make up its sounds. For instance, if it is scaly, a creature might have a reptilian component, such as a hiss or a rattle; it if has a large feline-like build, it could also growl etc. A dragon might have all the earlier characteristics along with a gas-like or fire sound. It is very unlikely that a single source or layer would be enough to cover all these elements. Even if it did, it wouldn’t allow you the flexibility to change the mix between these layers to illustrate the various moods or states of our monster, such as resting, attacking, being wounded etc.

f. Be Organized

Asset management and version tracking are especially important to game audio, where dealing with multiple revisions is common place, and the sound designer is often dealing with hundreds, if not thousands of assets. Being organized means:

• Coming up with a sensible naming convention and sticking to it. Try to find something easy to understand and easy to adhere to. For instance, your ambiences may start with the letters AMB; gun sounds might start with the letter GUN etc. Additional information might be added to the name based on the context.

• Create a spreadsheet containing a list of all the sounds that need to be created, the name of the output file, the status of the progress on the sound, the number of variations needed if any as well as date of the last revision. An example of a spreadsheet used for this purpose can be found on the website for this book.

• Work with a version tracking software. There are a number of solutions out there, and the choice might not be up to you. A good version tracking system will make sure that all members of the teams are working with the latest version of the software and that no duplication will occur.

• Create a detailed design document. An example of a design document can be found on the companion website for this book. Its purpose is to

86

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

ensure that there is a clear artistic direction for the sound design and scope of the project and that the basic implementation and limitations of the audio engine are clearly outlined.

g. Communicate

With other members of your team and the client. Communication with your client is especially crucial during the pre-production process and continuously throughout production. Most people that aren’t sound designers have a difficult time articulating what they are looking for in terms of sound design or what they are hearing in their head. It is your responsibility as a sound designer to help them express and articulate their needs. Perhaps the client doesn’t know exactly what they are looking for, and your creative input and vision is why you are part of the team. When talking about sound, use adjectives, a lot of them.

Is the sound design to be realistic, cartoonish, exaggerated, slick, understated?

h. Experiment, Experiment, Experiment

Keep trying out new processes, watching tutorials of other sound designers and, of course, keep your ears and eyes open. Ideally, get a small, high quality portable recorder and carry it with you at all times or often; you never know when something interesting will come up.

2. Basic Techniques

Explaining the inner working of the processes and effects mentioned in this chapter would fall far beyond the scope of this book, instead, we shall focus on the potential and applications for sound design, from a user, or sound designer’s perspective.

1. Layering/Mixing

Layering or mixing is one of the staples of sound design. The process of layering allows us to break down a sound into individual parts, which can be processed independently and customized to best fit the visuals. Most sounds tend to be much more complex than they initially appear to the casual listener, and, although we perceive them as a single event, they are often the combination of several events. The sound of a car driving by is often the combination of the sound of its tires on the road, especially on some material such as gravel; then there’s the sound of the engine, which is itself a rather complex quantity; additional sounds such as the body of the car or the shock absorbers, breaks squealing and more can also easily become part of the equation. The relationship between these sounds isn’t a static one either, meaning that the intensity of the sound of the tires on the road depends on the speed of the vehicle, for instance, and we all know an internal combustion engine’s sounds can be very different based on at which gear and rpm the vehicle is going.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

87

A gunshot sound is often broken down into three or more layers, such as the initial transient, which gives the gun its ‘snap’; the sound of an actual detonation, as the round is being pushed through the barrel and, often, a low end layer or sub, which gives the sound weight and power.

By breaking a sound down into individual layers at the design process, it is also much easier to create variations, something often required in video games. If a sound is comprised of three layers, for instance, we can obtain multiple permutations by applying mild pitch shifting to one or more layer for each permutation, replacing one of the samples in a layer with a different but similar sounding one and much more.

2. Pitch Shifting

Pitch shifting is one of the most commonly used techniques employed by sound designers and one of the most useful ones too. As previously outlined, pitch is often related to the size of an object. This is especially useful in games where we might be able to use a sample in various contexts to score similar objects but of different sizes. It can also be used to great effect in creature sound design, where the growl of a cat, when pitch shifted down, will imply a much larger creature and might not, when put to visual, remind the player of a cat at all but of a giant creature.

There are several considerations to keep in mind of when working with pitch shifting as a technique. The first being that higher sampling rates, 88.2Khz and above, are usually desirable when dealing with pitch shifting, especially down pitching. The reason is simple. If you pitch shift a recording made at 44.1Khz an octave down, you essentially low pass filter your frequency content in addition to lowering its pitch. Any information that was recorded at 22Khz, when pitched down an octave is now at 11Khz, which will have a similar effect to removing all frequencies above 11Khz with a low pass filter. The resulting file might end up sounding a little dull and lose a bit of its original appeal. Doing the same thing with a file recorded at 88.2Khz means that your Nyqusit frequency, which was at 44.1Khz, is now at 22.050Khz, which still gives us a full bandwidth file and will not suffer from the perceived lack of high frequencies you would encounter with a standard resolution sample rate of 44.1 or 48Khz.

Always record files you plan on pitch shifting at high sampling rates if possible.

Not all pitch shifters work in similar ways, and their output can sound quite different as a result. Choosing the right type of pitch shifting algorithm can make the difference between success and failure. Some algorithms can change the pitch without affecting the overall duration, some will preserve formants, others will alter the harmonic content and can act as distortion processes, some are better with transients and are best suited for percussive material.

Most pitch shifters fall into these few categories:

a. Playback Speed Modulation

These work by changing the playback speed of the file, in the same way older reel to reel tape players could alter the pitch of the material by slowing down or

88

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

speeding up the playback speed. Playing a tape at half speed would make the audio twice as long and drop the pitch by about an octave, and, conversely, playing a tape at twice the speed would make the audio half the length and raise the pitch by an octave. This is clearly not a transparent process, and outside of very mild changes the artifact of the pitch shifting process will be heard. This is a very commonly available algorithm, and usually the default pitch shifting method in game engines such as Unreal, or Unity. The algorithm is cheap computationally, and when within mild ranges it is an effective way to introduce mild variations in a sound.

b. Granular Synthesis

Granular synthesis is a technique first articulated by physicist Denis Gabor, then developed by pioneers such as Iannis Xenakis and Curtis Roads to name a few.

Granular synthesis is a technique in which a sound is broken down into very small chunks, known as grains, typically ranging from 20 to 60ms and then manipulated at this granular level. Pitch Synchronous Overlap and Add, PSOLA, is the most commonly used technique for pitch shifting using granular synthesis. By transposing individual grains rather than the entire sound file, as with the technique discussed previously, we can change the pitch independently from duration. This technique for sound design is especially useful for sustained, harmonically rich material. It can be applied to transient rich audio; however, transient deterioration and smearing might occur. This is due to the fact that in order to keep the resulting audio sounding smooth, it has to be duplicated and overlapped.

Overlap: the signal is enveloped and duplicated, then added back together, 180° out of phase to avoid audible amplitude modulation artifacts from the enveloping process Grain Envelope

Grain Duration

+

=

Figure 5.6

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

89

Each grain has to be enveloped in order to prevent pops and clicks. If no overlap is present the enveloping will eventually be heard as amplitude modulation, and the audio will appear to have a tremolo effect applied to it due to the grain envelopes. Duplicating the signal out of phase and adding both together will mitigate, if not eliminate, the amplitude modulation effect.

The ideal number of overlaps is ultimately dependent on the desired transformation. Because overlapping means playing two or more copies of the signal against itself, a comb filtering effect can sometimes be heard. The grain duration will also affect the end result. Longer grains, 50ms and above, will tend to sound smoother but will negatively affect transients more so than shorter grain sizes. Conversely, shorter grain sizes tend to be better at preserving transient information but do not sound as smooth, and, in some cases, sidebands may be introduced as a byproduct of the grain enveloping process.

c. Fast Fourier Transform-Based Algorithms

There are a number of pitch shifting based algorithms available via Fourier-based transforms, the earliest one being the phase vocoder introduced in 1966 by Flanagan, one of the first algorithms to allow for independent control over time and pitch. Fourier-based algorithms share some similarities with granular-based algorithms due to the segmentation process (breaking down sounds into small windows of time) enveloping and overlapping.

Fourier-based algorithms are fundamentally different from granular-based ones, however. Fourier-based transforms occur in the frequency domain, where each frame of audio and its spectrum are analyzed and manipulated in the frequency domain. Granular synthesis in opposition processes signal in the time domain.

3. Distortion

Distortion is another extremely powerful process for sound design. To clarify, we are talking about harmonic distortion, which is a process where overtones are added to the original signal by one of several methods. In purely engineering terms, however, distortion occurs when any unwanted changes are introduced in a signal as it travels from point A to point B. The latter is of no interest to us in this chapter.

Distortion has many uses and comes in many flavors, from mild to wild sonic transformations. Some of these flavors or distortion types can be a little confusing to tell apart, especially as some of the terms to describe them are used liberally. Not surprisingly, the earliest forms of distortion came from analog processes and equipment, and their sounds are still very much in use and sought after today. Here is a non- exhaustive list of various distortion types and some of their potential applications.

90

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

a. Saturation

Saturation plug ins generally attempt to emulate the behavior of a signal pushed harder than the nominal operational level into tape or tube circuitry. The process is gradual and generally appealing to our ears, often described as warm.

Saturation also sometimes involves a compression stage, often referred to as tape compression, which comes from the signal reaching the top of the dynamic range of the device through which it is passed. This type of distortion is usually associated with a process known as soft clipping, which describes what happens to an audio signal when overdriven through tape or a tube amplifier, as illustrated in the following illustration. It can be contrasted to hard clipping, which has a much harsher sound and can be better suited for use as a distortion pedal for guitar.

Figure 5.7

Figure 5.8

Every saturation plug in tends to have a personality of its own, but saturation tends to be used in one of several ways:

• Mild saturation: mild saturation can be used to add warmth to otherwise rather bland or somewhat clinical sounds that tend to be the

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

91

trademark of some of the cheaper software synthesizers out there. It is a good compromise between severely altering the original sound –

something you might not always desire – but still injecting some excitement to it. It can also be used as part of a signal chain and combined with other distortion plugins sequentially to achieve a more severe overall distortion, which is often better achieved in stages rather than with one plug in driven hard.

• Heavy saturation: by applying more saturation or driving the signal harder into the plug in, a more obvious color can be imparted, which can be used to emulate the sound of older recordings or gear or the sound of a signal going through a device, such as a boombox. Any sound will also start to appear more aggressive.

Saturation is a gradual process as noted earlier, that is, a signal with a decent dynamic range will therefore sound slightly different at softer levels, where it will appear cleaner, than at higher levels, where it will sound warmer and more colored.

b. Overdrive

Sonically, overdrive falls between saturation and full-on distortion. It does come from driving a signal into an analog circuit hotter than the designers intended, which can be done with an analog preamplifier, for instance, and it is often used by guitar players to generate a clipped signal when entering the next stage of the signal stage.

Overdrive, sonically, tends to be a more extreme process than saturation.

c. Distortion

Distortion is indeed a type of distortion. Unlike saturation, it isn’t a gradual process, and the sonic transformations are more drastic-sounding than saturation or overdrive. It is often associated with a process known as hard clipping and is the type of process often used by guitar players to achieve the aggressive tones associated with heavy metal styles.

Figure 5.9

92

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

Distortion will severely change the harmonic content of a sound and will make any sound appear much more aggressive and increase in intensity dra-matically. In terms of sound design its applications as a process are numerous.

Distortion can be used to make any audio source more edgy and terrifying sounding. That can be very effective for creature sounds, where the voice, snarls or growls of a monster can be made more malevolent and angrier by being distorted. It can be used as part of a transformation process as well, where it is used to transform the sound of an existing recording, such as a cat meowing and turn it into a much more intimidating creature, especially if layered with one or two other samples as to not make the initial recording readily identifiable.

d. Bit Crushing

Bit crushing is a native digital signal processing technique. Digital audio signals are expressed in terms of sampling rate – the numbers of samples per seconds at the recording or playback stage – and the bit depth, which is the number of bits used to express the numerical value of each sample. As the number of bits increases so does the range of potential values, increasing the resolution and accuracy of the signal. The sampling rate relates to the frequency range of the audio signal, which is the sampling rate divided by two, while the bit depth relates to the dynamic range. Bit crushing plugins in fact often combine two separate processes, bit depth reduction and sample rate reduction or more. Bit crushers work by artificially reducing the number of possible values with which to express the amplitude of each sample, with the consequence of increasing quantization errors and reducing the fidelity of the signal. As the bit depth or resolution is decreased from the standard 24 bits to lower rates, such as 12, eight or lower, noise is introduced in the signal, as well as a decidedly digital, very harsh, distorted quality. It is interesting to note that, especially at low bit depths, such as ten and under, the signal becomes noisiest as it is at its softest, while the louder portions of the signal will remain (relatively) noise free. It is especially noticeable and interesting from a sound design perspective on slow decaying sounds, such as the sound of a decaying bell for instance, where the artifacts created by the bit depth reduction become more and more obvious as the signal decays.

Bit crushing, because of its very digital and harsh-sounding quality, is very well suited for sound design application dealing with robotics, non-organic or partially organic characters.

4. Compression

Compression is not always thought of as a creative tool in sound design but rather a utilitarian process, often misunderstood and somewhat overlooked by beginners. Compression is harder to hear than a lot of other processes, such as a sharp equalizer boost or cut, and as a result it is often misused. At its core compression is a simple concept, yet its implications are profound

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

93

and not always intuitive. Dynamic range compression is used to ensure that audio signals exceeding a certain level, usually determined by the threshold, are brought down by a certain amount, mostly determined by the ratio setting.

Figure 5.10

In practice, however, compression tends to actually bring up the softer portions of a sound, especially if the compression stage is followed by a gain stage.

This can make the sound feel louder or generally thicker and more interesting.

Be careful, however, not to overdo it.

Figure 5.11

94

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

Compression can be used in creative ways beyond just making sure that signals do not exceed a certain level, such as:

a. Blending Through Bus Compression

When layering multiple sounds or samples together, it is sometimes difficult to achieve a sense of cohesion, especially if the sounds are coming from completely different sources. The result may sometimes sound like several sounds on top of each other rather than a unified sound. In such cases compression can help. By bussing all the audio tracks together (see your DAW’s user guide for assistance) and applying compression to the entire bus, therefore all the sounds together, we can achieve a better sense of cohesion. In such a scenario, it usually is best to apply mild rather than heavy compression. It is usually enough to achieve the desired results without altering the overall sound too much. Try a mild compression ratio: 2:1

to 4:1, a rather high threshold, which would need to be adjusted on a case per case basis and medium to slow attack time (50ms and above) are good places to start. The release time can be adjusted to taste. A short release time will tend to make the audio feel a bit more natural sounding by releasing the compressor sooner and letting the signal return to its natural state quickly, while a longer release time will keep the compressor in longer and impart a bit more color. Additionally, if your compressor has a knee control, which controls how abruptly the compressor kicks in, a medium setting, implying a more gradual transition from compressed to uncompressed audio, is also desirable. Every situation is different, however you can look for about 3dB of gain reduction on the compressor meter, and you can follow it up by about as much gain. The result, if successfully applied, will bring up the soft portions of the sound by the amount of gain reduction dialed in, which will help the sound feel more cohesive. As always, when A/B’ing before and after, make sure the overall loudness for both settings, compressed and bypassed, are similar. A lot of compressors have an auto gain setting, where the compressor will automatically apply as gain to match the gain reduction achieved. While these settings can be useful when learning to work with compression initially, I would recommend applying gain post compression manually, which gives the sound designer more control over the process. The amount of gain reduction obtained through compression is not dependent on one setting alone. Although the ratio is one of most important factors in the process, it is a combination of all the previously mentioned factors.

Lowering the threshold will increase the amount of gain reduction obtained, as will reducing the attack time.

b. Transient Control

While there are dedicated transient shapers plugins available to the sound designer today, compression is a great way to manage transients. Especially

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

95

useful with gunshots and percussive sounds, a slow attack time, over 50ms, will let the initial transients pass through untouched but then compress the rest of the audio signal. This will increase the dynamic range between the transients and the rest of the sound. The result will be a snappier sounding, percussive sound. If, on the other hand, transients are a bit too harsh and they need to be tamed, a short attack time, followed by gain reduction, will tend to smooth them out. Experiment with the release time to get the desired result.

c. Infation

Drastic compression/limiting can be used to inflate the perceived amplitude a signal. In the same way that mild compression can be used to slightly bring up the softer portion of a signal relative to its peak, likewise, drastic compression or limiting can be used to inflate these same portions and drastically change the overall quality of a sound. This can be a particularly useful technique for explosions and impacts. This usually means lowering the threshold to a place where most of the signal is affected, and higher compression ratios are followed by a fair amount of gain. This will significantly inflate the perceived loudness of a sound.

5. Equalization/Filtering

Equalization is not always thought of as a creative tool, and it is often used in sound design and music rather as a corrective tool. That is, it is often used to fix an issue with a sound, either with the sound itself – which might appear muddy or too dull for instance – or with the sound in the context of the mix, where some frequency range might need to be tamed in order not to clash with other elements in the mix. However, especially with digital equalization algorithms becoming increasingly more transparent and allowing for more drastic transformations before audible artifacts start to appear, equalization has indeed become a full-fledged creative tool.

a. Equalization for Sound Design

One aspect of understanding how to use equalization is to understand the various qualities associated with each frequency band. These ranges are meant as guidelines.

Note: even the most sophisticated EQ cannot boost – or cut for that matter –

what isn’t already there. If you wish to add bottom end to a sound that has none, a massive boost anywhere below 200Hz will only bring up the noise floor and therefore degrade the signal. In such cases a subharmonic generator plug in might be better suited to synthesize these frequencies.

96

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

Figure 5.12

The previous chart is intended as a reference or starting point only, and the borders between terms are intentionally left somewhat vague, as the terms themselves are subjective. As always, with any aspect of audio engineering, please use your ears, and keep in mind that every situation and every individual sound must be assessed on an individual basis.

Weight: EQ can be used to modulate the weight of a sound. A very common occurrence is on footsteps samples. A high pass filter between 160–250Hz can be used to make the sound of heavy footsteps more congruent with the visual of a smaller person, such as a child for instance. Likewise, adding bottom end will have the effect of adding weight.

b. Resonance Simulation

A very often-quoted rule of working with EQ is to boost broad and cut narrow.

In this case, this is a rule we are going to break. When trying to emulate the sound of an object inside a box or tube, applying a short reverberation plugin will help but often will not fully convince. That is because 2D and 3D resonant bodies tend to exhibit narrow spikes in certain frequencies known as modes.

The amplitude and frequency of these modes depends on many factors, such as the dimension of the resonant body, its material, shapes and the energy of the signal traveling through it. A very good way of recreating these modes is by applying very narrow boosts; usually two or three are enough to create the necessary effect. As to where these frequencies should be, the best way is to figure it out empirically by using a spectrum analyzer on a similar sound and looking for where the modes are located. For best results, the frequencies

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

97

ought to be heard individually and not overlap each other, so make sure to use a very narrow boost for each boost. You may apply as much gain as 15dB per boost, so turn the audio output of the track down ahead of time.

6. Harmonic Generators/Aural Exciters

Related to equalization but not quite equalizers are a family of processes and plug ins that will synthesize harmonics where none or few are present. They differ from equalization insofar that they indeed are capable of adding information where none is present. The main common use for exciters is to generate high frequencies such as the now famous Aphex Aural Exciter or for subharmonic generators, to generate additional bottom end, such Wave’s MaxxBass plug in.

The main applications in sound design for aural exciters are found in the enhancement of otherwise dull recordings, mixes and potentially for audio restoration. However, the generation of artificial bottom end can be very useful to sound designers. The main point here, of course, being to add weight or impact through additional bottom end. Adding a subharmonic generator in a mix can be done in one of two ways:

1. Directly as an insert on a track where the sound file is. This will of course add weight and impact, but the drawback is that very low frequencies can be sometimes difficult to manage and tame in a mix and may demand to be processed separately.

2. As a parallel process, using an aux/send configuration where a portion of the signal is sent to the plugin via a mixer’s send. The benefit of this configuration is that the wet signal can be processed independently of the original audio by following the plugin by a dynamic processor, such as a compressor, which may be helpful in keeping your bottom end from getting overwhelming. Additionally to compression, a steep high pass filter set to a very low frequency, such as 30 to 45Hz, might prevent extremely low frequencies from making their way into your mix and eating up dynamic range without actually contributing to the mix, as most speakers, even full range ones, will not be able to reproduce such low frequencies.

On the other hand, these types of processors can also be very useful when trying to bring back to life an audio file that has suffered from high frequency loss either through processing or recording. Where an equalizer might only bring up noise, an aural exciter can often manage to at least partially bring back lost frequency content and give the sound a bit more crispness.

7. Granular Synthesis and Granulation of Sampled Sounds

Granular synthesis as a process for sound synthesis and modification was first articulated by Nobel prize recipient, Hungarian born physicist Dennis Gabor, in his 1946 paper “The Theory of Communication”, which was followed

98

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

with “Acoustical Quanta and the Theory of Hearing”. Gabor theorized that a granular representation of sound on the micro-scale was apt to describe any sound in a novel way, by looking at it and manipulating it on a micro time scale of short 10ms to long 100ms windows of time (the length may vary). He suspected that, at that scale, sonic manipulations that were otherwise difficult or impossible would become available. It took several decades, however, for the technology and science to catch up with Gabor’s vision and for the tools to become widely available to sound designers. Granular synthesis is a vast topic, and anyone curious to find out more about it is encouraged to study on further.

Even at the time of this writing, granular synthesis remains a relatively underused technique by sound designers, though it does offer some very powerful and wide-ranging applications and is already implemented in a number of major tools and DAWs. While usually considered a rather exciting technique, it often remains poorly understood overall. Granular synthesis can be a little confusing. It has its own terminology, with terms like clouds, evaporation or coalescence, and some of its theory remains somewhat counter-intuitive when put to practice.

The basic premise of granular synthesis is deceptively simple. Operating on the micro time scale, that is, a time scale shorter than individual musical notes, granular synthesis breaks down sound into very short individual micro chunks, roughly 10ms to 100ms in length, known as grains. Each grain has its own envelope to avoid pops and clicks, and a number of grains are fired at a rate called density, either synchronously or asynchronously. While envelopes do prevent unwanted clicks, they can also be used in creative ways.

Figure 5.13

a. Granular Synthesis Terminology

The content of each grain can vary greatly, and while we will focus on the granularization of sampled sounds in this chapter, grains can also be made up basic waveforms, such as sine or triangular waves. The most common synthesis parameters and terms employed in granular synthesis are:

• Density: this is the number of grains per second. Generally speaking, a higher density will create a thicker sounding output. Increasing the density, however, often comes at the expense of the computational cost.

• Grain duration: this is the individual length of individual grains. The useful range is usually between 10ms, although at this duration grains

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

99

might sound like clicks, up to 200ms, which is a relatively long duration for a grain. It is difficult for humans to perceive pitch below 50ms.

(Roads’96)

• Clouds: composer, researcher and mathematician Iannis Xenakis described clouds in relation to granular synthesis as a collection of particles part of a similar musical structures. Clouds, opposed to streams, are generally somewhat diffuse in their pitch boundaries, and since granular synthesis parameters are often randomized, clouds are more apt a description.

• Streams: while sound clouds are in nature nebulous and their boundaries often hard to distinguish and ever changing, streams in comparison are very focused, narrow sonic patterns.

• Opacity: generally associated with clouds, opacity refers to the ability of a cloud to mask other sounds.

• Evaporation: by gradually reducing the density of an audio stream down to nothing, it is possible to create the illusion of the sound disappearing, in a very different way than a simple fade out. Evaporation, as opposed to a fade out, is not about a gradual drop in amplitude of the audio file but rather a quick and somewhat randomized deconstruction (depending on the synthesis parameters) of the overall audio files.

• Coalescence: coalescence is the opposite of evaporation. By gradually increasing the density from nothing, it is possible to create the effect of a sound appearing out of thin air. Both evaporation and coalescence are very useful tools for magic spells and other unusual animations.

Here are a few basic principles that should help guide you in your explorations:

• The higher the number of grains per seconds, the thicker the overall sound.

• Adding randomization to the pitch and amplitude of each grain creates a more diffuse sound, often referred to as clouds, while no randomization at all will create very focused sounds, sometimes referred to as streams; this is especially true if the content of the grain is a basic waveform.

• When applied to sampled sounds, a medium grain density, played at the same rate as the original audio file with medium grain size and no randomization, will approximate the original recording.

b. Sound Design Applications of Granular Synthesis

Time Stretching – Pitch Shifting

As outlined earlier in this chapter, granular synthesis can be used for pitch shifting and time stretching applications through a technique known as Pitch Synchronous Overlap and Add or PSOLA.

100

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

This technique is particularly well-suited to sustaining, non-percussive sounds but can also be adapted to transient rich material. A number of time-stretching algorithms available in DAWs and plugins will often offer a granular option if they implement pitch shifting and time stretching.

Sample Manipulation/Animation

Granular synthesis is often used musically to synthesize endlessly evolving soundscapes or to add movement to a pad or texture. Likewise, we can use granular synthesis to breathe additional life into otherwise stale recordings or simply to modify an existing sound file in order to make it fit a change in the context of the game or scene. With this technique it is possible to take a recording of a small water stream and transform it into roaring rapids and everything in between. We can therefore thicken or thin out a recording according to the following parameters.

To thicken a sound, you will want to increase the density of the grains, adjust the grain size to a relatively long duration and add a random offset to both pitch and amplitude. As always, in order to keep the results from sounding too artificial, it is best to work with a relatively smooth amplitude envelope for each grain, such as a gaussian shape. If you start to notice a hollow ring, characteristic of comb filtering, try reducing the density, and if your software allows it, try randomizing the length of each grain slightly.

To thin a sound, you’ll want to decrease the density of the grains, as well as shorten the duration of each one and decrease the random factor of each grain’s pitch and amplitude. By reducing the density and shortening the grains slightly, you can take the intensity level below its original levels.

This technique of sample animation works particularly well on sounds that tend to be granular per nature, such as the sound of gravel or coins, but it works well on a lot of sources, such as water, wind or fire.

8. DSP Classics

a. Ring Modulation/Amplitude Modulation

Ring modulation is a process involving multiplying two audio signals together, one of them typically a sine wave, but it can be anything, and the other is the signal to be modulated, in order to create a hybrid signal. Ring modulation could be considered a form of distortion, but unlike the distortion processes described earlier, ring modulation will destroy the harmonic relationship of the original sound. More specifically, ring modulation will remove the fundamental frequency/frequencies of the original signal and add sidebands, a pair of new frequency components where the fundamental previously was. It’s easiest to predict the resulting spectrum if both signals are sinewaves. With a carrier at 100Hz and a modulator at 10Hz the resulting spectrum will be an output at: (Frequency of Carrier – Frequency of Modulator) + (Frequency of Carrier –

Frequency of Modulator)

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

101

Figure 5.14

Because ring modulation will remove the signal’s original fundamental and effectively destroy the harmonic relationship of the original signal, it is often used, still today, as an effective way to mask someone’s voice while retaining enough intelligibility for speech to be understood. Perhaps one of the most famous example of sound design using ring modulation is the original Doctor Who’s robotic villains, the Daleks.

Ring modulation is a subset of amplitude modulation, which has a similar outcome with the difference that the original’s signal fundamental frequency will be preserved.

b. Comb Filtering/Resonators

Comb filters take their names from the visualization of their output and the sharp, narrow resonance that characterizes them. These are obtained by adding to the signal a delayed version of itself, resulting in both creative and destructive interferences.

Figure 5.15

102

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

The simplest way to implement comb filtering is by duplicating a signal and applying a short delay to it in the DAW of your choice. Resonances will appear at a delay time of 0.1ms or more (“Clear resonances around 10Khz will start to appear at a delay time of 0.1ms up to about 50Hz for delay times of 20ms”

(Roads ’96)). A more classic and full featured implementation, allowing for more control, is through a delay line with feedback, as in the following: Figure 5.16

Comb filters are the building blocks of resonators. They are useful in many other applications, most notably reverberation. It is possible to control the resulting output resonance by adjusting the delay time and the amplitude of the resonances by adjusting the amount of feedback. Resonant frequencies are created at 1/delay time in milliseconds, and the higher the feedback, the more obvious the effect. As always with algorithms involving feedback, do exercise caution and lower your monitoring level.

The sound design applications of comb filters and resonators are plenty.

They are quite good at recreating synthetic or robotic resonances. When the resonant resulting frequencies have a low fundamental, they create deep metallic, somewhat ominous sounds. As the frequency of the resonances increases, they can be a pleasant yet still synthetic addition to a voice.

9. Reverberation

a. Indoors vs. Open Air

Reverberation is one of the most crucial aspects of sound design or music production and is often overlooked, or a very utilitarian approach is taken.

That is, a reverb is added to a scene without much consideration for whether it is the best possible choice for that scene or whether the settings that come with the preset are the best ones for our purposes. Another common mistake of sound designers starting out is often to forgo reverb completely when it isn’t an obvious type of reverberation implied on screen. An obvious reverb would be what we would expect to hear within a large stone cathedral, where sounds will sustain for several seconds after they were initially heard. A much less but just as crucial type of reverb is the one that would come with a small living room for instance, where the sound would not be expected to sustain for several seconds, and reverb, although present, is far more subtle. Yet, when no reverb is applied, the listener’s brain may have a hard time accept-ing that the sounds being heard at that moment all belong together and are

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

103

coming from the same space. Another situation where reverb is often ignored is outdoor scenes, as some sound designers only think of reverb as an indoors phenomenon. It certainly tends to be a more obvious phenomenon indoors, but reverb is a product of thousands or more individual reflections, and most, if not all outdoors settings will offer reflective surfaces. In other words, unless you are dealing with a scene happening inside an anechoic chamber, some kind of reverb needs to be added to your scene.

Although reverberation may appear to the listener as a single, unified phenomenon, it can be broken up into two parts, the early reflections, which represent the onset of the reverb and the late reflections, which are the main body of the reverb. Most plugins will allow the sound designer some control over each one individually, especially in the context of virtual reality, where that kind of control can be crucial in recreating a believable, dynamic space.

Historically, reverb was created using an actual space’s reverberant qualities, also known as a chamber. Sound was routed through a speaker in the space and was picked up by a microphone located strategically in another corner or at a distance from the speaker. Throughout the 20th century, other means of creating reverberation were developed, such as with the use of springs, still popular to this day with many guitar players and often included in guitar amplifiers; metal plates and then eventually through electronic means. To this day, a lot of plugins attempt to recreate one of these methods, as they all tend to have their own distinct sound.

Figure 5.17

In terms of recreating an actual space, which is often the case when dealing with picture, animation and games, reverbs that emulate actual spaces are usually the best choice. However, even these can be created in multiple ways.

Some reverberation plugins use a technique known as convolution. The main

104

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

idea behind convolution reverb is that an actual space sonic characteristics are recorded by setting up one or often more microphones strategically around the space and recording a burst of noise or a signal sweeping through every frequency up to the desired sample rate. This is known as the impulse response, which is a bit like the space’s DNA or fingerprint. To apply the impulse response to the desired sound, the process known as convolution is applied. The benefits of this technique are many. This allows sound designers to have access to spaces otherwise unattainable, such as world famous opera houses or recording studios. Additionally, this technique tends to sound very realistic and therefore convincing. By recording simultaneous impulse responses from different locations in the original space we can also give the sound designer access to multiple auditory perspectives and crossfade between them to best match a perspective change in a scene.

The drawback of this technique, if any, is that we have limited control over the reverb itself once it has been recorded. Most convolution reverb plugins will have a more limited set of parameters than some of their algorithmic counterparts. Algorithmic reverberation plugins are usually created using a combination of delay lines, comb and allpass filters. These algorithmic reverbs can be just as effective as their convolution counterparts, with the added benefit of giving the sound designer access to more parameters and therefore more control. Neither of these two categories is inherently better than the other, however, each has an edge when it comes to certain aspects. A convolution reverb might give you access to the Sydney Opera House’s audi-torium, as well as a vintage spring reverb from a mythical guitar amplifier. A good convolution reverb can sometimes get you the sound you are after more realistically than an algorithmic one. Algorithmic reverbs may not be as easily and readily able to recreate a famous or less famous space, for that matter, but they might be just as effective in some regards and allow the sound designer to have more control and fine tuning over the sound. In that regard, they can be used for sound design purposes in more creative and not necessarily realistic sounding ones.

Generally, when dealing with indoors reverb, the most important matters to consider are room size and materials. Some materials reflect sound more than others, typically harder materials, such as stone or marble, absorb very little sound and therefore reflect most of it, while softer materials, such as heavy curtains or carpeting, will absorb more sound and make for tighter-sounding environments. Another common misconception is that materials absorb or reflect sound evenly across all frequencies. Generally speaking, although there are exceptions, higher frequencies will have a shorter decay time than medium or low frequencies.

There are many types of reverberation units out there, and although more are software these days, they all tend to recreate a specific reverb type. The following is a discussion of parameters found in most reverb plugins, but note that your particular plugin may not implement all of the following, or some parameters may have slightly different names.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

105

b. Reverb Parameters

Reverb time / decay time: this is the most obvious and perhaps important setting, though by no means the only parameter in getting just the right sound. It defines how long the sound will persist in the environment once it has occurred. It is defined, scientifically, by the RT60

measurement. That is the time it takes for sound pressure levels to decay by 60dB once the source has been turned off or has stopped. It is usually measured using noise, abruptly turned off. Noise is useful because it allows for the presence and therefore measurement of all frequencies at once. Longer reverberation times will sound pleasant for music but can get in the way of intelligibility of speech. Keep in mind that unless you are working on an actual simulation, the best reverb for the job many not exactly be that of the space depicted in the scene.

Size: if present, this parameter will affect the dimension of the space you are trying to emulate. A larger space will put more time between individual reflections and might make the reverb feel slightly sparser and wider in terms of its spatial presence.

Predelay: measured in milliseconds, this parameter controls the amount of time between the original signal and the arrival of early reflections.

This setting is often set to 0 by default in a lot reverbs, and although it can be a subtle parameter to hear, leaving a reverb on with a predelay of 0 means that the original signal (dry) and the reverberant signal (wet) essentially are happening at the same time, at least as far as early reflections are concerned. This is not only a physical impossibility but also will have the tendency to make mixes a bit muddy, as the listener’s ear is given no time to hear the dry signal first on its own, followed closely by the early reflections. While this might seem like nitpicking, it is a much more important setting than it may appear. A shorter predelay time will be associated with smaller rooms or the listener being closer to a wall.

Density: controls the number of individual reflections, often for both the early and late reflection stage at once, which tends to make the reverb thicker-sounding, when up or thinner, when down. Some older plugins tend to sound simply better with the density knob all the way up, as the individual reflections can sound a bit lo-fi when heard individually.

Width: this controls the spread of the reverb in the stereo field. Generally speaking, a 0% setting will create a monaural effect, while a setting over 100% will artificially increase the width.

High cut: this setting usually controls the frequency at which the high frequencies will start decaying faster than the rest of the signal. This parameter sometimes includes additional controls, which can be used to give the sound designer more control, such as how quickly the high

106

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

frequencies will start to decay compared to the rest of the reverberant signal. This is especially useful to make the sound smoother and more pleasant, as too much high frequency content tends to make the signal sound harsh, a sound often compared to frying bacon.

Low cut: similar to the high cut setting, this controls the frequency at which low frequencies will start decaying faster than the main body of the reverb. This is very useful to add or regain clarity to your signal as, especially with longer decay times, too much reverb in the lower octaves of the spectrum will muddy up a mix and diminish the clarity of the overall signal. Going overboard with that setting can, however, make the reverb a bit thin sounding. As always, use your ears.

c. Reverberation for Environmental Modeling

The most obvious application of reverberation is as part of environmental modeling. Environmental modeling is not limited to reverberation, but reverberation is certainly one of its most crucial aspects. Being able to give a viewer or player the sense that all the sounds heard are happening in the proper acoustical space is key to immersion. Keep in mind, however, that the best reverb for a given situation isn’t necessarily the one that would actually recreate the depicted acoustical space exactly. There are several reasons for this, but usually it has to do with the mix, often for speech intelligibility.

The space the scene takes place in might have a very long decay time and very dense reflections, which could get in the way of the intelligibility of speech or simply make the mix a bit muddy. The choice of a reverb for a given scene is therefore also an artistic decision, for the sake of the mix or dramatic impact.

Reverb as a Tool for Blending

In some cases, reverberation can be applied gently to a group of sounds in order to blend them together better. This can be particularly useful when using the layering techniques outlined in Chapter five and trying to blend multiple samples together into a single sound. By applying the same reverb to several sounds happening at once we can trick the ear into believing that these sounds belong together since they will now have similar reflections applied.

In such a case we are not looking to use reverberation as a way to recreate an actual acoustical space, and therefore we are looking for a somewhat subtle effect. As such, you should try short to very short reverb times; start at around 0.8 seconds and adjust as needed. If the reverb is to be applied directly to a submix as an insert, then start with a 10% wet to 90% dry ratio and adjust as needed. If you are using an aux/send pair, send very little to the bus going to the reverb and raise the send level until the desired effect is achieved, usually right before the reverb starts to draw attention to itself and is heard clearly.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

107

The effect should be subtle, and reverb might be only one of the tools you are using to achieve the desired results (another common tool in this scenario is compression). In fact, the effect should be transparent to a casual listener or anyone not familiar with the session or sound and should really only be noticed when taken off.

d. Reverberation as a Dramatic Tool

While reverb is a crucial tool to recreate a sense of space and realism, it can also be used to great effect as a way to create drama and punctuation by drenching a sound, usually percussive one, in a very long reverb. Most reverb plugins and manufacturers make realistic reverb algorithms, but some focus on non-realistic reverb types, such as extremely long reverb times; some reverb units offer infinite decay times, others allow the users to freeze any portion of the decay and some still focus on creating spaces that simply could not exist in the physical world. Do feel free to blend reverb, layering a realistic impulse response in a convolution unit with a more exotic reverb such as a pitch shifted signal or a frozen or infinite reverb.

Reverberation is also a crucial, although not always obvious, aspect of the way we perceive certain sounds, the most obvious perhaps being gunshots. While there are many factors that can affect a gunshot, such length of the barrel and caliber of the round fired, gunshots sound very different and sound significantly softer when you take away environmental reflections.

10. Convolution

Convolution is by now well-known for its role in reverberation, and it is one of the most studied Digital Signal Processing techniques in the engineering world. Reverberation, however, is a small subset of what convolution can achieve. Here also, an in-depth study of convolution goes far beyond the scope of this book, but there are a few points we can make about this technique that will help anyone unfamiliar with the process and eager to find more.

Convolution is a technique where a hybrid sound is created from two input audio files. Usually one of them is designated as the impulse response –

or IR – and the other the input signal. Although convolution is often mathematically expressed in other ways (see brute force convolution), it is usually implemented as the multiplication of the spectra of both files used in the process. Convolution therefore requires a Fast Fourier Transform to take place first; to obtain the spectral content of both sound files, their spectra are multiplied together, then an inverse FFT has to occur before we can use the resulting hybrid output. The artifacts resulting from any FFT process (transient smearing, echo, high frequency loss etc.), therefore apply to convolution as well.

108

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

Figure 5.18

So, what does convolution sound like? Well, it primarily depends on the files used in the process, of course, as well as the settings used in the process for the available parameters, but, generally speaking, the qualities of one sound will be applied to another, especially in areas where the spectrums overlap. The sound of a human voice convolved with a snare drum hit will sound like the sound a human voice through a snare drum. Another very common example is the sound of someone singing convolved with the sound of a rapid noise burst left to decay in a cathedral, which will sound like that person singing in that cathedral. This is why convolution is such a great way to create reverberation, usually meant to emulate real spaces. And still another common application of convolution relevant to game audio is for the spatialization of monaural sources, via Head Related Transfer Functions.

A lot of plugins dedicated to reverberation actually allow you to use your own impulse responses as long as they are properly formatted. This essentially turns your reverberation plugin into a general-purpose convolution engine, which you can use for any of the purposes outlined previously. Additionally, fluency with a computer music language such as MAXMSP, Csound or Chuck will give you access to very flexible ways of working with convolution, among other things, and while these tools might seem off-putting to some, mastering one such tool is highly recommended to the adventurous sound designer.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

109

a. Optimization

There are a few universal principles about working with convolution that are very useful to understand in order to get the best results: 1. Convolution is the spectral intersection of two audio files; the most interesting results will occur when both the IR and the input file share a lot of common spectral data. In other words, convolution sounds most interesting when both files share similar frequency information.

If not, such as in the instance where one might convolve a kick drum with the IR of a triangle, since there is very little overlap in the frequency content of both signals, the output will simply sound like both sounds mixed together.

2. A loss of high frequency information is a rather common side effect of convolution, and you might need to compensate for that phenomenon. The reason is simple; multiplication of two spectrums means that frequencies strongly present in both signals will come out strongly in the output, and frequencies that aren’t will end up being less present in the output. Most sounds do not tend to have a lot of information in the last octave of human hearing – or not as much as other frequency bands, anyway. As a result, it is common for the output to be duller than either file used.

3. As with all FFT based processes, there is a tradeoff between time and frequency resolution (see the FFT explanation earlier); when dealing with a transient rich sound, if you have access to such parameters, it is best to use a shorter window. Otherwise, longer windows, 1,024

samples and over, will be better suited for frequency-rich sounds.

So, what are the applications of sound design beyond reverberation? There again there are many potential options, but here are a few applications where convolution might especially come in handy and where other traditional techniques might fall short.

b. Speaker and Electronic Circuit Emulation

This is a very common scenario for any sound designer: recreating the sound of a small speaker, portable radio, PA system etc. The traditional method involves band-limiting the output using an equalizer, adding some type of distortion and perhaps compression to the signal. While this might create okay, perhaps even good results at times, this technique often seems to fall a bit short. That’s partly due to the fact that while EQ and distortion will get us part of the way there, they usually are approximations of the sound. A better approach is simply to convolve the audio that we need to treat with the IR

of a small speaker such as the one we are trying to emulate. Of course, this does require the proper IR, and while I would recommend indeed to become

110

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

familiar with the process of recording simple impulse responses to use for convolution, some manufacturers actually specialize in convolution-based tools whose purpose is to emulate electronic circuits and speakers, but a lot of convolution-based reverb plugins will offer some less traditional IRs, such as those of small speakers, common electronic circuitry and other non-traditional impulse responses. These are usually great starting points and often do not require much additional processing to obtain a realistic sound.

c. Filtering/Very Small Space Emulation

Another common situation where convolution will come in very handy is in the emulation of very, very small spaces, such as a full body space suit for instance, where the subject’s voice should reflect the very small space their helmet represents. There again, a traditional approach was to use filtering or equalization in order to simulate this sound. However, a fully enclosed helmet can be thought of as a very tiny room. One option to simulate this sound is to scale down or shorten an available impulse response to an extremely short reverb time, perhaps 0.2 or 0.1 seconds and apply an extremely short predelay time, in the order of 1–2ms. Another option is to record the impulse response of such a helmet or similar object and convolve it with the sound to be treated. Something as trivial as a bucket might be extremely helpful in this situation.

d. Hybrid Tones

One of the most exciting applications of convolution from the standpoint of sound design is the creation of hybrid tones. In this case, the only limit is the sound designer’s imagination. Convolving the sound of speech with a wind gust will give speech an other-worldly, ghost-like quality. Convolving the sound of two animal calls will create a sound sharing the quality of both, possibly sounding like neither. The key here is to experiment and carefully choose the files you are going to process.

11. Time-Based Modulation FX

Time based modulation effects are a wonderful way to add movement to your sound and make a static recording feel a lot more compelling. There are a number of time-based effects that are classics by now and ought to be in every sound designer’s inventory.

a. Chorus

A chorus is one or multiple delay lines whose length is modulated, usually by a low frequency oscillator, which causes pitch fluctuations. The intention is to add slightly delayed duplicated copies of the signal at slightly different pitches.

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

111

Chorus is used to help thicken and often widen a sound. It is a good way to widen mono sounds especially or to take an otherwise mono source and make it a stereo sound. It was and still is widely used as a way to make mono synth bass sounds much more interesting, and some early commercial synthesizers, such as the original Juno series by Roland, derived a lot of their magic from their built-in chorusing units. Chorus can be applied to any sound source you wish to impart any of these qualities to and can also give the sounds it is applied to a dreamlike, psychedelic quality.

Figure 5.19

b. Flanger

Flangers are similar to chorus, they are a variable delay line, constantly modulated, usually within a 1–10ms range. At these times, the perceived effect is not that of a delay but rather a filtering instead of individual repetitions. Unlike in a chorusing unit, flangers include a feedback path, mixing the delayed signal or signals with the original one, creating resonances similar to that of a comb filter.

The filtering of the sound will depend upon the delay times and is due to constructive and destructive interference when the waves are added together. The small delay time means the duplicated signal’s phase will be different from the original. When the two are layered, destructive interference will create notches, a frequency where the signal is attenuated significantly. Because the delay time is modulated, usually by an LFO, the notches are constantly changing in frequency, which creates a dynamic signal and is a good way to add movement to a sound.

Figure 5.20

112

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

c. Phasers

Phasers work by duplicating the signal and changing the phase of the duplicates. Like flangers and choruses, they alter the sound through patterns of cancellation and reinforcement, but phasers rely on allpass filters instead of delay lines, which gives the sound designer a bit more control. Phasers are a staple of robotic sound design and are often added as part of a signal chain to make human voices robotic by adding a soft resonance in the high frequencies.

There is definitely an association with futuristic and sci-fi sounds, which can be both a little commonplace and delightful.

Figure 5.21

d. Tremolo

Tremolo is a relatively simple effect, which has been widely used by musicians for quite some time to add movement to their sound. It is a classic with guitar players and electric piano players. It usually consists of a low frequency oscillator that modulates the amplitude of a signal, giving the user control over depth of modulation, rate of modulation and sometimes the shape of the waveform used by the LFO.

While it is obviously a form of amplitude modulation, because it happens at sub audio rates, tremolo does not create sidebands as we saw earlier with ring modulation.

In sound design, its applications can be both subtle – as a way to add some movement to a sound – and more drastic in order to create dramatic effects.

When working with tremolo in a subtle way, to simply add some movement to a sound, we tend to work with slow rates under 1Hz and set the depth to a rather small value using a smooth shape for the LFO, such as a sine. Some tremolo plugins allow you to set the phase between the left and right channel independently, which can be helpful when working with stereo sounds or trying to widen a mono sound in stereo.

Tremolo can also be a far more dramatic effect, used to recreate or emulate drastic forms of modulations found in real-world sounds or to create new ones.

A rapid form of amplitude modulation can be used to recreate the sound of a rattlesnake for instance, especially if the rate and depth can be automated over time. If the plugin used allows for extremely fast rates, tremolo can be used to emulate the sound of insect wings flapping. Tremolo can be used very effectively with non-organic sounds too, such as hovering or flying vehicles where adding a fast tremolo to the sound of an engine or fly by can increase the perceived sensation of movement and speed, especially if the rate of the tremolo

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

113

follows the speed of the vehicle or rpm. Additionally, if the tremolo sounds irregular or unsteady, it will give the impression the vehicle is struggling.

12. Foley Recording

No discussion on sound design can be complete without a mention of Foley.

Named after Jack Foley, another pioneer of sound design, Foley is the process of recording sounds using real-world objects to create part of the soundtrack of a film or game. Foley can sometimes be literal – by literally recording the sound of an object depicted in a game by recording that same object in the studio – or by using a completely different object altogether to create the desired effect. Foley is very much in the vein of the ‘contraption based’ sound design mentioned at the top of this chapter and can be both rewarding and a quick way to get the sound you are looking for, if you know how to get it, of course. Foley is a skill in itself in the world of sound design, one that is developed over time, but anyone with a little imagination can take advantage of these techniques and reap some benefits rather quickly.

Let’s not forget that bringing in objects in the studio and sometimes building them was the only alternative for sound design until recording technology became truly portable. Of course, today this is not much of a concern, but the reason Foley is still relevant tends to fall into two main categories: If you have a basic recording setup, which can be as simple as a portable handheld recorder and as complex as a full recording studio, recording certain sounds tends to be as fast, if not faster, than looking for them in a sound effect library and has the undisputed benefit of making your sound design unique.

A sound pulled from a sound effect library may have been heard before in other shows and games, while your recordings are always unique, giving your sound design its own identity and personal touch. If it seems like pointing a microphone at a source to record that sound will be faster than looking it up in a library, editing it and conforming it, it probably is. Ultimately, the more of your own recordings you can rely on, the more unique your sound worlds will be.

The other category of sounds that Foley is useful for are sounds that are not literal but can be recreated convincingly using certain objects and can be customized to a particular situation. There are so many candidates, tricks and techniques that a whole book dedicated to the art of Foley would just scratch the surface, but let’s take a look at a few examples that may spark your imagination and compel you to research this topic.

• The sound of a grapefruit being manually squeezed, at various speeds and pressure will make for very realistic gore sounds especially close mic’ed, while the sound of dry celery being broken will sound like bones getting crunched or broken.

• Similarly, the sound of overcooked elbow pasta being mixed together will sound quite gory.

114

THE ART OF EFFECTIVELY COMMUNICATING WITH SOUND

• Bird wings sounds can be achieved by flapping together a pair of plastic gloves such as dish washing gloves or some leather gloves.

• A smaller, creaky door, such as a cabinet door, will sound like the wail of a mammal if moved at the right speed. (You can also add water to the hinges to make it a bit creakier.)

• Hitting one end of a slinky with a microphone making contact at the other will sound like a laser blaster.

The list is endless as you can see, and as you watch more tutorials and read up on the topic you will grow your library of tricks and unique sounds.

Conclusion

Sound design is a vast topic and requires technical knowledge, mastery and artistic sensibility. Over the course of this chapter you have been introduced to some of the basic tools of the trade, as well as some suggestions for their uses. These suggestions are merely intended to be starting points for exploration and experimentation. You should dedicate time to learning more about the tools and techniques introduced here, as well as experiment as much as possible. Over time you will develop a catalog of techniques and aesthetics that will make your sound design unique.

6 PRACTICAL SOUND DESIGN

Learning Objectives

In Chapter fve we looked at the origins of sound design and some of the most commonly used techniques and processes used in the trade. In this chapter we look at a few more specifc examples on how to apply these techniques in the context of linear and interactive sound design. We will also introduce the concept of prototyping, which consists of building interactive sound objects such as vehicles or crowd engines and recreating their behavior in the game by building an interactive model of it, in a software such as MaxMASP or Pure Data, prior to integration in the game engine. The process of prototyping is extremely helpful in testing, communicating and demonstrating the intended behavior or possible behaviors of the interactive elements in a game. But frst we shall take a closer look at some of the major pitfalls most game sound designers run into when setting up a session for linear sound design, such as cut scenes, as well as some basics of signal fow and gain staging.

1. Setting Up a Sound Design Session and Signal Flow

Sound design is both a creative and technical endeavor. There is a ‘what’ element and a ‘how’ element. The ‘what’ is the result we intend to create, and the

‘how’, of course, the method we use to get there. This is a common struggle to most artists and one that the great painter Wassily Kandinsky had identified and articulated in his writings, a testimony to the universality of this struggle to all artists. A solid understanding of the signal flow in DAWs and gain staging overall is critical to obtaining good results. Students often end up struggling with the technology itself, as much as the sound design portion, complicating their tasks a great deal. Often, however, the technical matters can be overcome with a better understanding of the technical side, leaving the student to focus on the matter at hand, the creative.

116

PRACTICAL SOUND DESIGN

1. Signal Flow

The term signal flow refers to the order through which the audio signal encounters or flows through the various elements in a mixer or via external processors, from the input – which is usually the hard drive – or a mic input to the digital audio converters (DACs) and out to the speakers.

In this chapter we will use Avid’s Pro Tools as our DAW. The concepts discussed here, however, will easily apply to another software, especially as most DAW mixers tend to mimic the behavior and setup of classic analog mixers.

Let’s take a look at how the signal flows, from input to output, in a traditional DAW and how understanding this process will make us better audio engineers and therefore sound designers.

The following chart will help us understand this process in more detail:

Figure 6.1 Main elements of a mixer channel strip

a. Input

In most mixers the very first stage is the input. The input varies whether we are in recording mode, in which case the input will usually be a microphone or line input or whether we are in playback mode, in which case the input will be the audio clip or clips in the currently active playlist.

b. Inserts

The next stage your signal is going to run into are the inserts or insert section. This is where you can add effects to your audio, such as equalization,

PRACTICAL SOUND DESIGN

117

compression and whatever else may be available. Inserts are often referred to as an access point, allowing you to add one or multiple processors in your signal path. In most DAWs, the signal goes from the first insert to the last from top to bottom.

c. Pre-Fader Send

After the inserts, a pre - fader send is the next option for your signal. This is where you will send a copy of your audio to another section of your mixer, using a bus. A bus is a path that allows you to move one or multiple signals to a single destination on another section of the mixer. Sending out a signal at this point of the channel strip means the amount sent will be irrespective of the main fader, therefore changes in volume across the track set by the main fader will not affect the amount of audio going out on the pre-fader send. The amount of signal sent is only dependent on the level of the send and, of course, the level of the signal after the insert section.

If you were to send vocals to a reverb processor at this stage, fading out the vocals would not affect the level of the reverb, and you would eventually end up with reverberation only after fading out the vocals.

d. Volume Fader

The next stage is the volume fader, which controls the overall level of the channel strip or audio track. When the volume fader is set to a value of 0dB, known as unity, no gain is applied to the overall track, and all the audio is playing at the post insert audio level. Raising or lowering the fader by any amount will change the current gain value by as much.

Often it is here that you will find panning, to place the audio output in stereo or surround space, depending on the format you are working with.

e. Metering: Pre-Fader vs. Post Fader

Next to the volume fader, you will usually find a level meter. Please check with your DAW’s manual to find out exactly how the meter is measuring the level (Peak, RMS, LUFS etc.). Some DAWS will allow you to change the method for metering. Irrelevant of the method employed, you have the option to monitor signals pre-fader or post-fader. By default, most mixers will have their meters set to post fader mode, which means the fader will display the level after the volume fader and will therefore be affected by it. When monitoring pre-fader, the meter will display the level of the signal right after the last insert, giving you an accurate sense of the level at this stage. It’s probably a good idea to at least occasionally monitor your signals pre-fader, so you can be sure your signal is clean coming out of the insert section.

Please refer to your DAW’s documentation to find out how to monitor pre or post-fader.

118

PRACTICAL SOUND DESIGN

f. Post-Fader Send

Next we find the post-fader send. The level sent to the bus will be impacted by any changes in the level of the volume fader. This is the most commonly used type of send. In this case, if you are sending vocals to a reverb processor, fading out the vocals will also fade out the level of the reverb.

g. Output

Last, we find the output, which determines where the signal is routed to next, by default usually the master bus, where all the audio is summed. Often the output of an audio track should be routed to a submix, where multiple audio tracks that can or should be processed in the same way are mixed together, such as all the ambience tracks in a session or the dialog, music etc.

It’s probably a good rule of thumb to make sure that no track be routed directly to the master fader but rather to a subgroup or submix. Routing individual tracks directly to the master will make your mix messy and difficult to manage.

You may have already noticed that DAWs often do not display the information on a channel strip in their mixer in the order through which the signal flows from top to bottom. If unaware of this, it is easy to make mistakes that get in the way of the task at hand.

2. Working With Video

Sound designers are often asked to work to linear video clips when working in games. Models, such as AI characters, can be exported to video before they are implemented in the game engine, and the animations are often given to the sound designers as linear loops prior to their implementation in the game. Working to video is also a great way to experiment freely in the DAW of your choice, prior to exporting the sounds you created as assets to be imported in the game.

In other cases, you will be given a video clip of a cut scene, a cinematic sequence often used to move the plot forward between levels.

Either way, it is important to be aware of a few key issues when working to picture. Every DAW has slightly different way of importing video, so if you are unsure, please refer to the user manual; the points made here, however, will apply regardless of the DAW you are working in. As in the rest of this chapter, Avid’s Pro Tools will be used to illustrate these concepts.

a. Know Your Frame Rate

Frame rates for video are usually lower than the ones we work with in gaming. Frame rates ranking from 24 to 30 frames per second are common in video, film and broadcast. Find out what the frame rate is of the video you are working with, and make sure to set your DAW’s timeline to be displayed in Timecode format, rather than bars and beats.

PRACTICAL SOUND DESIGN

119

Figure 6.2

Timecode is a way to make sure that each and every frame in a piece of video will have a single address that can be easily recalled and is expressed in the following format:

HH:MM:SS:FF.

Hours, Minutes, Seconds and Frames.

It is important to understand that, although expressed in seconds and frames, time code is a positional reference, an address for each frame in the video file.

Do make sure your DAW’s session is running at the same frame rate as the picture. Setting up our timeline to time code format allows us to move through our session in a frame by frame way, using the nudge feature. Nudging allows you to scrub forward and backwards through the video and allows you to find out exactly and easily where the sync points for each event are in the picture, down to frame accuracy. In some cases, you might need to use a nudge value to half frame for events where synchronization is critical.

The first frame of the clip should be lined up with the address: 01:00:00:00

in the timeline; any material such as slates that provide information about the video clip or countdowns will therefore start prior to the hour mark. Lining up the first frame of video with the address 01:00:00:00 is not a requirement but rather a convention and will make it easier to keep track of time.

Once you have imported the video, set up your DAW to the proper time timecode format and lined up your movie, you’re almost ready to sound design. The next step is to set up the routing and gain staging of the session.

3. Clipping Is Easy – Mind the Signal Path

As you can see from Figure 6.1, the inserts are located pre-fader. A common mistake is to assume that if an audio track is clipping and the meter is in the red, that the problem can be solved by reducing the level with the main fader.

This will indeed turn the audio level down, and the meter may no longer be in the red, if they are monitoring the level post fader, which is often the default.

Doing this, however, only makes the signal quieter, and the clipping is still present, polluting your signal.

120

PRACTICAL SOUND DESIGN

Figure 6.3

The clipping may not be obvious, especially to tired ears and mixed in with other audio signals, but this can lead to harsh sounding mixes and make your task much more difficult.

A better solution would be to turn the gain down at the level of the first insert by inserting a trim plugin and turning the level down before it hits the first plugin and preventing any clipping to occur in the first place.

Use the Dynamic Range

The term dynamic range in the context of a mixing session or a piece of equipment refers to the difference– or ratio – between the loudest and the softest sound or signal that can be accurately processed by the system. In digital audio, the loud portion of the range refers to the point past which clipping occurs, introducing distortion by shaving off the top of the signal. The top of the dynamic range in the digital audio domain is set to 0dBFS, where FS stands for full scale.

Figure 6.4 shows the same audio file, but the right one shows the characteristic flat top of a clipped audio file, and the fidelity of the audio file will be severely affected.

Figure 6.4

PRACTICAL SOUND DESIGN

121

In the digital audio world, the bottom of the dynamic range depends on the number of bits the session or processor is running at. A rule of thumb is that 1 bit = 6dB of dynamic range. Keep in mind this is an approximation, but it is a workable one. A session at 24 bits will therefore offer a dynamic range of 144dB, from 0 to −144dBFS. This, theoretically, represents a considerable improvement over previous high-end large format analog mixing consoles.

Any signal below that level will simply blend into the background noise and probably will sound quite noisy as it approaches that level.

Figure 6.5

Clipping therefore ought not to be an issue. Yet is often is. A well-mastered modern music pop track, when imported into a session, will already bring your master fader dangerously close to the 0dB mark. While it might be tempting to lower the master fader at this stage, refrain from doing so. Always address gain staging issues as early as possible. Lowering the master fader may lower the level on the master bus meter, but in reality, it lends itself to a session where you are constantly fighting for headroom.

There again, a better solution is to lower the level of the music track, ideally at the first insert, and push its levels down by 10 to 15dB, with the volume fader for both the music track and the master fader still at unity. This will give you a lot more headroom to work with while leaving the volume fader at unity.

If the music track now peaks at −15dB, it is still 133dB above the bottom of your dynamic range, which, if working with a clean signal where no noise is already present, gives you more than enough dynamic range to work with.

As good practice, I recommend always keeping the mixer’s master fader at unity.

122

PRACTICAL SOUND DESIGN

4. Setting Up a Basic Session for Linear Mixes and Cut Scenes

Next we will organize the mix around the major components of our soundtrack, usually music, dialog and sound effects.

a. Music, Dialog and Sound Efects

Delivery of stems is quite common and often expected when working with linear media. Stems are submixes of the audio by category such as music , dialog and sound effects. Stems make it convenient to make changes to the mix, such as replacing the dialog, without needing to revisit the entire mix. Having a separate music bounce also allows for more flexible and creative editing while working on the whole mix to picture.

It also makes sense to structure our overall mix in terms of music, effects and dialog busses for ease of overall mixing. Rather than trying to mix all tracks at once, the mix ultimately comes down to a balance between the three submixes, allowing us to quickly change the relative balance between the major components of the mix.

b. Inserts vs. Efects Loops for Reverberation

Effect loops are set up by using a pre or post-fader send to send a portion of the signal to a processor, such as reverb, in order to obtain both a dry and wet version of our signals in the mixer, allowing for maximum flexibility. The effect we are routing the signal to usually sits on an auxiliary input track.

Figure 6.6

Additionally, when it comes to sound effects such as reverb and delays, which are meant to be applied to multiple tracks, it usually makes more sense to use effects loops and sends rather than inserting a new reverb plugin directly on every track that requires one. The point of reverberation when working with sound replacement is often to give us a sense for the space the scene takes place in,

PRACTICAL SOUND DESIGN

123

which means than most sound effects and dialog tracks will require some reverberation at some point. All our sounds, often coming from completely different contexts, will also sound more cohesive and convincing when going through the same reverb or reverbs. Furthermore, applying individual plugins to each track requiring reverb is wasteful in terms of CPU resources and makes it very difficult to make changes, such as a change of space from indoors to outdoors, as they must be replicated over multiple instances of the plugins. This process is also time consuming and difficult to manage as your mix grows in complexity.

As a rule, always set up separate aux send effect loops for reverberation processors and delays used for modeling the environment. In addition to the benefits mentioned earlier, this will also allow you to process the effects independently from the original dry signal. The use of equalization or effects such as chorus can be quite effective in enhancing the sound of a given reverb. As all rules, though, it can be broken but only if there is a reason for it.

c. Setting Up the Mix Session

The structure suggested here is intended as a starting point, and ultimately every audio engineer settles on a format that fits their workflow and the needs of the project the best. Different formats for delivery may have different needs in terms of routing and processing, but we can start to include all the elements outlined so far into a cohesive mix layout.

Figure 6.7 represents the suggested starting point for your mix. From top to bottom:

Figure 6.7

124

PRACTICAL SOUND DESIGN

d. Master Output and Sub Master

In this configuration, no audio from the mix is routed directly to the master fader. Rather there is an additional mixing stage, a master sub mix where all the audio from our mix is routed. The sub master is then sent to the master output sub master -> master output. This gives us an additional mix stage, the sub master, where all premastering and/or mastering processing can be applied, while the master output of the mix is used as a monitoring stage only, such as audio levels, spatial image and spectral balance.

Since all premastering or mastering is done at the master sub mix, our master outputs will be ‘clean’. Should we wish to use a reference track, this configuration means that we can route our reference track directly to the master out and compare it to the mix without running the reference through any of the mastering plugins as well as easily adjust the levels between our mix and the reference.

e. Submixes and Efects Loops

The next stage from the top is where we find the submixes by categories or groups for music, dialog and sound effect, as well as the effect loops for reverb and other global effects. All the audio or MIDI tracks in the session are summed to one of these, no tracks going out directly to the master or sub master output.

Each of the group will likely in turn contain a few submixes depending on the needs and complexity of the mix. Sound effects are often the most complex of the groups and often contain several submixes as illustrated in the diagram.

Figure 6.8

PRACTICAL SOUND DESIGN

125

The screenshot shows an example of a similar mix structure for stereo output realized in Avid’s Pro Tools, although this configuration is useful regardless of the DAW you are working with. The submixes are located on the left side of the screen, to the left of the master fader, and the main groups for music, dialog and sound effects are located on the right side.

• On each of the audio tracks routed to the groups a trim plugin would be added at the first insert, in order to provide the sound designer with an initial gain stage and prevent clipping.

• Each audio track is ultimately routed to a music, dialog or sound effect submix, but some, especially sound effects, are routed to subgroups, such as ambience, gunshots and vehicles that then get routed to the sound effect submix.

• Three effect loops were added for various reverberation plugins or effects.

f. Further Enhancements

We can further enhance our mix by adding additional features and effects to our mix to give us yet more control and features.

Dedicated Software LFE Submix

Adding weight to certain sounds, such as impacts and explosions, can be achieved using a subharmonic generator plugin that will generate low frequency components to any sound that runs through it. These plugins can be difficult to manage as they introduce powerful low-end frequencies that can in turn make the mix challenging to manage. Rather than applying these plugins as inserts on one or multiple tracks, use an effect loop instead, setting it up in the same way you would a reverb, and send any audio file you desire to add weight to it.

Using a dedicated submix for the plugin means that we can process the low frequencies introduced in our mix by the plugin independently from the dry signal, making it easy to compress them or even high pass filter the very lower frequency components out.

Group Sidechaining

Sidechaining is a commonly used technique in mixing where a compressor sits on track A but is listening (aka ‘is keyed’) to track B, compressing A only when the level of B crosses the threshold.

We can also use our subgroup structure to apply sidechain compression on an entire submix at once. A common example of group sidechaining involves the sound effects being sidechained to the dialog so that the mix naturally ducks the effects when dialog occurs. Another option would be to sidechain the music to the sound effect, if we want our sequence to be driven mostly by

126

PRACTICAL SOUND DESIGN

sounds effects where there is no dialog present. This type of group sidechaining is most common in game engines but is also used in linear mixing.

Monitoring

While the meters in the mixer section of your DAW give you some sense of the levels of your track, it is helpful to set up additional monitoring for frequency content of the mix, stereo image (if applicable) and a good LUFS meter to have an accurate sense of the actual loudness of your mix.

At this point, we are ready to mix. Additional steps may be required, based on the session and delivery requirements, of course.

2. Practical Sound Design and Prototyping

When dealing with interactive objects that the player can pilot or operate, our task becomes a little bit more difficult, as we now need to create sound objects that can respond in real time and in a believable fashion to the actions of the player. Often this might involve manipulating sounds in real time, pitching shifting, layering and crossfading between sounds. More complex manipulations are also possible; granular synthesis as noted in the previous chapter is a great way to manipulate audio. Of course, the power of granular synthesis comes at a computational cost that may disqualify it in certain situations.

1. Guns

Guns are a staple of sound design in entertainment, and in order to stay interesting from game to game they demand constant innovation in terms of sound design. In fact, the perceived impact and power of a weapon very much depends on the sound associated with it. The following is meant as an introduction to the topic of gun sound design, as well as an insight as to how they are implemented in games. There are lots of great resources out there on the topic, should the reader decide to investigate the topic further, and is encouraged to do so.

a. One Shot vs. Loops

There are many types of guns used in games, but one of the main differences is whether the weapon is a single shot or an automatic weapon.

Most handguns are single shot or one shot, meaning that for every shot fired the used needs to push the trigger. Holding down the trigger will not fire additional rounds.

Assault rifles and other compact and sub compact weapons are sometimes automatic, meaning the weapon will continue to fire as long as the player is pushing the trigger or until the weapon runs out of ammunition.

PRACTICAL SOUND DESIGN

127

The difference between one shot and automatic weapons affects the way we design sounds and implement the weapon in the game. With a one-shot weapon it is possible to design each sound as a single audio asset including both the initial impulse, the detonation when the user presses the trigger, as well as the tail of the sound, the long decaying portion of the sound.

Figure 6.9

In the case of an automatic weapon, the sound designer may design the weapon in two parts: a looping sound to be played as long as the player is holding onto the trigger and a separate tail sound to be played as soon as the player lets go of the trigger, to model the sound of the weapon decaying as the player stops firing. This will sound more realistic and less abrupt. Additional sounds may be designed and triggered on top of the loop, such as the sound of the shell casings being ejected from the rifle.

Figure 6.10

b. General Considerations

Overall, regardless of the type of weapon you are sound designing and implementing, when designing gun sounds, keep these few aspects in mind:

• Sound is really the best way to give the player a sense of the power and capabilities of the weapon they’re firing. It should make the player feel the power behind their weapon and short of haptic feedback, sound remains the best way to convey the impact and energy of the weapon to the player.

Sound therefore plays an especially critical role when it comes to weapons.

128

PRACTICAL SOUND DESIGN

• Guns are meant to be scary and need to be loud. Very loud. Perhaps louder than you’ve been comfortable designing sounds so far if this a new area for you. A good loudness maximizer/mastering limiter is a must, as is a transient shaper plugin, in order to make the weapon both loud and impactful.

• Guns have mechanical components; from the sound of the gun being handled to the sound of the firing pin striking the round in the chamber to that of the bullet casings being ejected after each shot (if appropriate), these elements will make the weapon sound more compelling and give you as a sound designer the opportunity to make each gun slightly different.

• As always, do not get hung up on making gun sounds realistic, even if you are sound designing for a real-life weapon. A lot of sound designers won’t even use actual recordings of hand guns or guns at all when working sound designing for one.

• The sound of a gun is highly dependent on its environment, especially the tail end of it. If a weapon is to be fired in multiple environments, you might want to design the initial firing sound and a separate environmental layer separately, so you can swap the appropriate sound for a given environment. Some sound designers will take this two-step approach even for linear applications. That environmental layer may be played on top of the gun shot itself or baked in with the tail portion of the sound.

Figure 6.11

• A simple rule of thumb for determining the overall loudness of gun is the ratio of the length of the barrel to the caliber of the bullet. The shorter the barrel and the bigger the caliber, the louder the gun.

• Most bullets travel faster than the speed of sound and therefore will create a supersonic crack. Some bullets are subsonic, designed specifically to avoid creating excessive noise.

c. Designing a Gunshot

One approach when sound designing a gun is to break down the sound into several layers. A layered approach makes it easy to experiment with various

PRACTICAL SOUND DESIGN

129

samples for each of the three sounds, and individually process the different aspects of the sound for best results.

Three separate layers are a good place to start:

• Layer 1: the detonation, or the main layer. In order to give your guns maximum impact, you will want to make sure this sample has a nice transient component to it. This is the main layer of the sound, which we are going to augment with the other two.

• Layer 2: a top end, metallic/mechanical layer. This layer will increase realism and add to the overall appeal of the weapon. You can use this layer to give your guns more personality.

• Layer 3: a sub layer, to add bottom end and make the sound more impactful. A subharmonic generator plugin might be helpful. This layer will give your sound weight.

When selecting samples for each layer, prior to processing, do not limit yourself to the sounds that are based in reality. For instance, when looking for a sound for the detonation or the main layer, go bigger. For a handgun, try a larger rifle or shotgun recording; they often sound more exciting than handguns. Actual explosions, perhaps smaller ones for handguns, may be appropriate too.

Figure 6.12

The Detonation/Main Body Layer

As always, pick your samples wisely. A lot of sound effects libraries out there are filled with gun sounds that are not always of the best quality, may not be the right perspective (recorded from a distance) or already have a lot reverberation baked in. You’ll usually be looking for a dry sample, as much as possible anyway, something that ideally already sounds impressive and scary. Look for something with a healthy transient. You might want to use a transient shaping

130

PRACTICAL SOUND DESIGN

plugin or possibly a compressor with a slow attack time as described in the previous chapter in order to emphasize the transients further. An equalization scoop around 300–400Hz might actually be a good way to make a bit more room for the low and mid frequencies to cut through.

The Top End/Mechanical Layer

When a shot is fired through a gun, some of the energy is transferred into the body of the gun and in essence turns the gun itself into a resonator. This is partially responsible for the perceived mechanical or metallic aspect of the sound. In addition, some guns will eject the casing of the bullet after every shot. The sound of the case being ejected and hitting the floor obviously makes a sound too. The mechanical layer gives you a lot of opportunity for customization. When sound designing a lot of guns for a game, inevitably they will tend to sound somewhat similar. This layer is a good place to try to add some personality to each gun. Generally speaking, you will be looking for a bright sound layer that will cut through the detonation and the bottom end layers. It should help give your gun a fuller sound by filling up the higher frequencies that the detonation and the sub may not reach. It also adds a transient to your gun sound, which will make it sound all the more realistic and impactful.

The Sub Layer

The purpose of the sub layer is to give our sounds more weight and impact and give the player a sense of power, difficult to achieve otherwise, except perhaps via haptic feedback systems. Even then, sound remains a crucial aspect of making the player ‘feel’ like their weapon is as powerful as the graphics imply.

A sub layer can be created in any number of ways, all worth experimenting with.

It can be created using a synthesizer by modifying or creating an existing bass preset and applying a subharmonic generator to it to give it yet more depth and weight. Another option is to start from an actual recording, perhaps an explosion or detonation, low pass filtering it and processing it with a subharmonic generator to give it more weight still. A third option would be to use a ready-made sub layer, readily found in lots of commercial sound libraries.

Avoid using a simple sine wave for this layer. It may achieve the desired effect on nice studio monitors but might get completely lost on smaller speakers, while a more complex waveform, closer to a triangle wave, will translate much better, even on smaller speakers.

Modeling the Environment

Guns and explosions are impossible to abstract from the environment they occur in. Indeed, the same weapon will sound quite different indoors and

PRACTICAL SOUND DESIGN

131

outdoors, and since in games it is often possible to fire the same gun in several environments, game sound designers sometimes resort to designing the tail end of the gun separately so that the game engine may concatenate them together based on the environment they are played into. In some cases, sound designers will also add an environment layer to the gun sounds simply because the reverb available in the game may not be quite sophisticated enough to recreate the depth of the sound a detonation will create when interacting with the environment. This environment layer is usually created by running the sound of the gun through a high-end reverberation plugin.

The environment layer may be baked into the sound of the gun – that is, bounced as a single file out of the DAW you are working with – or triggered separately by the game engine, on top of the gun sound. This latter approach allows for a more flexible weapon sound, one that can adapt to various environments.

Putting It all Together

Once you have selected the sounds for each layer, you are close to being done, but there still remain a few points to take into consideration.

Start by adjusting the relative mix of each layer to get the desired effect.

If you are unsure how to proceed, start by listening to some of your favorite guns and weapons sounds from games and movies. Consider importing one or more in the session you are currently working on as a reference. (Note: make sure you are not routing your reference sound to any channels that you may have added processors to.) Listen, make adjustments and check against your reference. Repeat as needed.

Since guns are extremely loud, don’t be shy, and use loudness maximizers and possibly even gain to clip the waveform or a layer in it. The real danger here is to destroy transients in your sound, which may ultimately play against you. There is no rule here, but use your ears to strike a compromise that is satisfactory. This is where a reference sound is useful, as it can be tricky to strike the proper balance.

In order to blend the layers together, some additional processing may be a good idea. Compression, limiting, equalization and reverberation should be considered in order to get your gun sound to be cohesive and impactful.

Player Feedback

It is possible to provide the player with subtle hints to let them know how much ammunition they have left via sound cues rather than by having to look at the screen to find out. This is usually done by increasing the volume

132

PRACTICAL SOUND DESIGN

of the mechanical layer slightly as the ammunition is running out. The idea is to make the gun sound slightly hollower as the player empties the magazine.

This approach does mean that you will need to render the mechanical layer separately from the other two and control its volume via script. While this requires a bit more work, it can increase the sense of immersion and realism as well as establish a deeper connection between the player and their weapon.

2. Prototyping Vehicles

When approaching the sound design for a vehicle or interactive element, it is first important to understand the range of actions and potential requirements for sounds as well as limitations prior to starting the process.

The implementation may not be up to you, so you will need to know and perhaps suggest what features are available to you. You will likely need the ability to pitch shift up and down various engine loops and crossfade between different loops for each rpm. Consider the following as well: will the model support tire sounds? Are the tire sounds surface dependent? Will you need to provide skidding samples? What type of collision sounds do you need to provide? The answers to these questions and more lie in the complexity of the model you are dealing with.

a. Specifcations

A common starting point for cars is to assume a two gear vehicle, low and high gear. For each gear we will create an acceleration and deceleration loop, which the engine will crossfade between based on the user action.

• Eng_loop_low_acc.wav Low RPM engine loop for acceleration.

• Eng_loop_low_de.wav

Low RPM engine loop for deceleration.

• Eng_loop_high_acc.wa

High RPM engine loop for acceleration.

• Eng_loop_high_de.wav High RPM engine loop for deceleration.

This is a basic configuration that can easily be expanded upon by adding more RPM samples and therefore a more complex gear mechanism.

The loops we create should be seamless, therefore steady in pitch and without any modulation applied. We will use input from the game engine to animate them, to create a sense of increased intensity as we speed up by pitching the sound up or decreased intensity as we slow down by pitching the sound down. As the user starts the car and accelerates, we will raise the pitch and volume of our engine sample for low RPM and eventually crossfade into the high RPM engine loop, which will also increase in pitch and volume until we reach the maximum speed. When the user slows down, we will switch to the deceleration samples.

PRACTICAL SOUND DESIGN

133

Figure 6.13

Let’s start by creating the audio loops, which we can test using the basic car model provided in the Unity Standard’s asset package, also provided in the Unity level accompanying this chapter.

b. Selecting Your Material

When working on a vehicle it is tempting to start from the sound of a similar looking or functioning real-world vehicle and try to recreate it in the game.

Sample libraries are full of car and truck samples that can be used for this purpose, or, if you are feeling adventurous, you can probably record a car yourself. A little online research can give you tips about what to look out for when recording vehicles. This can be a very effective approach but can be somewhat underwhelming ultimately without further processing. Remember that reality, ultimately, can be a little boring.

Another approach still is to look at other types of vehicles, such as propeller airplanes, boats and other vehicles and layer them together to create a new engine sound altogether.

Finally, the third option is to use sounds that have nothing to do with a car engine – gathered via recordings – or synthesize and create the loops required from this material.

Always try to gather and import in your sound design session more than you think you will need. This will allow you to be flexible and give you more options to experiment with.

c. Processing and Preparing Your Material

Once you have gathered enough sounds to work with it’s time to import them and process them in order to create the four loops we need to create.

134

PRACTICAL SOUND DESIGN

There are no rules here, but there are definitely a few things to watch out for:

• The sample needs to loop seamlessly, so make sure that there are no obvious variations in pitch and amplitude that could make it sound like a loop.

• Do not export your sounds with micro fades.

Use all the techniques at your disposal to create the best possible sound, but, of course, make sure that whatever you create is in line with both the aesthetics of the vehicle and the game in general.

Here are a few suggestions for processing:

• Layer and mix: do not be afraid to layer sounds in order to create the right loop.

• Distortion (experiment with various types of distortion) can be applied to increase the perceived intensity of the loop. Distortion can be applied or ‘printed’ as a process in the session, or it can be applied in real time in the game engine and controlled by a game parameter, such as RPM or user input.

• Pitch shifting is often a good way to turn something small into something big and vice versa or into something entirely different.

• Comb filtering is a process that often naturally occurs in a combustion engine; a comb filter tuned to the right frequency might make your sound more natural and interesting sounding.

Once you have created the assets and checked that length is correct, that they loop without issue and that they sound interesting, it’s time for the next step, hearing them in context, something that you can only truly do as you are prototyping.

d. Building a Prototype

No matter how good your DAW is, it probably won’t be able to help you with the next step, making sure that, in the context of the game, as the user speeds up and slows down, your sounds truly come to life and enhance the experience significantly.

The next step is to load the samples in your prototype. The tools you use for prototyping may vary, from a MaxMSP patch to a fully functioning object in the game engine. The important thing here is not only to find out if the sounds you created in the previous step work well when ‘put to picture’, it’s also to find out what are the best ranges for the parameters the game engine will control. In the case of the car, the main parameters to adjust are pitch shift, volume and crossfades between samples. In other words, tuning your model. If the pitch shift applied to the loops is too great, it may make the sound feel too synthetic, perhaps even comical. If the range is too small, the model might not be as compelling as it otherwise could be and lose a lot of its impact.

We will rely on the car model that comes in with the Unity Standard Assets package, downloadable from the asset store. It is also included in the Unity level for this chapter. Open the Unity project PGASD_CH06 and open the

PRACTICAL SOUND DESIGN

135

scene labelled ‘vehicle’. Once the scene is open, in the hierarchy, locate and click on the Car prefab. At the bottom of the inspector for the car you will find the Car Audio script.

Figure 6.14

The script reveals four slots for audio clips, as well as some adjustable parameters, mostly dealing with pitch control. The script will also allow us to work with a single clip for all the engine sounds or with four audio clips, which is the method we will use. You can switch between both methods by clicking on the Engine Sound Style tab. You will also find the script that controls the audio for the model, and although you are encouraged to look through it, it may make more sense to revisit the script after going through Chapters seven and eight if you haven’t worked with scripting and C# in Unity. This script will crossfade between a low and high intensity loop for acceleration and deceleration and perform pitch shifting and volume adjustments in response to the user input. For the purposes of this exercise, it is not necessary to understand how the script functions as long as four appropriate audio loops have been created. Each loop audio clip, four in total, is then assigned to a separate audio source. It would not be possible for Unity to swap samples as needed using a single audio source and maintain seamless playback. A short interruption would be heard as the clips get swapped.

Next, import your sounds in the Unity project for each engine loop, load them in the appropriate slot in the car audio script and start the scene. You should be able to control the movement of the car using the WASD keys.

Listen to the way your sounds sound and play off each other. After driving the vehicle for some time and getting a feel for it, ask yourself a few basic questions:

• Does my sound design work for this? Is it believable and does it make the vehicle more exciting to drive?

• Do the loops work well together? Are the individual loops seamless?

Do the transitions from one sample to another work well and convey

136

PRACTICAL SOUND DESIGN

the proper level of intensity? Try to make sure you can identify when and how the samples transition from one another when the car is driving.

• Are any adjustments needed? Are the loops working well as they are, or could you improve them by going back to your DAW and exporting new versions? Are the parameter settings for pitch or any other available ones at their optimum? The job of a game audio designer includes understanding how each object we are designing sound for behaves, and adjusting available parameters properly can make or break our model.

In all likelihood, you will need to experiment in order to get to the best results.

Even if your loops sound good at first, try to experiment with the various different settings available to you. Try using different loops, from realistic, based on existing sounding vehicles, to completely made up, using other vehicle sounds and any other interesting sounds at your disposal. You will be surprised at how different a car can feel when different sounds are used for its engine.

Other sounds may be required in order to make this a fully interactive and believable vehicle. Such a list may include:

• Collision sounds, ideally different sounds for different impact velocity.

• Tire sounds, ideally surface-dependent.

• Skidding sounds.

• Shock absorbers sounds.

There is obviously a lot more to explore here and to experiment with. This car model does not include options to implement a lot of the sounds mentioned earlier, but that could be easily changed with a little scripting knowledge.

Even so, adding features may not be an option based on other factors such as RAM, performance, budget or deadlines. Our job is, as much as possible, to do our best with what we are handed, and sometimes plead for a feature we see as important to making the model come to life. If you know how to prototype regardless of the environment, your case for implementing new features will be stronger if you already have a working model to demonstrate your work and plead your case more convincingly to the programming team or the producer.

3. Creature Sounds

Creatures in games are often AI characters that can sometimes exhibit a wide range of emotions, which sound plays a central role in effectively communicating. As always, prior to beginning the sound design process, try to understand the character or creature you are working on. Start with the basics: is it endearing, cute, neutral, good, scary etc.? Then consider what its emotional

PRACTICAL SOUND DESIGN

137

span is. Some creatures can be more complex than others, but all will usually have a few basic emotions and built in behaviors, from simply roaming around to attacking, getting hurt or dying. Getting a sense for the creature should be the first thing on your list.

a. Primary vs. Secondary Sounds

Once you have established the basic role of the creature in the narrative, consider its physical characteristics: is it big, small, reptilian, feline? The appearance and its ‘lineage’ are great places to start in terms of the sonic characteristics you will want to bring out. Based on its appearance, you can determine if it should roar, hiss, bark, vocalize, a combination of these or more. From these characteristics, you can get a sense for the creature’s main voice or primary sounds, the sounds that will clearly focus the player’s attention and become the trademark of this character. If the creature is a zombie, the primary sounds will likely be moans or vocalizations.

Realism and believability come from attention to detail; while the main voice of the creature is important, so are all the peripheral sounds that will help make the creature truly come to life. These are the secondary sounds: breaths, movement sounds coming from a creature with a thick leathery skin, gulps, moans and more will help the user gain a lot better idea of the type of creature they are dealing with, not to mention that this added information will also help consolidate the feeling of immersion felt by the player. In the case of a zombie, secondary sounds would be breaths, lips smacks, bones cracking or breaking etc. It is, however, extremely important that these peripheral or secondary sounds be clearly understood as such and do not get in the way of the primary sounds, such as vocalizations or roars for instance.

This could confuse the gamer and could make the creature and its intentions hard to decipher. Make sure that they are mixed in at lower volume than the primary sounds.

Remember that all sound design should be clearly understood or legible. If it is felt that a secondary sound conflicts with one of the primary sound effects, you should consider adjusting the mix further or removing it altogether.

b. Emotional Span

Often, game characters, AI or not, will go through a range of emotions in the game’s lifespan. These are often, for AI at least, dictated by the game state and will change based on the gameplay. A sentinel character can be relaxed, alert, fighting, inflict or take damage and possibly kill or die. These actions or states should be reflected sonically of course, by making sure our sound design for each state is clear and convincing. It may be overkill to establish a mood map (but if it helps you, by all means do), yet it is important to make sure that the

138

PRACTICAL SOUND DESIGN

sounds you create all translate these emotions clearly and give us a wide range of sonic transformations while at the same time clearly appearing to be emanating from the same creature.

The study or observation of how animals express their emotions in the real world is also quite useful. Cats and dogs can be quite expressive, making it clear when they are happy by purring or when they are angry by hissing and growling in a low register, possibly barking etc. Look beyond domestic animals and always try to learn more.

Creatures sound design tends to be approached in one of several ways: by processing and layering human voice recordings, by using animal sounds, by working from entirely removed but sonically interesting material or any combination of these.

c. Working With Vocal Recordings

A common approach to designing creature sounds is to begin with a human voice and emote based on the character in a recording studio. These sounds are usually meant to be further processed, but it is important to record a lot of good quality material at this stage. Do not worry too much about synchronization at this point; this is what editing is for. Try loosely matching animations, that is if any were provided, and record a wide variety of sounds. Your voice or that of the talent may not match the expected range of the character, perhaps lacking depth or having too much of it, but the raw sounds and emotions are more important at this point. Emotion is harder to add to a sound after the fact, and while it can be done, usually by drawing pitch envelopes and layering different sounds together, it is faster to work with a file that already contains the proper emotional message and process it to match the character on screen.

As always, record more material than you think you’re going to need. This will give you more to work with and choose from, always recording multiple takes of each line or sound.

Also make sure your signal path is clean, giving you a good signal to work with in the first place. This means watching out for noise, unwanted room ambiences, room tones etc.

Traditionally, large diaphragm condenser microphones are used for voice recording, but in noisy environments you may obtain cleaner results with a good dynamic microphone, though you might need to add some high-end back into the signal during the design and mix process.

Pitch Shifting in the Context of Creature Design Your voice talent may sound fabulous and deliver excellent raw material, but it is unlikely that they will be able to sound like a 50 meters tall creature or a ten centimeters fairy. This is where pitch shifting can be extremely helpful.

PRACTICAL SOUND DESIGN

139

Pitch shifting was detailed in the previous chapters, but there are a few features that are going to be especially helpful in the context of creature sound design.

Since pitch is a good way to gauge the size of a character, it goes without say that raising the pitch will make the creature feel smaller, while lowering it will inevitably increase its perceived size.

The amount of pitch shift to be applied is usually specified in cents and semitones.

Note: there are 12 semitones in an octave and 100 cents in a semitone.

The amount by which to transpose the vocal recording is going to be a product of size and experimentation, yet an often-overlooked feature is the formant shift parameter. Not all pitch shifting plugins have one, but it is recommended to invest in a plugin that does.

Formants are peaks of spectral energy that result from resonances usually created by the physical object that created the sound in the first place. More specifically, when it comes to speech, they are a product of the vocal tract and other physical characteristics of the performer. The frequency of these formants therefore does not change very much, even across the range of a singer, although they are not entirely static in the human voice.

Table 6.1

Formant

E

A

0h

0oh`

Frequencies in Hz

Men

Formant 1

270

660

730

300

Formant 2

2290

1720

1090

870

Formant 3

3010

2410

2440

2240

Women

Formant 1

310

860

850

370

Formant 2

2790

2050

1220

950

Formant 3

3310

2850

2810

2670

These values are meant as starting points only, and the reader is encouraged to research more information online for more detailed information.

When applying pitch shifting techniques that transpose the signal and ignore formants, these resonant frequencies also get shifted, implying a smaller and smaller creature as they get shifted upwards. This is the classic ‘chipmunk’ effect. Having individual control over the formants and the amount of the pitch shift can be extremely useful. Lowering the formants without changing the pitch can make a sound appear to be coming from a larger source or creature and inversely. Having independent control of the pitch and formant gives us the ability to create interesting and unusual hybrid sounds.

140

PRACTICAL SOUND DESIGN

A lot of pitch correction algorithms provide this functionality as well and are wonderful tools for sound design. Since pitch correction algorithms often include a way to draw pitch, they can also be used to alter the perceived emotion of a recording. By drawing an upward pitch gesture at the end of a sound, it will tend to sound inquisitive, for instance.

Distortion in the Context of Creature Design

Distortion is a great way to add intensity to a sound. The amount and type of distortion should be decided based on experience and experimentation, but when it comes to creature design, distortion can translate into ferocity. Distortion can either be applied to an individual layer of the overall sound or to a submix of sounds to help blend or fuse the sounds into one while making the overall mix slightly more aggressive. Of course, if the desired result is to use distortion to help fuse sounds together and add mild harmonics to our sound, a small amount of distortion should be applied.

Watch out for the overall spectral balance upon applying distortion, as some algorithms tend to take away high frequencies and as a result the overall effect can sound a bit lo-fi. If so, try to adjust the high frequency content by boosting high frequencies using an equalizer or aural exciter.

Note: as with many processes, you might get more natural-sounding results by applying distortion in stages rather than all at once. For large amounts, try splitting the process in two separate plugins, in series each carrying half of the load.

Equalization in the Context of Creature Design

As with any application, a good equalizer will provide you with the ability to fix any tonal issues with the sound or sounds you are working with.

Adding bottom end to a growl to make it feel heavier and bigger or simply bringing up the high frequency content after a distortion stage, for instance.

Another less obvious application of equalization is the ability to add formants into a signal that may not contain any or add more formants to a signal that already does. By adding formants found in a human voice to a non-human creature and sounds, we can achieve interesting hybrid resulting sounds.

Since a formant is a buildup of acoustical energy at a specific frequency, it is possible to add formants to a sound by creating very narrow and powerful boosts at the right frequency. This technique was mentioned in Chapter five as a way to add resonances to a sound and therefore make it appear like it takes place in a closed environment.

In order to create convincing formant, drastic equalization curves are required. Some equalizer plugins will include various formants as parts of their presets.

PRACTICAL SOUND DESIGN

141

Figure 6.15

d. Working With Animal Samples

Animal samples can provide us with great starting points for our creature sound design. Tigers, lions and bears are indeed a fantastic source of ferocious and terrifying sounds, but at the same time they offer a huge range of emotions: purring, huffing, breathing, whining. The animal kingdom is a very rich one, and do not limit your searches to these obvious candidates. Look far and wide, research other sound designer’s works on films and games and experiment.

The main potential pitfall when working with animal samples is to create something that actually sounds like an animal, in other words too easily recognizable as a lion or large feline for instance. This is usually because the samples used could be processed further in order to make them sound less easily identifiable. Another trick to help disguise sounds further is to chop off the beginning of the sample you are using. By removing the onset portion of a sample you make it harder to identify. Taking this technique further you can also swap the start of a sample with another one, creating a hybrid sound that after further processing will be difficult to identify.

Amplitude Modulation in the Context of Creature Design Amplitude modulation can be used in two major ways: to create a tremolo effect or to add sidebands to an existing sound. A rapid tremolo effect is a good way to bring up insect-like quality in creatures, such as the rapid wing flap of a fly. It can also be applied to other aspects of a sound and impart to other sounds a similar quality.

When applied as ring modulation, the process will drastically change the current harmonic relationship of the sound by adding sidebands to every

142

PRACTICAL SOUND DESIGN

frequency component of the original sound while at the same time removing these original components. In other words, ring modulation removes the original partials in the sound file and replaces them with sidebands. While the process can sound a little electronic, it is a great way to drastically change a sound while retaining some of its original properties.

Convolution in the Context of Creature Design

• Convolution can be a potentially very powerful tool for creature sound design. Although most frequently used for reverberation, convolution can be very effective at creating hybrid sounds by taking characteristics of two different sounds and creating a new, hybrid audio file as a result. The outcome will tend to be interesting, perhaps even surprising, as long as both files share a common spectrum. In other words, for convolution to yield its most interesting results, it is best if the files’

frequency content overlaps. You will also find that often, unless the algorithm used compensates for it, the resulting file of a convolution can come out lacking in high frequencies. This is because convolution tends to yield more energy in the areas in both files which share the most, while its output will minimize the frequency content where the energy in either or both files is less strong. High frequencies are often not as powerful in most sounds as other frequency ranges, such as mid-range frequencies.

When trying to create hybrid sounds using convolution, first make sure the files you are working with are optimal and share at least some frequency content. You may also find that you get slightly more natural results if you apply an equalizer to emphasize high frequencies in either input file, rather than compensating after the process.

Some convolution plugins will give you control over the window length or size. Although this term, window size, may be labelled slightly differently in different implementations, it is usually expressed as a power of two, such as 256 or 512 samples. This is because most convolution algorithms are implemented in the frequency domain, often via a Fourier algorithm, such as the fast Fourier transform.

In this implementation, both audio signals are broken down into small windows whose length is a power of two, and a frequency analysis is run on each window or frame. The convolution algorithm then performs a spectral multiplication of each frame and outputs a hybrid. The resulting output is then returned to the time domain by performing an inverse Fourier transform.

The process of splitting the audio in windows of a fixed length is not entirely transparent, however. There is a tradeoff at the heart of this process that is common to a lot of FFT-based algorithms: a short window size, such

PRACTICAL SOUND DESIGN

143

as 256 and under, will tend to result in better time resolution but poorer frequency resolution. Inversely, a larger window size will yield better frequency resolution and a poorer time resolution. In some cases, with larger window sizes, some transients may end up lumped together, disappearing or getting smeared. Take your best guess to choose the best window size based on your material, and adjust from there.

Experimentation and documenting your results are keys to success.

e. Working With Non-Human or Animal Samples

Perhaps not as obvious when gathering material for sound design for creatures and monsters is to use material that comes from entirely different sources than human or animal sources. Remember that we can find interesting sounds all around us, and non-organic elements can be great sources of raw material. Certain types of sounds might be more obvious candidates than others. The sound of a flame thrower can be a great addition to a dragon-like creature, and the sound of scraping concrete blocks or stone can be a great way to add texture to an ancient molten lava monster, but we can also use non-human or animal material for primary sounds such as vocalizations or voices.

Certain sounds naturally exhibit qualities that make them sound organic. The right sound of a bad hinge on a cabinet door, for instance, can sound oddly similar to a moan or creature voice when the door is slowly opening. The sound of a plastic straw pulled out of a fast food cup can also, especially when pitch shifted down, have similar characteristics.

The sound of a bike tire pump can sound like air coming out of a large creature’s nostrils and so on. It’s also quite possible to add formants to most sounds using a flexible equalizer as was described in the previous section.

Every situation is different of course, and every creature is too. Keep experimenting with new techniques and materials and trying new sounds and new techniques. Combining material, human, animal and non-organic, can create some of the most interesting and unpredictable results.

4. An Adaptive Crowd Engine Prototype in MaxMSP

Our next example is a simple adaptive crowd engine, built this time in MaxMSP. MaxMSP is a graphical programming environment for audio and visual media. This example is meant to recreate the crowd engines you can find in classic large arena sports games and demonstrate the basic mechanics of how the crowd sounds react to the action. 1

In order to create an evolving and dynamic ambience, we will rely on four basic loops, one for each state the crowd can be in: quiet, medium intensity, high intensity, and finally upset or booing.

144

PRACTICAL SOUND DESIGN

Rather than doing simple crossfades between two samples, we will rely on an XY pad instead, with each corner linked to an audio file. An XY pad gives us more options and a much more flexible approach than a simple crossfade.

By moving the cursor to one of the corners, we can play only one file at a time.

By sliding it toward another edge, we can mix between two files at a time, and by placing the cursor in the center of the screen, we can play all four at once.

This means that we could, for instance, recreate the excitement of fans as their teams is about to score, while at the same time playing a little of the boos from the opposite teams as they express their discontent. As you can see, XY pads are a great way to create interactive audio objects, certainly not limited to a crowd engine.

Figure 6.16

We will rely on four basic crowd loops for the main sound of the crowd:

• Crowd_Lo_01.wav: A low intensity crowd sample: the crowd is quiet and waiting for something to happen.

PRACTICAL SOUND DESIGN

145

• Crowd_Mid_01.wav: A medium intensity crowd sample: the crowd is getting excited while watching a play.

• Crowd_Hi_01.wav: A high intensity crowd sample: the crowd is celebrating a score or play.

• Crowd_Boo_01.wav: the crowd is unhappy and booing the action.

Each one of these samples should loop seamlessly, and we will work with loops about 30 seconds to a minute in length, although that figure can be adjusted to match memory requirement vs. desired complexity and degree of realism of the prototype. As always when choosing loops, make sure that the looping point is seamless but also that the recording doesn’t contain an easily remembered sound, such as an awkward and loud high pitch burst of laughter by someone close to the microphone, which would eventually be remembered by the player and suddenly feel a lot less realistic and would eventually get annoying. In order to load the files into the crowd engine just drag the desired file to the area on each corner labelled drop file.

As previously stated, we will crossfade between these sounds by moving the cursor in the XY pad area. When the cursor is all the way in one corner, only the sound file associated with that corner should play; when the cursor is in the middle, all four sound files should play. Furthermore, for added flexibility, each sound file should also have its own individual sets of controls for pitch, playback speed and volume. We can use the pitch shift as way to increase intensity, by bringing the pitch up slightly when needed or by lowering its pitch slightly to lower the intensity of the sound in a subtle but efficient manner. This is not unlike how we approached the car engine, except that we will use much smaller ranges in this case.

In order to make our crowd engine more realistic we will also add a sweeteners folder. Sweeteners are usually one-shot sounds triggered by the engine to make the sonic environment more dynamic. In the case of a crowd engine these could be additional yells by fans, announcements on the PA, an organ riff at a baseball game etc. We will load samples from a folder and set a random timer for the amount of time between sweeteners. Audio files can be loaded in the engine by dragging and dropping them in each corner of the engine, and sweeteners can be loaded by dropping a folder containing .wav or .aif files into the sweetener area.

Once all the files have been loaded, press the space bar to start the playback.

By slowly moving and dragging around the cursor in the XY pad while the audio files are playing, we are able to recreate various moods from the crowd by starting at a corner and moving toward another. The XY pad is convenient because it allows us to mix more than one audio file at once; the center position would play all four, while a corner will only play one.

Recreating the XY pad in Unity would not be very difficult; all it would require are five audio sources, (one for each corner plus one for the sweeteners) and a 2D controller moving on a 2D plane.

The architecture of this XY pad is very open and can be applied to many other situations with few modifications. Further improvements may include

146

PRACTICAL SOUND DESIGN

the addition of a granular synthesis or other processing stage, which could be used to further animate the audio generated by our engine and obtain a significantly wider range of variations and intensities, albeit at some computational cost. Perhaps a more obvious improvement would be to work with multiple loops for the crowd states, which would also give us more potential for variations. This architecture also does not have to be used for a crowd engine; it could easily be applied to ambiences, machines, vehicles and lots more situations.

Conclusion

Sound design, either linear or interactive, is a skill learned through experimentation and creativity, but that also requires the designer to be organized and aware of the pitfalls ahead of them. When it comes to linear sound design, organizing the session for maximum flexibility while managing dynamic range are going to be some of the most important aspects to watch out for on the technical side of things. When it comes to interactive sound design, being able to build or use prototypes that effectively demonstrate the behavior of the object in the game by simulating the main parameters is also very important. This will allow you to address any potential faults with the mechanics or sound design prior to implementation in the game and communicate more effectively with your programming team.

Note

1. In order to tryout this example, the reader will need to install Cycling74’s MaxMSP, a free trial version being available from their website.

7 CODING FOR GAME AUDIO

Learning Objectives

This chapter is intended to be studied along with the next chapter, Chapter eight, and it introduces the reader to the basics of scripting and programming. The reader is strongly encouraged to keep learning about the concepts discussed in this chapter and the next, as they are only introduced in these chapters, and anyone interested in a career in game audio would greatly beneft from further knowledge. These next chapters, however, will give the reader a lot of tools with which to work with for upcoming projects.

By the end of this chapter, the reader will have been introduced to the basics of object-oriented programming; will know how to create a class in C#

in Unity; will be able to play back an audio fle using scripting while randomizing pitch, volume and sample selection and more. Some audio-specifc issues will be introduced as well.

1. Why Learn to Code?

Coding may seem a tad daunting at first, and the benefits of dedicating time and effort to the task may not seem obvious when starting out in game audio.

Modern gaming development environments, however, do require a relatively high level of technical proficiency and some computer science literacy, and anyone who’s dedicated any time to working in an environment like Unity, Unreal or other game engines has probably reached the conclusion that knowing some scripting will be a huge asset.

Another reason to learn programming has to do with the ability to interface with a development team. Being able to have a conversation with a programmer and articulate your goals in terms a programmer can clearly understand is an invaluable skill.

The purpose of this chapter is to introduce students to the main concepts that they are going to encounter while working in game audio and is intended as a starting point from which to further explore these concepts and, hopefully,

148

CODING FOR GAME AUDIO

demystify some of the fundamentals of scripting. For the purpose of this book we will focus on C# and Unity, though a lot of the concepts explained here will translate quite easily to another language.

Unity uses Microsoft’s Visual Studio as a its programming environment.

Visual Studio is an IDE, an Integrated Development Environment. An IDE

is usually made up of three components: a text editor or source code editor, build tools and a debugger. We enter our code using the source code editor, use the build tools to compile it and the debugger to troubleshoot the code.

1. Syntax and Logic

When learning to code there are usually two main areas to address, the syntax and the logic.

The syntax is the grammar and orthography of the language you are study-ing. What are the keywords, the symbols to use and in what order? Learning the syntax is not really any different than learning a new language. We must get used to its spelling, grammar and way of thinking. Different computer languages have different syntax, but a lot of the C-based computer languages will have some elements in common.

The logic covers the steps that need to be undertaken to achieve our goal.

The logic can be outlined using plain language and should help the programmer establish a clear view of each of the steps that needs to be undertaken to achieve the task at hand and then how to translate and implement these steps in the programming language. This process will lead to the creation of an algorithm. Outlining the logic is an important step that should not be overlooked.

We all have an intuitive understanding of this process, as in many ways we do this every day multiple times a day in our daily lives.

2. Algorithms

We can define an algorithm as a precise set of instructions that must be followed in the order in which they are delivered. In fact, anyone who’s ever followed a cook-ing recipe has followed an algorithm and has an intuitive understanding for it.

This, for instance, is the soft-boiled egg boiling algorithm: 1. Place egg in a saucepan.

2. Fill the saucepan with water; cover the egg by an inch.

3. Set the stove top to high heat.

4. Place the saucepan on the stovetop.

5. Bring water to a boil.

6. Remove the saucepan from heat and cover.

7. Wait for four to six minutes.

8. Immerse in ice cold water for ten minutes.

9. Enjoy.

CODING FOR GAME AUDIO

149

In many ways, programming is not any different. Whenever starting to code a new task, ask yourself if you are able to clearly articulate each step of the process prior to starting the coding process. It is strongly recommended to outline an algorithm first and only after each step is clear to start coding. This will help save you time down the line and make sure that the logic is sound, eliminating a lot of potential causes for error.

3. Basic Object-Oriented Programming Concepts

a. Procedural vs. Object-Oriented

Programming languages fall into two rather broad categories: procedural and object-oriented. The difference is a rather profound one and may take a moment to fully appreciate. Procedural languages, such as C, tend to focus on a top-down approach to coding, where tasks to accomplish are broken down into functions and the code is driven by breaking down a complex task into smaller, easier to grasp and manipulate, bits of code. In procedural programming the data and the methods are separate, and the program flow is usually a direct product of the task at hand. The C programming language is an example of a procedural language.

Figure 7.1 A procedural, top-down approach to programming

150

CODING FOR GAME AUDIO

b. Encapsulation and Inheritance

In object-oriented programming, by contrast, data and tasks, also referred to as attributes and behaviors, are contained within a single object. The process of including both attributes and behaviors within a single object is known as encapsulation. Encapsulation is one of the most powerful features of object-oriented programming and greatly contributes to making the code you write easy to re-use. By creating objects in which attributes and behaviors are self-contained we can create complex systems easily and introduce a level of modularity that makes it convenient to re-use code. For instance, once an object has been created you can use it as many times as desired without or with very little need to write additional code.

Figure 7.2

When starting a script in an object-oriented language such as C#, one usually starts by creating a class. A class can be thought of as a template, in which the programmer defines the behaviors and attributes of an object.

When the object is used in the code, it is instantiated. Instantiation is what allows a programmer to write a class once but be able to use it multiple times in a program. In Unity, most of the classes we will create will inherit from Monobehaviour. Monobehaviour is therefore the parent class, also referred to as the base class.

Object-oriented programming goes further by making it possible to use an already existing object or class to create a new one, through a process known

CODING FOR GAME AUDIO

151

as inheritance. Inheritance is one of the pillars of object-oriented programming. In this case, the object used as a template is known as the parent object, and the new object, whose data and behavior are derived from the parent object, is known as the child. The child class, sometimes referred to as the subclass, contains all the data and behaviors of the parent class, also sometimes referred to as superclass. Inheriting the functionality of the parent class allows the programmer to create more specialized objects quickly.

Base Class

Vehicles

Wheeled Vehicles

Flying Vehicles

Cars

Trucks

Fixed Wings

Rotary

Coupe

Pickup

Jet

Chopper

Sedan

18 Wheeler

Propeller

Drone

Figure 7.3 Vehicles in a game

As we shall also see shortly, object-oriented languages also allow the programmer to control access to the data within a class, also known as members, so that only other objects that need to access that data may do so, while others simply are not allowed to access it, preventing potential errors and mishaps.

2. An intro to C#: Syntax and Basics

1. Our First Script

Unity supports the programming language C#, which has made it a widespread language for game development. Let’s start by taking a look at the syntax of C#. Some of this will also apply to other languages incidentally.

When creating a new script, Unity creates the following file, which opens

152

CODING FOR GAME AUDIO

by default in a Visual Studio, Microsoft’s IDE, Integrated Development Environment.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class exampleScript : MonoBehaviour

{

// Start is called before the frst frame update

void Start()

{

}

//Update is called once per frame

void Update()

{

}

}

At the top of the file, we notice three statements starting with the keyword using. This allows the compiler to access additional code, needed to run the code entered below. Removing these lines may cause the compiler to be unable to run the code successfully.

The first odd characters we might notice are semicolons at the end of each using statement. Semicolons are used to separate instructions to the computer and are sometimes called separators for that reason. Separators, as the name implies, are used to separate instructions. If a semicolon is forgotten an error will ensue, which Unity will display in the console.

Below the ‘using’ statements is the class declaration itself: public class exampleScript : MonoBehaviour

It is important that the class name, here ‘exampleScript’, matches the name of the text file created by Unity. This is done by default when creating a new script; Unity will name the class after the name of the script; do not change it after the fact from the finder, that will only confuse Unity and induce errors.

The colon between the class name and the word Monobehaviour is important. After a class name, at the top of a class declaration the colon means

‘extends’, or inherits from. According to the Unity manual, Monobehaviour is the base class from which every Unity script derives, although there are a few occasions where you will use another class when scripting. Monobehaviour does, among many other things, allow us to attach the script to an object. We can read the line:

public class exampleScript : MonoBehaviour

as meaning, in plain English:

the public class exampleScript extends from the base class Monobehaviour.

CODING FOR GAME AUDIO

153

Curly braces, when used after a class or method definition, indicate the start and end of a block of code. They can be used in other contexts to mean something similar, such as after a conditional statement (such as an IF statement, for instance). A missing curly bracket will also result in the compiler reporting an error. In this case, the curly brackets after Monobehaviour on line 6 signal the beginning of the class exampleScript and correspond to the last curly bracket in the script. Curly brackets are also used to delineate the start of both functions in this script, awake and update.

These functions are part of the Unity script Lifecycle. Every frame in a game repeats a cycle that calls a number of functions in a specific order. Knowing when these functions are called is crucial in order to make the best decisions when it comes to scripting.

Figure 7.4

154

CODING FOR GAME AUDIO

Awake() gets called only once in the script’s lifecycle, and the Unity documentation suggests it’s a good place to initialize variables, functions and other data prior to the start of the game or level.

Update() gets called once per frame and is a good place to put in any code that looks for changes in the game or any code that gets updated on a frame per frame basis.

The two forward slashes ahead of some of text lines are used to write comments. Any text following comments is ignored by the compiler and can be used by the programmer to add notes for future reference or as a reminder.

Comments are particularly useful when annotating code or making notes about future ideas to implement.

2. Variables, Constants, Data Types Operators, Arrays and Lists

a. Data Types

Computer languages use a strict classification of data types, which tells the compiler how to interpret the data, letting it know whether it’s a letter, word, number or another type. There are lots of data types, but for now we will focus on the most common ones, such as:

Integers: abbreviated ‘int’ in C#, used for whole numbers, no decimal point.

Floating point: abbreviated ‘float’ or ‘f ’ when declared with a number.

Floats are numbers with decimal points.

Booleans: abbreviated ‘bool’, Booleans are a logical data type that can be either true or false. They are false by default, unless specified otherwise.

Characters: abbreviated ‘char’, are a single (Unicode) character.

Strings: abbreviated ‘string’, is used for a sequence of characters or words.

Unity uses different data types for different purposes. For instance, the minimum and maximum distance range for an audio source are expressed as integers, while the source’s volume and pitch are expressed using floats. Finding out which data type to use is usually easy and solved by taking a look through the documentation.

b. Variables

Variables are used to store data or values by assigning them memory locations and a name, referred to as an identifier. As the name implies, the value of a variable can change within the lifespan of the program, either due to user input or based on internal game logic. Each variable must be declared and named by the programmer.

CODING FOR GAME AUDIO

155

When a variable is declared it can be also assigned a value at that time: foat sourceVolume = 0.9f;

int index;

The first statement declares a variable of type float, named volume and initialized with a value of 0.9. Naming variables can be tricky. While there are no hard rules on naming variables, you want the name to be descriptive and easy to understand. The naming convention used here is known as camel casing, where if the variable name is made of two words the first word will be lowercase while the first letter of the second word will be uppercase. This is common practice in the C# and Java programming language.

The second statement declares a variable of type integer named index but does not yet assign it a value. Variables can be of any data type, such as the ones we listed earlier in the chapter, but they can also be used to hold audio sources or audio clips:

public AudioClip woodenStep01;

The previous line declares a variable of type audio clip, named woodenstep01.

However, unless we load an actual audio file and assign it to the variable, either via script or by manually dragging an audio file on the slot created in the Unity editor (by making the variable public), no sound has been assigned at this point.

c. Arrays

Each variable can only hold a single value at a time. When working with larger data structures, declaring and initializing dozens of variables can quickly become tedious, hard to work with, and difficult to keep track of. This is where lists and arrays come in. Arrays allow us to store multiple bits of data, of a single type, in one container, making each data entry accessible via an index.

The length of the array remains fixed once defined.

Figure 7.5

When it comes to audio, a common case, amongst many others, where arrays are useful is footstep sounds. If we need to store four sounds for

156

CODING FOR GAME AUDIO

footsteps on wood, we could declare four individual audio clips variables and name them something appropriate, then assign a new clip at random each time a footstep is needed.

Four individual variables of type audio clip:

public AudioClip woodenStep01;

public AudioClip woodenStep02;

public AudioClip woodenStep03;

public AudioClip woodenStep04;

There are several drawbacks to using four individual variables. For one, it requires a bit of extra typing. Then, should we need to change the number of samples from four to six, we would need to edit the code and add another two variables. Keeping track of such changes can add unnecessary causes for errors, which can be hard to track down in the context of a larger project. A more elegant solution would be to declare an array of type audio clip, which can be more concisely written as:

public AudioClip[] woodenSteps;

This line creates an array of audio clips named woodenSteps, of length yet undetermined. Not declaring a specific length for the array in the script makes the code more flexible and easy to re-use. The practice of writing – or embed-ding data or values in code so that these cannot be changed – unless by altering the code itself is known as hard coding. This is considered poor practice, sometimes referred to as an AntiPattern, which is a way to solve a problem using a less-than-ideal solution. By making the array public, it will show up as a field in the inspector, and its length will be determined by the number of samples the developer will import in it by dragging them from the audio asset folder into the slot for the array in the inspector or specifying a length by typing it in directly into the slot for the array.

Note: an alternative to making the array public in order for it to show up in the inspector is to add [SerializeField] in front of the array declaration.

Figure 7.6

This makes the code flexible and easy to re-use. For instance, if we decide to change the numbers of footsteps in the game, the array will automatically resize as we drag more samples or decide to remove a few. Writing code that

CODING FOR GAME AUDIO

157

can be re-used easily is one of the staples of good programming habits, and we should always aim for nothing less.

By assigning our footsteps sounds to an array, we make it easy for the game engine and programmer to implement randomization of sample selection.

Individual entries in an array can be accessed by using the index number in which they are stored, as we shall see shortly.

The following line of code assigns entry number 3 (do keep in mind that the first entry in an array is 0, not 1) in our array of audio clips to the audio source named footStepAudioSource:

footStepAudioSource.clip = woodenSteps[2];

or we could assign the audio clip randomly using the following: footstepAudioSource.clip = woodenSteps[Random.Range(0, woodenstep.

Length)]

Rather than hardcoding a value for the top of the range, we simply call .Length, which will return the length of the array. This makes the code easier to re-use and allows us to change the length of the array or numbers of samples we use without having to touch the code.

d. Lists

Lists are similar to arrays but are sized dynamically, that is to say that unlike arrays, lists can change in length after they have been declared or that we do not need to know their length prior to using them.

In order to use lists, we must type the following at the top of our scripts, along with the rest of the using segments.

using System.Collections.Generic;

In order to declare a list, we need to first specify the data type that we want to store in the list, in this case audio clips, then we need to name it, in this case footSteps. The next step is to call the new keyword.

public List<AudioClip> footSteps = new List<AudioClip>(); Items in a list are accessed in the same way as in arrays, using an index.

footStepSource.clip = footSteps[0];

This line assigns the audio clip that corresponds to the first entry in the list footSteps to the audio source footStepSource. So, when should one use lists instead of arrays? Generally speaking, lists are more flexible, since they can be

158

CODING FOR GAME AUDIO

dynamically resized. If it is not possible to determine in advance the number of entries you will need to store/access, or if you are going to need to dynamically change the number of entries in the data, lists are best, otherwise, arrays are fine. In this book, we shall work with both.

e. Access Modifers

It is good practice to limit access to part of your code, such as variables or functions, to ensure that they do not get used or set to another value acciden-tally. This is done through a set of keywords known as access modifiers. Access modifiers may be applied to classes, methods or members. The most common of these are:

• public

• private

• protected

• static

public: this keyword doesn’t restrict access at all, and additionally, specific to Unity, any variable made public will show up as a field in the Unity inspector. A value entered in the inspector will take precedent over a value entered in code. This is a very convenient way to work and make changes easily without having to hard code any values; however, this alone is not a reason to make a variable public:

public foat sourceVolume = 0.9f;

Making a variable public for the sake of having it show up as a field in the Unity editor, however, may not be the best approach, as any variable can be made to show up in the inspector as a field by entering the following code above it:

[SerializeField]

foat sourceVolume = 0.9f;

This yields the same results in the inspector, without the need to make the variable public and thus shields our variable from being accessed inadvertently.

private: access is restricted only within the class. Other classes may not access this data directly.

protected: protected member will only be accessible from within its class, and derived classes (through inheritance).

static: the static keyword can be a bit confusing initially. Static members are common to all instances of a class and, unlike other members, their value

CODING FOR GAME AUDIO

159

is identical across all instances. Non static variables – or members – will exist in every instance of a class, but their value will be different in each instance.

Static members, in contrast, will have the same value across all instances.

Therefore, changing the value of a static member in one class instance will change it across all instances. Additionally, static members are in some way easier to access as they can be accessed without the need to instantiate an object of the class first. That means that a static function can be accessed without the need to create first an instance of a class. By the same logic, however, this also means that any class made static cannot be instantiated.

3. Accessing a Function From Another Class

Oftentimes you will need to access a function or variable defined in another class. This is a very common situation that can be somewhat confusing to beginners. Accessing a function from another class can be done in one of several ways.

If the function you are trying to access is a static method, the following example uses pseudo code and calls the function function1(), which is static and public, from another class, named ‘call’. Because the function is static and public, we can access function1() from another class by calling the name of the class it is defined in, followed by the name of the function: public class GenericClass

{

public static void function1()

{

// code

}

}

public class call()

{

public void GenericFunction()

{

GenericClass.Function1(); // calling the static function function1() in the class GenericClass

}

}

If the function you are trying to access isn’t a static one, accessing is from another class is only a slightly different process.

public class GenericClass

{

public void function1()

{

160

CODING FOR GAME AUDIO

// code

}

}

public class call()

{

public void GenericFunction()

{

GenericClass.instance.Function1(); // calling the static function function1() in the class GenericClass

}

}

In this case we simply call the function by referencing the class and using the instance keyword.

3. Playing Audio in Unity

1. Our First Audio Script

Let’s get started with sound and scripting with a simple scenario: we want a sound to play as soon as the level starts and loop it. Of course, we could do this without needing to use a script by checking the PlayOnAwake and Loop checkboxes of the audio source itself, but that would defeat the purpose of this example, and without using scripting, we are extremely limited should we wish to perform additional tasks.

Let’s outline the basic steps necessary to achieve the desired outcome:

• Create an empty object to add an audio source to as a component or add an audio source as a component to an already existing object.

Don’t forget to make sure PlayOnAwake isn’t checked.

• Assign an audio clip to the audio source and adjust parameters as desired.

Make sure the volume property is properly set (above zero).

• Create a new C# script, and give it a descriptive name. This will create a new class as well.

• Determine where in the execution of the script we wish to trigger the sound, i.e., when do we want the sound to play? In this case on Awake().

• Gain access to a reference to that audio source in the script through the GetComponent() method, and access its Play() method.

• Add the script to the same object that we added the audio source to.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

CODING FOR GAME AUDIO

161

public class loopableAmbience : MonoBehaviour

{

void Start()

{

GetComponent<AudioSource>().loop = true;

GetComponent<AudioSource>().Play();

}

We call the class ‘loopableAmbience’ and are using the provided Start() function to access the audio source, since we want the audio to play as soon as the level starts. In order to access the audio source component we use the GetComponent() function and specify the component type using the <> brackets, in this case, an audio source. First, we set the audio source to loop by setting its loop property to true. Then, in order to start the audio source, we use the play() function. In essence the line:

GetComponent<AudioSource>().Play();

could read as: access the component of type audio source and play it.

This example is about as basic as can be, and we can improve it in several ways. Let’s begin by giving the user a little bit more control from the script by setting a value for the pitch and volume parameters of our audio source. If we specify a value for pitch and amplitude in code, we would have to modify this script to change these values for a different sound, or write a different one altogether. This process, known as hard coding, is not a very flexible solution.

Instead we can declare two variables for pitch and amplitude and assign them a value from the inspector. This will make our script for loopable ambiences easily reusable across multiple objects.

Here’s an updated version of the code:

Public class loopableAmbience : MonoBehaviour

{

[SerializeField]

[Range (0f, 1f)]

private foat sourceVolume;

[SerializeField]

[Range(0f, 1f )]

private foat sourcePitch;

private AudioSource ambientLoop;

void Start()

{

ambientLoop = GetComponent<AudioSource>(); ambientLoop.loop = true;

162

CODING FOR GAME AUDIO

ambientLoop.pitch = sourcePitch;

ambientLoop.volume = sourceVolume;

ambientLoop.Play();

}

}

By using [SerializeField] above the variable declarations, we get access to them in the editor without the need to make them public. Additionally, by adding

[Range (0f, 1f)] below it, we create a slider to enter these values, rather than the default number box. We’ve declared an audio source called ambientLoop in a more sophisticated manner than simply dropping an audio source in the level and checking its PlayOnAwake property. However, we might still wish to add another bit of functionality to it before moving on, such as the ability to randomize pitch and amplitude. Pitch and amplitude randomization are very common tools in game audio as a way to maximize the use of samples, allowing us to re-use them without sounding too repetitive. In order to do so, we’re going to call the Random.Range() function and allow the user to add a random offset to both pitch and amplitude. The main thing to keep in mind when using pitch and amplitude randomization is that finding the right range for the random values is critical. For instance, too much pitch randomization may give sounds a pitch and make our sample sound too musical or plain distracting. Too little randomization and the effect is lost altogether. Experimentation is usually required.

Public class loopableAmbience : MonoBehaviour

{

[SerializeField]

[Range(0f, 1f )]

private foat sourcePitch, sourceVolume, volOfset, pitchOfset; private AudioSource ambientLoop;

void Start()

{

ambientLoop = GetComponent<AudioSource>(); ambientLoop.loop = true;

ambientLoop.pitch = sourcePitch + Random.Range(0f, pitchOfset); ambientLoop.volume = sourceVolume + Random.Range(0f, volOfset); ambientLoop.Play();

}

}

This method adds a random number between 0 and the value specified by each slider to the volume and amplitude values for the audio source. If the volume was set to 1 in the first place, there is no additional room for amplitude, but it

CODING FOR GAME AUDIO

163

is a starting point that allows us some control over the amount of randomization for each audio source’s pitch and volume properties. If you are new to this technique try to load different sound clips in the audio source and experiment with small to large random offsets and notice their effect on each sound.

2. Play() vs. PlayOneShot()

So far we have relied on the Play() method to play an audio file. Another way of triggering sounds is with the PlayOneShot() method, which works slightly differently from the Play() method, the Unity API describes it as follow:

public void PlayOneShot(AudioClip clip, foat volumeScale = 1.0F); and it can be used in a somewhat similar fashion to Play() but with a few major differences. Here’s a simple example of code using PlayOneShot(): using UnityEngine;

using System.Collections;

[RequireComponent(typeof(AudioSource))]

public class PlayAudio : MonoBehaviour

{

public AudioClip mySoundClip;

AudioSource audio01;

void Awake()

{

audio01 = GetComponent<AudioSource>();

}

void Start()

{

audio01.PlayOneShot(mySoundClip,

0.90f);

}

}

This code will play the clip mySoundClip upon start but will do so using PlayOneShot() rather than Play(). You’ll notice a few differences in the way we use PlayOneShot() compared to Play(): For one, the PlayOneShot() method takes a few arguments: the audio clip to be played and a volume parameter, which makes it a convenient way to scale or randomize the amplitude of an audio source. Other properties of the audio source will be inherited from the audio source passed to the function: audio01.PlayOneShot(mySoundClip, 0.90f);

In this case, the audio source audio01 will be used to play the clip mySoundClip.

164

CODING FOR GAME AUDIO

A major difference between Play() and PlayOneShot() is that when using PlayOneShot(), multiple clips can be triggered by the same audio sources without getting cut off by each other. This makes PlayOneShot() extremely useful for repeating audio sources such as machine guns for instance. A drawback of this method, however, is that it is not possible to stop the playback of an audio source once the playback starts, making this method best suited for shorter sounds rather than long ones.

3. Using Triggers

Triggers are a staple of gaming. They are used in many contexts, not just audio, but they are especially useful for our purposes. A trigger can be defined as an area in the game, either 2D or 3D, which we specifically monitor to find out whether something, usually the player, has entered it, is staying within its bounds or is exiting the trigger area. They allow us to play a sound or sounds for each of these scenarios, depending on our needs as developers. A simple example would be to play an alarm sound when the player walks in a certain area in a level, which would also call hostile AI and start a battle sequence for instance.

Triggers in game engines are usually in the shapes of geometric primitives, such as spheres or cubes, but more complex shapes are possible in most engines. In order to add a trigger to a level in Unity, one must first add a collider component to an empty game object, though it is also possible to add a collider to an existing game object. When adding a collider, we must choose its shape, which will be the shape of our trigger, whether 2D or 3D, cube, sphere etc.

Once the appropriate collider component has been added, we can adjust its dimensions using the size number boxes for the x, y and z axis and position it on the map as desired. It is not yet a trigger, however, it will remain a collider until the ‘isTrigger’ checkbox is checked.

Note: triggers will detect colliders; you therefore must make sure that any object you wish to use with a trigger has a collider component attached.

The white cube pictured below in Figure 7.7 will act as a trigger since its collider component has its isTrigger property checked.

Figure 7.7

CODING FOR GAME AUDIO

165

Once the ‘isTrigger’ box is checked the collider is ready to be used. We can access the collider via code by attaching a script to the same object as the collider and using the functions:

• OnTriggerEnter(): for detecting movement into the trigger, the collider touching the trigger.

• OnTriggerStay(): gets called for almost every frame the collider is touching the trigger.

• OnTriggerExit(): gets called when the collider has stopped touching the trigger.

In the following example we will use the OnTriggerEnter() and OnTriggerExit() functions to turn a sound on and off as the player enters and leaves the trigger respectively. In order to make sure the sound is indeed triggered by the player and not anything else, such as an AI entity, we must gather information regarding the collider that touches or untouches the trigger. In other words, we want to ask any collider that enters the trigger if it is the player. One simple way to do this is by using the tagging system in Unity.

By tagging the first-person controller in the game with the word ‘Player’ we can simply check the tag of any object that collides with the trigger, ignoring all other tags.

Let’s outline the basic steps necessary to achieve the desired outcome:

• Create an empty object.

• Add a collider component to it, check its ‘isTrigger’ box.

• Adjust its size and location.

• Add an audio source component to the object.

• Create a script to:

• Assign an audio clip to the audio source component.

• Access appropriate trigger function ( OnTriggerEnter() for instance).

• Check if the collider entering or leaving the trigger is indeed the player.

• Play the desired audio clip when the player enters the trigger.

• Stop the audio clip from playing upon leaving the trigger.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class audioTrigger : MonoBehaviour

{

private AudioSource triggerAudio;

[SerializeField]

private AudioClip triggerClip;

166

CODING FOR GAME AUDIO

void Start()

{

triggerAudio = GetComponent<AudioSource>(); triggerAudio.clip = triggerClip;

}

private void OnTriggerEnter(Collider other)

{

if (other.CompareTag(“Player”)) {

triggerAudio.Play();

}

}

private void OnTriggerExit(Collider other)

{

if

(other.CompareTag(“Player”)){

triggerAudio.Stop();

}

}

As you enter the area where the trigger is located, as long as the tag

‘Player’ was added to the first-person controller you are using you should be able to hear the sound start to play and then stop as you leave the trigger area.

4. Sample Randomization

Another common issue in game audio has to do with sample randomization.

The ability to play a sample at random from a pool of sounds is very useful.

We can do this either with lists or arrays. In this next example, we’ll modify the previous example to trigger a sound at random when we enter the trigger. Additionally, we will make sure that the engine does not trigger the same sound twice in a row, as that can be very distracting.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class RandomTrigger : MonoBehaviour

{

private int currentClip, previousClip;

private AudioSource triggerAudio;

[SerializeField]

private AudioClip[] triggerClip;

void

Start()

{

CODING FOR GAME AUDIO

167

triggerAudio = GetComponent<AudioSource>();

}

private void OnTriggerEnter(Collider other)

{

if

(other.CompareTag(“Player”))

{

while (currentClip == previousClip)

currentClip = Random.Range(0, triggerClip.Length); triggerAudio.clip

=

triggerClip[currentClip];

triggerAudio.Play();

previousClip

=

currentClip;

}

}

private void OnTriggerExit(Collider other)

{

if

(other.CompareTag(“Player”))

{

triggerAudio.Stop();

}

}

}

We could also implement sample randomization with a list rather than an array. Lists are more flexible and can dynamically alter the length of the number of samples we work with. This would be helpful in the context of a game such as Simon, for instance, where we don’t know ahead of time how many entries we will need to keep track of.

5. Detecting Keyboard Events

Checking for user input is a very common operation. Typically, user input will occur in the form of keyboard, joystick or gamepad. Unity supports all these methods and offers us multiple ways to check for user input. Here we will create a simple script that will allow us to turn sounds on and off by pressing the 1 key on the keyboard and turn off the same sounds by pressing the 2 key.

In this example we will use Input.GetKeyDown() to check whether the user is pressing the right key.

Typically, user input code is placed within the Update() function.

void Update()

{

if

(!enablePlayMode)

{

Debug.Log(“NotPlaying”);

168

CODING FOR GAME AUDIO

if

(Input.GetKeyDown(KeyCode.Alpha1))

{

enablePlayMode

=

true;

StartSound();

}

}

else if (enablePlayMode)

{

if

(Input.GetKeyDown(KeyCode.Alpha2))

{

enablePlayMode

=

false;

StopSound();

}

}

}

GetKeyDown() takes a single argument, an integer that represents the last key pressed on the keyboard. A complete listing of each key and corresponding keycode can be found on the Unity website.

Note: it’s not usually a good idea to link a keystroke directly to an action such as accessing the play method of an audio source; instead, it is better to call a function from which to call the audio source’s play method. This is because the purpose of a key or the action it needs to trigger may change with context. For instance, the ‘W’ or forward key can be used to control a character’s movement, which means walking but also possibly swim-ming if the gameplay allows it. A more modular implementation is usually recommended.

6. Audio-Specifc Issues

a. Timing – Frame Rate vs. Absolute Time

Frame rates are impossible to predict accurately across computers and mobile platforms and may vary wildly based on the hardware used. Therefore, we should not rely on frame rate when dealing with events whose timing is important, which is often the case in audio. Consider fades, for instance. We could initiate a fade-in by increasing the amplitude of an audio source by a certain amount at each frame until the desired amplitude has been achieved, however, since the time between frames will vary from one computer to another, it is difficult to predict exactly how long the fade will take. A better solution would be to use an absolute timing reference and increase the volume by a specific amount at regular intervals. Unity has a time class that can help us, and more specifically the deltaTime variable, which can be accessed to let us know how much time has elapsed since the last frame as a float. To be exact, deltaTime measures the amount of time since the last Update() function was called. The

CODING FOR GAME AUDIO

169

variable deltaTime can be used as a multiplier and specify an absolute timing reference for the duration of the fade.

Fades bring us to another point that is often relevant to audio, that is, that we might at times need to keep track of an audio source or object for that matter over multiple frames, which requires us to use a special type of function, known as coroutines. Coroutines always have return type of IEnumerator and are called slightly differently from other functions.

Coroutines are different from other functions in so far that they can pause execution at each frame, relinquish control to Unity and pick up where they left off at the next frame.

Let’s try an example of a script that can be used to do fade-ins and fade-outs as the player enters or leaves a trigger, instead of abruptly stopping or starting the audio source.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class TriggerFades : MonoBehaviour

{

[SerializeField]

private AudioSource triggerSource;

[SerializeField]

private AudioClip triggerClip;

[SerializeField]

private foat fadeTime = 1f;

bool inCoRoutine;

void Awake()

{

triggerSource = GetComponent<AudioSource>(); triggerSource.clip = triggerClip;

}

private void OnTriggerEnter(Collider other)

{

inCoRoutine = true;

StartCoroutine(FadeIn(triggerSource,

fadeTime));

}

private void OnTriggerExit(Collider other)

{

StartCoroutine(FadeOut(triggerSource,

fadeTime));

}

public static IEnumerator FadeOut(AudioSource triggerSource, foat fadeTime)

{

170

CODING FOR GAME AUDIO

foat startVolume = triggerSource.volume;

while (triggerSource.volume > 0f)

{

triggerSource.volume -= (Time.deltaTime/fadeTime); Debug.Log(Time.deltaTime);

yield return null;

}

triggerSource.Stop();

triggerSource.volume = 0f;

}

public static IEnumerator FadeIn(AudioSource triggerSource, foat fadeTime)

{

foat startVolume = 0.0f;

triggerSource.Play();

triggerSource.volume = startVolume;

while (triggerSource.volume < 0.95f)

{

triggerSource.volume

+=

(Time.deltaTime/fadeTime);

yield

return

null;

}

}

}

b. Linear vs. Logarithmic Amplitude

The volume slider of an audio source in Unity is a linear value from 0 to 1. Audio engineers and content creators are used to working with DAWs, which map the amplitude of a track to a logarithmic slider, which gives us much more resolution and as such a better way to control the level of our tracks. It also provides a more accurate representation of how human beings perceive sound, more akin to the decibel scale that we are used to. Mixers in Unity do give us the ability to work with logarithmic sliders; however, some might find the linear volume mapping of audio sources awkward to work with.

Another issue with randomization while working with a linear amplitude scale is randomization. A random value of plus or minus 0.2 will sound different whether the audio source it is applied to has a starting value of 0.8 or 0.2. Working with a decibel scale can help with these issues as well.

We can remedy this by a simple script, which will remap the linear amplitude of an audio source from 0 to 1, to a decibel scale using the formula: dB = 20 * Log10(linear)

CODING FOR GAME AUDIO

171

Where:

• dB is the resulting value in decibels.

• Linear is the value of the audio source from 0.0001 to 1, which will translate to a range from −80 to 0dB (0 is not an acceptable value).

We can convert the value of a number in dB back to a linear amplitude using this formula:

Linear = 10 Pow (dB/20)

Where:

• Linear represents the value of an audio source from 0.0001 to 1.

• dB is the value in dB to be converted back to linear.

Armed with this knowledge we can write a separate class whose purpose will be to handle these conversions for us. This is usually known as a utility class: using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class AudioUtility

{

public static foat dbToVol(foat dB) // takes a value in dB and turns it into a linear value

{

return Mathf.Pow(10.0f, 20 / dB);

}

public static foat VolTodB(foat linear) //takes a linear value and turns it into dB

{

return 20.0f * Mathf.Log10(volume);

}

}

You’ll notice that two static functions were created, dbToVol(), which will take a value expressed in decibels and turn it back into a linear value and VolTodB(), which will perform the opposite function. Each takes a float as an argument, and since it is located in a separate utility class, it will need to be accessed from another function. Since they are both static functions they will not need to be instantiated when accessed from another class.

To use the functions from another class one must simply type: foat linearVol = AudioUtils.dbToVolume(−20);

audioSource01.volume = linearVol;

172

CODING FOR GAME AUDIO

Conclusion

In this chapter you were introduced to the basics of scripting in Unity and C#.

Some of these concepts ought to take a moment to sink in, and you should experiment with them, modify the code, break it, fix it and always attempt to learn more about the many topics introduced here. Further exploration and experimentation is key. In the next chapter we will build upon these concepts and revisit a few in the context of more practical situations, learn how to work with triggers and much more.

8 IMPLEMENTING AUDIO

Common Scenarios

Learning Objectives

Great sound design is only as good as the way it is implemented and mixed in the game. An amazing sound will lose a lot of its impact and power if triggered at the wrong time or at the wrong pitch, volume or distance. Audio implementation is the area of game development that focuses on the mechanics behind the sounds and music coming out of the speakers or headphones, and is responsible for creating or properly exploiting the features needed for the sounds to be properly presented in the mix and create a successful interactive soundtrack. Implementation is increasingly becoming a creative discipline as much as it is a technical one and can often augment the impact and success of the sound design. By the same logic, poor audio implementation can also greatly diminish the impact of a soundtrack and the work of the sound design and music team. In this chapter we build upon the concepts covered in Chapter seven and learn to apply these in practical scenarios coming from common gaming situations. We will start by adding a simple sound to a room using the Unity editor only in the simplest of ways and build gradually from there introducing and developing the concepts learned in Chapter seven. We will cover triggers, collisions, raycasting and much more.

1. Before You Start: Preparing Your Assets

We introduced the topic of asset preparation in the chapter on sound design, but as we now tackle the topic of implementation, it is worth revisiting. In order to properly prepare your assets, you should be aware of how these assets are going to be implemented in the game and what their intended purpose is.

A spreadsheet with a list of all the sounds in the level is an invaluable ally. This will tell the sound designer which sounds are loops and should be seamless, which sounds are 3D and should most likely be mono etc.

174

IMPLEMENTING AUDIO

Making sure the assets are ready does involve a checklist:

• Naming convention.

• File format, sampling rate, bit depth, number of channels.

• Number of variations, if any.

• Loop or one shot.

• Consistency quality control: are the levels of the sound consistent with other similar sounds?

• Trim/fades: is the sound properly trimmed and, if appropriate, faded in/out?

A batch processor is highly recommended. It will save you a lot of time both in terms of mouse clicks and in terms of potential human errors when dealing with dozens if not hundreds of audio files. A good batch processor will help you address all the issues cited earlier, from naming conventions to the inclusion of micro fades.

Once you are sure of your assets, you are ready to import them into the game engine and begin the process of implementing them and testing them in the context of the game. You will sometimes find that in-game some sounds might not work how you had expected them to initially and possibly require you to re-think them. The creative process is often iterative, and keeping your work organized is a must.

In the following chapter we will try to tackle some common scenarios you are likely to encounter when dealing with audio implementation, such as:

• Creating and working with loops for ambience and backgrounds.

• Using triggers for loop for 2D and 3D ambiences.

• Working with random emitters to create a richer soundscape.

• Collisions.

• Surface and velocity-dependent collisions.

• Distance crossfades.

• Sound concatenation.

• Raycasting for occlusion simulation.

• Adding sound to animation clips.

• Working with prefabs.

2. Ambiences and Loops

Ambiences and environmental sounds in the real world are quite a bit more complex than they might appear to the casual listener. Ambiences are usually comprised of several layers or sounds, some constant, such as a fan or an AC

unit, others intermittent, such as birds or the honking of cars in a city. In order to create an immersive experience, we must create a multilayered landscape that provides the user with a rich, dynamic soundscape that will combine multiple implementation techniques. The foundational layer for ambience sounds

IMPLEMENTING AUDIO

175

often relies on one or multiple audio loops playing concurrently, possibly at several individual locations in the same space. Creating loops for ambiences isn’t very difficult technically, but the aesthetic challenge of creating something that can be both interesting and at the same time un-obtrusive is a difficult one.

1. Creating Ambiences and Loops

Loops are a staple for creating ambiences in games. At its simplest, a loop is just a short audio file, six to 12 seconds, created to add the sound of a hum or a room tone to a space. Loops can, however, be combined in order to create more sophisticated ambiences. Before looking at possible combinations, let’s take a moment to consider what makes a good loop and how to easily create seamless loops.

a. Seamless Loops

There are a few things to keep in mind when creating or selecting material for seamless loops:

• Length: how long should your loops be? The answer here is only as long as you need them to be. This, of course, will depend on how the loop will be used in the game. For simple ambiences, shorter loops such as eight to 12 seconds might be a good place to start. Remember we are always trying to keep the RAM footprint of our sounds to a minimum and trying to get the most out of the least.

• Mono vs. stereo: as always, when confronted with this choice, consider whether you need the loop to be localized in 3D or not. In other words, sounds that ought to emanate from a place within the level should be mono. Sounds for which 3D localization is not desirable can be rendered as stereo. Wind and rain are good examples of ambient loops that would sound unnatural if they appeared to come from a single direction. These are usually best left 2D and rendered in stereo.

You can always force a stereo sound to play back in mono from the Unity editor if unsure or both versions are somehow needed.

• Sample choice: how does one choose appropriate audio files for looping? Look for a sample that is relatively even over the life of the loop.

Avoid including any portion that includes sound that could stand out upon playback and draw attention to itself and remind the user that they are listening to a loop. The sound of someone sharply and loudly laughing among a crowd ambience or a particularly loud bird call, for instance, are good examples of elements to avoid.

• Layering: your loops do not need to be bland or boring, and you can achieve interesting results by layering multiple audio files, so long as it does not conflict with the previous rule. Create loops of slightly different lengths. Asynchronous loops create a more dynamic ambience by looping at different times and avoid repetition fatigue.

176

IMPLEMENTING AUDIO

Figure 8.1

• The big picture: ambient loops often act as the foundational layer of your sound design, upon which all other sounds will exist. While it is difficult to predict which sounds are going to be triggered in a game at any given time, you can help maintain consistency in your mix by keeping your loops within a similar ‘spectral niche’ by ensuring the frequency content is consistent across all loops. For instance, avoid creating loops with a lot of low end, as they might clash with the music or other sounds that are more important to the player and could be partially masked by it. A high pass filter in the 100–200Hz range can be very effective in that regard.

b. Creating a Simple Loop – Looping Techniques

As long as you are working with a sample that is relatively consistent and that abides by the first rule outlined earlier, you can turn most sounds into a seamless loop with little effort:

1. Import your audio file into your DAW of choice. Make sure to work with a sample that is at least a few seconds longer than you need the length of the loop to be.

Figure 8.2

IMPLEMENTING AUDIO

177

2. Somewhere near the middle of the loop, split the audio region in two.

Do not add fades or micro fades to either one. This would break the waveform continuity required for a seamless loop to work.

Figure 8.3

3. Reverse the order of the regions by dragging the first region so it starts after the second one, giving yourself a few seconds overlap or ‘handles’ between the two, which you will use for a crossfade.

Figure 8.4

4. At the place where both regions overlap, use your best judgement to find a good spot to crossfade between the two regions. Make sure to use an equal power fade, rather than an equal gain fade. Equal power fades maintain the energy level constant across the fades; equal gain fades do not and may result in a perceived drop of amplitude in the middle of the fade. This step requires the most experimentation and is worth spending some time on. Some material is easier than others to work with.

Figure 8.5

178

IMPLEMENTING AUDIO

5. Once you are satisfied with the crossfade, select both regions exactly, down to the sample, and set your DAW to loop playback mode to listen to the results. The transition between your exit and entry points should be seamless, as the wave form should be continuous. You are done and ready to export your loop as an audio file. Always make sure to mind your audio levels, though.

c. Creating Variations

Creating variations of sounds is a challenge especially common to game developers, which we must confront in many areas, ambiences being one of them.

Most sounds we create are layered – or probably should be – in order to be interesting. Once you’ve created an interesting ambience by layering a few audio layers, you are at a good place to start thinking about generating variations.

First establish the sound, through experimentation and by using any of the techniques outlined in the sound design chapter or of your own making.

Once you have obtained satisfactory results, work on variations by using some of these techniques:

• Pitch shift one or more of the layers. The range you choose for pitch shifting depends on many factors, but what you are trying to achieve is variations without the pitch shift becoming distracting or musical when the samples are played in a row.

• Swap one or more of the layers with a similar but different sample. It may be a new file altogether or a different portion of the same file/region.

• Add subtle effects to one of the layers, for one or more variations, such as mild distortion, modulation effects etc.

• Alter the mix slightly for each layer from one variation to the next.

Again, be careful not to change the overall mix and the focus of the sound.

• Combine all the previous techniques and more of your making to create as many variations as possible.

This list is by no means exhaustive, and over time you will likely come up with more techniques, but when in doubt, you can always refer back to this list.

2. Implementing Our Loops in a Unity Level

Once the loops have been created, the next question is, of course, how do we best implement them in the game?

a. Challenges

Let’s start with 2D sounds. The geographical placement of these in the level matters little, as they will be heard evenly throughout the scene, only able to be

IMPLEMENTING AUDIO

179

panned in the stereo field if the designer desires it. They can be attached to an empty game object and moved anywhere out of the way where it’s convenient.

3D sounds can require a bit more attention. Let’s start with a simple example: two rooms, a 2D ambience playing across both, the sound of outside rain and a single audio source set to 3D spatial blend in the center of each room.

Figure 8.6

Here we come face to face with one of limitations of the Unity audio engine. Audio sources are defined as spheres within the level, which, of course, doesn’t bode well with the geometry of most rooms, which tend to be rectangular. Remember that audio sources are not stopped by objects that may be located in front of them, and sound travels through walls unaffected. Later, we will look at ways to compensate for this, but for now, when using a single audio source to cover an entire room we are left with a few options: 1. In order to avoid blind spots, we extend the radius of the sphere, which means that it will also be heard if a player is standing close to a wall in the next room. This may be acceptable in some cases, but it is usually not okay.

2. We restrict the radius of the sphere so that it only covers the room it is located in, but we are left with audio blind spots in the corners. Again, this may be an acceptable option sometimes, but it is not generally okay.

3. We add smaller audio sources in the corners to cover the audio blind spots. However, if we simply duplicate the audio source in the center of the room and shrink its radius to fit the corners, we are inevitably left with areas where the sources will overlap, creating audio phasing, similar to comb filtering, which is definitely not okay.

180

IMPLEMENTING AUDIO

As you can see, none of these solutions is entirely, if at all, satisfactory.

One solution is to extend the radius of each audio source so that it covers the entire room and therefore spills over in the next room, but we will use triggers to turn the sound on/off between two rooms and control the bleeding of sounds into the other room. We will set up the sound of the rain to be heard evenly throughout the level, and it will remain a 2D stereo ambience. Inside each room we will set up a simple ambience that will play in 3D located in the center of each room.

This solution works relatively well when it comes to controlling the bleeding of audio sources from one room to another, but it is not a very exciting ambience as the sound remains somewhat monolithic throughout the room.

This may be okay for a top-down 2D type game or a casual game, but it will definitely seem a little weak in the context of a more developed first-person game and even more so in VR.

b. Spatial Distribution

An alternative approach is spatial distribution. Spatial distribution of ambient loops is the idea that a single ambient loop in a room will not always suffice to provide a satisfactory solution to our problem and that we can create better, more immersive ambiences by distributing several loops across the space. By distributing ambiences around the room we create a much more interesting soundscape, one that evolves with the space and over time, especially if these ambiences are of slightly different length.

When it comes to implementation of ambiences in Unity, it is possible to attach an audio source to an already existing object, but for the sake of organization I recommend creating empty game objects and adding an audio source and script to these. With a good naming convention, it will make it much easier to find your audio sources in the hierarchy, and you can easily turn those into prefabs, which makes them very easy to re-use.

The spatial arrangement or configuration of spatially distributed ambient audio sources is endless, but a common configuration is similar to a quad set up, putting one audio source toward each corner of a room with overlap so that there are no audio blind spots.

For this technique to work and avoid phasing issues, it is important that the audio sources that overlap each play a different sound clip and that they each be set to 3D spatial blend. The parameter adjustments of each audio source will depend on many factors, such as room size, type of sound etc. The thing to keep in mind is that you are usually looking for a smooth, gradual transition in the room tone as you walk around and that no audio source should stand out as the player moves from one position to another. The main factors to keep in mind are of course going to be the adjustment of the minimum and maximum radius of each audio source, their individual volume and the shape of the fall-off curve. Adjust each one until the results are smooth and

IMPLEMENTING AUDIO

181

Figure 8.7 Quad confguration

satisfactory. If needed you can also adjust the placement of each audio source in the space.

When setting an audio source’s spatial blend property to 3D, the default setting for the spread parameter is zero, which makes the audio source very narrow overall in the sound field. A very narrow audio source can make panning associated with movements of the listener feel a bit abrupt and unnatural, at best distracting. You can use, and probably should use, the spread parameter to mitigate that effect by increasing the value until the sound feels more natural when you are moving about the space. Experimentation is encouraged. Too small a value and the benefits may be negligible, too big a value and the panning will become less and less obvious as the audio source occupies an increasingly wider area in your sound field.

c. Working With the Time Property to Avoid Phasing Issues

There may be times where you will find it difficult to prevent two or more audio files playing in overlapping areas at the same time, which will usually result in phasing issues. Phasing will make the sound appear hollow and unnatural. One

182

IMPLEMENTING AUDIO

way to prevent or mitigate the phasing is to randomize the start time of the playback of the audio clip in at least one audio source. This can be done with the time property, which can be used to change or report the start time of the playback position of an audio clip, although the time property is applied to an audio source.

audioSource.clip = impact;

audioSource.time = Random.Range(0f, impact.length);; audioSource.Play();

This example code uses the length property of an audio clip, which will return its duration and is used as the upper range for randomizing the start time of the playback.

3. Random Emitters

Ambient loops are a great way to lay down the sonic foundation of our level, but in order to create a rich, dynamic environment we need more than just loops.

Another very helpful tool is random emitters. The term emitter is used somewhat loosely in the interactive audio industry, but in this case, we will use it to describes sound objects which are usually 3D, which can play one or often multiple sound clips in succession, picked at random, and played at random intervals. They are often meant to play somewhat sparingly, although that is in no way a rule. For instance, in an outdoors level we might use random emitters for the occasional bird calls rather than relying on an ambient loop. Random emitters represent a number of benefits over loops. It would take a rather long piece of audio in order for our bird calls not to sounds like a, well, loop, when played over and over.

Probably several minutes, perhaps more if the player spends a lot of time in the environment. That of course means a large memory footprint for a sound that, while it may be useful to contribute to immersion, does not play a significant part in the game itself. If the bird calls are spaced well apart, most of that audio may end up being silence. Another issue is that a long ambient loop is static; it cannot change much to reflect the action in the game at that moment. By using a random emitter, we control the amount of time between calls and therefore the density of the birds in the level, and it can be adjusted it in real time easily via script. Furthermore, each bird call can be randomized in terms of pitch and amplitude or even distance from the listener, and by placing a few random emitters around the level, we can also create a rich, 360-degree environment. Combined with ambient loops, random emitters will start to give us a realistic and immersive soundtrack.

Bird call long loop: a few audio events separated by silence. Looping predictably.

Audio Events

Figure 8.8

IMPLEMENTING AUDIO

183

a. A Simple Random Emitter Algorithm

Let’s break down what we want our intermittent emitter to do: 1. Wait for a random amount of time, specified within a minimum and maximum range in seconds. (It would be awkward if instead of starting with the silence portion we started by playing a sample. If multiple versions of the script were added to the level, it would mean that our level would start with multiple sounds all playing at once.) 2. Pick a sample at random from an array. Optional: avoid repeating the last sample played by placing it at the start of the array – entry 0- and only picking samples from index 1 and up.

3. Randomize pitch and amplitude, possibly distance from the listener.

4. Play the audio clip.

5. Do it again.

Because we are likely to be using random emitters in more than one place, as always, we want our code to be as easy to re-use as possible. To that extent we will add a few additional features in our script. For one, we will check to see if an audio source component already exists, and if none is found, our script will automatically attach one to the same object as our script. We will make sure all the most important or relevant settings of the audio source, whether one is already present or not, can be set from the script and then passed to the audio source. We will give the user control over:

• The number of sounds the user can load in the script.

• Whether it’s a 2D or 3D source.

• The minimum and maximum amount of time between sounds.

• The source volume and randomization range.

• Additionally, we will also randomize the maximum distance range of the audio source, which will further add a sense of realism by appearing to modulate the distance from the listener at each iteration.

We will create a function that will perform these tasks and use a coroutine to keep track of how much time to wait between samples by adding the random offset the computer picked to the length of the sample selected.

b. Coroutines

The lifespan of a function is usually just one frame. It gets called, runs, then returns, all in a single frame. This makes it difficult to use functions to work with actions that require the game engine to keep track of something over multiple frames. For this purpose, we can use coroutines.

A coroutine is akin to a regular function, but its lifespan can encompass multiple frames, and the coroutine keeps track of where it last left off and picks up from that same spot at the next frame cycle.

184

IMPLEMENTING AUDIO

Coroutines always have a return type of IEnumerator and include a yield return statement. Coroutines are called using the StartCoroutine(‘NameOfCoroutine’) statement. In this example, we will use the yield return new WaitForSeconds() statement to introduce a random pause in the execution of our code.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class IntermittentSounds : MonoBehaviour

{

[SerializeField]

private AudioSource _Speaker01;

private AudioLow passFilter _lpFilter;

[Range(0f, 1f )]

public foat minVol, maxVol, SourceVol;

[Range(0f, 30f )]

public foat minTime, maxTime;

[Range(0, 50)]

public int distRand, maxDist;

[Range(0f, 1.1f)]

public foat spatialBlend;

public AudioClip[] pcmData;

public bool enablePlayMode;

private AudioRollofMode sourceRollofMode = AudioRollofMode.Custom; void Awake()

{

_Speaker01 = GetComponent<AudioSource>();

if (_Speaker01 == null)

{

_Speaker01

=

gameObject.AddComponent<AudioSource>();

}

}

void Start()

{

_Speaker01.playOnAwake = false;

_Speaker01.loop = false;

_Speaker01.volume = 0.1f;

}

// Update is called once per frame

void Update()

{

if

(!enablePlayMode)

IMPLEMENTING AUDIO

185

{

Debug.Log(“NotPlaying”);

if

(Input.GetKeyDown(KeyCode.Alpha1))

{

enablePlayMode

=

true;

StartCoroutine(“Waitforit”);

}

}

else if (enablePlayMode)

{

if

(Input.GetKeyDown(KeyCode.Alpha2))

{

StopSound();

}

}

}

public void SetSourceProperties(AudioClip audioData, foat minVol, foat maxVol, int minDist, int maxDist, foat SpatialBlend)

{

_Speaker01.loop = false;

_Speaker01.maxDistance = maxDist – Random.Range(0f, distRand); _Speaker01.rollofMode = sourceRollofMode;

_Speaker01.spatialBlend = spatialBlend;

_Speaker01.clip = audioData;

_Speaker01.volume = SourceVol + Random.Range(minVol, maxVol);

}

void PlaySound()

{

SetSourceProperties(pcmData[Random.Range(0, pcmData.Length)], minVol, maxVol, distRand, maxDist, spatialBlend);

_Speaker01.Play();

Debug.Log(“back in it”);

StartCoroutine(“Waitforit”);

}

IEnumerator Waitforit()

{

foat waitTime = Random.Range(minTime, maxTime);

Debug.Log(waitTime);

if (_Speaker01.clip == null) //used for the frst time, before a clip has been assigned, just use the random time value.

{

yield

return

new

WaitForSeconds(waitTime);

}

186

IMPLEMENTING AUDIO

else // Once a clip has been assigned, add the cliptlength’s to the random time interval for the wait between clips.

{

yield return new WaitForSeconds(_Speaker01.clip.length +

waitTime);

}

if

(enablePlayMode)

{

PlaySound();

}

}

void StopSound()

{

enablePlayMode = false;

Debug.Log(“stop”);

}

}

At the top of the script we begin by creating a number of variables and link them to sliders the user can adjust to determine their value. These variables represent the various parameters we wish to set our audio source to: minimum and maximum distance, pitch, pitch randomization, as well as minimum and maximum time between sounds. By taking these values out of the code and making them available to the user, it is much easier to make our code re-useable. We will then create a function whose purpose is to apply these settings to our audio source.

After the variable declaration we use the awake function to check to see if an audio source is already present. This script will work if an audio source is already present but will also add one if none is found: void Awake()

{

_Speaker01 = GetComponent<AudioSource>();

if (_Speaker01 == null)

{

_Speaker01 = gameObject.AddComponent<AudioSource>();

}

}

After making sure an audio source is present or adding one if none is found, we use the Start() function to initialize some basic properties of our audio source, such as turning off PlayOnAwake and looping.

For the purposes of this example, we can use the 1 key on the keyboard to turn on the emitter or 2 to turn it off. Pressing the 1 or 2 keys on the keyboard sets a Boolean variable to true or false, controlling when the script should be

IMPLEMENTING AUDIO

187

running. The code checking for key input was put in the update loop, as it is usually the best place to check for user input. The reader is encouraged to customize this script to fit their needs of course. By pressing 1 on the keyboard we also start a coroutine called WaitForIt. The point of the coroutine is to let the class wait for an amount of time chosen at random from the minimum and maximum values set by the user, then trigger a sample.

The SetSourceProperties() function is how we are able to set the parameters of our audio source to the values of each variable declared at the top of the class.

Having a dedicated function whose purpose to set the audio source’s parameters is key to making our code modular. This allows us to avoid hard coding the value of the source’s parameters and instead use the editor to set them.

Next comes the PlaySound() function. PlaySound() calls SetSourceproperties() to set the parameters of our audio source to the settings selected by the user, triggers the audio source and then calls the coroutine WaitForIt() in order to start the process again and wait for a certain amount of time before resetting the process.

If PlaySound() calls SetSourceProperties() and plays our audio source, where does PlaySound() get called from? The answer is from the WaitForIt() coroutine. Several things happen in the coroutine.

1. The coroutine sets a specific amount of time to wait between the minimum and maximum range set by the user:

foat waitTime = Random.Range(minTime, maxTime);

2. The coroutine checks to see if a sound has been assigned to the audio source. Essentially this line of code is to check whether we are running this script for the first time, in which case there would be no audio clip associated with the audio source.

if (_Speaker01.clip == null)

{

yield return new WaitForSeconds(waitTime);

}

The second time around and afterwards, a clip should have been assigned to the audio source and the coroutine will wait for the duration of the clip + the amount of time selected at random before calling another sound.

{

yield return new WaitForSeconds(_Speaker01.clip.length + waitTime);

}

3. The coroutine checks to see that our Boolean variable enablePlayMode it set to true, and if it is, calls the PlaySound() function.

188

IMPLEMENTING AUDIO

This script can be dropped on any game object and will create an array of audio clips that can be filled by the sound designer by dragging and dropping a collection of audio files on the array or by individually filling each sound clip slot after defining a length for the it. The sliders can be used to adjust pitch, pitch minimum and maximum random offset, volume, as well as volume randomization minimum and maximum offset, 2D vs. 3D, as well as maximum distance and distance randomization.

4. Ambiences, Putting It All Together

We can supplement loops and random emitters with intermittent triggers in order to create more immersive and dynamic environments. An intermittent trigger is one that will not always result in an action when the trigger is entered. A good example would be a trigger in an outdoors level that would once in a while play the sound of a twig cracking under the player’s footsteps in a certain area. We can make a trigger intermittent by generating a random number every time the player enters the trigger but only follow through with any action if the number is over or below a certain threshold. In this example a sound will only be played if a random number generated upon entering the trigger is less-than or equal to one. We can change the odds of the sound playing by changing the range in the inspector. If the range is set to 2, the odds of the sound playing are about 50%; changing that number to 10 will only make the sound play about 10% of the time.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class IntermittentTrigger: MonoBehaviour

{

[SerializeField]

private int range;

[SerializeField]

private AudioSource triggerSource;

[SerializeField]

private AudioClip triggerClip;

// Start is called before the frst frame update

void Start()

{

triggerSource = GetComponent<AudioSource>(); triggerSource.clip = triggerClip;

}

private void OnTriggerEnter(Collider other)

{

if (Random.Range(0, range) <= 1)

IMPLEMENTING AUDIO

189

triggerSource.Play();

}

private void OnTriggerExit(Collider other)

{

triggerSource.Stop();

}

}

We have now looked at a number of ways to work with sounds in Unity; by combining these techniques together we can start to put together convincing and immersive environments, using different tools for different purposes.

• 2D Loops: useful for sounds that do not require spatial treatment, such as wind, rain or some room tones.

• 3D Loops: useful for loops requiring spatial treatment.

• Intermittent Emitters: for one-shot sporadic sounds such as birds, insects, water drops etc.

• Triggers, to play sounds upon entering a room, a space or to turn on and off ambiences and/or other sounds.

• Intermittent triggers.

5. Sample Concatenation

Concatenation of sample is a very useful technique in game audio. We concatenate dialog, gun sounds, ambiences, footsteps etc. Concatenation refers to the process of playing two samples in succession, usually without interruption. In that regard the Intermittent Emitter script does sample concatenation, but we can write a script dedicated to sample concatenation that can be used in a number of scenarios. Let’s take a look at a few examples that can easily be applied to game audio.

a. Creating Variations With Footsteps Samples

Footsteps are notorious for being some of the most repetitive sounds in games, sometimes downright annoying. There are a number of reasons why that may be the case, from poor sound design to mix issues. One common com-plaint about footsteps sounds is that they tend to be repetitive. Most games do recycle a limited number of samples when it comes to footsteps, often on randomizing a limited number of parameters for each instance of one in the game, such as amplitude and pitch. Another way to combat repetition without the additional overhead of more audio files is to break each footstep sample in two: the heel portion and the toe portion of the sample. If we store four samples for each surface we would go from:

Gravel_fs_01.wav

Gravel_fs_02.wav

190

IMPLEMENTING AUDIO

Gravel_fs_03.wav

Gravel_fs_04.wav

to:

Gravel_fs_heel_01.wav

Gravel_fs_heel_02.wav

Gravel_fs_heel_03.wav

Gravel_fs_heel_04.wav

and:

Gravel_fs_toe_01.wav

Gravel_fs_toe_02.wav

Gravel_fs_toe_03.wav

Gravel_fs_toe_04.wav

This allows us to at each instance randomize both the heel and toe sample, each individually randomized in pitch and amplitude, thus creating more variations.

Instance 1: Gravel_fs_heel_02.wav + Gravel_fs_toe_03.wav Instance 2: Gravel_fs_heel_04.wav + Gravel_fs_toe_01.wav A similar application would be for explosions and impacts. We can similarly think of an explosion as being a detonation, followed by debris and environmental effects, such as reflections and reverberation.

b. Case 1: Swapping Audio Clips

We can concatenate samples in one of two ways, by scheduling the second sample at exactly the end of the first sample or by checking to see if the audio source is still busy playing the first sample and as soon as it no longer is, swapping the audio clip with the second one and starting playback again.

To check if an audio source is busy currently playing a sound, we can simply check its isPlaying property. It will return true only when the audio source is playing. In this first example we’ll use this method for an explosion example.

Let’s break this one down in terms of an algorithm: 1. Load sample A in the audio source.

2. Set the audio source to play.

3. Set a flag (Boolean) to let the program know A has already been played.

4. Once A is done playing, the audio source is free again; load sample B.

5. Set the audio source to play.

IMPLEMENTING AUDIO

191

We’ll break the logic into two functions:

PlayFirst() will load audio file number one into the audio source, set it to play and set our Boolean variable to true to let the software know we’ve played audio file one already.

PlaySecond() will load audio file and reset our flag to false.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class Concatenation: MonoBehaviour

{

public AudioSource audioSource01;

public AudioClip sound01, sound02;

public bool isDone;

void Awake()

{

audioSource01 = GetComponent<AudioSource>(); PlayFirst();

}

void Update()

{

if (audioSource01.isPlaying == false && isDone) PlaySecond();

}

void PlayFirst()

{

audioSource01.clip = sound01;

audioSource01.Play();

isDone = true;

}

void PlaySecond() {

audioSource01.clip = sound02;

audioSource01.Play();

isDone = false;

}

}

This method works but does have a few drawbacks; most notably, there is a short interruption at the moment the audio source loads another clip. This may be okay for a lot of situations, but for more time-critical needs and a smooth transition, it makes sense to use two audio sources and to delay the second by the amount of time it takes to play the first sound. We could modify this script rather easily to make it work with two audio sources rather than

192

IMPLEMENTING AUDIO

one, but let’s consider another approach, sound concatenation using the PlayScheduled() function, which gives us much more accurate timing and should be used for application where timing is crucial, such as music and music loops.

Also, you can see how easy it would be to modify this script to make it play samples at random and be used in the footstep example mentioned earlier.

c. Case 2: Using PlayScheduled ()

It is possible to schedule events very accurately in Unity using the PlayScheduled() method. PlayScheduled() takes a single parameter, a double (more precise than floats) that represents the time at which to schedule the event. In this example we will work with multiple audio sources and use PlayScheduled() to make sure they are triggered at the right time.

In order to achieve this seamlessly, we have to know the exact length of the audio clip we are starting with. The length of an audio clip in terms of playback may vary, however, based on pitch modulation or randomization.

The best way to accurately calculate the duration of a single clip is to divide its length in seconds by the playback pitch. We could do this in a number of ways: foat clipLength = audioClip.length/audioSource.pitch; The previous line will return the length in seconds of the audio clip. When it comes to sound and music, however, that might not be enough resolution to keep the playback smooth and music loops in time. For that reason, you might want to increase the level of precision by using a double, rather than a float: double clipLength = (double)audioSource.clip.samples/audioSource.pitch; A couple of things are worth noting here. For one, we actually look for the length of the audio clip in samples, rather than in seconds, which makes our measurement more accurate. You will also notice that we inserted the keyword (double) in front of the expression audioSource.clip.samples. This is known as casting , which is a way to turn the audioSource.clip.samples into a double, rather than a float.

By turning the length into a double, we also make sure the data is ready to be used by the PlayScheduled() function.

Now we can schedule our events accurately, ensuring one will play right after the other:

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class SchedEvent: MonoBehaviour

{

public AudioSource audioSource01, audioSource02;

IMPLEMENTING AUDIO

193

public AudioClip sound01, sound02;

void Start()

{

//Please assign the audio sources manually in the editor.

audioSource01.clip = sound01;

audioSource02.clip = sound02;

}

// Update is called once per frame

void Update()

{

if

(Input.GetKeyDown(KeyCode.Alpha3))

{

double clipLength = (double)(audioSource01.clip.samples /

audioSource01.pitch);

Debug.Log(clipLength);

audioSource01.Play();

audioSource02.PlayScheduled(AudioSettings.dspTime

+

clipLength/

44100);

}

}

}

In this example the transition between clips ought to be a lot smoother than in the previous example, partly because we are using two audio files rather than one but also because of the more accurate timing allowed by PlayScheduled() over other alternatives. You will notice that the clip length is divided by 44,100, which is assumed to be the sampling rate of the project. This will convert the length of the audio clip from samples to seconds. When working with other sample rates, be sure to adjust that number to reflect the current rate.

6. Collisions

Without collision detection, making even the simplest games, such as Pong, would be impossible. Unity’s physics engine will call a specific function when a collision is detected, which we can access through scripting and use to trigger or modify sounds or music.

a. Detecting Collision

In order for Unity to detect a collision between two objects, a collider component is required on the game objects. Colliders are invisible and tend to be of a simpler shape than the object they are attached to, often geometric primitives such as cubes, spheres or a 2D equivalent. Keeping the collider simple in shape is much more computationally efficient and in most cases works just as well.

194

IMPLEMENTING AUDIO

The green outline shows the colliders used for the body of the car. Primitive geometric shapes are used for greater efficiency.

Figure 8.9

In the cases where an approximation of the shape of the object by a primitive shape is not satisfactory, it is possible to use mesh colliders to match the shape of the object in 3D or polygon collider 2D for 2D objects. These colliders considerably increase the computational load on the game engine and should be used sparingly and only when necessary.

Unity makes a few distinctions between colliders. Static colliders are colliders that will not move, such as the geometry of a level, walls, floor etc. RigidBody colliders, which require a RigidBody component, will react to forces applied to them and the material they collide with and may bounce, glide, drop etc.

Kinematic RigidBody colliders are meant for geometry, which is intended to be mostly static but may move under certain conditions, such as a door or gate for instance.

Collisions can be a complex topic, so please refer to the Unity documentation for more information.

In this next example we will trigger a collision between a game object, a cube and the floor by walking through a trigger. The floor will consist of a static collider, while the cube is a kinematic RigidBody collider. Upon entering the trigger, the cube’s ‘isKinematic’ property will be set to off, which will allow it to drop. We will use tags to make sure that the player is the object walking through the trigger and that the cube does indeed collide with the floor.

We will access the OnCollisionEnter function – which gets called by the physics engine upon collision – and use it to trigger a sound at the moment of impact.

using UnityEngine;

using System.Collections;

IMPLEMENTING AUDIO

195

//This class plays a sound upon collision of a rigid body with the foor of a level.

public class CollisionExemple : MonoBehaviour {

void OnCollisionEnter(Collision impact){

//If the impact is with an object tagged ‘Floor’, a sound gets triggered if(impact.gameObject.CompareTag(“Floor”))

GetComponent<AudioSource>().Play();

}

void OnTriggerEnter(Collider target){

//If the object entering the collider is a player if(target.CompareTag(“Player”))

// The object’s physics are activated

GetComponent<Rigidbody>().isKinematic = false;

}

}

You will notice that the order in which the functions are declared has little impact. In this case, CollisionEnter() is defined prior to onTriggerEnter() although the trigger has to be called first in order to switch isKinematic to false, which drops the cube and lets it fall to the ground.

Furthermore, we can get additional information about the collision, such as velocity, which can be extremely useful when dealing with game objects with Rigidbodies or physics properties whose behavior mimics a real-world condition such as gravity. Here we encounter another limitation of the current audio technology in game engines. An object with physics properties will theoretically be able to create a massive amount of different sounds. A trash can may bounce, roll or drag, all at various velocities and on various surfaces, with a close to infinite potential for variations in sounds. Obviously, we cannot store such a massive amount of sounds, and much less justify the time and effort to create so many possible variations. Still, if we obtain enough information on an event, we can let the engine choose between various samples for low, medium and high velocity collisions and if needs be with different sounds for different surfaces. With pitch and amplitude randomization, this may prove to be just enough to make the process convincing.

b. Velocity-based Sample Selection

Objects with physics properties will often behave in unpredictable ways, which makes the job of triggering the right sample for the event difficult.

One way we can address the issue when it comes to collisions, is to select an audio sample based on the velocity of the impact. A simple way to obtain this information in Unity is by calling relativeVelocity when an impact is detected.

relativeVelocity will return the velocity of a collision relative to the speed of both objects, which is very useful when dealing with two moving objects, such as a projectile colliding with a moving vehicle for instance.

196

IMPLEMENTING AUDIO

The following bits of code perform two actions. The first script , pro-

jectile.cs, will instantiate a RigidBody object, in this case, a sphere, (but it could be any object in a level), and it will propel it forward at a random speed within a given range when the user presses the fire button (by default left click):

using UnityEngine;

using System.Collections;

public class Projectile: MonoBehaviour

{

public Rigidbody projectile;

public Transform Spawnpoint;

void Update()

{

if

(Input.GetButtonDown(“Fire1”))

{

Rigidbody

clone;

clone = (Rigidbody)Instantiate(projectile, Spawnpoint.position, projectile.rotation); clone.velocity = Spawnpoint.TransformDirection(Vector3.forward *

Random.Range(10f, 90f ));

}

}

}

This script is attached to an invisible game object, which also acts as an invisible game object and spawn point in this example, and every time the user presses fire, an object, which can be any RigidBody selected by the user, is instantiated from the spawn point and propelled straight ahead.

This next script, attached to the wall located in front of the spawn point, detects any collision and finds out the velocity of the collision. Based on that velocity, it will choose one of three samples from an array, one for low velocity impacts, another under ten another for medium velocity impacts, one for magnitudes within ten to 30 and another for high velocity impacts for anything above 30.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class CollisionDetection: MonoBehaviour

{

AudioSource source;

public AudioClip[] clips;

void Awake (){

source = GetComponent<AudioSource>();

IMPLEMENTING AUDIO

197

}

void OnCollisionEnter(Collision other)

{

Debug.Log(other.relativeVelocity.magnitude);

if (other.relativeVelocity.magnitude > 0.1f && other.relativeVelocity.

magnitude < 10f )

{

source.PlayOneShot(clips[0],

0.9f);

}

else if (other.relativeVelocity.magnitude > 10.01f && other.relative Velocity.magnitude < 30f )

{

source.PlayOneShot(clips[1],

0.9f);

}

else

source.PlayOneShot(clips[2],

0.9f);

Destroy(other.gameObject);

}

}

This script is attached to the wall, and once the RigidBody collides with it will be destroyed right away.

7. Raycasting and Smart Audio Sources

a. Implementing Occlusion With Raycasting

Let’s try a new challenge and try building an audio source that could detect whether there is a wall or significant obstacle between it and the listener that could apply a low pass filter and volume cut if one is detected. This would be a great first step toward achieving a further level of realism in our projects via the recreation of occlusion, the drop of amplitude and frequency response in a sound that occurs naturally as it is separated from the listener by a partial or fully enclosing obstacle. It might also be helpful if our audio source automatically turned itself off when the listener is beyond its maximum range since it cannot be heard beyond that range. We’ll call this a smart audio source, one that is capable of raycasting to the listener, of simulating occlusion, detecting the distance to the player and turning itself off if it is beyond the range of the listener.

Let’s start with finding out the distance between the listener and the audio source:

First, we will need to identify and locate the object the listener is attached to. There is more than one way to do this, but in the Start() function we will use the GameObject.Find() function to locate the object called ‘Player’, since

198

IMPLEMENTING AUDIO

in this case we are going to use a first-person controller and the listener will be on the player’s camera. The object to which the listener is attached must be named or changed to ‘Player’ in the inspector located above the transform component of the game object, or Unity will not be able to find it, and the script will not work. The word ‘Player’ was chosen arbitrarily. In this example, we also assign the object named ‘Player’ to the game object created earlier in the same line:

listener = GameObject.Find(“Player”);

Then, at every frame we will keep track of the distance between the audio source and the listener object. Since we need to check on that distance on a per frame basis, the code will go in the update() function. Instead of doing the math in the update function itself, we’ll call a function that will return the distance as a float. We will call the function CheckDistance(): private foat CheckForDistance(GameObject obj, foat distance)

{

foat dist = Vector3.Distance(obj.transform.position, transform.position); if (dist > distance)

_AudioSpeaker.Stop();

return

dist;

}

The function takes in two arguments: a game object – a reference to the object that carries the listener – and a float, which represents the maximum distance of the audio source’s range and returns a float, representing the distance between the source and player. Note: in this case, the maximum value passed to CheckForDistance() is not directly obtained from the audio source parameter and is passed as a value by the user. When the distance between the source and the player exceeds the range of the distance passed to the function as the second argument, we tell the audio source to stop.

The code that turns the audio source back on is located in the Update() function.

In order to create basic occlusion and have the audio low pass filtered when the listener and the audio source are separated by a wall, we will need to introduce a new technique: raycasting. Raycasting is a very powerful tool. It allows is us to create a line in any given direction, starting from a desired set of coordinates a desired length, that will detect any intersection with colliders in the scene. In the script SmartAudioSource.cs we will raycast from the location

IMPLEMENTING AUDIO

199

of the smart audio source to the listener – only when the listener is within the maximum distance of our audio source so as to conserve resources – and look for any collider in our path.

Raycasting requires a few steps:

1. Determine a point of origin. If you are attaching the ray to an object, you can use transform.position to establish that object as a point of origin.

2. A direction, as a set of 3D or 2D coordinates, depending.

3. A desired length.

Raycasting can be used for a number of purposes. For instance, rather than raycasting from the audio source to the listener, by raycasting outwards from the listener in every direction we can obtain information on the distance between the player and the walls and adjust reverberation information accordingly for additional realism.

Figure 8.10

b. Avoiding the Pebble Efect

If we are not careful, any object with a collider attached to itself, such as another player or even a projectile, could be detected by the raycasting process and trigger the occlusion process. This is sometimes known as the Pebble Effect, and it can be quite distracting. In order to make sure that we are in fact dealing with a wall and not a passing game object, such as a

200

IMPLEMENTING AUDIO

projectile, we will rely on the object tagging system and check its tag. If the object is tagged ‘geometry’ (chosen arbitrarily) the script will update the frequency of the low pass filter component attached and bring it down to 1000Hz, at the same time lowering the amplitude of the audio source by 0.3 units.

The raycasting occurs in the GetOcclusionFreq() function, which takes two arguments, a game object – which is a reference to the object with the listener attached – and a float, which is the length of our raycast.

First, we must find the coordinates of the listener so that we know where to raycast to:

Vector3 raycastDir = obj.transform.position – transform.position; The next statement does several things at once, nested within the if statement, we instantiate the ray:

If(Physics.Raycast(transform.position, raycastDir, out occluderRayHit, distance) We do so by calling Physics.Raycast, which requires the following arguments:

• The initial coordinate from which to cast the ray, in this case, by using transform. position we are using the current coordinates of the object this script is attached to.

• The coordinates from which we are raycasting to, our destination.

• A RayCastHit, which will provide us with information back on the raycast.

• A distance, the max distance for our ray to be cast.

Additionally, it is also possible to use a layer mask as an optional argument to filter out results in more complex environments. Raycasts will return true if the ray intersects with a collider, so we can nest our raycast in an if statement, which will return true if a collider intersects with a collider: private foat GetOcclusionFreq(GameObject obj, foat distance)

{

Vector3 raycastDir = obj.transform.position – transform.position; if (Physics.Raycast(transform.position, raycastDir, out occluderRayHit, distance)) // raycast to listener object

{

// occlude if raycast does not hit listener object if

(occluderRayHit.collider.gameObject.tag

==

“Geometry”)

{

Debug.Log(“OCCLUDE!”);

return

1000;

IMPLEMENTING AUDIO

201

}

}

return 20000f; // otherwise no occlusion

}

As you can see, the code also checks to see; once a collider has been detected by the ray, we check to see if that object is tagged ‘Geometry’. This is to avoid the pebble effect and ensure that the audio source does not get low pass filter if another player or a projectile intersects with the ray.

The Update() function is where we put it all together: void Update()

{

if

(_AudioSpeaker.isPlaying)

{

_lpFilter.cutofFrequency = GetOcclusionFreq(listener, 20);

}

else if (_AudioSpeaker.isPlaying == false && CheckForDistance(listener, 20) < maxDistance)

_AudioSpeaker.Play();

CheckForDistance(listener,

maxDistance);

}

The first if statement checks to see if our audio source is playing and, if so, constantly updates the value of the low pass filter by calling GetOcclusionFreq(). The second if statement, however, checks to see if the audio source should be playing at all, based on whether the listener is within earshot of the audio source. For that, we call CheckForDistance(). CheckForDistance() will return the distance between the listener and the audio source, and if we are too far to hear it, the function will turn off the audio source. Here, we check to see if we are back within the range of our audio source and, if so, turn it back on.

Lastly, we call CheckForDistance() before leaving the update function. This will turn off the audio source if we are too far away to hear it.

There is a lot to this script, and it is worth spending some time with it and really understand what is going on. You will likely find ways to modify it and make it more efficient for the situations you need to address.

8. Animation Events

When working with animations, specifically animations clips, the best way to sync up sounds to a specific frame in the timeline is through the use of animation events. Animation events allow us to play one or

202

IMPLEMENTING AUDIO

multiple sounds in sync with a specific frame in the animation timeline by calling a function in the script attached to the object. As an option, animation events can also take a parameter in the form of a float, int, string or object.

In this example we’ll add a footstep sound to the third-person character controller from the Unity standard assets. We’ll focus on the running animation, since it is one of the most commonly used. First, we need to write a simple script that will play the actual sound whenever the character’s feet make contact with the ground in the animation.

using UnityEngine;

using System.Collections;

public class Run: MonoBehaviour {

public AudioClip[] footsteps;

AudioSource Steps;

void Start () {

Steps = GetComponent<AudioSource> ();

}

void playFootstepSound()

{

if (Steps.isPlaying == false) {

Steps.clip = footsteps [Random.Range (0, footsteps.Length)]; Steps.pitch = Random.Range (1f, 1.2f );

Steps.volume = Random.Range (0.8f, 1.2f);

Steps.Play

();

}

}

}

Adding a footstep sound to a walking animation for a third-person controller: (The same can be applied to most animations.)

1. Locate the appropriate animation. For our example we will look for the running loop, from the standard asset package we imported earlier. The animations are going to be located in: Standard Assets/

Characters/ThirdPersonCharacter/Animations. The running loop is called HumanoidRun.

2. In the inspector, make sure to select the Animation tab.

3. Scroll down to the Events tab, and open it up.

4. Using the area next to the play button at the top of the animation window, scroll to the frame in the animation you would like to add a sound to.

5. Back in the events tab, in the inspector, click on the Add Events button, located to the left of the timeline of the events section.

IMPLEMENTING AUDIO

203

6. Under the Function tab, write the name of the function you created in the script earlier, attached to the third-person controller.

7. Make sure to add the script and an audio source to the character controller.

Figure 8.11

Press play!

204

IMPLEMENTING AUDIO

9. Audio Fades

Fades are gradual changes in volume over time that tend to have two main parameters: target volume and duration. Fades are useful for elegantly transitioning from one music track to another, but a short fade can also help smooth out the sound of sample as it plays, especially if the audio sample is meant to be a seamless loop and therefore will not contain a micro fade to prevent pops and clicks, and may sound a tad jarring when first triggered.

We do fades by gradually increasing or decreasing the volume value of an audio source over time. However, we must be careful to not rely on the frame rate as a timing reference, since the frame rate may vary with performance and is therefore not an absolute timing refence. Instead, it is better to rely on Time.deltaTime. Time.deltaTime gives us timing independent from frame rate.

It will return the time since the last frame, and when doing animations, or in this case fades, multiplying our fade increment by Time.deltaTime will ensure that the fade’s timing is accurate in spite of any potential frame rate variations by compensating for them.

Since many files would likely benefit from fades, it makes sense to write the code so that it will be easily available to all audio sources. Rather than writing a block of code for fades in every script that plays an audio file, we shall write a separate class and make the code available to all objects in the scene by making the functions both public and static.

Since fades occur over the course of multiple frames, it makes sense to use a coroutine, and since we wish to make that coroutine available to all audio sources, at any time, we will place our coroutine in a public class and make the coroutine itself both public and static. Making it static means that we do not need to instantiate the class it belongs to in order to call the function. It also ensures that the implementation will be identical, or consistent across all class methods. Static classes do have some drawbacks, they cannot be inherited or instantiated, but in this case this implementation should serve us well.

We’ll create a new class Fades.cs, which will contain three functions for fades: a fade-in, fade-out and transitioning to a target volume function. We’ll start by creating the fade-out function:

public static IEnumerator FadeOut(AudioSource audioSource, foat FadeTime)

{

foat startVolume = audioSource.volume;

while (audioSource.volume > 0)

{

audioSource.volume -= Time.deltaTime / FadeTime; yield

return

null;

}

audioSource.Stop();

}

IMPLEMENTING AUDIO

205

The function, being static and public, is easy to access from other classes. In order to fade out the volume of our audio source we will gradually decrease the volume over time. As mentioned previously, however, rather than simply relying on the frame rate of the computer, which can be erratic and is based on performance, we want to make sure our fades are controlled by Time.

deltaTime, which returns the time elapsed since the last frame and therefore allows us to compensate for any frame rate discrepancies: audioSource.volume -= Time.deltaTime / FadeTime; If we assume a frame rate of 60 frames per second, the time for each frame is 1/60 = 0.0167 seconds. Assuming a start from a volume of 1 and looking for a fade to occur over two seconds, each increment would be: 1 * 0.017 / 2 = 0.0085

To check our math, a fade from 0 to 1, over two seconds or 120 frames, incrementing the volume by 0.0085:

120 * 0.0085 = 1.02

Note: the decimal portion, .02, is due to rounding errors.

The fade in function works similarly.

public static IEnumerator FadeIn(AudioSource audioSource, foat FadeTime)

{

audioSource.Play();

audioSource.volume = 0f;

while (audioSource.volume < 1)

{

audioSource.volume += Time.deltaTime / FadeTime; yield

return

null;

}

}

The function for transitioning to a new value is slightly more complex but based on the same idea:

private static int changeIncrement = 15;

public static IEnumerator FadeAudioSource(AudioSource player, foat duration, foat targetVolume)

{

//Calculate the steps

int Steps = (int)(changeIncrement * duration);

foat StepTime = duration / Steps;

206

IMPLEMENTING AUDIO

foat StepSize = (targetVolume – player.volume) / Steps;

//Fade now

for (int i = 1; i < Steps; i++)

{

player.volume += StepSize;

yield return new WaitForSeconds(StepTime);

}

//Make sure the targetVolume is set

player.volume = targetVolume;

}

10. Distance Crossfades

Often, when sounds are heard from a large distance, such as a thunderstorm, it is difficult to accurately recreate the sound of the event heard from afar and from close up with a single sample. Rather, we employ two sounds for afar and close-up and crossfade between them as we move toward or away from the sound source. This is known as a distance crossfade.

Figure 8.12

IMPLEMENTING AUDIO

207

In order for us to implement a distance crossfade we need a few elements.

1. Two audio sources, one for the sound from afar, and another for the sound up close.

2. Keep track of the distance between the listener and the origin of the audio source.

3. Map the distance to a normalized range between 0 and 1, which can be used to control the volume of each audio source.

We will start by writing a new class, DistanceXFade: public class DistancexFade: MonoBehaviour

{

[SerializeField] AudioSource soundAfar, soundClose;

[SerializeField] AudioClip closeUpSound, farAwaySound; public foat minDistance, maxDistance;

public foat dist;

[SerializeField] GameObject listener;

void Awake() {

listener = GameObject.Find(“Player”);

soundClose.clip = closeUpSound;

soundAfar.clip = farAwaySound;

soundClose.maxDistance = maxDistance;

soundAfar.maxDistance = maxDistance;

}

We begin by declaring two audio sources, soundAfar and soundClose and two audio clips closeUpSound and farAwaySound for each one. We also declare a few floats, minDistance and maxDistance, which are going to represent the minimum and maximum range of the audio source. The float dist will be used to keep track of the distance between the listener and the audio source, while the GameObject listener will hold a reference to the player, which assumes the listener will be assigned to it.

Next, in Awake() we proceed to initialize our audio sources and find the player.

We are using GameObject.Find() to look for a game object by name, which means that the object on which the listener is attached must be named ‘Player’, or, if using a different name, that field needs to be changed to match the name you gave it. Next we assign the appropriate clips to our audio sources and assign the max distance specified by the user to our audio source. Allowing the user to specify the max distance for each source makes the code easy to re-use across different contexts.

void Start()

{

soundAfar.Play();

soundClose.Play();

208

IMPLEMENTING AUDIO

}

void Update()

{

CheckForDistance(listener, maxDistance);

if (soundAfar.isPlaying == false && CheckForDistance(listener, maxDistance)

< maxDistance)

soundAfar.Play();

}

foat CheckForDistance(GameObject obj, foat distance)

{

dist = Vector3.Distance(obj.transform.position, transform.position); if (dist > distance)

soundAfar.Stop();

Vector3 raycastDir = obj.transform.position – transform.position; Debug.DrawRay(transform.position, raycastDir, Color.black); MapToRange();

return dist;

}

We start both audio sources in the Start() function, though that could easily be changed to a trigger or to respond to a game event.

Next, during Update(), therefore once per frame, we call CheckForDistance().

This function, which we will look at next, will determine the distance between the audio source and the player. The if statement that follows checks to see if the audio sources are currently playing and whether the player is within maximum range of the audio source. If the audio source isn’t playing (it can be turned off when we are outside range) and we are within range, the audio source will be turned back on.

CheckForDistance() is next, and the first line of code assigns the distance between the player and the sound source to the variable dist. CheckForDistance takes two arguments; the first is a reference to the player and the second is the maximum distance for the audio sources. If the player is farther than the maximum range and therefore unable to hear them, CheckForDistance turns both audio sources off. The next two lines are used to draw a raycast between the audio sources and the listener, which is only for debugging purposes and can be turned off when running the scene.

Once we’ve established the distance between the listener and the source, we call MapToRange(), which will then map the distance between the listener and the source to a range between 0 and 1, which can be used to control the volume of each audio source.

In order to map the distance to a range between 0 and 1 we do a little math.

If the player is within the range of the audio source, we map the distance to a percentage using this simple formula:

(Current Distance−M

inimum Distance)

F(x) = (Maximum Distance −Miinimum Distance)

IMPLEMENTING AUDIO

209

This will return a value between 0 and 1 depending on the distance – 0 when on top of the audio source and 1 being at the limit of the range. We can now map this value to control the volume parameter of each audio source using the next function, UpdateVolume().

Since we want the value of the close-up source to be at one when we are on top of it and at the same time the far away source to have a value of zero, we will assign the value returned by MapToRange() to the far away audio source, and the close-up will assign the volume of the close-up audio source, to (1-range).

void UpdateVolume(foat ratio) {

foat closeRoot = Mathf.Sqrt(1 – ratio);

foat farAwayRoot = Mathf.Sqrt(ratio);

soundAfar.volume = farAwayRoot;

soundClose.volume = closeRoot;

Debug.Log(ratio);

}

You will also notice that we actually use the square root of the percentage value, rather than the value itself. That’s optional, but it is to compensate for a drop of overall perceived amplitude while we stand at the halfway point between the two sources. Our perception of amplitude is not linear, and mapping volume curves to linear functions may result in sometimes awkward results. Most common when using a linear fade is a drop of the overall perceived amplitude at the halfway point, by about 3dB, rather than a constant amplitude across the fade. This technique of using the square root value rather than the raw data can be applied to panning and other fades as well.

Note: when working with a distance crossfade in Unity or any similar game engine, do keep in mind that the process will only be successful if the right candidates are selected for each perspective. Finding or creating two sounds that are meant to represent the same object but from a different perspective can be a little tricky, especially if they have to blend seamlessly from one to another without the player being aware of the process. Other factors are to be considered as well, the main one being that you may wish for the sounds to have different spatial signatures. In the case of a thunderstorm, the faraway sound would likely be 3D or partially 3D so that the player can easily identify where the storm is coming from, but up close and ‘in’ the storm the sound is often 2D, with rain and wind happening all around you. You may also wish to adjust the spread parameter differently for each. The spread parameter controls the perceived width of the sound. Sound heard from a distance tends to have narrower spatial signatures than the same sound up close. These changes may affect the perceived amplitude of each sound in the game – the 3D one with a narrower spread may appear softer than it was previously, especially

210

IMPLEMENTING AUDIO

when compared to the close-up sound. You may need to add a volume multiplier to each audio file so that you may control the levels better.

11. Working With Prefabs

Working with scripts, audio sources and other components that must be added to multiple objects, the process of populating a level can quickly become time-consuming and prone to errors. Unity offers an asset type known as a prefab, which works as a template. A prefab allows the user to combine an object made of multiple assets – or components – into a template that’s easy to instantiate multiple times.

Creating a prefab is simply a matter of creating an object from the hierarchy back into the asset folder. When wishing to re-use the prefab, one can simply drag the newly created asset back into a scene. The object will be displayed in blue, indicating it is a prefab. Prefabs can also be instantiated from script, making it easy to quickly create complex objects from a few lines of code at runtime.

With prefabs, we can create a complex audio object, containing scripts, an audio source and additional processing such as a low pass filters and store it as a prefab, easy to instantiate multiple times and across scenes.

a. Creating a Smart Intermittent Emitter Prefab With Occlusion

1. Create a new empty GameObject by right clicking on the hierarchy and selecting Create Empty, or by selecting the GameObject menu and selecting Create Empty.

2. If you haven’t done so yet, import the IntermittentSourceOcclusion.

cs script Assets/Import New Assets and add the script to the newly created game object as a component.

3. After making any adjustments to the parameters of either components you wish to save across multiple instances, simply click the empty object you created in step one and drag it into the asset folder.

b. Instantiating a Prefab From Scripting

Instantiating prefabs from scripting is done using the Instantiate() method, which is overloaded and can take different arguments based on the situation.

It is often useful to instantiate an object at a specific location in the 2D or 3D

world, and this can be easily done with the instantiate method.

The Instantiate() method always requires a reference to the prefab that is to be instantiated. In the following example we’ll instantiate a prefab at a specific location in a 3D level:

// Reference to the Prefab. Drag a Prefab into this feld in the Inspector.

public GameObject myPrefab;

// This script will simply instantiate the Prefab when the game starts.

IMPLEMENTING AUDIO

211

void Start()

{

// Instantiate at position (0, 0, 0) and zero rotation.

Instantiate(myPrefab, new Vector3(0, 0, 0), Quaternion.identity);

}

c. Destroying an Object Instantiated From a Prefab

When a prefab is instantiated, it becomes just another game object. Unless action is taken to remove it from the scene when no longer needed, it will linger on and use up resources for no reason This could potentially seriously damage the performance of your game and drastically slow down the frame rate if we are not careful with our code and do not keep track of instantiated prefabs. Whenever instantiating an object, you should also have a strategy in mind to remove it or destroy it when no longer needed.

The following code instantiates a prefab and waits three seconds before destroying it, using the Destroy() method: using UnityEngine;

public class PrefabInstance : MonoBehaviour

{

// Reference to the Prefab. Drag a Prefab into this feld in the Inspector.

public GameObject myPrefab;

double life;

// This script will simply instantiate the Prefab when the game starts.

void Start()

{

// Instantiate at position (0, 0, 0) and zero rotation.

myPrefab = (GameObject)Instantiate(myPrefab, new Vector3(10, 0, 0), Quaternion.identity);

life = Time.time + 3.0;

}

void Update() {

if (life <= Time.time)

{

Destroy(myPrefab);

}

}

}

We are using the Time.time to determine the amount of time that has gone since the game started. Time.time only gets defined once all Awake() functions have run.

It would be easy to customize the code to instantiate objects around the listener and dynamically create audio sources from any location in the scene.

212

IMPLEMENTING AUDIO

d. Instantiating Audio Emitters at Random Locations in 3D

With prefabs we can easily write a script that will instantiate an audio source at random coordinates around the listener at random intervals whose range is to be determined. This next script will instantiate a new prefab, with an audio source attached to it, that will stay active for three seconds, then be destroyed.

The script will wait until the previous object is destroyed and until we are out of the coroutine before instantiating a new object, ensuring we are only ever creating one prefab at a time.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

public class Intermittent3D : MonoBehaviour

{

// Reference to the Prefab. Drag a Prefab into this feld in the Inspector.

public GameObject Bird;

GameObject temp;

public AudioClip emitter;

AudioSource speaker;

double life;

[SerializeField] [Range(0f, 30f )] foat minTime, maxTime; public bool inCoroutine;

void Start()

{

StartCoroutine(Generate());

}

void PlaySound()

{

temp = (GameObject)Instantiate(Bird, new Vector3(Random.Range (−10, 10), Random.Range(−10, 10), Random.Range(−10, 10)), Quaternion.identity);

life = Time.time + 3.0;

}

void Update()

{

if (life <= Time.time)

{

{

Destroy(temp);

GoAgain();

}

}

}

IMPLEMENTING AUDIO

213

void GoAgain()

{

if

(!inCoroutine)

{

StartCoroutine(Generate());

}

}

IEnumerator Generate()

{

inCoroutine = true;

foat waitTime = Random.Range(minTime, maxTime);

Debug.Log(waitTime);

yield return new WaitForSeconds(waitTime);

PlaySound();

inCoroutine = false;

}

}

This script could easily be modified in a number of ways. The audio sources could be generated randomly around the listener’s location, for instance, and the script could be started by a trigger, for instance. Further improvements could include the ability to play a sound at random from a list, and we could pass the clips to the prefab from the interface by dragging them onto the script directly. As always, the possibilities are endless.

Conclusion

Scripting can be a difficult skill to master, or it may appear that way at first, but with practice and perseverance, anyone can master the skills outlined earlier. Remember to go slow, build complexity slowly, one element at a time, and always try to have a map of exactly what it is you are trying to accomplish and the necessary steps to get there. Languages and systems change all the time. Today you might be working in C# in Unity, but there’s no telling what your next project will be. Being aware of audio-specific issues when it comes to implementation and the ability to break down a complex task into a series of smaller manageable steps is a skill you will be able to translate to any situation and game engine. So, where to go from here? If audio implementation is a topic of interest to you, delve deeper into C# and possibly learn C++, a skill that employers always seem to be looking for. Learn to work with other game engines and middleware. Learn about manager classes, classes written specifically to handle one aspect of the game mechanics, such as audio, which will allow you to create larger structures and more organized code. Look for additional material on the companion website, and have fun!

9 ENVIRONMENTAL MODELING

Learning Objectives

In this chapter we study the various ways we can use technology to recreate a rich, believable audio environment that matches the virtual worlds we are attempting to score and recreate in order to immerse the user. Designing and implementing great sounds is only part of the equation; another is to also create a world where the propagation and behavior of these various sounds is believable and realistic, at least as far as the expectations of the player are concerned. In this chapter we take a look at the elements to consider when dealing with environmental modeling and how to apply them within the context of a modern game engine such as Unity. Note: the technical aspect such as scripting of these concepts has at times been described in other parts of the book. In such cases the reader is encouraged to consult these chapters for further details.

1. What Is Environmental Modeling?

The work of a game audio sound designer doesn’t stop with designing great sounds. Just as important is the ability to create a satisfying audio environment within which those sounds can live and thrive and provide the player with as much information as possible on their surroundings and happenings.

Successful environmental modeling is also an important factor for immersion, by helping create a rich and consistent picture of the environment surrounding the gamer.

In Chapter two we outlined some of the ways that sound can provide us with many different cues that can assist the player with critical information on their surroundings. In this chapter we will look at the ways that we can create such environment for our sounds to live in and propagate and ultimately inform and immerse the player.

Calculating and recreating the exact acoustical properties of a space requires a significant amount of computational power, still exceeding the

ENVIRONMENTAL MODELING

215

real time capabilities of most machines. Most game engines tend to take the more practical approach of giving us tools that allow us to approximate the acoustics of the worlds we aim to create and transport our players to, rather than simulating the exact acoustical properties of a space. Remember that as game designers our task is not to recreate reality but to create a convincing and engaging environment within which players may find themselves immersed, informed and entertained, serving both gameplay and storyline. In fact, in some cases recreating the actual properties of the space we are in might actually prove counterproductive. A space may require a long, very harsh sounding reverb, but that might make the mix difficult to listen to, and if dialog is present, it might become unintelligible.

Over large distances emulating the speed of sound and therefore the delay between a visual cue (such as a lightning bolt) and the aural cue (thunder) might prove distracting rather than immersive, even if technically accurate.

There are countless examples where in the context of a gaming experience reality is not the best option. Therefore, when it comes to environmental modeling, focus on the gameplay and narrative first and foremost and realism second. Once you’ve established a set of rules for your level, be consistent and stick to them. Changing the way a sound or environment behaves after it’s already been introduced will only serve to confuse the player and break immersion.

Environmental modeling can be tricky, especially with more complex levels, where the behavior of sound might be difficult to establish even for a veteran audio engineer. This chapter outlines some of the most important elements that the game developer ought to address. This is in no way intended to be a scientific approach but rather an artistic one. There is science to what we do, but as sound and game designers we are, at our core, artists.

1. Reverberation

Too often, Environmental Modeling in a game is summarized to a hastily applied reverberation plugin, often ill chosen. While reverberation alone is not the only aspect of creating a convincing audio environment, it is indeed a crucial cue and one that deserves our full attention. Reverberation provides the user with two main clues as to their surroundings: an impression of distance from the sounds happening in the space and a sense of the space itself, in terms of size and materials. A complete lack of reverberation will make it much harder for the player to properly estimate the distance of objects and will sound artificial. As previously seen, however, distance is not just a factor of reverberation; loss of high frequency content and perceived width of a sound also come into play.

Another common misconception is that reverberation is an indoors only phenomenon, which is of course untrue. Unless our game takes places in an anechoic chamber, any other environment will require the addition of reverberation.

216

ENVIRONMENTAL MODELING

a. Pre-Computed vs. Real Time Computation

Reverberation is a relatively costly process, and resources are always a concern when dealing with real time applications. There may be instances in which you might be able to ‘print’ or bounce audio files with reverb out of your DAW

and import them in Unity that way, with rendered reverberation. This may be possible in a few cases, but generally speaking, reverb must be implemented in the level itself in order to sound convincing. Most reverb plugins tend to work in real time, computing reverb at each frame of the game, but some third-party software packages offer non real time solutions that can be quite convincing. Often, these software solutions will need to render a separate reverb map, similar to the way lighting is usually rendered in games. This process of rendering a separate reverb map prior to running the game is known as baking. We shall, however, focus on the solutions available within Unity at the moment of this writing.

b. Absorption Coefcients

Environmental modeling needs not be realistic. Ultimately, as with every aspect of the game, it is second to the narrative. Some reverberation processors, however, usually non real time ones, allow the user to match the materials used for the geometry’s surfaces used in the level to acoustical properties based on real life behavior. These values are usually based on tables used by acousticians, known as the absorption coefficients of various materials. Absorption coefficient tables provide us with a detailed way to understand how various kinds of materials absorb or reflect sounds at a wide range of frequencies. An absorption coefficient of 0 means that all sound was reflected and none absorbed, while a coefficient of 1 will mean that all sound was absorbed and none reflected.

Although Unity at the time of this writing does not natively support such a solution, (third party developers for Unity do provide such solutions) you might want to consult such tables, freely available online if you are unsure as to the reflective properties of various common materials or if you are looking for a reference point to start from.

c. Environmental Modeling With Reverberation in Unity

Unity’s reverberation algorithm allows us enough flexibility to model most situations we will come across in our work, although the way it breaks down parameters may appear confusing initially when coming from a music or audio production background. We will shine a light on these parameters and how to use them shortly, but first, let’s look at how Unity allows us to apply reverberation. Unity offers several ways to add reverberation to a scene: 1. Via an audio reverb filter, which is applied as a component to an object with an audio source.

ENVIRONMENTAL MODELING

217

2. Via a reverb zone, which is added to an area in a scene.

3. Via an SFX reverb effect, which is added to a mixer group.

They differ slightly in terms of the options and the parameters they offer, based on how they are intended to be used, but their features are somewhat similar. The Unity reverberation model breaks the signal down between low and high frequencies, expressed in a range from −10,000 to 0. The model also allows independent control of early reflections and late reflections as well as the overall thickness or density of the reverb. This implementation makes it a practical algorithm to model indoors and outdoors spaces.

d. Unity’s Reverberation Parameters

Figure 9.1

A reverb zone component.

Reverb preset: reverb zones only

Unity comes with a number of presets that can be used as starting points in our work; you will find them under the drop-down menu under this heading. In order to create your own, you must select the user setting.

Min distance: reverb zones only

Reverb zones are rendered in a scene in a similar manner to audio sources, as spheres. The minimum distance represents the inner radius of the sphere, where the reverb effect is applied in full.

218

ENVIRONMENTAL MODELING

Max distance: reverb zones only

This represents the outer radius of the reverb zone. At the edge of the radius – or maximum distance – little to no reverb will be heard. As we move closer toward the center of the sphere, more reverb will gradually be heard in the scene.

Dry level

Controls the mix of the dry signal, from −10,000 to 0.

Room

Controls the amount of overall gain fed to the reverb.

Room LF

Room effects level at Low frequencies, from −10000 to 0.

Room HF

Room effects level at high frequencies, from −10000 to 0.

Decay time

The reverb time is defined relative to low frequencies, from 0.1 to 20 seconds.

Decay HF ratio

Decay time for the high frequencies expressed as a ratio of the low frequencies, from 0.1 to 2.0.

Reflections level

Level of early reflections, expressed relative to the room effects from –10,000

to 1,000.

Reflections delay

Controls the early reflections delay time relative to the room effects, from 0.0 to 0.3

Reverb level

Controls the late reflection levels relative to the room effects, from

–10,000 to 2,000.

Reverb delay

Controls the late reflection delay time, relative to the early reflections, in seconds, from 0.0 to 0.1.

HF reference

Determines the high frequency reference point, from 1,000Hz to 20,000Hz

ENVIRONMENTAL MODELING

219

LF reference

Determines the low frequency reference point, from 20Hz to 1,000Hz Diffusion

Controls the density of the reverb, in terms of the amounts of reflections/

echoes.

Density

Controls the density of the reverb, with regard to modes or resonances.

Next, let’s try to understand how some of these parameters may be used most effectively in the specific context of environmental modeling.

2. Best Practices for Environmental Modeling

a. Late vs. Early Refections

Remember from Chapter five that early reflections are the collection of the first reflections to reach the listener’s ears, prior to the main bulk of the reflections, known as late reflections (the portion of the sound most people associate with the actual sound of reverberation). The early reflections are a good indicator of the room size and shape, as well as the position of the listener in the space. If you are trying to model a large room, such as a train station for instance, a longer pre-delay is appropriate. A small living room will benefit from short to very short pre-delay time. Dense urban environments also generally have strong early reflections due to the large number of reflective surfaces. The smaller the space, the shorter the pre-delay time for the early reflections. The closer to a reflective surface, the shorter the pre-delay as well.

No matter how small the space, however, you should never leave the pre-delay time at a value of zero (which, sadly, a lot of reverb plugins tend to default to and users never change). One reason to never leave the pre-delay time set to zero milliseconds is that it is physically impossible for the early reflections to occur at exact same time as the dry signal, as sound travels fast but not that fast. Another reason is that not leaving any time between the dry signal and the early reflections will make your mix muddier than it needs to be. The human brain expects this short delay and uses it to make sense of the environment.

b. Refections Level

The reflections level parameter controls the level or loudness of the early reflections. This parameter tends to be influenced mostly by the materials that the space is covered in, rather than the size. A room with highly reflective materials, such as tiles, marble or cement, will demand a higher value for this

220

ENVIRONMENTAL MODELING

parameter than a room covered in more absorbent materials such as drapes.

Adjust it accordingly. You can always refer to an acoustic chart of absorption coefficients for various materials if unsure when tackling a new scene, but your intuition might serve you just as well, and ultimately your ears should decide.

Having strong early reflections might be accurate, but could sound distracting and make the mix too harsh or sound modulated. As always, do what you feel best serves the story.

c. Density and Difusion

Unity allows control over two kinds of reverb thickness and color: the density and diffusion settings. This allows the user to shape the overall tone of the reverb. The diffusion setting controls the number of echoes or delays making up the reverberation. It is recommended to leave this setting at a rather high value for best results and a lusher sounding reverb. As you decrease the amount of echoes that make up the reverberation, you will make the reverb thinner, which means it will allow for more room in the mix for other elements, but, on the other hand, lowering the diffusion also tends to expose each individual reflection more to the listener and starts to decrease the overall perceived quality and fidelity. The diffusion setting is a bit more subtle and acts as a little bit as an overall tone or ‘color’ control. At higher settings the diffusion will yield a rather smooth and interesting sounding reverb. Below a certain point, however, reducing this setting too much will result in the sound of the reverb being a bit more neutral, perhaps even bland and will at lower settings impart a cartoonish ‘boingg’-like quality reminiscent of a bad spring.

Use both these parameters to tune your reverb to the appropriate settings, but I do not recommend lowering either setting too much for best results.

d. High Frequencies vs. Low Frequencies

Most real-world materials generally are better at absorbing high frequencies than they are at absorbing low frequencies. For that reason, most real-world reverbs will have a faster rate of decay at high frequencies than lower ones.

When high frequencies linger on, they generally sound harsh and can easily pollute a mix and make it sound unpleasant and harsh to listen to. The decay HF ratio allows the engineer to control how the high frequencies decay over time when compared to low frequencies. A value of 1 makes both high and low frequencies decay at the same rate, while a value below 1 will shorten the high frequencies’ decay time, and, inversely, a value over 1 will extend them.

You may use this setting to help you model the various materials and their frequency absorption coefficients but also adjust the tone of the reverb itself.

A room covered in heavy fabrics will absorb high frequencies faster than one covered in tiles. Use the decay HF ratio setting to model this. Additionally, you can change the point of the crossover from high to low frequencies by adjusting the LF and HF reference parameters.

ENVIRONMENTAL MODELING

221

3. Reverb Zones, Efects Loops and Audio Reverb Filters

As noted previously, Unity offers multiple ways to add reverb to your levels.

Reverb zones, which are similar to audio sources, represent spherical areas with a minimum and maximum range within which the reverberation is applied.

Reverb may also be added via a mixer, as an insert or effect loop, via a traditional aux-send structure.

Lastly, we can also add an audio reverb filter as a component, as long as that object also has an audio source attached. In this case the reverb will only be applied to the audio source it follows in the inspector.

a. Reverb Zones

Reverb zones work similarly to audio sources. They can be added as a standalone game object or as a component to an existing object. I would suggest creating an empty game object to add the reverb zone to or creating it as a standalone object and clearly naming it, as it will make keeping track of where your reverb objects are much easier. Once added to an object you will find a sphere similar to that of an audio source, with a minimum and maximum distance. As you might expect by now, the maximum distance tells us when the reverb will start to be heard in relation to the position of the listener, and the minimum distance denotes the area where the reverb will be heard at its peak.

Right below the minimum and maximum distance you will have the option to select a setting from several presets, which I recommend you explore, as they can be used as starting point and modified by selecting the ‘user’ setting.

Adding a Reverb Zone to a Level

1. Create an empty game object from the GameObject menu: GameObject/

Create Empty.

2. In the inspector, rename the object Reverb Zone.

3. Adjust the minimum and maximum distance for the zone, and select a preset from the dropdown menu.

4. Still in the inspector, click the Add Component button; select Audio Reverb Zone.

5. Using the Move, Scale and Rotating tools, place the Reverb Zone in its desired position.

The benefit of working with reverb zones is that they are easy to map to geographical areas in your level and can overlap, which can be used to create more complex reverb sounds and make transitions from one acoustical space to another much easier, such as when dealing with transitions from one place to

222

ENVIRONMENTAL MODELING

another, each with different reverberant qualities, where we want the reverb to smoothly change as we move from one to the other.

Every audio source also has a Reverb zone mix , which can be used to control how much of that audio source is sent to the reverb zone. This parameter can also be controlled using a curve on the distance graph, which can be used to control the wet/dry mix ratio based on distance. This makes it very convenient to easily map the amount of wet vs. dry signal you wish to hear when moving away from an audio source in a given space.

A major drawback of reverb zones is that they are spherical, a shape that does not bode well with most geometry in a level. Adding a lot of individual reverb zones can also become a little unwieldy to manage and can translate to a lot of CPU activity.

Figure 9.2

Efect Loops for Reverberation

The method to choose when implementing reverb in your project depends on several factors. Working with reverb zones does have some advantages but also a few drawbacks when compared to an effect loop-based reverb configuration.

On the plus side, reverb zones are easy to use and can be dropped easily on any part of a level. They are relatively easy to match to a geographical area since they get placed on a scene in the same way as an audio source, clearly outlining its range. On the other hand, they suffer from the same shortcoming of being spherical in shape, and make covering an entire rectangular room without spilling over the walls impossible. Additionally, a large level may require the use of many different reverb zones, which can become difficult to manage and keep track of and can become somewhat inefficient. Another drawback of working with reverb zones is that it makes it difficult to have independent control over the dry and wet signals, which, as we will see when discussing obstruction and exclusion, can be desirable.

ENVIRONMENTAL MODELING

223

b. Adding Reverb as an Efect Loop Using the Mixer

Note: the following is explained in further detail in Chapter twelve, but it is also included in this chapter for convenience.

1. If a mixer isn’t already present in your project, add one, under the Assets/Create/AudioMixer.

2. Create two new groups by clicking the + button to the right of the word Groups in the Audio Mixer window; name one SFX and the other Reverb. We will route the dry signal through the SFX group and apply the reverb on the reverb group. Your configuration here may vary widely based on your mix configuration. Both groups ought to be children of the master group, which always sits at the output of the mixer.

3. In the reverb group, click the Add . . . button at the bottom of the group, then select the option receive. Adding a receive component allows us to grab a signal from a different group and run it through the selected group and whatever processes happen to be on that group.

4. Still in the Reverb group, after the Receive component add an SFX

REVERB effect in the same manner you added a receive component in Step 3.

5. Since we are going to run our dry signal from the SFX group I recommend turning the Dry Level slider all the way down on the SFX reverb component. This will ensure that we only have the wet signal playing through the reverb group.

6. Now we need to send the audio from the SFX group to the reverb group. In order to do so, we will create a send on the SFX group, by clicking Add . . . then selecting Send from the dropdown menu.

7. In the send component, using the dropdown menu to the right of the word Receive select the receive plug in from the reverb group labelled Reverb/Receive (or something different if you named your groups differently). You may now select the desired amount of reverb you wish to hear by using the send level slider. To make sure the send is working I recommend using an obvious setting initially, by raising the slider close to the 0dB level then adjusting to the desired level. BE CAREFUL! Do make sure to turn the volume down a bit prior to raising the send level to make sure that you don’t get any loud surprises.

8. Lastly, route the output of at least one audio source in the level to the SFX group. Mind your monitoring levels, as always, and press play.

You should now hear a lot of reverb. You may adjust the Send to the reverb on the SFX group while in Play mode, by enabling Adjust in Play Mode.

It’s also possible to change the amount of reverb by creating several snapshots, each with the appropriate send value, or by changing the send value via script directly.

224

ENVIRONMENTAL MODELING

c. Audio Reverb Filters

Audio reverb filters can be added as any other components via the component menu

Component -> audio -> audio reverb filter

Or by selecting the game object you wish to add the Audio Reverb Filter to and clicking the Add Component button in the inspector then selecting audio

-> audio reverb filter.

2. Distance Modeling

1. Filtering as a Product of Distance

As mentioned in the introduction to this chapter, air, over long distances, acts as a gentle low pass filter. Combined with the right reverberation effect/setting, this can create a convincing impression of distance, especially for loud sounds that can be heard from afar. Thankfully Unity allows us to add a low pass filter as a component and control its cutoff frequency easily from the audio source component.

a. Adding a Low Pass Filter That Will Modulate its Cutof Frequency

Based on Distance

1. Add an audio source to an empty game object or to the game object that you wish the sound to be attached to: component -> audio -> audio source

2. Add an audio low pass filter component to that object: component -> audio -> audio low pass filter

3. Make sure the game object you added the audio source and low pass filter components to is still selected. In the inspector, find the audio source component, open the 3D source settings, and at the bottom of the distance graph, click on the Low-Pass text at the bottom. This should now only display the low pass filter graph.

4. Keep in mind the x axis in this graph represents distance, while the Y axis, in this case, represents the filter cutoff frequency. Moving the graph up and down with the mouse by simply clicking and dragging anywhere in the line should also adjust the frequency of the low pass filter in the low pass filter component.

5. Move the line to the frequency you wish the filter’s cutoff to be when the listener is close to the audio source (usually fully open or closed) then double click the line where you wish the filter to be at its lowest cutoff frequency. This should create a second anchor point. Move the anchor point to the desired cutoff frequency. You’re done!

ENVIRONMENTAL MODELING

225

Figure 9.3

You may have to adjust the curve and actual cutoff frequency through trial and error.

The rule here is, there is no rule. Adjust the curve and cutoff frequencies of the low pass filter until the transition is smooth and feels natural as the player walks toward the audio object. The point of low pass filtering here is to accen-tuate the sense of distance by recreating the same filtering that occurs naturally.

b. Width Perception as Product of Distance

The spread parameter controls the perceived width of an audio source in the sound field. Out in the real world, when one is moving toward a sound source, the perceived width of the sound tends to increase as we get closer to it and get narrower as we get further away from it. Recreating this phenomenon can be very helpful in terms of adding realism and overall smoothness to any sound.

The spread parameter of Unity’s audio sources component allows us to address this phenomenon and vary the perceived width of a sound for the

226

ENVIRONMENTAL MODELING

listener. By default, an audio source in Unity has a width of 1, and the max value is 360. The spread parameter is expressed in degrees. As we increase the spread value the sound ought to occupy more space in the audio field.

The spread parameter will also affect how drastically the panning effects will be for 3D sounds sources as the listener moves around the audio source. At low values, if the audio source is set to 3D, the panning effects will be felt more drastically, perhaps at times somewhat artificially so, which can be distracting. Experimenting with this value will help mitigate that effect.

The spread parameter can also be controlled using a curve in the distance box in the 3D sound setting of an audio source like we did with the low pass filter component. Increasing the perceived width of a sound as we move toward it will likely increase the realism of your work, especially in VR applications where the player’s expectations are heightened.

To modulate the spread parameter based on distance: 1. Select an object with the audio source you wish to modulate the width of, or add one to an empty game object: component -> audio -> audio source.

2. In the inspector, find the audio source component, open the 3D source settings and at the bottom of the distance graph, click on the spread text at the bottom. This should now only display the spread parameter in the distance graph.

3. Keep in mind the x axis in this graph represents distance, while the y axis, in this case, represents the spread of the sound or width.

Moving the graph up and down with the mouse by simply clicking and dragging anywhere in the line will adjust the width of the audio source.

4. Move the line to the width you wish the sound to occupy when the listener is close to the audio source (usually wider), then double click the line where you wish spread to be at its narrowest. This should create a second anchor point. Move the anchor point to the desired width.

You’re done!

Keep in mind that as the spread value increases, panning will be felt less and less drastically as you move around the audio source, even if the audio source is set to full 3D. When the spread value is set to the maximum, panning might not be felt at all, as the sound will occupy the entire sound field.

Although Unity will by default set the spread parameter to a value of one, this will make every audio source appear to be a single point in place, which is both inaccurate with regard to the real world, and might make the panning associated with 3D sound sources relative to the listener jarring. Adjusting this parameter for your audio sources will contribute to making your work more immersive and detailed, especially, although not only, when dealing with VR/

AR applications.

ENVIRONMENTAL MODELING

227

Figure 9.4

c. Dry to Wet Ratio as a Product of Distance

We know that, in the real world, as we get closer to an audio source, the ratio of the dry to reflected signal changes, and we hear more of the dry or direct signal, as we get closer to the source and less of the reflected sound or reverberated signal. Implementing this will add an important layer of realism to our work.

A lot of models have been put forth for reverberation decay over distance by researchers over the years. One such was put forth by W.G. Gardner for Wave Arts inc. (1999), which suggests that for a dry signal with a level of 0dB

the reverb signal be about −20dB when the listener is at a distance of zero feet from the signal.

The ratio between both evens out at a distance of 100 feet, where both signals are equal in amplitude, the dry signal dropping from 0 to −40dB and the reverberant signal from −20 to −40dB. Past that point, the proposed model suggested that the dry signal drop by a level of −60dB at a distance of 1,000

228

ENVIRONMENTAL MODELING

feet, while the reverberant signal drops to a level of −50dB or an overall drop of 30dB over 1,000 feet. In other words:

1. At a distance of zero feet, if the dry signal has an amplitude of 0dB, the wet signal should peak at –20dB.

2. At a distance of 100 feet, both dry and wet signals drop to –40dB; the ratio between both is even.

3. At a distance of 1000 feet, the dry signal drops to −60dB while the wet signal plateaus at –50dB.

It is important to note that this model was not intended to be a realistic one but a workable and pleasant one. A more realistic approach is costly to compute and is usually not desirable anyway; if too much reverb is present, it may get in the way of clarity of the mix, intelligibility, or spatial localization.

Figure 9.5 Illustration of dry vs. wet signal as a product of distance Unity’s audio sources include a parameter that allows us to control how much of its signal will be processed by an existing audio reverb zone or zones, the reverb zone mix slider. A value of zero will send no signal to the global audio reverb bus dedicated to reverb zones, and the signal will appear to be dry. A value of 1 will send the full signal to the global bus. The signal will be much wetter and the reverb much more obvious.

This parameter can be controlled via script but also by drawing a curve in the distance graph of an audio source as we did with the low pass filter

ENVIRONMENTAL MODELING

229

and spread parameter. When working with reverb zones, this can be a good way to quickly change the dry to reflected signal ratio and increase immersion.

If you are using a mixer setup for reverberation in your scene, you must use automation, discussed in the adaptive mixing chapter.

Figure 9.6

d. Distance Simulation: Putting It All Together

All and all, convincing distance simulation of audio sources in games is achieved by combining several factors, each addressing a specific aspect of our perception of sound. These are:

• Volume: the change in level of an audio source with distance.

• Reverb: the ratio of dry to reverberant signal with distance.

• Frequency content: low pass filtering the audio source with distance,

• The change of perceived width of the audio source with distance.

230

ENVIRONMENTAL MODELING

Figure 9.7

Most game engines will give you the ability to control these parameters, and their careful implementation will usually yield satisfying and convincing results.

By carefully implementing these cues, you will create a rich and subtle environment and give the player a consistent and sophisticated way to gauge distance and establish an accurate mental picture of their surroundings via sound.

3. Additional Factors

1. Occlusion, Obstruction, Exclusion

As we’ve seen so far, Unity does not account for geometry when it comes to sound propagation, which is to say that an audio source will be heard through a wall from another room as if the wall wasn’t there, as long as the listener is within range of the audio source. The issue of obstacles between the audio source and the listener uses a combination of raycasting from the audio source to the listener and low pass filtering. This scenario was used to recreate the phenomenon known as occlusion.

There are, however, a number of situations that we should consider when dealing with a physical barrier or barriers between the listener and an audio source and whether the direct, reflected sound or both are obstructed.

ENVIRONMENTAL MODELING

231

a. Occlusion

Occlusion occurs when there is no direct path or line of sight, for either the direct or reflected sound to travel to the listener. As a result, the sound appears to be muffled, both significantly softer as well as low pass filtered. This can be addressed by a combination of volume drop and low pass filtering, as seen in with the smart audio source script. In order to detect an obstacle between the audio source and the listener, we can raycast from the audio source to the listener and look for colliders with the tag ‘geometry’ (the name of the tag is entirely up to the developer; however, it is recommended to use something fairly obvious). If one such collider is detected, we can update the volume and the cutoff frequency of a low pass filter added as a component to the audio source.

Figure 9.8

b. Obstruction

Obstruction occurs when the direct path is obstructed but the reflected path is clear.

The direct path may therefore be muffled, but the reflections ought to be clear. A common scenario would be someone standing behind a column listening to someone speaking on the other side. It’s important to know that, in spite of the obstacle, not all the direct sound is usually stopped by the obstacle.

The laws of physics, refraction in particular, tell us that frequencies whose wavelength is shorter than the obstacle will be stopped by the obstacle and not reach the listener, while frequencies whose wavelength is greater than that of the obstacle will travel around the obstacle. Since low frequencies have very

232

ENVIRONMENTAL MODELING

Figure 9.9

long wavelength, a 20Hz sound has a wavelength of approximately 17 meters or 55.6 feet; they tend not to be obstructed while high frequencies are much more easily stopped.

Obstruction, as with many aspects of our work, needs not be real-world accurate in order to be convincing and can be approximated by low pass filtering the direct sound but leaving the reflected sound unaffected.

c. Exclusion

Exclusion occurs when the direct path is clear but the reflected path is obstructed.

Figure 9.10

ENVIRONMENTAL MODELING

233

A common scenario would be walking past an open door leading to a reverberant space, such as a large church or cathedral, while the preacher is speaking facing the open doors. If you are on the outside, the path of the direct sound is unobstructed, while the path of the reflected sound is mostly contained within the space.

This can be approximated by lowering the level, possibly filtering the reflected sound and leaving the direct sound unaffected.

Out of these three cases, occlusion, obstruction and exclusion, obstruction is usually the most noticeable and therefore the most critical to implement. The reader is encouraged to refer back to Chapter eight, in the section on smart audio sources in order to look for an instance of occlusion implementation.

2. Distance Crossfades

Sounds that can be heard from a distance, such as a waterfall or thunder, present us with a few unique challenges. That is partly due to the fact that sounds can appear quite different from a distance than they do up close. As we get from afar to very close, naturally loud sound sources, such as a waterfall, tend to exhibit differences in three categories: amplitude, spectral content and spatial perception.

In addition to the obvious effect of distance over amplitude, spectral differences will also appear as a sound gets further and further away. It will sound more and more filtered; high frequencies tend to fade and while low frequencies remain. Indeed, especially over long distances, air acts as a low pass filter. The amount of filtering is a factor of distance, and atmospheric conditions such as air temperature, humidity level and atmospheric conditions. In addition to the overall amplitude dropping and the low pass filtering with distance, so do the details of amplitude modulation present in a sound. That is to say that the differences between the peaks and valleys present in the amplitude of a sound also tend to fade away, and the sound may appear to be slightly ‘washed out’, partly due to the combination of loss of high frequencies and the ratio of dry to reverberant sound increasing with distance. Reverberation can indeed have a smoothing effect on the dynamic range of a sound.

In addition to amplitude and spectral changes, sounds that can be heard over large distances also change in how they appear to be projected spatially.

In the case of a waterfall, for instance, from a distance the sound is clearly directional, and you could use the sound itself to find your way to the waterfall. From up close, however, the same sound may not be so easy to pinpoint and, in fact, might not be localizable at all, as it might appear to completely envelop the listener. In other words, from a distance the waterfall might appear to be a 3D sound, but from up close it would turn into a 2D sound.

The transition is of course gradual, and as the listener gets closer to the source of the sound, the apparent width of the sound will appear to get larger.

234

ENVIRONMENTAL MODELING

Rather than try to manipulate a single recording to fit both up close and afar sounds, it is usually much more satisfying and believable to crossfade between two sounds – a faraway sound and a close-up one – and change the mix in relation to the distance of the listener to the source. This technique is known as a distance crossfade. To implement it in Unity requires two audio sources and keeping track of the distance of the listener to the source. Distance crossfade implementation was discussed in detail in Chapter eight.

3. Doppler Efect

The doppler effect is the perceived shift in pitch as a sound source moves relative to a listener. This is an extremely common occurrence, one that we’re all familiar with. Perhaps the most common example is that of an emergency vehicle with sirens on, driving fast past a person standing on a sidewalk. As the vehicle moves toward us, the pitch of the siren seems to increase, then decrease as the vehicle moves away. This can of course provide us with important information as to the location of moving objects relative to the listener in games. The change in pitch is due to the wavelength of the sound changing as the vehicle or sound source is moving.

Sound moving away from Listener

Listener

Apparent wavelength

Sound moving toward Listener

Apparent wavelength

Listener

Figure 9.11

As the vehicle moves toward the listener, the oncoming sound waves are compressed together, reducing the wavelength and therefore increasing the pitch. Conversely, as the vehicle moves away, the movement from the vehicle stretches the waveform and extends the wavelength, lowering the pitch.

Note: the relationship between frequency and wavelength is given to us by the formula:

frequency in Hz = speed of sound/wavelength

ENVIRONMENTAL MODELING

235

The change in observed frequency can be calculated from the following formula:

Figure 9.12

Where:

ƒ = observed frequency in Hertz

c = speed of sound in meters per seconds

Vs = velocity of the source in meters per seconds. This parameter will have a negative value if the audio source is moving toward the listener, positive if moving away from the listener.

ƒo = emitted frequency of source in Hertz

Although the math is helpful to understand the underlying phenomenon, it is provided only as a reference to the reader, since in the game audio business, we are storytellers first and foremost, and accuracy is always second to the narrative.

Unity offers doppler shift control on each audio source individually, and a global control is found in the project settings: Edit->Project Settings->Audio-> Doppler Factor.

The Doppler factor acts as a global setting to emphasize or de-emphasize the Doppler effect on every audio source in the game, the default value being one. The higher the value, the more pronounced the effect will be overall.

Values under 1 will reduce the perceived effect.

Figure 9.13

236

ENVIRONMENTAL MODELING

Additionally, each audio source’s doppler effect, labelled Doppler Level, can be adjusted individually from the audio source’s 3D settings: Figure 9.14

The default value for Doppler Level is 1. Increasing this value will increase or exaggerate the perceived shift in pitch for moving audio sources, and, conversely, lowering will make the effect less obvious to nonexistent.

When thinking about how to use the Doppler feature in Unity or any other game engine, remember our motto from Chapter two: inform and entertain.

Use the Doppler effect to let the player know when critical elements are in motion and in which direction they are moving, either toward or away from the player. This can be applied to enemy vehicles or drones, large projectiles and anything else the user would benefit from.

Adjusting the value of the Doppler effect for each audio source is to be done on an individual basis in the context of the game and mix. Experimentation is key. Usually you’ll be looking for a balance where the doppler effect is easily noticeable, yet not distracting or even comical. Remember our conversation

ENVIRONMENTAL MODELING

237

on immersion in Chapter two; if the effect is too obvious and jumps out in the mix, it will break immersion.

Conclusion

Environmental modeling is as important to game audio as sound design and implementation. Increasingly, as we develop more and more immersive, realistic looking levels and games, the ability for our sounds to exist within an environment that makes their propagation and behavior believable has become all the more important. Being able to address the main issues of distance simulation, spatialization, occlusion and Doppler shift will make every experience you design all the more enjoyable for the user and make your work stand out.

10 PROCEDURAL AUDIO

Beyond Samples

Learning Objectives

In this chapter we will look at the potential and practical applications of procedural audio, its benefts and drawbacks, as well as how to tackle this relatively new approach to sound design. Rather than an in-depth study of the matter, which would be beyond the scope of this book, we will examine the potential benefts and drawbacks of this technology and carefully take a look at two specifc models to illustrate these concepts. First, we will look at how to model a wind machine using mainly subtractive synthesis, then we will look at a physical model of a sword, realized using linear modal synthesis. Due to basic limitation with Unity’s audio engine, both models will be realized in MaxMSP but can be easily be ported to any synthesis engine.

1. Introduction, Benefts and Drawbacks

With the advent of PCM playback systems throughout the 80s and 90s, video game soundtracks gained a great deal in terms of realism and fidelity when compared to the limited capabilities of early arcades and home entertainment systems. All of sudden, explosions sounded like actual explosions rather than crunchy pink noise, video games started to include actual dialog rather than type on the screen punctuated by cute chirp sounds and the music sounded like it was played by actual instruments. The improvements were so significant that almost no one seemed to have noticed or minded that they came at the expense of others: flexibility and adaptivity.

We sometimes forget that an audio recording is a snapshot of a sound at an instant, frozen in time, and while we can use a number of techniques to make a recording come to life, we can only take it so far before we simply need another sample. Still, it wasn’t too much of an issue until two technological advancements made the limitations of sample playback technology obvious.

The first one was the advent of physics in games. Once objects started to fall, bounce, scrape and everything in between, the potential number of sounds

PROCEDURAL AUDIO

239

they could generate became exponentially larger. Other technical limitations, however, remained. Ram budgets didn’t change, and audio hardware and software didn’t either. All of sudden, the limitations of our current technologies became quite apparent as sound designers and game developers had to develop new techniques to deal with these developments and to come up with enough sounds to cover all the potential situations that could arise. Even though a single object could now make dozens, if not hundreds of potential different sounds, we were still confronted with the same limitations in terms of RAM and storage. Even if these limitations weren’t there and we could store an unlimited number of audio assets for use in a game, spending hundreds of hours coming up with sounds for every possible permutation an object could make would be an entirely unproductive way to spend one’s time, not to mention a hopeless task.

The other major development that highlighted the issues associated with relying on playing back samples is virtual reality. Experiencing a game through a VR headset drastically changes the expectations of the user. Certainly, a higher level of interactivity with the objects in the game is expected, which creates a potentially massive numbers of new scenarios that must be addressed with sound. PCM playback and manipulation again showed its limitations, and a new solution was needed and had been for some time: procedural studio.

1. What Is Procedural Audio?

The term procedural audio has been used somewhat liberally, often to describe techniques relying on the manipulation and combination of audio samples to create the desired sound. For the sake of simplicity, we will stick to a much more stringent definition that excludes any significant reliance on audio recordings.

The term procedural assets, very specifically, refers to assets that are generated at runtime, based on models or algorithms whose parameters can be modified by data sent by the game engine in real time. Procedural asset generation is nothing new in gaming; for some time now textures, skies and even entire levels have been generated procedurally, yet audio applications to this technology have been very limited.

a. Procedural Audio, Pros and Cons

Let’s take a closer look at some of the pros and the cons of this technology before taking a look at how we can begin to implement some of these ideas, starting with the pros:

• Flexibility: a complete model of a metal barrel would theoretically be able to recreate all the sounds an actual barrel could make – including rolls and scrapes, bounces and hits – and do so in real time, driven by data from the game engine.

240

PROCEDURAL AUDIO

• Control: a good model will give the sound designer a lot of control over the sound, something harder to do when working with recordings.

• Storage: procedural techniques also represent a saving in terms of memory, since no stored audio data is required. Depending on how the sound is implemented, this could mean savings in the way of streaming or ram.

• Repetition avoidance: a good model will have an element of random-ness to it, meaning that no two hits will sound exactly alike. In the case of a sword impact model, this can prove extremely useful if we’re working on a battle scene, saving us the need to locate, vary and alter-nate samples. This applies to linear post production as well.

• Workflow/productivity: not having to select, cut, and process variations of a sound can be a massive time saver, as well as a significant boost in productivity.

Of course, there are also drawbacks to working with procedural audio, which must also be considered:

• CPU costs: depending on the model, CPU resources needed to render the model in real time may be significant, in some cases making the model unusable in the context of an actual game.

• Realism: although new technologies are released often, each improving upon the work of previous ones, some sounds are still difficult to model and may not sound as realistic as an actual recording, yet. As research and development evolve, this will become less and less of an issue.

• A new paradigm: procedural audio represents a new way of working with sound and requires a different set of skills than traditional recording-based sound design. It represents a significant departure in terms of techniques and the knowledge required. Some digital signal processing skills will undoubtedly be helpful, as well as the ability to adapt a model to a situation based on physical modeling techniques or programming. Essentially, procedural audio requires a new way of relating to sound.

• Limited implementation: this is perhaps the main hurdle to the widespread use of this technology in games. As we shall see shortly, certain types of sounds are already great candidates for procedural audio techniques; however, implementation of tools that would allow us to use these technologies within a game engine is very limited still at the time of this writing and makes it difficult to apply some of these techniques, even if every other condition is there (realism, low CPU

overhead etc.).

It seems inevitable that a lot of the technical issues now confronting this technology will be resolved in the near future, as models become more efficient

PROCEDURAL AUDIO

241

and computationally cheaper, while at the same time increasing in realism.

A lot of the current drawbacks will simply fade over time, giving us sound designers and game developers a whole new way of working with sound and an unprecedented level of flexibility.

Candidates for Procedural Audio

Not every sound in a game might be a good candidate for procedural audio, and careful consideration should be given when deciding which sound to use procedural audio techniques for. Certain sounds are natural candidates, however, either because they can be reproduced convincingly and at little computational cost, such as hums, room tones or HVAC

sounds or because, although they might use a significant amount of resources, they provide us with significant ram savings or flexibility, such as impacts.

b. Approaches to Procedural Audio

When working on procedural audio models, while the approach may differ from traditional sound design techniques, it would be a mistake to consider it a complete departure from traditional, sampled-based techniques, but rather it should be considered an extension. The skills you have accumulated so far can easily be applied to improve and create new models.

Procedural audio models fall in two categories:

• Teleological Modeling: teleological modeling relies on the laws of physics to create a model of a sound by attempting to accurately model the behavior of the various components of an object and the way they interact with each other. This is also known as a bottom-up approach.

• Ontological modeling : Ontological modeling is the process of building a model based on the way the object sounds rather than the way it is built, a more empirical and typical philosophy in sound design. This is also known as a top-down approach.

Both methods for building a model are valid approaches. Traditional sound designers will likely be more comfortable with the ontological approach, yet a study of the basic law of physics and of physical modeling synthesis can be a great benefit.

Analysis and Research Stage

Once a model has been identified, the analysis stage is the next logical step.

There are multiple ways to break down a model and to understand the mechanics and behavior of the model over a range of situations.

242

PROCEDURAL AUDIO

In his book Designing Sound (2006), Andy Farnell identifies five stages of the analysis and research portion:

• Waveform analysis.

• Spectral analysis.

• Physical analysis.

• Operational analysis.

• Model parametrization.

Waveform and Spectral Analysis

The spectral analysis of a sound can reveal important information regarding its spectral content over time and help identify resonances, amplitude envelopes and a great deal more. This portion isn’t that different from looking at a spectrogram for traditional sound design purposes.

Physical Analysis

A physical analysis is the process of determining the behavior of the physical components that make up the body of the object in order to model the ways in which they interact. It is usually broken down into an impulse and the ensuing interaction with each of the components of the object.

The impulse is typically a strike, a bow, a pluck, a blow etc. Operational analysis refers to the process of combining all the elements gathered so far into a coherent model, while the model parametrization process refers to deciding which parameters should be made available to the user and what they should be labelled as.

2. Practical Procedural Audio: A Wind Machine

and a Sword Collision Model

The way to apply synthesis techniques to the area of procedural audio is limited mainly by our imagination. Often, more than one technique will generate convincing results. The choice of the synthesis methods we choose to implement and how we go about it should always be driven by care for resource management and concern for realism. Next we will look at how two different synthesis techniques can be used in the context of procedural audio.

1. A Wind Machine in MaxMSP With Subtractive Synthesis

Noise is an extremely useful ingredient for procedural techniques. It is both a wonderful source of raw material and computationally inexpensive. Carefully shaped noise can be a great starting point for sounds such as wind, waves, rain, whooshes, explosions and combustion sounds, to name but a few potential applications. Working with noise or any similarly rich audio source naturally lends itself to subtractive synthesis. Subtractive synthesis consists in carving away frequency material from a rich waveform using filters, modulators and envelopes

PROCEDURAL AUDIO

243

until the desired tone is realized. Using an ontological approach we can use noise and a few carefully chosen filters and modulators to generate a convincing wind machine that can be both flexible in terms of the types of wind it can recreate as well as represent significant savings in terms of audio storage, as wind loops tend to be rather lengthy in order to avoid sounding too repetitive.

We can approximate the sound of wind using a noise source. Pink noise, with its lower high frequency content will be a good option to start from, although interesting results can also be achieved using white, or other noise colors.

Figure 10.1

Figure 10.2

244

PROCEDURAL AUDIO

White noise vs. pink noise. The uniform distribution of white noise is contrasted.

Broadband noise will still not quite sound like wind yet, however. Wind tends to sound much more like bandpass filtered noise, and wind isn’t static, either in terms of amplitude or perceived pitch. Both evolve over time. Wind also tends to exhibit resonances more or less pronounced depending on the type of wind Figure 10.3 The spectrogram reveals how the frequency content of this particular wind sample evolves over time

Figure 10.4

PROCEDURAL AUDIO

245

we are trying to emulate. A look at a few spectral analyses of wind recordings can be used to extract more precise data which can be used to tune the parameters of the model, such as the center frequency of our bandpass filter(s), the amount of variation over time of amplitude and pitch and many more.

From a starting point of pink noise, in order to make our wind more convincing, first we need to apply a resonant bandpass filter to our noise source.

The center frequency of our bandpass filter will determine the pitch of the wind. One way to find the right center frequency for the bandpass filter is to take a look at a spectral analysis of a few wind samples in the same vein as the sound we are trying to emulate, use these as a starting point and adjust until your ears agree. Once we’ve achieved convincing settings for center frequency and bandwidth of the bandpass filter, we must animate our model so that the output is not static. For our model to be realistic we ‘re going to need to modulate both the overall amplitude of the output, as well as the center frequency of the bandpass filter. The frequency of the bandpass filter, which would be the perceived ‘pitch’ of the wind, needs our attention first. Using a classic modulation technique, such as an LFO with a periodic waveform, would sound too predictable and therefore sound artificial. Therefore, it makes more sense to use a random process. A truly random process, however, would cause the pitch of the wind to jump around, and the changes in the pitch of the wind would feel disconnected from one another, lacking the sense of overall purpose. In the real world the perceived pitch of wind doesn’t abruptly change from one value to another but rather ebbs and flows. The random process best suited to match this kind of behavior would be a random walk, a random process where the current value is computed from the previous value and constricted to a specific range to keep the values from jumping randomly from one pitch to another.

The center frequency for the overall starting pitch of the wind will be determined by the center frequency of the bandpass filter applied to the noise source, to which a random value will be added at semi-regular intervals. By increasing the range of possible random values at each cycle in the random walk process we can make our wind appear more or less erratic. The amount of time between changes should also not be regular but determined by a random range, which the sound designer can use to create a more or less rapidly changing texture.

A similar process can be applied to the amplitude of the output, so that we can add movement to the volume of the wind model as well. By randomizing the left and right amplitude output independently we can add stereo movement to our sound and increase the perceived width of the wind.

Making the Model Flexible

In order to make our model flexible and capable of quickly adapting to various situations that can arise in the context of a game, a few more additions would be welcome, such as the implementation of gusts, of an intense low rumble

246

PROCEDURAL AUDIO

for particularly intense winds and the ability to add indoors vs. outdoors perspective.

Wind gusts are perceived as rapid modulation of amplitude and/or frequency; we can recreate gusts in our model by rapidly and abruptly modulating the center frequency, and/or the bandwidth of the filter.

In a scenario where the player is allowed to explore both indoors and outdoors spaces or if the camera viewpoint may change from inside to outside a vehicle, the ability to add occlusion to our engine would be very convenient indeed. By adding a flexible low pass filter at the output of our model, we can add occlusion by drastically reducing the high frequency content of the signal and lowering its output. In this setting, it will appear as if the wind is happening outside, and the player is indoors.

Rumble can be a convincing element to create a sense of intensity and power. We can add a rumble portion to our patch by using an additional noise source, such as pink noise, low pass filter its output and distort the output via saturation or distortion. This can act as a layer the sound designer may use to make our wind feel more like a storm and can be added at little additional computational cost.

The low rumble portion of the sound can itself become a model for certain types of sounds with surprisingly little additional work, such as a rocket ship, a jet engine and other combustion-based sounds. As you can, the wind-maker patch is but a starting point. We could make it more complex by adding more noise sources and modulating them independently. It would also be easy to turn it into a whoosh maker, room tone maker, ocean waves etc. The possibilities are limitless while the synthesis itself is relatively trivial computationally.

2. A Sword Maker in MaxMSP With Linear Modal Synthesis

Modal synthesis is often used in the context of physical modeling and is especially well suited to modeling resonant bodies such as membranes and 2D and 3D resonant objects. By identifying the individual modes or resonant frequencies of an object under various conditions, such as type of initial excitation, intensity and location of the excitation, we can understand which modes are activated under various conditions allowing us to model the sound the object would make, allowing us to build a model of it.

The term modes in acoustics is usually associated with the resonant characteristics of a room when a signal is played within it. Modes are usually used to describe the sum of all the potential resonant frequencies within a room or to identify individual frequencies. They are also sometimes referred to as standing waves, as the resonances created tend to stem from the signal bounc-ing back and forth against the walls, thus creating patterns of creative and destructive interference. A thorough study of resonance would require us to delve into differential equations and derivatives; however, for our purposes we can simplify the process by looking at the required elements for resonance to occur.

PROCEDURAL AUDIO

247

Resonance requires two elements:

• A driving force: an excitation, such as a strike.

• A driven vibrating system: often a 2D or 3D object.

When a physical object is struck, bowed or scrapped, the energy from the excitation source will travel throughout the body of the object, causing it to vibrate, thus making a sound. As the waves travel and reflect back onto themselves, complex patterns of interference are generated and energy is stored at certain places, building up into actual resonances. Modal synthesis is in fact a subset of physical modeling. Linear modal synthesis is also used in engineering applications to determine a system’s response to outside forces.

The main characteristics that determine an object’s response to an outside force are:

• Object stiffness.

• Object mass.

• Object damping.

Other factors are to be considered as well, such as shape and location of the excitation source, and the curious reader is encouraged to find out more about this topic.

We distinguish two types of resonant bodies (Menzies):

• Non-diffuse resonant bodies: that exhibit clear modal responses, such as metal.

• Diffuse resonant bodies: exhibit many densely packed modes, typically wood or similar non-homogenous materials.

Modeling non-diffuse bodies is a bit simpler, as the resonances tend to happen in more predictable ways, as we shall see with the next example: a sword impact engine.

Modal synthesis is sometimes associated with Fourier synthesis, and while these techniques can be complementary, they are in fact distinct. The analysis stage is important to modal synthesis in order to identify relevant modes and their changes over time. In some cases, Fourier techniques may be used for the analysis but also to synthesize individual resonances. In this case, we can take a different approach; using a spectral analysis of a recording of a sword strike we can identify the most relevant modes and their changes over time. We will model the most important resonances (also referred to as strategic modeling) using highly resonant bandpass filters in MaxMSP. The extremely narrow bandwidth will make the filters ring and naturally decay over time, which will bypass the need for amplitude envelopes. The narrower the bandpass filter, the longer it will resonate. Alternatively, envelope sine waves can be used to model individual modes; sometimes both methods are used together.

248

PROCEDURAL AUDIO

Note: we are using filters past their recommended range in the MaxMSP

manual; as always with highly resonant filters, do exercise caution as the potential for feedback and painful resonances that can incur hearing damage is possible. I recommend adding a brickwall limiter to the output of the filters or overall output of the model in order to limit the chances for potential accidents.

Spectral Analysis

We’re going to start our modeling process by taking a look at a spectrogram of a sword strike. This will help us understand exactly what happens when a sword hit occurs:

Figure 10.5

Looking at this information can teach us quite a bit about the sound we are trying to model. The sound takes place over the course of 2.3 seconds, and this recording is at 96Khz, but we shall only concern ourselves with the frequencies up to 20Khz in our model. The sound starts with a very sharp, short noise

PROCEDURAL AUDIO

249

burst lasting between 0.025 and 0.035 seconds. This is very similar to a broadband noise burst and is the result of the impact itself, at the point of excitation.

After the initial excitation, we enter the resonance or modal stage. A sword falling in the category of non-diffuse bodies exhibits clear resonances that are relatively easy to identify with a decent spectrogram. The main resonances fall at or near the following frequencies:

• 728Hz.

• 1,364Hz.

• 2,264Hz.

• 2,952Hz.

• 3,852Hz.

All these modes have a similar length and last 2.1 seconds into the sound, the first four being the strongest in terms of amplitude. Additionally, we can also identify secondary resonance at the following:

• 5,540Hz, lasting for approximately 1.4 second.

• 7,134Hz, lasting for approximately 0.6 second.

Further examination of this and other recordings of similar events can be used to extract yet more information, such as the bandwidth of each mode and additional relevant modes. To make our analysis stage more exhaustive it would be useful to analyze strikes at various velocities, as to identify the modes associated with high velocity impact and any changes in the overall sound that we might want to model.

We can identify two distinct stages in the sound: 1. A very short burst of broadband noise, which occurs at the time of impact and lasts for a very short amount time (less than 0.035 seconds).

2. A much longer resonant stage, made of a combination of individual modes, or resonances. We identified seven to eight resonances of interest with the spectrogram, five of which last for about 2.2 seconds, while the others decay from 1.4 seconds to 0.6 seconds approximately.

Next we will attempt to model the sound, using the information we extracted from the spectral analysis.

Modeling the Impulse

The initial strike will be modeled using enveloped noise and a click, a short sample burst. The combination of these two impulse sources makes it possible to model an impulse ranging from a mild burst to a long scrape and everything in between. Low-pass filtering the output of the impulse itself is a very common technique with physical modeling. A low pass-filtered impulse can be used to model impact velocity. A low-pass filtered impulse will result in

250

PROCEDURAL AUDIO

fewer modes being excited and at lower amplitude, which is what you would expect in the case of a low velocity strike. By opening up the filter and letting all the frequencies of the impulse through, we excite more modes, at higher amplitude, giving us the sense of a high velocity strike.

Scrapes can be obtained by using a longer amplitude envelope on the noise source.

Modeling the Resonances

This model requires a bank of bandpass filters in order to recreate the modes that occur during the collision; however, we will group the filters into three banks, each summed to a separate mixing stage. We will split the filters according to the following: initial impact, main body resonances and upper harmonics, giving us control over each stage in the mix.

Making the Model Flexible

Once the individual resonances have been identified and successfully implemented, the model can be made flexible in a number of ways at low additional CPU overhead.

A lot can be done by giving the user control over the amplitude and length of the impulse. A short impulse will sound like an impact, whereas a sustained one will sound more like a scrape. Strike intensity may be modeled using a combination of volume control and low-pass filtering. A low-pass filter can be used to model the impact intensity by opening and closing for high velocity and low velocity impacts. Careful tuning of each parameter can be the difference between a successful and unusable model.

Similarly to the wind machine, this model is but a starting point. With little modification and research we can turn a sword into a hammer, a scrape generator or generic metallic collisions. Experiment and explore!

Conclusion

These two examples were meant only as an introduction to procedural audio and the possibilities it offers as a technology. Whether for linear media, where procedural audio offers the possibility to create endless variations at the push of a button or for interactive audio, for which it offers the prospect of flexible models able to adapt to endless potential scenarios, procedural audio offers an exciting new way to approach sound design. While procedural audio has brought to the foreground synthesis methods overlooked in the past such as modal synthesis, any and all synthesis methods can be applied toward procedural models, and the reader is encouraged to explore this topic further.

Note: full-color versions of the figures in this chapter can be found on the companion website for this book.

11 ADAPTIVE MIXING

Learning objectives

In this chapter we will identify the unique challenges that interactive and game audio poses when it comes to mixing and put forth strategies to address them. By the end of this chapter the student will be able to identify potential pitfalls of non-linear mixing, set up mixers and mixer groups in order to optimize the mix process, use code to automate mixer parameters and use snapshots to create a mix that adapts to the gameplay and the environment.

1. What’s in a Mix? Inform and Entertain (Again)

As we did with sound design, before looking at the techniques available to us for mixing, we should stop and ask ourselves what makes a good mix in a game. Indeed, if we are unable to identify the objectives we are trying to achieve, we will never reach them. The question is not a simple one; there are a lot of factors to consider, and as always with interactive media, the relative unpredictability of gameplay complicates the matter somewhat. So, what are the goals one should strive for in considering mixing for game audio?

The mix is how we present information to the player. A good mix will make our experience all the more immersive and cinematic and make our sound design shine by highlighting the most important sounds and presenting them clearly to the player.

The following is a non-exhaustive list but will act as a good starting point for our conversation.

1. Mix Considerations

1. Clarity: as with any mix, linear or not, real time or not, achieving clarity is an essential aspect of our work. Many sounds sharing similar characteristics and spectral information will likely play on top of each other; our job is to make sure that all sounds are heard clearly and

252

ADAPTIVE MIXING

that, no matter what, the critical sounds for the gameplay are heard clearly above all else.

2. Dynamic range: a good mix should have a decent dynamic range, giving the player’s ears time to rest during low intensity moments and highlighting and enhancing the gameplay during action-packed sequences. A good dynamic range management will make it easier to hear the details of a well-crafted soundtrack, immersing the player further.

3. Prioritization: at any given moment, especially during the more intense portions of the game, the engine might attempt to trigger a large number of audio sources. The question for us is which of these sounds are the most relevant to the player and can provide them with information to play the game better, giving them a better gaming experience. For instance, a bit of critical dialog may be triggered at the same time as an explosion. While both need to be heard, the dialog, although much softer than the explosion, still needs to be heard clearly, and it is the responsibility of the developer to see to it.

4. Consistency: a good mix should be consistent across the entire game.

The expectations developed during the earlier portions of the game in terms of quality and levels should be met throughout. Audio levels between scenes should be consistent and of course so should sounds by categories such as dialog, footsteps, guns etc.

5. Narrative function: the mix needs to support and enhance the storyline and gameplay. It needs to be both flexible and dynamic, reflect-ing both the environment and plot developments. This can mean something as obvious as the reverb changing when switching to a different environment, but it is often much more subtle. Simple moves like making the beginning of a sound slightly louder when it is introduced for the first time can tell the player to pay attention to something on the screen or the environment without being too obvious about it.

6. Aesthetics: this is harder to quantify, but there are certain things to look out for when thinking about the overall aesthetics of the mix.

Does it sound harsh when played at high levels; is the choice of effects such as reverbs, delays and other processes optimized to serve the soundtrack as well as possible? Is it pleasant to listen to over long periods and at all levels, or is the bottom end clear and powerful yet not overpowering? These and many more questions are the ones that relate specifically to the aesthetics of a mix.

7. Spatial imaging: 3D and virtual/mixed reality environments require special attention to the spatial placement of sounds. Our mix needs to accurately represent the location of sounds in 3D space using the technologies at our disposal to the best of our abilities.

8. Inform: How do we create a mix that informs the player, providing them with important cues and establishing a dialog between the user

ADAPTIVE MIXING

253

and game itself? If all the points mentioned so far have been carefully weighted into your mix, very likely you’ve already succeeded in doing so.

• Are the important sounds prioritized in the mix?

• Does the mix reflect the environment accurately or appropriately? In this way the player is able to gain information on the space the scene takes place in.

• In a 360-degree environment, sounds can be used to focus the attention of the player. Do make sure that sounds used in such a way are clearly heard, designed to be able to be easily localized; remember the chime vs. buzzer principle. Sounds with brighter spectrums and a sharp attack are easier to localize than low-frequency hums.

With so many variables involved, it isn’t very surprising that mixing is a skill that is acquired over time, likely by working on both linear and non-linear material. It is important to understand that a good mix is a dynamic one and that we should always be in control of it. Let’s begin by breaking down the mix into three main categories – music, dialog and sound effects – and understand each’s function in the context of a mix.

2. Music, Dialogue and Sound Efects

The soundtrack of games and movies can be broken down in terms of its three most important structural components: music, dialog and sound effects. Each serves a different purpose, and the mix is – or should – ultimately be dominated by one of these three elements at any given point based on the desired perspective and emotional impact.

Music serves to underscore or manipulate the emotional perspective of the game. It tells us how to feel and is usually the most emotionally impactful aspect of the soundtrack. The music throughout the development of a game or movie is often quite dynamic, from very soft to very loud, and we might need to make sure that, while preserving the original intentions of the composer and the needs of the scene, we keep the music within a comfortable range, only breaking out of it when there is a need to make a certain point.

The dialog is the main narrative element of the soundtrack, and as such it is usually treated as the most important aspect of the soundtrack when present.

Nothing should get in the way of dialog unless intentional, taking precedence over music and sound effects. In games, dialog falls into two broad categories, critical, which contains the important narrative elements – in other words it moves the plot forward – and non-critical, which does not contain important information and can therefore be treated as chatter.

The sound effects serve a number of purposes. They greatly contribute to the feeling of overall immersion by giving us a detailed and rich aural picture of our environment; they take the place of sense that we cannot experience

254

ADAPTIVE MIXING

over a screen and speakers, and, crucial to gaming, provide us with information on the objects and the environment that we evolve in, such as location, texture, movement etc. Sound effects can also become part of the emotional or narrative aspect of a game or a scene. Indeed, none of these categories are absolute. A good sound designer will sometimes blur the lines between the music and sound effects by using sounds that blend with and perhaps even augment the musical score.

Note: when present, narration can sometimes be considered a fourth component of the soundtrack, to be treated independently of the dialog.

At any given moment, the mix should be driven or dominated by one of these categories – and usually only one. The same principle applies to movies.

If there is dialog, the music and the sound effects should not get in its way, and we should consider taking them down in the mix. The choice of which category should dominate and when usually depends on the gameplay itself.

In video games you will hear the terms states or game states used quite often. Game states can be used to mean any number of things, as they are a technique for implementing artificial intelligence in games, sometimes described as finite state machine. Game states, as they relate to mixing, are usually derived from the significant changes in the gameplay such as switching from an exploration mode to battle mode. These changes in game states can be useful references for our mix to follow and adapt, and they ideally organically stem from the game itself.

3. Planning and Pre-Production

Planning is an essential part of the mixing process.

A mix can be approached like a complex problem that we need to solve, the problem being: how do we get a large number of audio files playing all at once, in an impossible-to-predict sequence, to sound like an ordered, easy to understand and pleasant mix, rather than a cacophony of overlapping noises?

a. SubMixing

A classic and effective approach to tackling a complex problem is to break down complexity into smaller, manageable bits, which when put together create the desired result or mix. Breaking down complexity means, for instance, that rather than thinking of a mix as a large number of audio files playing all at once, we start by considering how we can group multiple tracks into a small number of groups by routing audio sources into a few carefully chosen subpaths – or in Unity’s case, in groups. The process starts by grouping sounds that belong together into subcategories such as music, dialog and sound effects, then by dividing audio sources still further such that sound effects might be made up of several subgroups such as ambiences, explosions, Foley etc.

This means that rather than trying to mix 40 sounds or more at once, we start

ADAPTIVE MIXING

255

by focusing on each of these submixes and therefore only a few sounds at a time. After we are satisfied with the submix, such as ambience, Foley etc., we can consider how the overall ambience sits in the mix and make adjustments in the overall context of the mix.

This is a recursive process, one that requires making constant adjustments as more elements are brought in.

b. Routing

Careful routing is essential in order to get the most flexible mix. Establishing a good routing structure is critical. It usually starts from the basic categories that constitute a traditional soundtrack, music, dialog and sound effects and gets further subdivided based on the sounds present in the soundtrack. At this stage you can effectively architect the mix and plan the various places where you will place dynamic compressors and set up side chain inputs. Every mix, every game is slightly different, but the following diagram should make for a good starting point from which to work.

Figure 11.1

256

ADAPTIVE MIXING

As you can see, music, dialog and sound effects get their own subgroup, all routed to the main output, at the top of which sits a limiter, to prevent any signal to exceed 0dBFS and cause distortion. The limiter should probably have its output ceiling or maximum level output set slightly below 0dBFS – such as −0.3dBFS – and a quick attack time to catch fast transients and prevent them from getting past the limiter.

c. Dynamic Range

It is impossible to study mixing without at some point discussing the concept of dynamic range. The term can be confusing because dynamic range can be employed in either one of two contexts:

1. The difference between the loudest and softest points of a recording, expressed in dB.

2. The ratio between the softest and loudest signal intensity a particular piece of equipment may accurately capture or playback.

Concept 1 is relevant to us because a good mix should have a decent dynamic range, in order to create contrast, surprise and give the player a pleasant experience by giving their ears a break once appropriate. Without adequate dynamic range the mix becomes tough to listen to very quickly for a number of reasons. Without giving the player ear’s the opportunity to rest, it will inevitably become tiring, leading them to possibly turn off the soundtrack altogether. A mix without dynamic range or an insufficient one will also tend to deteriorate the transients of the sounds it plays back, shaving them off. This will blur the attack portion of percussive sounds and make it harder for our ears and brain to process. Over-compression and poor gain staging leading to levels on individual groups and/or the master level clipping are amongst the main culprits.

If the dynamic range is too large, however, and the difference between the softest sounds and loudest sounds is too great, likely the player will adjust their monitoring levels based on the loudest sounds played and will simply not hear the softest sounds in the mix, usually ambiences but possibly more, including dialog.

A good balance between the two is therefore required and needs to be achieved in order to create the best possible experience.

Measuring Dynamic Range in the Digital Audio Domain: In the digital audio domain, the dynamic range of a given device is related to the bit depth at which you record or playback a session. The top of the dynamic range, the loudest possible point the system is capable of reproducing, is 0dBFS, where FS stands for full scale. 1 bit is roughly equal to 6dB

of dynamic range; that means that a session at 24bit depth has a potential dynamic range of up to 144dB, from 0dB to −144dB. At 16 bits, which some game engines still operate at, the dynamic range is lesser, upwards of 96dB.

ADAPTIVE MIXING

257

Figure 11.2

A compressor typically sits on the output of each of the three main subgroups as well. These tend to serve either one of two purposes: they can be used as a regular bus compressor, taking care of lowering loudness peaks in the signal routed through them, as well as blending all the sounds together via mild compression. They can also work as side chain compressors or ‘ducking’ compressors, usually taking their key input from the dialog and applying compression on the music and sound effects busses when dialog is present. For that reason and other potential ones, the dialog bus is usually split in two submixes: critical and non-critical dialog. Only the critical dialog would trigger the sidechain inputs on the compressors located on the music and sound effects busses. Typically, the compressor on the dialog will not have a key input and will work as a regular bus compressor.

The music bus will usually be a simpler setup, as while music soundtrack can get quite complex in terms of branching and adaptivity, the music or stems that comprise the soundtrack are usually already mixed. In some instances, if one is available a multiband compressor will sometimes help mix complex stems together. Since dialog may be triggered on top of the music, a compressor with a side chain input listening to the dialog will usually sit atop the music bus.

The sound effect bus is usually the busiest and most complex due to the number and variety of sound effects that make up a soundtrack. Just like the music bus, the sound effect bus will usually have a compressor keyed to the dialog group, sitting atop the bus, but the subgroup structure is usually much more complex. It is impossible to come up with a one-fits-all template, and

258

ADAPTIVE MIXING

each game has to be considered individually, but if we were to set up a mix for a first-person shooter, we might consider the following subgroups:

• Ambiences: room tones, outdoors sounds, city sounds.

• Player sounds: footsteps, player Foley, player vocalizations.

• Vehicles.

• Weapons: usually further divided into two main subgroups: friendly and enemies. Hearing your enemies’ positions, activity and fire is argu-ably more important than being able to hear the weapons from your own team, but further subgroups are possible.

• Explosions: explosions are dangerous, to our mixes anyhow. They tend to be the loudest elements of a game, and great care must be applied to avoid overwhelming the sound effect bus and possibly even the master bus. In order to do so, a limiter usually sits on top of the explosion bus.

• Enemies: footstep sounds, Foley, vocalizations.

Routing in mixing is usually done via busses, which are circuits or pathways that allow the mix engineer to route several audio tracks to a single destination. Unity uses a system of groups, which acts as a destination for multiple audio sources and send and receive modules to send signals from one group to another.

d. Passive vs. Active Mix Events

You will sometimes find mix events divided into subcategories, active and passive. The difference between the two highlights some of the inner mechanisms behind game audio mixing and perhaps game audio in general. Audio in games, generally speaking, is usually event-driven. That is to say that audio events, whether it’s playing an audio file or modifying a mix parameter, responds to something happening in the game, an event. In essence, most audio is triggered in response to an event in the game: shooting, walking into a trigger etc. An active mix event is one that is in direct response to something happening in the game, such as an enemy character spawning or a player walking into a trigger.

Passive mix events happen when the mix changes in response not from an event in the game but as a result of the mix structure itself, such as dialog ducking down the music by triggering a compressor on the music. The game engine has no awareness that the compressor on the music is being triggered.

This highlights another difficulty of mixing for games and interactive audio systems: self-awareness – or the lack thereof. Most games engines do not monitor their own audio outputs, either in terms of amplitude or spectral data.

Since the game is mixing the audio for the soundtrack, it is akin to trying to teach someone how to mix by giving them basic instructions and then turning off the speakers. This is indeed challenging, especially with the introduction of concepts such as audio-driven events. These are events in the game triggered

ADAPTIVE MIXING

259

by a change in the audio, such as leaves being swept up as the volume of the wind increases over a certain threshold. While audio-driven events remain relatively rare in games, we can look forward to a greater synergy between the game and soundtrack over the next few years in the industry.

2. The Unity Audio Mixer

Unity does provide us with the ability to add mixers to our projects. Multiple mixers per project, in fact, as unlike traditional linear production where we tend to work around a single mixer, game developers for non-linear media often rely on multiple mixers for larger projects. The decision to use one or multiple mixers should be considered in relation to the complexity of the game and mix, flexibility, gains vs. CPU overhead and much more.

To create a new mixer, navigate to the project folder window in Unity and right click in the assets window. Select Create->Audio Mixer from the contextual menu. An audio mixer window tab should appear under the project window; if it doesn’t, you can bring up the mixer window from the Window menu: Window->Audio->Audio Mixer.

When a mixer is created Unity will provide a master group, where all audio for this mixer will be routed, and it lets you add groups on an as-needed basis.

Groups allow us to run multiple audio sources into a single destination and are in essence submixes.

Additional mixers may be created by clicking the + button to the right of the mixers icon on the left side panel.

Figure 11.3

1. Adding Groups to the Unity Mixer

Groups can be added by clicking on the + button to the right of the groups label on the left panel. All subsequent subgroups will eventually run into the master group, but it is also possible to route the output of a group into other, cascading multiple subgroups. The Unity mixer defines two types of groups: child groups – which are routed into another – and sibling groups, that run in

260

ADAPTIVE MIXING

parallel to an existing group. You can change the routing hierarchy by dragging a group in the groups panel of the mixer window on top of the desired destination group or, when creating a new group, right clicking on an existing group and selecting either add child group or add sibling group. You can use the same contextual menu to rename, duplicate as well as delete groups.

Figure 11.4

The letters at the bottom of each group allow the developer to mute the group by clicking the M button, solo the group with the S button and bypass effects using the B button.

Figure 11.5

In the earlier snapshot, the ambiences group is a child of the SFX group. The audio output of the ambiences group will be routed to the SFX group, itself routed to the master group.

2. The Audio Group Inspector

When a group is selected in the hierarchy or directly by clicking on it in the mixer, the audio group inspector for that group will become active in the inspector window. Here you will find the different components that make

ADAPTIVE MIXING

261

up a mixer group. Whenever a group is created the following units are added automatically:

Inspector Header: here you will find the name of the group. By right-clicking anywhere in this window a contextual menu will appear with two options.

Copy all effects settings to all snapshots: this will copy all of the group’s settings on this group top all other snapshots in the mixer, allowing you to pass on to the group’s settings to all other snapshots.

Toggle CPU usage display: will turn on CPU performance metering for all effects present in the group.

Pitch Slider: this slider controls the pitch of all the audio routed through this group.

Attenuation Unit: every group can only have one attenuation unit, which acts as a gain stage control, ranging from −80dB, which is silence, to +20dB.

Each attenuation unit has a VU meter, which displays both the RMS

value of the signal as well as its peak hold value. The RMS value is displayed by the colored bar itself while the peak value is displayed by a gray line at the top of the range.

3. Working With Views and Colors in the Unity Mixer

Managing large mixing sessions is always a bit of a challenge once your session has reached a certain size and you are dealing with a large number of groups.

Visually, Unity provides us with two tools that can help us manage how we view and organize the information from the mixer window.

The first is the ability to color code our groups, which we can access by right clicking on the top of any group and selecting one of the available colors at our disposal.

Figure 11.6

This will add a colored strip at the top of each group right below the name and help visually break the monotony of the mixer window.

The other visual tool that Unity puts at our disposal is the ability to display only the relevant groups at any given time in our mix, hiding the ones we are

262

ADAPTIVE MIXING

not focused on, in order to minimize visual clutter. This is done with the views feature, located in the bottom left panel of the mixer window.

Creating Views in Unity

1. You can create a new view simply by clicking on the + button to the right of the word Views.

2. Right-clicking on the newly created view will allow you to rename, duplicate or delete a new group.

3. With a view selected, click on the eye icon to the left of each group name in the groups window. That group should now disappear.

4. Adding Efects to Groups in Unity

Working with effects in the mixer opens a world of possibilities. With effects we can better control our mix and use in real time some of the same types of plugins that we are used to working with in DAWs. Keep in mind that any effects added to the mixer will increase the load on the CPU and need to be monitored and carefully thought through. Any effect that could be rendered as an audio file prior to being imported in Unity should be. That being said, real time effects are a very powerful way to make our mix more dynamic, engaging and fun.

You may add effects to each group by clicking at the bottom of the group itself on the Add . . . button. The available effects are: (Note: some of these effects were described in more general terms in the sound design chapter)

• Duck volume: this really is a compressor optimized for side chain input; in order to control the audio levels of the group it is added to using the level of another one, such as a compressor on the sound effect group, listening to input from the dialog group for instance.

• Low pass: a low-pass filter with resonance control.

• High pass: a high-pass filter with resonance control.

• Echo: a simple delay line.

• Flange: time-based modulation effect.

• Distortion: a simple distortion effect.

• Normalize: normalization is a process that adds gain to a signal in order to raise its peak amplitude. Unity developers intended for the normalize effect to preprocess a signal prior to sending it to a compressor.

• ParamEQ: an equalizer with independent frequency and bandwidth control and a visual representation of the curve applied to the sound.

• Pitch shifter: Unlike changing the pitch value of an audio source, which changes the pitch and the duration of the audio accordingly, the pitch shifter plugin provides independent time and pitch control. For instance, raising the pitch by an octave will not make the sound twice as short as it otherwise would be. This will prevent the ‘chipmunk’

ADAPTIVE MIXING

263

effect but also does require more processing power than a simple pitch shift. Use sparingly.

• Chorus: another time-based modulation effect, often used for thicken-ing sounds.

• Compressor: a full-featured dynamic range processor.

• SFX reverb: a full-featured procedural reverb, which we will look at in more detail shortly.

• Low pass simple: a low-pass filter without resonance, cheaper computationally than the low pass.

• High pass simple: a high-pass filter without resonance, cheaper computationally than the high pass.

5. Inserts vs. Efect Loops

Under the same menu you will also find send and receive, although these are not effects per se but rather a way to send a signal to another group. Signals tend to travel from top to bottom in a mixer, going through the various components of the group it is routed to. There will be times, however, where you will wish to send the signal to another place in the mixer or maybe send a copy of the signal to another group. In traditional mixing, this is done using sends and busses. A send is a circuit that lets the signal flow out of the current group it is routed to, and it uses busses to travel to various destinations in the mixer. Unity does not rely on the same terminology, shying away from using the word bus. Rather, Unity uses send and receive.

Most audio effects in a mixer can be added in one of two ways, as an insert or as an effect loop. While there are instances where either solution could be appropriate, there usually are good reasons to go with one or another method.

Adding an effect as an insert simply means inserting the effect on the group that the audio is routed to by simply pressing the Add . . . button at the bottom of the mixer and selecting an effect. The effect is now inserted into the signal path. This method is appropriate for effects we only wish to apply to one particular group. This is usually the case for equalization and compression, although there are instances where you might wish to use the effect loop method for compression, something known as parallel compression or ‘New York’ compression. Working with inserts is fine for these types of situations but starts to become more difficult to manage when the same effect needs to be applied to more than one group, such as reverberation. If we wish to apply reverberation to multiple groups, then we need to insert multiple copies of the plugin on each group, which is costly computationally and inefficient in terms of workflow. Inefficient computationally because inserting multiple versions of the same plugin, especially reverberation, is going to increase the computational load on the CPU but also inefficient in terms of workflow because any change made to one reverb will also need to be applied to all other instances if we wish for the effect to be consistent across all channels. When working with a complex mix, a simple task may turn into a much more difficult and therefore time-consuming one. A much better solution in this case is to set up a

264

ADAPTIVE MIXING

separate group just for reverberation, insert one instance of a reverb plugin on it, then route all the audio requiring reverberation to be applied to be routed directly to that group, creating an effect loop.

Figure 11.7

6. Setting Up an Efect Loop for Reverberation in

Unity Using Send and Receive

Follow these simple steps to set up an effect loop you can use to route any group to a reverb plugin:

1. Create a new, dedicated group for reverberation and name it appropriately.

2. By clicking on Add . . . at the bottom of your new group, select Receive.

3. By clicking on Add . . . at the bottom of your new group, select SFX

Reverb. Note: the signal in a group flows from top to bottom; it is important that the receive be added prior to the SFX Reverb plug in or if added after the fact, be moved up before it.

4. Select another group on the mixer that you wish to add reverb to, and using the Add . . . button select Send. In the inspector for that group you should now see the send component. Click the popup menu next to the word Receive and select the group you added the reverb to. The send level slider allows you to adjust how much signal you are sending to the group, therefore how much reverb you will hear on the sounds from that group. Feel free to experiment!

Some effects, such as reverberation, will allow the user to have independent control over the dry, unprocessed signal and wet signals. This raises the CPU

usage a bit but does allow us to have much more control over our mix. To turn on that feature right-click on the SFX Reverb label in the reverb’s unit in the inspector and select Allow Wet Mixing.

ADAPTIVE MIXING

265

Figure 11.8

Note: you may not use the Send/Receive technique on a group that is a child of another one, as that may result in a feedback loop. In other words, the output the of group on which reverb was applied cannot be routed to the group you are sending from. The receive group needs to be routed to the master group or another group that runs in parallel to the group we are sending from.

This technique is highly recommended for reverberation, echoes/delays and any processor that you wish to apply in the same way to multiple groups.

Remember, by creating a send, you are sending a copy of the channel to another group; the original signal still flows through its original group. This gives the developer individual control over the dry signal (the original signal) and the wet signal (the copy going to the group reverb).

Note on Adjusting Levels During Gameplay

When Unity is in play mode, any change made to the game or any of its components will be lost as soon as you hit stop, and, as was pointed out earlier, you will need to make a note, mental or otherwise, if you wish to implement Figure 11.9

266

ADAPTIVE MIXING

these changes after the fact. The Unity mixer is the exception. When in play mode, if you bring the mixer window in focus you will notice a button appearing labeled Edit in Play Mode. When pressed, changes you make to the mixer while playing will be remembered, allowing you to make adjustments to the mix as you play the game in real time.

7. Ducking in Unity

Ducking is especially useful when it comes to automating certain aspects of the mix. Ducking occurs when a compressor placed on a group, say group A, listens for a signal from another group, group B. When group B is active, the compressor will duck a volume on group A, making the signal from group A easier to hear. A common example of this is in radio, where the DJ’s voice will turn the music down when it comes on. The most common application of ducking in games is for dialog, which will often duck the volume on the music and sound effect groups. The control signal in the case of the DJ’s voice, also known as the key. Setting up a ducking compressor is very much like setting up an effect loop.

Usually this effect is achieved with a compressor equipped with a key signal input; Unity provides us with a dedicated tool for this, the duck volume plugin, which is in fact a regular compressor with a key input built in.

Setting Up a Ducking Compressor in Unity

1. On the group that you wish to duck the volume of, place a duck volume plugin by clicking on Add . . . at the bottom of the group and selecting Duck Volume.

2. On the group you wish to use as your key, click Add . . . and select Send.

3. In the inspector for the group you just added the send plug in to, locate the send component, and click the popup menu next to the receive option and select the group you added the duck plug in to in step 1.

4. Adjust the send level by raising the slider closer to 0dB.

5. While the key signal is playing, adjust the duck volume plug in in order to obtain the desired results by adjusting the threshold and ratio.

You will likely need to adjust both the send coming out of the dialog group as well as the settings on the duck volume processor a few times before settling on the proper settings; use your ears, as always, and try your mix at a few places throughout the game.

3. Snapshots, Automation and Game States

Once you have planned and implemented the routing of your mix, the next thing to consider is to make it adapt to the various situations that will arise and require changes or adjustments as the game develops. There comes

ADAPTIVE MIXING

267

the idea of game states. Game states is a term borrowed from AI, where finite state machines systems are used to implement AI logic in non-player characters.

In video games, game states have come to be used to describe a relatively large change in the game. An example in a FPS might be:

• Ambient.

• Exploratory mode.

• Battle mode 1.

• Battle mode 2.

• Boss battle.

• Death.

Some game engines or third-party audio middleware will explicitly implement game states, while others, such as Unity, depend on whether the programmer implemented an explicit game mechanic. Either way, game states are very useful for mixing as they can give us a sense of the various situations we are likely to encounter and can prepare for. Battle mode might require the music to come up and ambiences to come down in volume for instance, and the opposite might be true for exploration states, where the ambience is more important and the music less intense anyway. In order to implement these changes, we can rely on snapshots.

1. Working With Snapshots

Snapshots is a term borrowed from music production, where snapshot automation was developed in order to automate the changes needed during a mix on large format mixing consoles, which became too complex to work in real time. A snapshot of the mixer and all of its setting could be stored at various points in the song and recalled in real time during mixdown. This technique, borrowed from a very linear world (most music was still recorded to tape when this technology came of age), is turning out to be quite useful in video games. By using snapshots, we can adjust our mix to match the developments of the game.

Working with snapshots in Unity is a simple process. To create a snapshot, follow these steps:

1. Adjust the mixer to the desired setting.

2. Once dialed in, in the left panel of the mixer window, press the + sign to the right of the word Snapshots.

3. Name the snapshot, then press enter; you’re done!

Recalling a snapshot can be achieved either by clicking on the snapshot’s name in the mixer window – which is really only a valid method when mixing – or via script, as we shall see shortly.

268

ADAPTIVE MIXING

2. Recalling Snapshots via Scripting

Snapshots can be easily recalled with scripting, using the TransitionTo() method, which will interpolate a transition to the new snapshot over the time specified by the user. The following example will demonstrate. This simple script will interpolate a transition between three snapshots labelled ambient, combat and victory in response to the user pressing the keys 1, 2 and 3 respectively.

First, create a new mixer or use an existing one, and create three snapshots, one for each of the three we outlined earlier: ambient, combat and victory.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

using UnityEngine.Audio;

public class Automation : MonoBehaviour

{

public AudioMixerSnapshot ambient;

public AudioMixerSnapshot battle;

public AudioMixerSnapshot victory;

public foat transTime = 1f;

void

Update()

{

if

(Input.GetKeyDown(KeyCode.Alpha1))

{

ambient.TransitionTo(transTime);

}

if

(Input.GetKeyDown(KeyCode.Alpha2))

{

battle.TransitionTo(transTime);

}

if

(Input.GetKeyDown(KeyCode.Alpha3))

{

victory.TransitionTo(transTime);

}

}

}

You’ll notice right away that we added a new namespace using UnityEngine.

Audio; which we need in order to use AudioMixerSnapshot. Next, after the class declaration we declare three new variables of type AudioMixerSnapshot, and by making them public they will show up in the inspector in the slot for the script. Prior to running this script, we need to assign an actual audio snapshot to each of the variables we just declared by clicking on the slot next to them in the inspector and selecting one of the three snapshots we created earlier in this example as demonstrated in the following illustration.

ADAPTIVE MIXING

269

The transition time has been set to one second by default but may be changed by the user, in this case, simply by changing the value in the slot labelled transTime.

To see the example at work, make sure the mixer is showing upon entering play mode, and press the 1, 2 and 3 keys; you should see the sliders for the three subgroups move over the course of a second. Of course, in most cases the changes in the mix would not come from keystrokes by the user (although they might in some cases) but rather would be pushed by the game engine. It would be very easy to change this script to respond to another input, such as entering a trigger, an object or player getting spawned etc.

Figure 11.10

Note: transitions between snapshots are done using linear transitions by default. That can be changed by right clicking on any unit in the audio group inspector and selecting one of the other options.

Note on Edit in Playmode: this option will only appear while the editor is in play mode. When the game is running, the mixer is not editable and is controlled by the current snapshot or default one if none have been created. By enabling Edit in Playmode, the current snapshot is overridden and the game developer can now make changes and adjustments to the current snapshot.

270

ADAPTIVE MIXING

Figure 11.11

3. Editing Mixer and Plugin Parameters via Scripting

Snapshots are a great way to manage a mix, but there might be time when you will need to control a single parameter and adjust it individually. In this case snapshots may not be a very good option. Rather, Unity allows you to control a single parameter from the mixer via the SetFloat() method, which takes only two arguments: a string, the name of the parameter to change and a float, which is the value for the parameter. However, before using SetFloat(), the parameter you wish to control individually has to be exposed. An exposed parameter will respond to values passed onto it by SetFloat() but will be removed from the snapshot, although as we shall see shortly, it can be returned to the snapshot if needed.

4. Exposing a Parameter: Controlling a Volume Slider

1. In order to expose a parameter, open the mixer you wish to expose a parameter of.

2. Select the group you wish to control the volume slider of.

3. In the inspector, locate the attenuation component and right click on the word Volume.

4. Select Expose ‘Volume of (name of group)’ to script. You will get con-firmation that the parameter is now exposed by the arrow pointing right that will appear next to the name of the exposed parameter.

ADAPTIVE MIXING

271

5. At the top right of the mixer window, you will notice a textbox that should now say Exposed Parameters (1). Clicking once on it will reveal the newly exposed parameter. Double click on the parameter to rename it.

Once the parameter has been exposed, we can now control it with a script, using SetFloat(). This simple script will change the value of a slider when the user presses the 8 or 9 keys on the keyboard.

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

using UnityEngine.Audio;

public class ExposeParameter : MonoBehaviour

{

public AudioMixer mainMixer;

void Update()

{

if

(Input.GetKeyDown(KeyCode.Alpha8))

{

mainMixer.SetFloat(“BoomVolume”,

−10f

);

}

if

(Input.GetKeyDown(KeyCode.Alpha9))

{

mainMixer.SetFloat(“BoomVolume”, 0f );

}

}

}

A very simple script indeed. Note that the mixer, which contains the exposed parameter you wish to change has to be explicitly named, which is why we are including a reference to it at the top of the script by creating a public AudioMixer variable. Since it is public, this variable will show up as a slot in the script in the inspector and has to be assigned by the developer by either dragging the proper mixer onto the slot itself, or by clicking the little disc next to the words Main Mixer in the inspector.

4. Good Practices

One of the most common questions that comes up, especially with beginners, is how to figure out what output levels should we shoot for in our mix. How loud should the dialog be? How much dynamic range is too much and will make the user reach for the volume slider in order to compensate, or how much is not enough and will make the mix fatiguing to listen to over time?

272

ADAPTIVE MIXING

Often, however, the person asking the question is somewhat disappointed by the answer. We do have some guidelines, of course, and most of these actually come from broadcast, which has been much more concerned than gaming when it comes to standardizing mixes, however even the best and most accurate guidelines are just that, guidelines. Most matters in a mix are context dependent and need to be adjusted in reaction to other elements.

The first issue has to do with loudness. Loudness is actually a rather difficult thing to quantify, as it is a subjective measurement, having to deal with the way humans perceive and relate to sound at various intensities. Our perception of loudness varies across different frequency ranges, with humans being most sensitive to frequencies toward the center of our hearing range and dropping off toward the edges. This was first and best outlined by the equal loudness contour graph, also sometimes known as the Fletcher-Munson curves.

Figure 11.12

An in-depth study of the equal-loudness contour is beyond the scope of this book, but in addition to showing us that our perception of loudness falls off with the edges of the frequency range (very low and high frequencies are harder to hear than mid frequencies), it also tells us that that perception is also, in itself, dependent on the amplitude of the signal and that as a signal gets louder, it becomes easier to perceive the edges of our frequency ranges, low and high frequencies, in relation to the mid frequencies.

A lot of mixers are equipped with peak meters, which measure the instan-taneous output value of each sample going through that particular group or channel strip. While peak meters are useful when it comes to making sure we

ADAPTIVE MIXING

273

are not clipping our output, they do not relate to loudness very well and are not an accurate measurement of it. A better solution is to use the relatively new standard LUFS unit, loudness unit full scale, which aims at measuring actual loudness in the digital audio domain by breaking down the frequency ranges in which energy is found in a sound and weighting them against the Fletcher-Munson curves. Another commonly found unit is LKFS, loudness K-weighted full scale, a term that you will find in the ITU BS.1770 specifications and the ATSC A/85 standards. Both LUFS and LKFS measure loudness and are often used interchangeably as units. The European Broadcast Union (EBU) tends to favor LUFS over LKFS, but they are otherwise very similar. Both of these units are absolute and, depending on the format for which you mix, a target level of

−23LUFS or −24LKFS is often the target level for broadcast.

NOTE: 1 LUFS or LKFS unit = 1dB

These standards were designed for broadcasts, not gaming, but they are proving useful to us. Doing a bit of research in this area will at the very least get you to a good starting place – a place that you may decide to stick to or not in your mix, depending on the game, mix, situation.

Note: while there are plugins out there that will allow you to monitor levels in Unity in LUFS, they need to downloaded separately. The reader is encouraged to.

Mix Levels

So how do we tackle the issues of levels and dynamic range? As you may have guessed, by planning.

1. Premix.

A good mix starts with a plan. A plan means routing and also preparing assets and target levels. Of course, don’t forget the basics:

• Make sure that all the sounds that belong to same categories or that are to be triggered interchangeably are exported at the same level.

This will prevent you from having to make small adjustments to compensate for level discrepancies that will eat up your time and resources.

• Set up starting levels for various scenes in your mix. You may start by using broadcasting standards as a guide if you are unsure of where to begin. Most broadcasters in the US will look for an average level at −24dB LKFS with a tolerance of + or – 2dB. If you do not have a LUFS or LKFS meter, try placing your dialog at −23 or

−24dB RMS for starter and make sure that your levels stay consistent across. If there is dialog, it can be a great way to anchor your mix around and as a reference for other sounds.

274

ADAPTIVE MIXING

• Don’t forget that the levels you set for your premix are just that, a premix. Everything in a mix is dependent on context and will need to be adjusted based on the events in the game.

2. Rest your ears.

Over time and as fatigue sets in, your ears are going to be less and less accurate. Take frequent breaks; this will not only make sure your ears stay fresh but also prevent mistakes that may occur from mixing with tired ears, such as pushing levels too hot or making the mix a bit harsh overall.

3. Mix at average loudness levels, but check the extremes.

While mixing, monitor the mix at average medium levels, but do occasionally check it at soft and louder levels. When doing so, you will listen for different things, based on the Fletcher-Munson curves of loudness. When listening to your mix at low volume, you should notice that relative to the rest of the mix, high and low frequencies should appear softer than they were at average listening levels, but can you still hear them? Are all the important components of your mix still audible, or do you need to adjust them further?

When listening to your mix loud, the opposite effect will occur.

Relative to the rest of the mix the lows and high frequencies should now appear louder. What we must watch out for in this case is if the bottom end becomes overpowering, or does the increased perception in high frequencies make the mix harsh to listen to over time?

4. Headphones are a great way to check stereo and 3D spatial imaging.

While mixing on headphones is usually not recommended, they are a very useful tool when it comes to checking for stereo placement and 3D audio source location. Are sounds panned, in 2D or 3D where you mean for them to? Speakers, even in very well-treated rooms, are sometimes a little harder to read in that regard than headphones.

More specific to gaming is the fact that a lot of your audience will play the game on headphones, possibly even earbuds, so do also check for the overall cohesion of your mix when checking the spatial imaging of your mix on headphones.

5. Check your mix on multiple systems.

Even if you’ve checked your mix on headphones, and assuming that you know your speakers very well, you should check your mix on several other playback systems. Of course, the mix should sound good on your studio monitors, but remember than most people will experience your mix on much less celebrated sound systems. Check your mix on built in computer speakers or TV speakers, try if you can a second pair of speakers. Of course, your mix will sound quite different on different systems, but your primary concern should not be the differences across speakers but whether or not the mix still holds up on other systems.

ADAPTIVE MIXING

275

Conclusion

Mixing is as much art as it is science. Learning all the tricks available in Unity or any other package for that matter is important – but is only useful if one is able to apply them in context, to serve the story and the game. Try, as much as possible, to listen to other games, picking apart their mixes, noting elements you like about them and those you like less. As you mix, always try to listen to your work on different systems, speakers, on headphones and make adjustments as you go along.

Mixing is skill learned over time through experience, but keeping in mind some of the guidelines outlined in this chapter should give you some good places to start. And as always and as with any other aspect of the game audio, the mix should both inform and entertain.

12 AUDIO DATA REDUCTION

Learning Objectives

In this chapter we focus on the art and science of audio data reduction and optimization or how to make audio fles smaller in terms of their RAM footprint while retaining satisfactory audio quality. In order to achieve the best results, it is important to understand the various strategies used in data reduction, as well as how diferent types of audio materials respond to these techniques. As always, technical knowledge must be combined with frst-hand experience and experimentation.

1. Digital Audio: A Quick Review

Audio is digitized by taking an analog, continuous signal such as a sound picked up by a microphone and by measuring these continuous signals at regular time intervals, known as the sampling rate. The Nyquist theorem says the sampling rate must be equal to twice the highest frequency we wish to accurately capture. For games, the sampling rate is often 44.1Khz or 48Khz.

1. Pulse Code Modulation

At each sample, a voltage value is converted into a numerical one within a given range of available numbers. The greater the range, the more accurate the process. That range is given to us by the bit depth or the number of bits that the session is running at. At the time of this writing, 24bits is the standard in music production. Increasing the bit depth and having more values to choose from, makes our measurement and recreation of the waveform more faithful to the original. At 16 bits, each sample has an available range of 2 to the 16th power, or 65, 536 values. 16 bits represented a huge improvement from the 256 values available in the early days of gaming, using 8bit systems. At 24bits the accuracy is further improved by giving us a range of 16, 777, 215 values for each sample to fit in. As we saw in

AUDIO DATA REDUCTION

277

the previous chapter, there is a relationship between the bit depth and the dynamic range, whereas each bit gives us approximately 6dB of dynamic range.

At lower bit depth, therefore with smaller numerical ranges to work with, the system will start to make significant enough mistakes in trying to reproduce the waveform. These mistakes will be heard in the signal as noise and are referred to as quantization errors. Noise stemming from quantization errors, especially at lower bit depth, such as 8bit, is very different from analog tape hiss. Unlike hiss, which is a relatively constant signal and therefore relatively easy for the listener to ignore, quantization errors tend to ‘stick’ to the signal, following the dynamic range of the waveform, being more obvious in the softer portions and less so in the louder ones. In other words, on a signal with a fair amount of dynamic range, quantization errors will add constantly evolving digital noise, making it impossible to ignore and very distracting. For that reason, in the early days of video game, working with 8bit audio, the audio was often normalized and compressed to reduce dynamic range and mask the quantization errors as best as possible. Thankfully, however, the days of 8bit audio are long behind us.

The process of digital audio encoding is a complex one, but the importance of the sample rate and bit depth become quite obvious when the process is beginning to be understood. Once a value for the current sample at the signal at the input has been identified, usually after a sample and hold process, the value is encoded as a binary signal by modulating the value of a pulse wave, a down state representing a zero and an up state a value of one. This process is referred to as pulse code modulation, and you will find the term PCM used quite liberally in the literature to describe audio files encoded in a similar manner, such as WAV and AIF files but also many others.

2. File Size Calculation

When it comes to uncompressed audio, the file size of a recording depends on the following factors:

• Length,

• Number of channels.

• Bit depth.

• Sample rate.

In order to calculate the overall size of a file, the following simple formula can be used.

Note: the final result needs to be converted from individual bits to megabytes.

File Size = Sample Rate * Bit Depth * Length * Channel Number

278

AUDIO DATA REDUCTION

In order to convert from bits to Megabytes the result must be divided as follow:

Final Result in bits / 8 = Result in bytes

Final Result in Bytes / 1024 = Result in kilobytes Final Result in Kilobytes / 1024 = Result in Megabytes For instance, a stereo file one minute in length, at 16bits 44.1Khz sample rate will be: 10.584 megabytes.

The same file at 24bits will have a file size of 15,876MB

Reducing the file size of audio recordings is trickier than it may first appear.

Anyone who’s ever tried to zip an audio file before sharing it realized that the gains obtained from the process, if any, are abysmal. That’s because audio data does not respond well to traditional, generic compression schemes such as zip and requires a specific approach.

The underlying principle behind audio data reduction is a simple one: trying to recreate the original signal while using fewer bits and at the same time retaining satisfactory audio quality.

File size reduction is expressed in terms of a few key terms. One such is the compression ratio, which expresses the ratio between the original file size and the file size upon reduction.

Another term you are likely to encounter is bit rate; not to be confused with the bit depth of a recording or digital audio system, the bit rate expresses the number of bits (or kilobytes, megabytes) per seconds needed to construct the signal.

2. Data Reduction Strategies

Audio data may be reduced by either one of these two processes: removing either redundant data or irrelevant data. In practical terms, there are four ways to go about these techniques.

• Reducing the sample rate.

• Reducing the bit depth.

• Detecting and reducing redundancy.

• Perceptual coding – removal of ‘irrelevant’ information.

Additionally, data reduction schemes fit in one of two categories: lossless and lossy.

Lossless schemes generally focus on redundancies, allowing them to re-arrange the data without actually throwing any away that cannot be gotten back upon decompression. In other words, once the file has been decompressed it is an exact duplicate of the original, uncompressed file. Zip files are a common example of lossless data reduction schemes. When it comes to audio, there again lossless formats must be designed with the needs and

AUDIO DATA REDUCTION

279

requirements of audio data in place, and a generic lossless codec such as zip files will not deliver any significant gains. Apple lossless is an example of a redundancy-based codec.

There are several ways to think of redundancy-based strategies in very simple terms. For instance, let’s take the hypothetical term: rrrghh555500000001

It could be encoded as such:

r3gh254071

reducing the number of characters needed to express that same quantity from 18 to only ten.

Techniques that rely on redundancy are sometimes called source coding techniques.

The average gains from data reduction in audio are relatively small compared to other techniques, about a 2:1 ratio, but they remain significant.

1. Speech vs. Generic Audio

Speech is something we are naturally very sensitive to, more so than any other types of sounds. As such great care must be given to dialog, which must always be heard clearly. Speech does present some advantages when it comes to data reduction. Generic audio, such as sound effects or music, generally requires a higher bitrate than speech. This is because, although each file is to be considered on a case per case basis, sound effects and certainly music tend to require a higher sample rate than speech for the quality to be maintained; dynamic ranges tend to be greater than speech, and the frequency content tends to be more complex also. That being said, nothing should get in the way of intelligibility.

2. Bit Rates

As mentioned previously, the bit rate refers to the amount of data, usually in kilobytes per seconds, that is needed in order to render the file. It is also a measure of quality; the higher the bit rate, the better the quality. The bit rate alone, however, is not everything when it comes to the quality of an audio file.

At the same bit rate different formats will perform differently. It is also worth noting that there are in fact two types of bit rates: constant bit rates (CBR) and variable bit rates (VBR).

As the name implies, constant bit rate keeps the data rate steady throughout the life of the audio file. Audio files are complex quantities, however, and some parts of an audio file may be easier to encode than other, such as silence as opposed to an orchestra hit for instance, but CBR files do not account for these differences in the way the available data is distributed.

280

AUDIO DATA REDUCTION

On the other hand, with a VBR file the data rate may be adjusted relative to a target rate or range, and bits can be dynamically allocated on an as-needed basis. The result is a more accurate encoding and rendering of the audio, and the process yields better results while maintaining a similar file size. One of the few drawbacks of VBR is compatibility with older devices.

The most common bit rates are 256kps, 192kps and 128kps. Artifacts will start to be heard clearly at 128kps, and it is not recommended to go below this figure of you can at all avoid it, regardless of the format. A little experimentation with various kinds of material is recommended so that the user can form their own opinion as to the best option for their needs.

3. Perceptual Coding

Perceptual coding is a family of techniques that rely on psycho-acoustics and human perception to remove parts of the signal that are not critical to the sound, making it easier to re-encode the signal with fewer bits afterwards. These technologies center around the acoustics phenomenon known as masking.

Masking can occur both in the time and the frequency domain and refers to a situation where if two signals are close together in frequency and/or time, one may prevent the other from being heard, and therefore the masked signal can be removed without significant loss of quality. Overall, masking based techniques obtain better results in the frequency domain than in the time domain and usually rely on a Fourier transform to analyze the audio, identify the bits of data that may be removed when compared to a human perceptual model and for the re-synthesis process. Artifacts relating to the Fourier transforms may become apparent at lower bit rates, such as loss of transients, high frequency and energy.

The Trade-Of

There is a bit of a trade-off when it comes to game audio and data reduction.

Reducing the amount of data of a given audio file will save us a lot of memory – or RAM space – however, playing back compressed audio data does put an increased demand on the system’s CPU, which may result in CPU

peaks if a lot of audio files are played at once. On the other hand, playing back uncompressed PCM data is an easier task on the CPU, but it does in turn require more storage space and available RAM.

4. Common File Formats

The following is a discussion of some of the formats you are most likely to encounter but is certainly not an exhaustive list.

a. MP3

The MP3 format, also known as MPEG-1 Audio Layer III, is perhaps the most famous of the perceptual-based compressed audio formats and one of

AUDIO DATA REDUCTION

281

the earliest as well. It remains one of the most commonly used standards for digital audio distribution and streaming to this day. MP3 is a lossy format, and depending on the type of material and the chosen bit rate, the artifacts of compression will become more or less obvious. At lower bit rates, 128Kps and lower, the artifacts will include smearing of transients and of stereo image as well as a dullness in highs and lows, the extremes of the frequency range.

The format supports meta data and may include the artist’s name and track information. The MP3 format, like all compressed formats, doesn’t necessarily perform evenly across different types of materials, from spoken word to a symphonic recording or a heavy metal track. Generally speaking, complex material, such as distorted electric guitars, is more difficult to encode accurately at lower bit rates, and sounds may end up sounding noisy.

Pros: compatible with a wide range of devices and streaming formats.

Cons: shows sign of aging; other formats have appeared since that perform better in terms of quality.

b. Advanced Audio Coding

AAC was developed as a successor to the MP3 format and as such tends to deliver better results than MP3 at similar bit rates. Like its predecessor, it is a lossy format, also centered on perceptual coding that supports up to 48

audio channels at up to 96Khz sample rate and 16 Low Frequency Effects channels (up to 120Hz only) in a single stream. The format is supported by a number of streaming and gaming platforms at the time of this writing such as YouTube, iPhones, Nintendo DSi, Nintendo 3DS and PlayStation 3, to name a few.

Pros: better quality than MP3 at similar bit rates, wide support, and high sample rates are supported.

Cons: although AAC has gained wide acceptance, it is not as widely supported as MP3, and some target platforms may not accept AAC.

c. Ogg Vorbis

Unlike MP3, Ogg Vorbis is open source and patent free and was developed as an alternative and for that reason had a lot of early adopters in the gaming world. It is a lossy format based on perceptual coding and tends to deliver superior results to MP3 at identical bit rates. Ogg Vorbis compression is supported within Unity, and it is recommended over MP3.

Pros: better quality than MP3 files at similar bit rates, open source and patent free, wide support in gaming.

Cons: very few, support on some devices and streaming format may still be an issue, however.

282

AUDIO DATA REDUCTION

d. AC-3 Dolby Digital

This format was developed by Dolby Labs and is widely used in home theatre and film. Its ability to work with multichannel formats such as 5.1 and its robust audio quality has made it a standard for broadcast, DVDs and Blu-Rays.

Dolby Digital Live is a variant of the format developed for real time encoding in gaming applications, supporting 6 channels at 16bits, 48Khz with up to 640kbits/second data rate.

e. Adaptive Diferential Pulse Code Modulation

ADPCM is a lossy format providing up to 4:1 compression. ADPCM allows the sound designer some control over the process of data reduction in order to get the best results but not as much as other formats, such as Ogg Vorbis.

Unity does support ADPCM.

3. Data Reduction Good Practices

When it comes to obtaining the best results, data reduction may seem as much of an art as a science, and some experimentation is usually a great way to get a sense for how various material will fare after data reduction. Not all material compresses well; some will do well at high compression ratios, while others will simply demand a high bit rate. There are some guidelines to watch out for that will ensure that no matter what you get the best results possible.

1. Not all material compresses well: watch out for material with a lot of transients or with wide frequency range as they require a lot of bits, comparatively to simpler signals, in order to sound convincing.

2. Always work at the highest quality possible until the very last minute.

In other words, keep your material at the highest resolution possible such as 48Khz or 96Khz and 24bits, until the data reduction process.

Never ever perform data reduction on files that have already gone through a similar process, even if the file has been resaved as .AIF or

.WAV uncompressed format. Saving an MP3 as a WAV file will make the file significantly larger, but it will not improve the audio quality.

3. Denoise prior to data reduction process. Ideally you will work with clean audio files, although in the real world, we all know that it isn’t always the case. Clean audio will always sound better after data reduction than noisy audio. If you are dealing with noisy audio, use a denoiser plug in in the signal first.

4. Pre-Processing. Some material will actually require some preprocessing in order to get the best result. Some of the pre-processes may include: a. Audio Clean up: de-noising is a given; by reducing the level of noise in your signal, you will end up with much cleaner audio once compressed. But the process may also include equalization to fix any issues

AUDIO DATA REDUCTION

283

in the tonal balance of the file or additional broadband noise reduction techniques in order to remove unwanted elements or distortion.

b. High frequency emphasis: it is not uncommon for files encoded at lower bit rates to result in somewhat dull output, sounding almost low-pass filtered. If this happens it may be a good idea to preprocess with an equalizer and boost the high frequency content gently, even if the file sounds a bit harsh initially. Once converted back, the high frequency boost may help compensate for the loss of high frequency content.

c. Reduce dynamic range: in the days of 8bit audio, thankfully gone, quantization noise was one of the main issues when dealing with going from 16 to 8bits. Since quantization noise has a tendency to be more obvious in the softer portion of an audio file, all dynamic range was severely limited in order to make sure the signal was always close to maximum output level or 0dB Full Scale. Although it is very unlikely you will be dealing with 8bit audio, this approach is still recommended for any audio with a bit rate lower than 16.

Reducing the dynamic range of an audio file can be achieved via compression or limiting. A good mastering limiter or audio maximizer is ideal. Do try to preserve transients.

d. Variable bit rate: when dealing with difficult to encode material, such as transient rich audio or complex spectrums, use VBR encoding whenever possible. You may want to experiment with several settings in order to obtain the best possible results.

e. Try whenever possible to design certain sounds with the data reduction process in mind if you are dealing with a stringent platform or data requirements. Be strategic: perhaps your ambiences and room tones can have little high frequency content information, making them easier to accurately reproduce at low sample and bit rates and save transient and high frequency rich sounds for more important sounds, such as those that provide the player with important information and that may need to be accurately reproduced in 3D space.

4. Data Reduction Options in Unity

1. File Options

The options for data reduction in Unity are found in the inspector when an audio file is selected as shown in the following figure. Note: Unity’s documentation can be a little light with regard to some of the audio features of the engine.

Force To Mono: sums multichannel audio to a mono source.

Normalize: when multiple channels are downmixed the resulting audio will often sound softer than the pre mixdown file. Checking this box will perform a peak normalization pass on the audio resulting in increased headroom.

284

AUDIO DATA REDUCTION

Figure 12.1

Load in background: this option allows for the audio to be loaded on a separate thread, leaving the main thread and process unblocked. This is meant to ensure the main thread will run unimpeded and will not stall. When this option is checked any play message will be deferred until the clip is fully loaded.

Ambisonic: check to flag the file is an ambisonic audio file. Unity does require the user to download a third-party plugin in order to render ambisonic files, but the format is supported.

2. Load Type

This section is used to determine how each audio asset will be loaded and running from runtime. There are three options available to us: decompress on load, compressed in memory and streaming.

AUDIO DATA REDUCTION

285

Decompress on load: with this option selected, compressed audio will be decompressed as soon as it is loaded and expanded back to its uncompressed size. By doing so you avoid the CPU overhead associated with playing back compressed audio files and improve performance, although your will end up with much larger audio assets. An Ogg Vorbis file can be up to ten times larger once decompressed, and for an ADPCM file that’s about 3.5 times, so you will want to make sure that you have the appropriate resources and RAM to deal with the uncompressed audio. Failing to do so will incur performance or audio drops or both. The Unity manual recommends checking this option only for smaller files.

Compressed in memory: with this option selected the audio is loaded in memory, compressed and only decompressed during playback. While this option will save memory, it does incur a slight increase in CPU

activity. The Unity manual recommends this option only for longer files that would require large amounts of memory to play uncompressed. Decompression occurs on the Mixer thread, which can be monitored in the ‘DSP CPU’ pane of the audio section of the profiler window.

Buffered: this option uses very little RAM by buffering audio on the fly as needed in order to maintain consistent playback. The audio is decompressed on a separate thread, in the ‘Streaming CPU’ pane of the audio section of the profiler window. The main issue with streaming is to be able to achieve the desired data rate for uninterrupted playback, which is why it is recommended to only stream large audio files such as a music soundtrack and to limit the number of files streamed, given time based on the expected transfer rate on the medium the game is authored for.

Preload audio data: when checked, the audio clip will be preloaded when the scene loads, which is standard Unity behavior. By default, all audio clips will therefore have finished loading once the file is playing. If unchecked the audio will be loaded at the first . Play() or . PlayOneShot() message sent to an audio source. Note: pre-loading and unloading audio data can also be done via script using AudioSource.

LoadAudioData(); and AudioSource.UnloadAudioData(); .

3. Compression Formats Options

Here we decide on the desired format for our audio. The options available here may vary based on the build target and installed SDKs.

PCM

ADPCM

Ogg Vorbis

MP3

286

AUDIO DATA REDUCTION

Quality: this slider determines the amount of compression applied to MP3 and Ogg Vorbis formats. The inspector will display the before and after compression once the user has adjusted the slider and pressed the Apply button at the bottom right of the inspector screen.

Sample Rate Setting

The main thing to keep in mind when dealing with sample rate issues when it comes to data reduction is of course the frequency content of the sample to be compressed. Since the sample rate / 2 = frequency range of the recording, any sound with little to no high frequency content is a good candidate for sample rate optimization. Low drones, ambiences and room tones are good candidates for sample rate reduction since they contain little high frequency information.

These are the options for addressing the sample rate aspect of data reduction in Unity.

Preserve sample rate: the sample rate is unaffected and no change is applied to the original recording.

Optimize sample rate: unity will automatically adjust the sample rate to match the highest frequency detected in the recording.

Override sample rate: this option allows the user to select the desired sample rate via a pop-up menu.

Conclusion

Audio data reduction is a complex topic but one that can be tackled more easily if we know what to pay attention to. The choice of an audio format and the amount of compression to use depends on many factors:

• Target platform: which formats are available on the platform.

• Memory requirements: how much data reduction is needed.

• CPU load: playing back compressed audio adds overhead to the CPU

load.

• The complexity of the audio material itself: based on the type of material you need to reduce the size of, what are the best options?

As always, use your ears. Do keep in mind that the side effects of compressed audio associated with listening fatigue will take a moment to set in. Consider how the overall soundtrack feels after playing the game for a time, and make adjustments as needed.

INDEX

Note: page numbers in italic indicate a figure and page numbers in bold indicate a table on the corresponding page.

2D audio sources 51–52, 62, 68

ambient lights 33

2D levels 26–27, 30

ambisonic recording 11

2D sounds 178–179, 189

ambisonics 65–67, 66, 68

2.5D audio sources 51–52

amplifiers 77

2.5D games 26

amplitude 75, 170–171

3D audio, implementation of 58–67

amplitude modulation 76, 100–101,

3D audio sources 51–52

233; creature design and 141–142

3D levels 26–27

animal samples 141, 143

3D sounds 179, 189

animation clips 35, 36

5.1 Dolby Digital 66

animation controllers 36

5.1 standard 62–65

animation events 36, 201–203

7.1 surround systems 66

animation system 35–37

360-degree surround 65–67

AntiPattern 156

Aphex Aural Exciter 97

AAC (advanced audio coding) format 281 area lights 33

absolute time 168–170

arrays 155–157, 155

absorption coefficients 216

asset delivery checklist 22–23

AC-3 Dolby Digital 282

asset management 22–23, 85–86

access modifiers 158–159

assets folder 26

Acorn, Allan 1, 2

Atari 1

active mix events 258–259

Atari 2600 2, 8, 18, 24

adaptive crowd engine prototype

attenuation shapes 47–52

143–146

attributes 150

adaptive mixing 251–275;

audio: object types 34; role of, in games considerations for 251–253; good

7–17; see also game audio

practices 271–274; music, dialogue, audio assets: gathering and preparing

and sound effects 253–254; planning

82–86; high quality 83–84; importing and pre-production 254–259

174; management and organization

ADPCM (adaptive differential pulse

of 22–23, 85–86; preparation of code modulation) format 282

173–174

aeoliphone 70

audio clips 34, 44, 46, 190–192

aesthetics 252

audio data reduction 276–286;

algorithmic reverb plugins 104

common file formats 280–282;

algorithms: in coding 148–149; Fourier-file size calculation 277–278; good based 89; random emitter 183

practices 282–283; options 283–286; ambiences 174–182, 188–189; creating perceptual coding 280; pulse code

175–178; spatial distribution 180–181, modulation 276–277; strategies

181; time property 181–182

278–282; trade-offs 280

288

INDEX

audio developers 5–6

bit rates 278, 279–280, 283

audio effects 52–53

blending 106–107

audio emitters 212–213

blind spots 179

audio engine 40–41, 43–69

Blue Print 17

audio fades 168–170, 204–206

Booleans 154

audio filters 52

broadband noise 244

audio group inspector 260–261

broad frequency spectrum 62

audio implementation 173–213; Brown, Tregoweth 71

ambiences and loops 174–182,

Burtt, Ben 71

188–189; animation events 201–203;

bus compression 94

asset preparation 173–174; collisions Bushnell, Nolan 1

193–197; distance crossfades

206–210, 206; fades 204–206;

C 149

intermittent triggers 188–189; prefabs

C# 148, 150; accessing functions from

210–213; random emitters 182–188,

another class 159–160; access modifiers

182; raycasting 197–201; sample

158–159; arrays 155–157, 155; audio concatenation 189–193; smart audio script 160–171; data types 154; first

sources 197–201

script in 151–154; introduction to Audiokinetic 17

151–171; lists 157–158; syntax audio listeners 43–45

151–154; variables 154–155

audio localization 53–69

Cage, John 18

audio mixers 53, 116–118, 123, 170,

camel casing 155

259–266, 272–273; see also mixing

cameras 29

audio playback technology, evolution of

Cartesian coordinates 26–27

3–5

cartoons 71

audio programming and implementation 5 center speakers 64

audio reverb filters 224

chambers 103

audio script 160–171

channel-based audio 58, 62

audio settings 40–41

character controllers 28, 28, 29

audio source parameters 46–47

characters 154

audio sources 34, 45, 49, 179–180; 2D,

CheckDistance() function 198

3D, and 2.5 51–52; directional 50;

CheckForDistance() function 198,

smart 197–201; square/cube 50, 51;

201, 208

volumetric 51

child classes 151

audio-visual contract 76

Chime Vs. Buzzer Principle 11–12

augmented reality 4, 5; categories of 14; Chion, Michel 76

immersion and 14–17

chorus 110–111, 111

aural exciters 97

clarity 251–252

automation 266–271

classes 150–151, 151; accessing Avatar 36

functions from other 159–160

Awake() function 154, 160, 207

class names 152

axes 37–38

clients, communication with 86

Azimuth 53, 54

clipping 119–121, 120

clouds 99

baking 216

coalescence 99

base class 150, 151

coding 147–172; algorithms 148–149; batch processors 80, 174

audio script 160–171; C# 151–171;

behaviors 150

detecting keyboard events 167–168; believability 137

encapsulation 150; inheritance

B format 67

150–151; logic 148; object-oriented

binaural renderings 58–61

programming 149–151; perceptual

bit crushing 92

280; reasons to learn 147–151;

bit depth 3

reusable code 156–157; sample

INDEX

289

randomization 166–167; syntax 148; data types 154

using triggers 164–166

DAWs 116–118

coin-operated games 2

deadlines 22

Colavita visual dominance effect 16

decay time 105

colliders 32, 32, 38, 164–166, 193–195, deltaTime variable 168–169

194, 200–201

density 98, 105, 220

collision detection 32, 38, 39, 193–195

design documents 85–86

collisions 193–197

Destroy() method 211

colons 152

dialogue 122, 253–254

colors, working with, in Unity mixer

diffuse resonant bodies 247

261–262

diffusion 220

comb filtering 89, 101–102, 102

digital audio 276–278

communication 86

digital audio converters (DACs) 116

complex sounds, breaking into layers

digital audio encoding 277

73–74, 74

digital audio signals 92

compressed audio formats 83

Digital Signal Processing techniques 107

compression 92–95, 93; bus 94;

directional audio sources 50

dynamic range 93; inflation 95; directional lights 33–34

transient control 94–95

distance: Doppler effect and 234–237; compression formats 285–286

dry to wet ratio as product of

compressors 77, 84–85, 257

227–229; factors in 75–76; filtering Computer Space 1

as product of 224–230; low pass

concatenation 189–193

filtering with 55; perception of

condenser microphones 80–82

10; simulation 229–230; spherical consistency 15–16, 22, 252

spreading over 48–50, 48, 49; width

constant bit rates (CBR) 279

perception as product of 225–226

context 8

distance crossfades 206–210, 206, 233–234

convolution 107–110, 108; creature distance cues 53, 54–56

design and 142–143; filtering/very

distance modeling 224–230

small space emulation 110; hybrid

distortion 89–92, 91; bit crushing 92;

tones 110; optimization 109; speaker creature design and 140; overdrive and electronic circuit emulation

91; saturation 90–91, 90

109–110

distortion/saturation plugins 78

convolution-based reverb plugins 78

Dolby Atmos 58

coroutines 169, 183–188

Dolby Digital Live 282

CPU resources 240

Doppler effect 234–237

creature sounds: amplitude modulation

Doppler factor 235

and 141–142; animal samples 141,

drop files 145

143; convolution and 142–143;

dry to reflected sound ratio 55

distortion and 140; emotional span

dry to wet ratio 227–229

137–138; equalization and 140; non-DSP classics 100–102

human samples 143; pitch shifting DTS:X 58

and 138–140; primary vs. secondary ducking 266

sounds 137; prototyping 136–143;

dynamic microphones 80–82

vocal recordings 138

dynamic mix 13, 21–22

crossfades 206–210, 233–234

dynamic range 77, 120–121, 120, 252, crosstalk 61

256–258, 257, 283

curly braces 153

dynamic range compression 93

cut scenes 122–126

effects, adding to groups 262–263

data 8

effects loops 122–125, 222–223; data reduction: good practices 282–283;

inserts vs. 263–264; setting up for options 283–286; strategies 278–282

reverberation 264–266

290

INDEX

Electronic Arts 4, 20

Fourier-based transforms 89

electronic circuit emulation 109–110

Fourier synthesis 247

emotional involvement 17

frame rates 118–119, 168–170

encapsulation 150, 150

frequency chart 96, 96

entertainment 8, 12–14

front left and right speakers 64

environmental modeling 4, 9–10, 21,

full bandwidth recordings 83

214–237; best practices for 219–220; full sphere, surround format 65–67

definition of 214–215; density and fully immersive systems 14

diffusion 220; distance crossfades functions: accessing, from another class

233–234; distance modeling 224–230;

159–160; see also specific functions

Doppler effect 234–237; effects loops

222–223; exclusion 230, 232–233, Gabor, Denis 88, 97–98

232; guns and explosions 130–131; game audio: challenges in 17–23; coding high frequencies vs. low frequencies

for 147–172; evolution of 3–5;

220; late vs. early reflections 219;

genesis of 1–3; role of 7–17

obstruction 230, 231–232, 232;

game engine: definition of 24–29; level

occlusion 230, 231, 231; reflections elements 29–34; paradigm 24–42; sub

level 219–220; reverberation systems 35–42

215–219, 222–223; reverberation for

game levels 26–27; elements of 29–34

106; reverb zones 221–222

game mechanics 11–12

equalization 77, 95–97; creature GameObject.Find() function

design and 140; resonance

197–198, 207

simulation 96–97

game objects 20; see also objects

equalizers 77

gameplay: adjusting levels during

equal loudness contour graph 272,

265–266; increasing complexity in 4

272, 273

game states 254, 266–271

evaporation 99

Gardner, W.G. 227

event functions 153–154, 153

generic audio 279

event scheduling 192–193

geometry 9–10

exclusion 10, 230, 232–233, 232

Gerzon, Michael 65

experimentation 86

GetComponent() method 160, 161

GetKeyDown() function 167–168

fades 168, 169–170, 204–206

GetOcclusionFreq() function 200, 201

fall-off curve 48

grain duration 98–99

Farnell, Andy 242

granular synthesis 88–89, 88, 97–100, Fast Fourier Transform (FFT) 107, 108

98; pitch shifting 99–100; sample fast Fourier transform-based

manipulation/animation 100;

algorithms 89

terminology 98–99; time stretching

fatigue avoidance 18–19

99–100

file formats 280–282

gravity gun 20

file size calculation 277–278

Grindstaff, Doug 71–72

filtering 95–97, 110, 233; low pass 55, groups: adding effects to 262–263;

76, 87, 224–225, 249–250; as product adding to audio mixer 259–260; audio

of distance 224–230

group inspector 260–261

first-personal controller 28

group sidechaining 125–126

flangers 111, 111

guns: detonation/main body layer

Fletcher-Munson curves 272, 272, 273

129–130; environmental modeling

floating point numbers 154

130–131; general considerations Foley, Jack 113

127–128; gunshot design 128–129; Foley recording 113–114

one shot vs. loops 126–127, 127;

footsteps 76, 189–190

player feedback 131–132; prototyping formants 139, 139, 140

126–132; sublayer 130; top end/

forward slash 154

mechanical layer 130

INDEX

291

Half Life 2 20

lavalier microphones 81–82

hard clipping 90, 90, 91

Law of Two and a Half 76

harmonic generators 97

layering/mixing 86–87, 94, 175

harmonic processors 78

layers 85

headphones 274

level meters 117

Head Related Transfer Functions 11

levels: 2D 26–27, 30; 3D 26–27;

head related transfer functions (HRTFs)

adjusting during gameplay 265–266;

58, 58–62, 59, 108

game 26–27, 29–34; mix 273–274

high cut parameter 105–106

LFE submix 125

high frequencies 220, 233, 283

Lifecycle script 153

high pass filtering 75–76

lighting 28, 33–34

home gaming consoles, first 2

Limbo 16

horizontal axes 37–38

linear amplitude 170–171

horizontal plane, localization on 56–57

linear animation 41–42

HRTFs see head related transfer

linear fall-off curve 48

functions (HRTFs)

linear mixes 122–126

humanoids 36

linear model synthesis 246–250

hybrid tones 110

listeners 34, 43–45

lists 157–158

IDE see Integrated Development

LKFS unit 273

Environment (IDE)

load type 284–285

IEnumerator 184

local coordinates 27

immersion 8; characteristics that create

localization: audio 53–69; cues 56–58;

15; definition of 14–17; maintaining 16

on horizontal plane 56–57; on vertical implementation, challenges 17–18

plane 58–59

impulse reponse 104

location, perception of 10–11

inflation 95

logarithmic amplitude 170–171

information, provided by audio 8–12,

logarithmic fall-off curve 48

252–253

logic 148

inheritance 150–151

loops 174–182, 176, 189; creating input 116

175–178; implementing 178–182; input system 37–38

inserts vs. effect 263–264; seamless

inserts 116–117, 122–123, 263–264

175–176; spatial distribution Inside 16

180–181, 181; time property Instantiate() method 210–211

181–182; see also effects loops

integers 154

lossless data reduction 278–279

Integrated Development Environment

loudness 54–55, 272–273, 272

(IDE) 148, 152

loudness K-weighted full scale

interactive elements 19–20

(LKFS) 273

interaural intensity difference (IID) 11,

loudness maximizers 77, 131

57, 57, 58

loudness unit full scale (LUFS) 273

interaural time difference (ITD) 11, 57,

low cut parameter 106

57, 58

low frequencies 220

intermittent emitters 189

low frequency effects (LFE) 64

intermittent triggers 188–189

low pass filtering 55, 76, 87, 224–225,

inverse square law 54–55

249–250

isKinematic property 38

LUFS-based loudness meters 78–79

isPlaying property 190

LUFS unit 273

isTrigger property 39

MacDonald, Jimmy 71

Kandinsky, Wassily 115

Magnavox Odyssey 2

keyboard events, detecting 167–168

MapToRange() function 208–209

kinematic RigidBody colliders 194

mass, of sound 74–75

292

INDEX

Massachusetts Institute of Technology

non-player controllers (NPCs) 28

(MIT) 1

non static variables 159

master output 124

Nutting Associates 1

materials 31

Nyquist theorem 276

MaxMSP: adaptive crowd engine

prototype 143–146; sword maker

object-based audio 58–61, 62, 67,

example in 246–250

68–69

MaxxBass plugin 97

object-oriented programming 149–151

Mecanim 35

objects 30; audio 34, 43–45; colliders

Menzies 247

32, 32; lights 33–34; materials 31;

meshes 30

meshes 30; models 30–31; particle

.meta extension 44

systems 32; prefabs 34; shaders

metering tools 78–79, 117

31; skyboxes 32; sprites 30; terrain

microphones 80–82; dynamic vs.

31–32; textures 31, 31; transform

condensers 80–82; placement of 82

component 30; triggers 33

mixer parameters 270

obstruction 10, 230, 231–232, 232

mixers 53, 116–118, 170, 223, 259–266, occlusion 10, 197–199, 210, 230,

272–273

231, 231

mixing 13–14, 21–22, 86–87; adaptive Ogg Vorbis 281

251–275; considerations for 251–253; OnCollisionEnter() function 194

dynamic range 256–258; good

ontological modeling 241

practices 271–274; inserts vs. effect OnTriggerEnter() function 165

loops 263–264; music, dialogue, and OnTriggerExit() function 165

sound effects 253–254; passive vs.

OnTriggerStay() function 165

active mix events 258–259; planning opacity 99

and pre-production 254–259; premix

output 118

273–274; routing 255–256; snapshots overdrive 91, 91

and 266–271; submixing 254–255;

overlapping 89

Unity audio mixer 259–266

overriding 34

mix levels 273–274

mix sessions 123

Pac Man 2

modal synthesis 246–250

parameters: editing via scripting 270;

models 30–31

exposing 270–271; see also specific

modes 96–97, 246

parameters

monitoring 126

parent class 151

Monobehaviour 152

particle systems 32

mono signals 61–62

passive mix events 258–259

MP3 83, 280–281

PCM audio 18

multichannel audio 62–65, 68

peak meters 272–273

multi-modal integration 76

pebble effect 199–201

multi-player games 42

perceptual coding 280

Murch, Walter 21, 72, 76

percussive sounds 75, 83

music 13–14, 122, 253–254

peripheral vision 9

music bus 257

phasers 112, 112

phasing issues 181–182

naming conventions 22–23, 85, 155, 180 physical analysis 242

narration 254

physics 4, 20, 38, 238–239

narrative function 252

physics engine 38–40

networking 42

pink noise 243–244, 245

No Country for Old Men 72

pitch 74

noise 84

pitch shifting 87–89, 178; creature

non-diffuse resonant bodies 247

design and 138–140; fast Fourier

non-immersive systems 14

transform-based algorithms 89;

INDEX

293

granular synthesis 88–89, 88, 99–100; ragdoll physics 4, 20

playback speed modulation 87–88

RAM 239

Pitch Synchronous Overlap and Add

random emitters 182–188, 182; (PSOLA) 88, 99–100

algorithm 183; coroutines 183–188

playback speed modulation 87–88

randomization 18–19, 99, 162–163; player controllers 28, 28, 29

linear amplitude and 170–171; sample player feedback 131–132

166–167

PlayFirst() function 191

raycasting 39, 197–201; avoiding

Play() method 160, 163–164

pebble effect 199–201; implementing PlayOneShot() method 163–164

occlusion with 197–199

PlayScheduled() function 192–193

realism 72, 137, 197

PlaySecond() function 191

real-time computation 216

PlaySound() function 187

rear left and right speakers 64

plugin parameters 270

reflections 56; late vs. early 219; level point lights 33

219–220

Pong 1–2, 32

relativeVelocity 195

post-fader sends 118

repetition 18–19

precedence effect 11

resonance 246–247, 250

predelay parameter 105

resonance simulation 96–97

pre-delay time to reverb 75, 219

resonant bodies 247

prefabs 34, 210–213; creating smart resonators 101–102

intermittent emitter prefab with

reverberation 78, 84, 102–107, 103;

occlusion 210; destroying objects

absorption coefficients 216; audio

instantiated from 211; instantiating reverb filters 224; as blending tool

audio emitters 212–213; instantiating

106–107; as dramatic tool 107; effects from scripting 210–211

loops for 222–223; for environmental pre-fader sends 117

modeling 106, 215–219; indoors vs.

premix 273–274

open air 102–104; inserts vs. effects

pre-production 254–259

loops for 122–123; parameters primary sounds 137

105–106, 217–219; pre-computed vs.

prioritization 252

real time computation 216; setting private keyword 158

up effect loop for 264–266; in Unity procedural assets 239

216–219

procedural audio 4–5, 238–250; reverb plugins 78, 103–104

approaches to 241–242; candidates reverb time/decay time 105

for 241; definition of 239–242;

reverb zones 217–218, 221–222, 229

introduction to 238–239; practical rigidbodies 38, 40

242–250; pros and cons of 239–241;

RigidBody colliders 194

sword maker example 246–250; wind

ring modulation 100–101

machine example 242–246

Roads, Curtis 88

procedural programming languages

routing 255–256

149–150, 149

Russel, Steve 1

procedural sound synthesis 5

programming see coding

sample concatenation 189–193

programming languages 149–150

sample manipulation/animation 100

protected keyword 158

sample playback 3–4

Pro Tools 116, 125

sample randomization 166–167

prototyping 19–20, 126–146; adaptive sample rates 3, 87, 92, 286

crowd engine 143–146; creatures

sample selection, velocity-based

136–143; guns 126–132; vehicles

195–197

132–136

sampling rate 276

public keyword 158

saturation 90–91, 90

pulse code modulation 276–277

Schaeffer, Pierre 71

294

INDEX

scripting: editing mixer and plugin

sound effect library 84

parameters via 270; recalling snapshots sound effects 4, 122, 253–254;

via 268–269; see also coding

procedural audio and 5

seamless loops 175–176

sound FX librarian software 84

secondary sounds 137

sound layers 85, 86–87; blending semicolons 152

106–107

semi-immersive systems 14

sound recording, Foley 113–114

Send/Receive technique 264–266

soundscapes 21

separators 152

sound sources see audio sources

SetFloat() method 270, 271

soundtracks: evolution of 4; music

SetSourceProperties() function 187

13–14; role of, in games 7–17

shaders 31

Space Invaders 2

shotgun microphones 80–82

Spacewar! 1

side chain compressors 257

spatial audio 5

sidechaining 125–126

spatial awareness 9–10

signal flow 115–118, 116

spatial distribution, of ambient loops

signal path 119–121

180–181, 181

silence 73

spatial imaging 252

size parameter 105

spatialization, optimizing sound design

skyboxes 32

for 68–69

smart audio sources 197–201

spatial width 56

snapshots 266–271; recalling vis

speakers: center 64; emulation of scripting 268–269; working with 267

109–110; front left and right 64; soft clipping 90, 90, 91

rear left and right 64

sound: information provided by 8–12;

spectral analysis 242, 248–249

mass or weight of 74–75; pitch of 74

spectral balance 140

sound cones 50, 50

spectrum analyzer software 79

sound design: art of 70–86; basic

speech 279

considerations 72–76; clipping spherical spreading 48–50, 48, 49

119–121, 120; effective 72–74;

spotlights 33

entertainment and 12–13; spread parameter 225–226

environmental 21; frequency chart sprites 30

for 96, 96; guidelines 74–76; history

square/cube audio sources 50, 51

of 70–72; microphones for 80–82;

Stalling, Carl 71

optimizing for spatialization 68–69; StartCoroutine() statement 184

practical 115–146; preparation for

Start() function 161, 186, 197–198, 208

82–86; prototyping and 126–146; Star Trek 71–72

session setup 115–118, 122–126; Star Wars 71, 72

technical 5; tools for 76–80; working

states 254, 266–271

with video 118–119

static colliders 194

sound designers 4; role of 9

static keyword 158–159

sound design techniques: amplitude

stems 122

modulation 100–101; comb filtering

stereo 62

101–102; compression 92–95, 93; Stochastic techniques 18

convolution 107–110, 108; distortion

streams 99

89–92; DSP classics 100–102;

strings 154

equalization/filtering 95–97; Foley

subharmonic generators 97, 125

recording 113–114; granular

sub master 124

synthesis 97–100, 98; harmonic submixes 118, 124–125, 124, 254–255

generators/aural exciters 97; layering/

Subotnick, Morton 13–14

mixing 86–87; pitch shifting 87–89;

sub systems 35–42; animation 35–37;

reverberation 102–107; time-based

audio engine 40–41, 43–69; input

modulation FX 110–113

37–38; linear animation 41–42; sound effect bus 257–258

physics engine 38–40

INDEX

295

subtractive synthesis 242–246

UREI 1165 limiting amplifier 77

subwoofer 64–65

user feedback 11–12

surround channel-based formats 62–65

user input, detecting 167–168

sweeteners 145

utilities 80

sword maker example 246–250

syntax 148; C# 151–154

variable bit rates (VBR) 279–280, 283

variables 154–155

teams, communication with 86

variations, creating 178, 189–190

technical sound design 5

vehicles: material selection 133; teleological modeling 241

processing and preparing materials

terrain 31–32

133–134; prototyping 132–136; textures 31, 31

specifications 132–133

third-party implementation tools 17

velocity-based sample selection

third-person controller 28, 29

195–197

time-based modulation FX 110–113;

version control 22

chorus 110–111, 111; flangers 111;

version tracking 85–86

phasers 112, 112; tremolo 112–113

vertical axes 37–38

timecode 119

vertical plane, localization on 58–59

Time.deltaTime 204, 205

very small space emulation 110

time property 181–182

video: frame rates 118–119; working

time stretching 99–100

with 118–119

Time.time 211

video games: first 1–3, 18; role of audio

timing 168–170

in 7–17; see also game audio

transform component 30

views, working with, in Unity mixer

transforms, Fourier-based 89

261–262

transient control 94–95

virtual reality 4, 5, 13, 239; categories of transients 75, 77

14; immersion and 14–17

TransitionTo() method 268

visual field 8–9

tremolo 112–113

Visual Studio 148, 152

tremolo effect 141

vocal recordings, working with 138

Trespassers: Jurassic Park 4, 20

volume faders 117

triggers 33, 39, 164–166, 188–189

volume sliders 270–271

trigger zones 33

volumetric sound sources 51

Unity3D project structure 25–29

WaitForIt() function 187

Unity Editor 26

Warner Brothers 71

Unity game engine 6, 10, 148; ambisonic waveform analysis 242

recording and 11; animation system weight, of sound 74–75

35–37; audio engine 40–41, 43–69;

wet to reverberant signal ratio 75

audio mixer 259–266; data reduction white noise 243–244

options in 283–286; ducking in 266; width parameter 105

input system 37–38; linear animation width perception 225–226

41–42; physics engine 38–40; playing

wind machine example 242–246

audio in 160–171; reverberation in Wirth, Werner 15

216–219; scenes vs. projects 26

world coordinates 27

Unity Hub application 25

world geometry 27–28

Unity projects: creation of 25–26; level

Wwise 17, 22

basics 26–29

Universal Audio LA-2A leveling

Xenakis, Iannis 18, 88

amplifier 77

XY pads 144–146, 144

Unreal engine 17, 19

Update() function 154, 167–169,

yield return statements 184

198, 201

UpdateVolume() function 209

zip files 278–279

Document Outline

Cover

Half Title

Title Page

Copyright Page

BRIEF CONTENTS

DETAILED CONTENTS

1 Introduction

1 The Genesis of Audio in Games

2 From Sample Playback to Procedural Audio

3 How to Use This Book

What This Book Is

What This Book Isn’t

2 The Role of Audio in Interactive and Immersive Environments

1 Inform, Entertain, Immerse

1 Inform: How, What

a Geometry/Environment: Spatial Awareness

b Distance

c Location

d User Feedback and Game Mechanics

2 Entertain

a Sound Design

b Music and the Mix

3 Defining Immersion

2 Challenges of Game Audio

1 Implementation

2 Repetition and Fatigue Avoidance

3 Interactive Elements and Prototyping

4 Physics

5 Environmental Sound Design and Modeling

6 Mixing

7 Asset Management and Organization

3 The Game Engine Paradigm

1 What Is a Game Engine

The Unity3D Project Structure

1 Level Basics 101

a 2D, 3D and Cartesian Coordinates

b World Geometry

c Lighting

d Character Controllers

e Cameras

2 Elements of a Level

a Everything Is an Object

b Transform

c Sprites

d Meshes

e Models

f Textures

g Shaders

h Materials

i Terrain

j Skyboxes

k Particles Systems

l Colliders

m Triggers/Trigger Zones

n Lighting

o Audio

p Prefabs

2 Sub Systems

1 Animation

2 Input

3 Physics

Rigidbodies and Collision Detection

Physics Materials

Triggers

Raycasting

4 Audio

5 Linear Animation

6 Additional Sub Systems

4 The Audio Engine and Spatial Audio

1 Listeners, Audio Clips and Audio Sources

1 The Audio Listener

Audio Clips

Audio Sources

2 Audio Source Parameters

3 Attenuation Shapes and Distance

a Spherical Spreading

b Sound Cones – Directional Audio Sources

c Square/Cube

d Volumetric Sound Sources

e 2D, 3D or 2.5D Audio?

4 Features of Unity’s Audio Engine

a Audio Filters

b Audio Effects

c Audio Mixers

2 Audio Localization and Distance Cues

1 Distance Cues

a Loudness

b Dry to Reflected Sound Ratio

c Low Pass Filtering With Distance

d Spatial Width

2 Localization Cues

a Localization on the Horizontal Plane

b Localization on the Vertical Plane

3 Implementing 3D Audio

a Object-based Audio and Binaural Renderings

b Working With HRTFs

c Multichannel Audio and Ambisonics

4 Optimizing Sound Design for Spatialization

a Putting It all Together

b Working With 2D and Multichannel Audio

c Working With Ambisonics

d Working With Object-Based Audio

5 Sound Design – The Art of Effectively Communicating With Sound

1 The Art of Sound Design

1 A Brief History of Sound Design

2 Sound Design – Basic Considerations

a Effective Sound Design

b Sound Design Guidelines

3 Getting the Right Tools

a Equalization

b Dynamic Range

c Reverberation

d Harmonic Processors

e Metering Tools

f Utilities

4 Microphones

a Microphone Choice: Dynamic vs. Condensers

b Mic Placement

5 Sound Design – Before You Start

a Always Use High Quality Material

b Don’t Get Too Attached

c Build and Learn

d Listen for the Expected and the Unexpected

e Layers

f Be Organized

g Communicate

h Experiment, Experiment, Experiment

2 Basic Techniques

1 Layering/Mixing

2 Pitch Shifting

a Playback Speed Modulation

b Granular Synthesis

c Fast Fourier Transform-Based Algorithms

3 Distortion

a Saturation

b Overdrive

c Distortion

d Bit Crushing

4 Compression

a Blending Through Bus Compression

b Transient Control

c Inflation

5 Equalization/Filtering

a Equalization for Sound Design

b Resonance Simulation

6 Harmonic Generators/Aural Exciters

7 Granular Synthesis and Granulation of Sampled Sounds

a Granular Synthesis Terminology

b Sound Design Applications of Granular Synthesis

8 DSP Classics

a Ring Modulation/Amplitude Modulation

b Comb Filtering/Resonators

9 Reverberation

a Indoors vs. Open Air

b Reverb Parameters

c Reverberation for Environmental Modeling

d Reverberation as a Dramatic Tool

10 Convolution

a Optimization

b Speaker and Electronic Circuit Emulation

c Filtering/Very Small Space Emulation

d Hybrid Tones

11 Time-Based Modulation FX

a Chorus

b Flanger

c Phasers

d Tremolo

12 Foley Recording

6 Practical Sound Design

1 Setting Up a Sound Design Session and Signal Flow

1 Signal Flow

a Input

b Inserts

c Pre-Fader Send

d Volume Fader

e Metering: Pre-Fader vs. Post Fader

f Post-Fader Send

g Output

2 Working With Video

a Know Your Frame Rate

3 Clipping Is Easy – Mind the Signal Path

Use the Dynamic Range

4 Setting Up a Basic Session for Linear Mixes and Cut Scenes

a Music, Dialog and Sound Effects

b Inserts vs. Effects Loops for Reverberation

c Setting Up the Mix Session

d Master Output and Sub Maste

e Submixes and Effects Loops

f Further Enhancements

2 Practical Sound Design and Prototyping

1 Guns

a One Shot vs. Loops

b General Considerations

c Designing a Gunshot

2 Prototyping Vehicles

a Specifications

b Selecting Your Material

c Processing and Preparing Your Material

d Building a Prototype

3 Creature Sounds

a Primary vs. Secondary Sounds

b Emotional Span

c Working With Vocal Recordings

d Working With Animal Samples

e Working With Non-Human or Animal Samples

4 An Adaptive Crowd Engine Prototype in MaxMSP

7 Coding for Game Audio

1 Why Learn to Code?

1 Syntax and Logic

2 Algorithms

3 Basic Object-Oriented Programming Concepts

a Procedural vs. Object-Oriented

b Encapsulation and Inheritance

2 An Intro to C#: Syntax and Basics

1 Our First Script

2 Variables, Constants, Data Types Operators, Arrays and Lists

a Data Types

b Variables

c Arrays

d Lists

e Access Modifiers

3 Accessing a Function From Another Class

3 Playing Audio in Unity

1 Our First Audio Script

2 Play()﻿ vs. ﻿PlayOneShot()

3 Using Triggers

4 Sample Randomization

5 Detecting Keyboard Events

6 Audio-Specific Issues

a Timing – Frame Rate vs. Absolute Time

b Linear vs. Logarithmic Amplitude

8 Implementing Audio: Common Scenarios

1 Before You Start: Preparing Your Assets

2 Ambiences and Loops

1 Creating Ambiences and Loops

a Seamless Loops

b Creating a Simple Loop – Looping Techniques

c Creating Variations

2 Implementing Our Loops in a Unity Level

a Challenges

b Spatial Distribution

c Working With the Time Property to Avoid Phasing Issues

3 Random Emitters

a A Simple Random Emitter Algorithm

b Coroutines

4 Ambiences, Putting It All Together

5 Sample Concatenation

a Creating Variations With Footsteps Samples

b Case 1: Swapping Audio Clips

c Case 2: Using ﻿PlayScheduled﻿()

6 Collisions

a Detecting Collision

b Velocity-based Sample Selection

7 Raycasting and Smart Audio Sources

a Implementing Occlusion With Raycasting

b Avoiding the Pebble Effect

8 Animation Events

9 Audio Fades

10 Distance Crossfades

11 Working With Prefabs

a Creating a Smart Intermittent Emitter Prefab With Occlusion

b Instantiating a Prefab From Scripting

c Destroying an Object Instantiated From a Prefab

d Instantiating Audio Emitters at Random Locations in 3D

9 Environmental Modeling

1 What Is Environmental Modeling?

1 Reverberation

a Pre-Computed vs. Real Time Computation

b Absorption Coefficients

c Environmental Modeling With Reverberation in Unity

d Unity’s Reverberation Parameters

2 Best Practices for Environmental Modeling

a Late vs. Early Reflections

b Reflections Level

c Density and Diffusion

d High Frequencies vs. Low Frequencies

3 Reverb Zones, Effects Loops and Audio Reverb Filters

a Reverb Zones

b Adding Reverb as an Effect Loop Using the Mixer

c Audio Reverb Filters

2 Distance Modeling

1 Filtering as a Product of Distance

a Adding a Low Pass Filter That Will Modulate its Cutoff Frequency Based on Distance

b Width Perception as Product of Distance

c Dry to Wet Ratio as a Product of Distance

d Distance Simulation: Putting It All Together

3 Additional Factors

1 Occlusion, Obstruction, Exclusion

a Occlusion

b Obstruction

c Exclusion

2 Distance Crossfades

3 Doppler Effect

10 Procedural Audio: Beyond Samples

1 Introduction, Benefits and Drawbacks

1 What Is Procedural Audio?

a Procedural Audio, Pros and Cons

b Approaches to Procedural Audio

2 Practical Procedural Audio: A Wind Machine and a Sword Collision Model

1 A Wind Machine in MaxMSP With Subtractive Synthesis

Making the Model Flexible

2 A Sword Maker in MaxMSP With Linear Modal Synthesis

Spectral Analysis

Modeling the Impulse

Modeling the Resonances

Making the Model Flexible

11 Adaptive Mixing

1 What’s in a Mix? Inform and Entertain (Again)

1 Mix Considerations

2 Music, Dialogue and Sound Effects

3 Planning and Pre-Production

a SubMixing

b Routing

c Dynamic Range

d Passive vs. Active Mix Events

2 The Unity Audio Mixer

1 Adding Groups to the Unity Mixer

2 The Audio Group Inspector

3 Working With Views and Colors in the Unity Mixer

Creating Views in Unity

4 Adding Effects to Groups in Unity

5 Inserts vs. Effect Loops

6 Setting Up an Effect Loop for Reverberation in Unity Using Send and Receive

Send and Receive

Note on Adjusting Levels During Gameplay

7 Ducking in Unity

Setting Up a Ducking Compressor in Unity

3 Snapshots, Automation and Game States

1 Working With Snapshots

2 Recalling Snapshots via Scripting

3 Editing Mixer and Plugin Parameters via Scripting

4 Exposing a Parameter: Controlling a Volume Slider

4 Good Practices

Mix Levels

12 Audio Data Reduction

1 Digital Audio: A Quick Review

1 Pulse Code Modulation

2 File Size Calculation

2 Data Reduction Strategies

1 Speech vs. Generic Audio

2 Bit Rates

3 Perceptual Coding

The Trade-Off

4 Common File Formats

a MP3

b Advanced Audio Coding

c Ogg Vorbis

d AC-3 Dolby Digital

e Adaptive Differential Pulse Code Modulation

3 Data Reduction Good Practices

4 Data Reduction Options in Unity

1 File Options

2 Load Type

3 Compression Formats Options

Sample Rate Setting

Index