Project Aim update (Audio Immersion)

After a lot of thought a final direction for the honours project has been identified. This is a rough statement of intent.

An investigation into immersive game audio.

What role does audio play in the immersive quality of 'First Person Shooter' games? 

This project aims to identify the relationship between audio and immersion in games. The project will include research and critical analysis of theories and techniques involved with creating and implementing immersive game audio. Using this acquired knowledge I hope to be able to quantify the immersive quality of audio in a test level produced in UDK.

Using known theories and techniques I hope to create a gaming environment which is focused on using aural feedback to produce an immersive experience which can be measured in some way.​

Next to do:

• List the objectives and tasks surrounding such a project
• Begin to search for relevant literature

Game Audio Tutorial - Book - Chapter 3 Part 3

Exercise Set 3 Continued:

304a_03 Footloose 2 : Bigger, Cornier, and more Callous

In this exercise there was 4 custom floor materials which all had to go through the same coding process as the previous task. Custom sound cues with appropriate footstep sounds also had to be generated and attached to each material.

    

304b_00 Bang ! Physics impacts for dummies

This exercise involved converting various 'Static Mesh' objects within the game to become movable object This was done by converting them into a 'K_Actor'. The 'K_Actor' allows an object to be movable and is governed by the game physics, which can be tailored to each object with regards to its estimated size and weight.

Sound cues had to be created for each of the objects then linked via the Kismet to play back when the object was impacted. This was done by linking the 'K_Actor' collision event to the sound cue. Information from the collision event (Rigid Body Collision) includes details of impact velocity, so this information was then linked to the 'Volume Multiplier' variable of the 'Play Sound' action. As the number outputted by the impact velocity was quite large, a 'Divide Float' had to be included in the chain. This divided the large number from the impact velocity into a smaller number that the 'Volume Multiplier could work with.

305_00 Moving sounds Wooded area

Here the 'Matinee' editor was introduced which is a sequence editor, where in this exercise it was used to move a sound around the level. A path was mapped out over an area which was then linked to a timeline within the Matinee editor window. This movement circuit was linked to the starting of the level so that it moved the sound around the defined path at level start up. In this case a sound cue containing randomised bird sounds was used and the final effect was that it sounded as if the birds were flying around the player, adding some movement to the sound as opposed to a static sound.

305a_00 Matinee Doors

Here, a 'Matinee' sequence was created to trigger the movement of a door opening. In order for this to work the static mesh door had to be converted to an ‘InterpActor’ which allows it to be controlled by the 'Matinee'. The parameters of the 'InterpActor' include options to add sound cues for each of the movement sounds, custom sound cues were built for each door in the exercise and door movements were synchronised to move in time with the length of each sound via the 'Matinee' editor.

305b_00 Pawn Attached Mover

This was similar to the previous exercise only this time the moving object was a horizontal moving platform. This platforms movement had to be activated when the player (Pawn) stepped on it. This was done in the Kismet window using the ‘Mover’ node linked to the platform which defines what happens to the platform as the player steps on and off. This was then linked to the Matinee sequence via a switch.

306_00 Lights, Camera, Action
This exercise deals with how cut scenes are dealt with in games. When the exercise level starts there is a cut scene containing dialogue between two robots, the problem is as the camera pans between different angles the sound also pans in relation to which way the camera is facing. This creates an off putting mix of both dialogue and environmental sounds.
The idea was to create a separate mix of both dialogue and environment sounds which would be played back via the 'Matinee'. All original sound cues had to be duplicated and defined as as belonging to the 'Cinematic Sound Class' in order for them to be played back in the 'Matinee' sequence. These were then placed on the timeline within the 'Matinee' (with dialogue synced to key markers associated with camera movement). Within the Kismet a 'Set Sound Mode' action was added which was triggered when the cut scene begins, this toggles the game sound off and the cinematic mix that was created to play instead.
  

Game Level Concept #2

After an extensive search for a pre-made UDK game level a suitable candidate has been found for the purposes of this honours project.

The following level was found here : http://www.indiedb.com/games/dungeon-escape/downloads/dungeon-escape-udk-source-files 

Dungeon Escape
This UDK level was created by Chris Holden who was contacted and has since given approval to use his level for this project. The level is set in a dungeon where the player has to navigate through various rooms and chambers to find keys and unlock chests of treasure with an aim to escape the dungeon.

Notes on the level:

• One of the attractions to this level is the low light theme which will help bring the aural side of the level more into player focus.
• There is no combat in this level but use of specific audio cues could still be used to induce the presence of threat.
• Some issues with player movement (too fast, this could be solved with some simple coding)
• No background story, who are you? what are you doing here? and what do you need to do? are all questions that should be answered (Some kind of story/intro will be key to drawing the player into this level, even a description of what the player needs to do at the bare minimum)
• Very little player direction/instruction which is a distraction to becoming engaged with the game. The player needs to know that they are making progress and achieving goals (ideal situation to use audio as a driver to help the player navigate and complete the level)
• It is not clear exactly how to complete the level

Next:
To fully understand every process that has been coded into the level more playing and testing is required. A full write up/walkthrough will follow this post

Research - Reseach Gate

Set up an account with Research Gate using my student credentials which allows free access to academic research papers. This should be of use in obtaining certain publications which are normally paid for.

Game Audio Tutorial - Book - Chapter 3 Part 2

Exercise Set 3 Continued:

303_00 Dynamic Ambient Zones

In this exercise two 'Reverb Volumes' had to be placed within the same space and linked to a 'trigger' within the Kismet that would switch between the two. There was two rooms in the level divided by a movable door, one of the rooms had a radio playing which was the sound source that needed to be managed when the door was closed from the opposite side. The door closing had to be linked in the kismet to trigger the switch between the room sounds. With the door closed and the radio in the other room, adjustments had to be made to one of the 'Reverb Volumes' to make is seem like the sound was filtering through the walls and door. Low pass filer and volume reduction were added to imitate the actual sound propagation through the walls and door.

304_00 Clumsy

The ‘Trigger Volume’ node within the Kismet was used in this exercise.This node was attached to various objects to trigger a sound when they were contacted/Bumped into by the player. The 'Trigger Volume' was created in the same way as the 'Reverb Volume' only more attention to the dimensions and size of the volume had to be considered when trying to model to the shape of an object. Suitable sound cues had to be created for each object and were referenced by the play sound node connected to the trigger volume. Including a variety of sound for each collision was the key to making the system sound better, by randomising sounds, modulating pitch/volume and including a re-trigger delay any repetitive sound was averted. This was identified as a simple but laborious way to play a collision sound for static objects due to the nature of sculpting the wire frame around each object one at a time.

304a_00 Footloose

This exercise was all about understanding how footstep sounds are played back through UDK. The sound of footsteps for each surface are defined within the code (Unreal Script) and linked to a 'Physical Material' which can then be associated with a building 'Material' in the game. Here, there were several different floor types that had been created for the tutorial and suitable footstep sound cues had to be attached to each. This was done using the 'Materials Editor' which gives the option to define what 'Physical Material' should be referenced when the player steps on the surface, this then calls the appropriate sound cue for playback.

304a_02 Custom Materials for Custom Footsteps

This was a more advanced version of the previous exercise where a new 'Physical Material' had to be created for custom footsteps. This involved first using one of the 'Building Materials' provided and applying it to the floor (in this case a wooden style floor) then creating a new 'Physical material' for the surface to reference via the content browser.
From here a new line of code had to be added to the 'Unreal Script' which identified the material type and the location of the custom footstep sound cue that was created.

Project update Meeting 17th Oct

This is a summary of the project update meeting with James:

My initial idea of constructing a game level within UDK and utilising the functions of the game engine to implement my own audio was seen as a good starting point. However the decision to focus on the issue of audio fidelity and whether or not it is, a) important, and, b) is there a demand for higher quality audio in games, was up for debate. 

James was very helpful in pointing out the flaws to this concept by stating the restrictions of game engines and how they handle sound. Although my original train of thought was that with computing hardware becoming more and more memory rich why couldn't audio be preserved at high fidelity. James also pointed out that, with these new advances also come better graphics and bigger games, all of which leads to the use of more memory. So in reality the ratio of memory budget allocation between audio and other assets is still similar to what it has been for years.

To the point then, this idea has been scrapped. The focus of this project still lies with game audio utilising the UDK engine, however and a new direction will be developed for consideration in the coming days/weeks.

Game Audio Tutorial - Book - Chapter 3

Exercise set 3

300b_00 Reverb
In this exercise the ‘Reverb Volume’ was introduced, which defines an area within the game were reverb can be applied. In this case I had define an area (this was done using the 'builder brush' which is also used for building structures within UDK) encapsulating a large cave and then apply a suitable reverb based on pre-set models within UDK. The effects of the reverb are then heard when the player is inside the 'Reverb Volume'.

301i_00 Sound Sources
Here, sounds had to be assigned to various sources within a room with the focus of experimenting with the 'Distance Model' settings, which are a selection of pre-set roll off curves which are applied to the volume attenuation. This was a deviation from the standard linear roll off which had been used up to this point. A logarithmic style roll of was applied along with low pass filter attenuation to attain a more convincing sound as the player moved closer or further away from the sound sources.

301j_00 Multichannel ambience
This was an introduction to the use of multi-channel sounds and the file naming conventions associated with importing such files. 4 mono files from a quad surround set up were imported which when named correctly were merged into a single surround sound audio file. This differences were compared between a single surround cue in the level and 4 separate mono cues placed in a quad configuration. The surround cue gelled together much more when playing in the level compared to the separate cues which suffered from excessive panning issues.

301k_00 Filter over distance
In this set up there was a robot who was projecting dialogue placed on top of a hill in a large environment, the objective was to apply both 'Volume attenuation' (with appropriate 'Distance Model') and low pass attenuation to make the sound more realistic with regards to distance from player to robot. A 'Reverb Volume' was also created around the environment to give a sense of the spaciousness.

301m_00 Propagation Crossfade 
Here, the 'Distance Crossfade' node within the sound cue editor was introduced. It is used as a more detailed way to attenuate the sound over a distance. Instead of just using low pass attenuation this node calls on more than one audio file of the sound source. Each audio file can be filtered with greater detail with respect to the scene in a DAW and brought back into UDK. The Distance Crossfade node is used to playback these files at certain distances from the source whilst cross fading between them depending on distance from the source. This results in a more realistic sound coming from a source as the player gets nearer or further away.

302a_00 Ambient Zones exercises
This exercise addresses issues relating to occlusion. The level included an outside area and an inside hanger area where ambient sounds had to be set. The problem with this is that when walking into the hanger the outside sound is still very present as by default walls and other solid objects have no impact on the propagation of sound in UDK. This was controlled using the ‘Reverb Volume’ to create a space in the hanger which could, when entered and exited effect the sound heard from inside and outside the hanger to make it more realistic. Settings used were to control sound levels and low pass filtering on sounds heard on either side of the ‘Reverb Volume’, there was also a fade time setting which defines the fade time when stepping between the different zones.

302f_00 Ambient Zones 02
This exercise was a more complex version of the previous that included 5 rooms each with their own sound source and corridors linking them. Using the skills learned so far a realistic balance of sound levels was created when walking around the level. This included placing ‘Reverb Volumes’ to simulate an obstruction such as a pillar or wall between player and sound source.

Game Audio Tutorial - Book - Chapter 2 - Part 3

Exercise Set 2 Continued:

206b_00 Concatenation of multiple sample rates
The Concatenator node was used to reassemble sounds that were chopped into sections. To save memory whole sound files were chopped into sections with each section having different sample rate applied depending on the frequency content. In this case, gunshot and explosion sounds were used. As the initial blast of each sound had a lot of high frequency content these were sampled at the highest rate to preserve the quality. The tail end of each sound could be chopped into sections of decreasing sample rates as the frequency content faded to a low rumble. The Concatenator node was then used to play the split files back as one sound. Here is the memory savings made which on the grander scale of a full game would soon add up.
Explosion sound file before: 61.28Kb
Explosion sound file after: 46.93Kb
Gunshot sound file before: 38.85Kb
Gunshot sound file after: 22.21Kb

207_00 Envelopes
The Kismet was used here to manipulate the ambient sound emitting from an engine room of a spacecraft. 3 separate engine sound files were looping and had to have an envelope applied to them to give the sense of life by adding movement and variation. The play sound node was used to play back each sound and the fade in and out, 'variables' were exposed and controlled by a 'random float' which randomly generates fade in and out times using a minimum and maximum value to determine the threshold of in and out times.

208a_00 Kismet [Trigger Volume] take damage and [Oscillator]
In this exercise 3 sound cues referencing the same 1 second triangle wav had to be created in order to make them sound like different pulsating alarm sounds. These cues were then triggered when a generator was shot. 3 generators had to be shot 10 times each to unlock a door in the level and each generator needed its own alarm sound. The 'oscillator' node was introduced which was used to oscillate the pitch of the triangle sound to get the rising and falling sound of the pitch that would be expected from an alarm.

209_00 Muffin Trucker
The goal of this exercise was to take the pre set sounds for a small game and edit them to bring the memory usage down from 22.6 Mb to 500Kb so the game could be used for the mobile market. This task involved using all the techniques learned from the tutorial so far.

All sounds assets were edited which included removing any unnecessary material (cutting any long fade outs/ins) exporting at lower sample rates depending of frequency content and prominence of sound in the level. In doing so the assets were reduced in size from 22.6MB to 2.94 Mb but after importing the assets back into UDK, the on-board file compression tool was used to reduce the package further. This was in the form of OGG compression which then took the package size down to 369.59Kb with little noticeable quality, especially if played on a mobile device. The following video is a before and after ccomparison of this exercise.

Game Level Concept #1

Originally the thought of process was to create a level from the ground up using the building tools and assets provided as part of the UDK game engine. After having played around with some of the tools within UDK it has become apparent this idea is going to take too much time away from achieving the project objectives.

The decision has been made to try and source a level which can adapted for the purposes of the honours project, thus saving valuable time which can be spent on achieving the main goals of the project.

NEXT:

The UDK package comes with some level layouts which could be adapted for the purposes of the honours project so these will be looked at in more detail.

There will also be a search on the web for UDK content created by others which could be of use. Initial sources where such content could be found have been identified in the following websites.

http://www.indiedb.com  (Indi game host)

https://forums.epicgames.com/forum.php  (UDK developers forum)

Game Audio Tutorial - Book - Chapter 2 - Part 2

Exercise Set 2 Continued:

204a_01 Randomized spooky one shot
Here a sound cue was created which played back a random scary sound, this was linked to a 'trigger' event within the kismet. The sound was to be triggered when the player enters the room. It was learned that the sound cue must include an attenuation node to be heard spatially in the game; otherwise the sound will come from everywhere.

204a_02 Randomized spooky ambience
The was based in the same room as the previous exercise only this time the sound cue had a variety of sounds to play back in a random order. Nodes in the sound cue editor used were: 'Random' (which randomly selects one of the sounds that is attached to it). Delay (which puts a random time delay between plays of each sound and is governed by a minimum and maximum time limit). Looping (which tells the sound cue to continually play the sound. Attenuation (Which places the sound in a specified area in the game, the boundaries of this area can be adjusted and can be attenuated over a distance to fade out using volume and low pass filter).

204a_03 : Re-use of randomized [SoundCue]s
For this exercise the ‘non loop’ ambience used in the wooded area 203_00  exercise were re-created using a sound cue. The sound cue could then be placed multiple times at each tree in the location to make it seem like the bird sounds were coming from the tree tops rather than just from all around the area.  

205_00 Layering for Variation
Here a sound cue had to be created to represent varying impacts sounds to be heard when the player attacks a robot. There were 4 different impact sounds and 3 debris sounds set to play back on a slight delay. The 'modulator' node was introduced to add a random pitch variation on each sound played back to reduce repetitive sound. The 'mixer' node was used to combine the impact sound and debris sound at the same time, the final sound cue looked like this.

This sound cue was then activated using a trigger in the kismet window which was also linked to the robots animation of being hit.

206a_00 Broken TV Robot
The 'Concatenator' node was introduced in this exercise which is used in the sound cue editor to play back a sequence of sound files in succession. It was used in this case to piece together a randomised response from a robot after the player tries in interact with it. A single wav file containing the robots response was edited into sections then re-imported, these sections were then set up to play back in a random sequence each time the player interacted with the robot. This method proves useful to both save memory and introduce variation. 

Game Audio Tutorial - Book - Chapter 2 - Part1

After working through chapter one of the Game Audio Tutorials and learning some fundamental knowledge about the basics of UDK audio implementation I moved on to chapter two.

Exercise set 2:

200_00 Sample rate room

In this exercise it was highlighted that audio can be imported at different sample rates in order to reduce memory usage. The the audio samples that were already used in the exercise had to be exported and re-edited using reduced sample rates and cleaner cuts, reducing file size and sample length respectively. For this a frequency analyser (to asses what frequency content was present) and an EQ plug in (low pass filter with steep cut off to preview how the sample rate cut off would sound) were utilised along with an understanding of the Nyquist theory to determine how low the sample rate could be reduced  to save memory whilst retaining audio quality.

201a_00 Multiple use through pitch change

Here another memory saving technique was investigated by making use of a single audio file to be used for various sounds in the same room by using pitch shifting. This included using a machinery ambient sound to represent 3 different sizes of the same machine.

202_00 Multiple use through filters

This exercise was a continuation of the previous only this time as well as using only 1 sample to represent gas escaping from various pipes around the room, there also had to be an application of a low pass filter attenuation. This is where the sound is filtered over a distance using min and max radius settings, this used in conjunction with the volume attenuation over distance reflected a more realistic acoustic environment.

203_00 [AmbientSoundNonLoop] : Wooded area

This was another memory saving exercise where a ‘Non Loop’ actor was introduced. This is a sound playback device that uses multiple samples which are played back randomly in a loop. In this case the sound of birds calling from trees was triggered to simulate a more realistic an non repetitive sound-scape.

204_00 Adding a SoundCue

The 'sound cue' was introduced which is a device similar to the kismet but just for sound and can be used to playback an audio in a variety of ways. This was presented as a more effective and controlled way to play back the looping bird ambience created in the previous exercise. Sound Cues can be placed in multiple locations and edited globally as opposed to the 'Non Loop' actor where if an edit was made to one it would have to be done to all individually. 

Game Audio Tutorial - Book - Chapter one Exercises

In order to understand how game audio is implemented and specifically within UDK I have chosen to use "The Game Audio Tutorial" book as a primary research tool to help gain an insight into the process. It is hoped by understanding the processes involved I can better direct a research question aimed at game audio implementation. 

Here I worked through the first chapter of the book which contains hands on tutorials. 


Exercise set 1
101_00 Changing an [AmbientSoundSimple]
101_01 Adding an [AmbientSoundSimple]
101_02 Importing your own sound
101_03 Looping machinery
102_00 Ambient Sound Radii
103_00 Kismet [Trigger] ‘Used’ to toggle sounds
103_01 Kismet [Trigger] ‘Touch’ to toggle sound
105c_00 [AmbientSoundSimple] Test Area
105d_00 Kismet [Trigger Volume] ‘Touch’ Switching for Occlusion
105d_01 [Trigger Volume]s Exercise
105d_02 Listen


Notes:
Exercise set 1 was mainly centred around getting to grips with the way UDK works with sound on a basic level.
The key elements I learned were as follows:
• Using the UDK content browser to import/access content such as actors/materials/sounds.
• Dealing with ambient sound nodes, here I learned how properties of an ambient sound node work. This included adding the node to the game map, understanding the attenuation radius marker (how the sound is heard over a distance), changing the sound that is referenced by the node using the content browser. 
• Using multiple instances of the same sound node to fill a room with sound whilst avoiding sound spilling into other areas. I learned from this exercise that sound projected by a node is unimpeded by walls, floors and ceilings and by using more than one instance of the sound a greater control can be gained on the sound spilling into other areas. Although this comes at the cost of using more voices.
• Using a trigger actor in conjunction with the kismet control panel (a flow chart that sets up series of commands for directing specific action within the game). This included using the triggers within the kismet to toggle between an outside ambient environment sound to an outside ambient sound as the player walks between them.


Basic understanding of the game engine was gained from this task. Next step towards strengthening my skills and understanding in this area is to complete the next chapter of exercises.