By Jennifer Walden
Dealing with locomotion, such as walking and especially running, is a challenge for VR content developers — but what hasn’t been a challenge in creating VR content? Climbing, on the other hand, has proved to be a simple, yet interesting, locomotion that independent game developer Crytek found to be sustainable for the duration of a full-length game.
Crytek, known for the Crysis game series, recently released their first VR game title, The Climb, a rock climbing adventure exclusively for the Oculus Rift. Players climb, swing and jump their way up increasingly difficult rock faces modeled after popular climbing destinations in places like Indonesia, the Grand Canyon and The Alps.
Crytek’s director of audio, Simon Pressey, says their game engine, CryEngine, is capable of UltraHD resolutions higher than 8K. They could have taken GPS data of anywhere in the world and turned that into a level on The Climb. “But to make the climbing interesting and compelling, we found that real geography wasn’t the way to go. Still, we liked the idea of representing different areas of the world,” he says. While the locations Crytek designed aren’t perfect geographical imitations, geologically they’re pretty accurate. “The details of how the rocks look up close — the color, the graininess and texture — they are as close to photorealistic as we can get in the Oculus Rift. We are running at a resolution that the Rift can handle. So how detailed it looks depends on the Rift’s capabilities.”
Keep in mind that this is first-generation VR technology. “It’s going to get better,” promises Pressey. “By the third-generation of this, I’m sure we’ll have visuals you can’t tell apart from reality.”
The Sound Experience
Since the visuals aren’t perfect imitations of reality, the audio is vital for maintaining immersion and supporting the game play. Details in the audio actually help the brain process the visuals faster. Even still, flaws and all, first-gen VR headsets give the player a stronger connection to his/her actions in-game than was previously possible with traditional 2D (flat screen) games. “You can look away from the screen in a traditional game, but you can’t in VR. When you turn around in The Climb, you can see a thousand feet below you. You can see that it’s a long way down, and it feels like a long way down.”
One key feature of the Oculus Rift is the integrated audio — it comes equipped with headphones. For Pressey, that meant knowing the exact sound playback system of the end user, a real advantage from a design and mix standpoint. “We were designing for a known playback variable. We knew that it would be a binaural experience. Early on we started working with the Oculus-provided 3D encoder plug-in for Audiokinetic’s Wwise, which Oculus includes with their audio SDK. That plug-in provides HRTF binaural encoding, adding the z-axis that you don’t normally experience even with surround sound,” says Pressey.
He explains that the sounds start as mono source-points, positioned in a 3D space using middleware like Wwise. Then, using the Oculus audio SDK via the middleware, those audio signals are being downmixed to binaural stereo, which gets HRTF (head related transfer function) processing, adding a spatialized effect to the sounds. So even though the player is listening through two speakers, he/she perceives sounds as coming from the left, the right, in front, behind, above and below.
Since most VR is experienced with headphones, Pressey feels there is an opportunity to improve the binaural presentation of the audio [i.e., better headphones or in-ear monitors], and to improve 3D positional audio with personalized HRTFs and Ambisonics. “While the visuals are still very apparently a representation of reality, the audio is perceived as realistic, even if it is a totally manufactured reality. The headphone environment is very intimate and allows greater use of dynamic range, so subtle mixes and more realistic recordings and rendering are sort of mandatory.”
Pressey leads the Crytek audio team, and together they collaborated on The Climb’s audio design, which includes many different close-up hand movements and grabs that signify the quality of the player’s grip. There are sweaty, wet sounding hand grabs. There are drier, firmer hand grabs for when a player’s hands are freshly chalked. There are rock crumbles for when holds crumble away.
At times a player needs to wipe dirt away from a hold, or brush aside vegetation. These are very subtle details that in most games wouldn’t be sounded, says Pressey. “But in VR, we are going into very subtle detail. Like, when you rub your hands over plants searching for grips, we are following your movement speed to control how much sound it makes as you ruffle the leaves.” It’s that level of detail that makes the immersion work. Even though in real life a sound so small would probably be masked by other environmental sounds, in the intimacy of VR, those sounds engage the player in the action of climbing.
Breathing and heartbeat elements also pull a player into the game experience. After moving through several holds, a player’s hands get sweaty, and the breathing sound becomes more labored. If the hold crumbles or if a player is losing his/her grip, the audio design employs a heartbeat sound. “It is not like your usual game situation where you hear a heartbeat if you have low health. In The Climb you actually think, “I’ve got to jump!” Your heart is racing, and after you make the jump and chalk your hands, then your heartbeat and your breathing slow down, and you physically relax,” he says.
Crytek’s aim was to make The Climb believable, to have realistic qualities, dynamic environments and a focused sound to mimic the intensity of focus felt when concentrating on important life or death decisions. They wanted the environment sounds to change, such as the wind changing as a player moves around a corner. But, they didn’t want to intentionally draw the player’s attention away from climbing.
For example, there’s a waterfall near one of the climbs, and the sound for it plays subtly in the background. If the player turns to look at it, then the waterfall sound fades up. They are able to focus the player’s attention by attenuating non-immediate sounds. “You don’t want to hear that waterfall as the focus of your attention and so we steer the sound. But, if that is what you’re focusing on, then we want to be more obvious,” explains Pressey.
The Crytek audio team records, designs and edits sounds in Steinberg’s Nuendo 7, which works directly with Audiokinetic’s Wwise middleware that connects directly to the CryEngine. The audio team, which has been working this way for the past two years, feels the workflow is very iterative, with the audio flowing easily in that pipeline from Nuendo 7 to Wwise to CryEngine and back again. They are often able to verify the audio in-game without needing to request code support. If a sound isn’t working in-game, it can be tweaked in Wwise or completely reworked in Nuendo. All aspects of the pipeline are version controlled and built for sharing work across the audio team.
“It’s a really tight workflow and we can do things quickly. In the game world, speed is everything,” says Pressey. “The faster you get your game to market the sooner you recoup on your very heavy R&D.”
Two factors that propelled this workflow are the collaboration between Crytek, Audiokinetic and Steinberg in designing software tailored to the specific needs of game audio pros, and Crytek’s overhaul of CryEngine where they removed the integrated FMOD-based audio engine in favor of using an external audio engine. Running the audio engine separate from the game engine not only improves the game engine efficiency, it also allows updates to the audio engine as needed without fear of breaking the game engine.
Within hours of Wwise releasing an update, for example, Pressey says their system can be up to date. “Previously, it could’ve been a long and complicated process to incorporate the latest updates. There was always the risk of crashing the whole system by making a change because the code was so mixed up with the rest of the system. By separating them we can always be running the latest versions of things without risking anything.”
Having that adaptability is essential for VR content creation since the industry is changing all the time. For example, Sony’s PS4 VR headset release is slated for this fall, so they’re releasing a new SDK about every week or so, according to Pressey.
CryEngine is freely available for anyone to use. VR games developed with CryEngine will work for any VR platform. CryEngine is also audio middleware agnostic, meaning it can talk to any audio middleware, be it Wwise, FMOD or proprietary middleware. Users can choose a workflow that best suits the needs of their game.
Pressey finds creating for VR to be an intensely experimental process, for every discipline involved in game development. While most members on the Crytek team have solved problems relating to a new IP or a new console, Pressey says, “We were not prepared for this amount of new. We were all used to knowing what we were doing, and now we are experimenting with no net to fall back on. The experience is surprisingly different; the interaction using your eye and head tracking is much more physical. It is more intimate. There is an undeniable and inescapable immersion, in that you can’t look away as the game world is all around you. You can’t switch off your ears.” The first time Pressey put on a VR headset, he knew there was no going back. “Before that, I had no real idea. It is the difference between reading about a country and visiting it.”
Crytek will be presenting a new VR release titled Robinson — The Journey at E3 this month, and Pressey gives us a few hints as to what the game experience might be like. He says that VR offers new ways of storytelling, such as nonlinear storytelling. “Crytek and the CryEngine team have developed a radically new Dynamic Response System to allow the game to be intelligent in what dialog gets presented to the player at what time. Aspects of a story can be sewn together and presented based on the player’s approach to the game. This technology takes the idea of RPG-like branching storylines to a new level, and allows narrative progression in what I hope will be new and exciting territory for VR.”
The Climb uses this Dynamic Response System in a limited capacity during the tutorial where the instructor is responsive to the player’s actions. “Previously, to be that responsive, a narrative designer or level designer would have to write pages of logic to do what our new system does very simply,” concludes Pressey.
Jennifer Walden is an audio engineer and writer based in New Jersey.