Tag Archives: Oculus Rift

Editing 360 Video in VR (Part 2)

By Mike McCarthy

In the last article I wrote on this topic, I looked at the options for shooting 360-degree video footage, and what it takes to get footage recorded on a Gear 360 ready to review and edit on a VR-enabled system. The remaining steps in the workflow will be similar regardless of which camera you are using.

Previewing your work is important so, if you have a VR headset you will want to make sure it is installed and functioning with your editing software. I will be basing this article on using an Oculus Rift to view my work in Adobe Premiere Pro 11.1.2 on a Thinkpad P71 with an Nvidia Quadro P5000 GPU. Premiere requires an extra set of plugins to interface to the Rift headset. Adobe acquired Mettle’s Skybox VR Player plugin back in June, and has made it available to Creative Cloud users upon request, which you can do here.

Skybox VR player

Skybox can project the Adobe UI to the Rift, as well as the output, so you could leave the headset on when making adjustments, but I have not found that to be as useful as I had hoped. Another option is to use the GoPro VR Player plugin to send the Adobe Transmit output to the Rift, which can be downloaded for free here (use the 3.0 version or above). I found this to have slightly better playback performance, but fewer options (no UI projection, for example). Adobe is expected to integrate much of this functionality into the next release of Premiere, which should remove the need for most of the current plugins and increase the overall functionality.

Once our VR editing system is ready to go, we need to look at the footage we have. In the case of the Gear 360, the dual spherical image file recorded by the camera is not directly usable in most applications and needs to be processed to generate a single equirectangular projection, stitching the images from both cameras into a single continuous view.

There are a number of ways to do this. One option is to use the application Samsung packages with the camera: Action Director 360. You can download the original version here, but will need the activation code that came with the camera in order to use it. Upon import, the software automatically processes the original stills and video into equirectangular 2:1 H.264 files. Instead of exporting from that application, I pull the temp files that it generates on media import, and use them in Premiere. (C:\Users\[Username]\Documents\CyberLink\ActionDirector\1.0\360) is where they should be located by default. While this is the simplest solution for PC users, it introduces an extra transcoding step to H.264 (after the initial H.265 recording), and I frequently encountered an issue where there was a black hexagon in the middle of the stitched image.

Action Director

Activating Automatic Angle Compensation in the Preferences->Editing panel gets around this bug, while trying to stabilize your footage to some degree. I later discovered that Samsung had released a separate Version 2 of Action Director available for Windows or Mac, which solves this issue. But I couldn’t get the stitched files to work directly in the Adobe apps, so I had to export them, which was yet another layer of video compression. You will need a Samsung activation code that came with the Gear 360 to use any of the versions, and both versions took twice as long to stitch a clip as its run time on my P71 laptop.

An option that gives you more control over the stitching process is to do it in After Effects. Adobe’s recent acquisition of Mettle’s SkyBox VR toolset makes this much easier, but it is still a process. Currently you have to manually request and install your copy of the plugins as a Creative Cloud subscriber. There are three separate installers, and while this stitching process only requires Skybox Suite AE, I would install both the AE and Premiere Pro versions for use in later steps, as well as the Skybox VR player if you have an HMD to preview with. Once you have them installed, you can use the Skybox Converter effect in After Effects to convert from the Gear 360’s fisheye files to the equirectangular assets that Premiere requires for editing VR.

Unfortunately, Samsung’s format is not one of the default conversions supported by the effect, so it requires a little more creativity. The two sensor images have to be cropped into separate comps and with plugin applied to each of them. Setting the Input to fisheye and the output to equirectangular for each image will give the desired distortion. A feathered mask applied to the circle to adjust the seam, and the overlap can be adjusted with the FOV and re-orient camera values.

Since this can be challenging to setup, I have posted an AE template that is already configured for footage from the Gear 360. The included directions should be easy to follow, and the projection, overlap and stitch can be further tweaked by adjusting the position, rotation and mask settings in the sub-comps, and the re-orientation values in the Skybox Converter effects. Hopefully, once you find the correct adjustments for your individual camera, they should remain the same for all of your footage, unless you want to mask around an object crossing the stitch boundary. More info on those types of fixes can be found here. It took me five minutes to export 60 seconds of 360 video using this approach, and there is no stabilization or other automatic image analysis.

Video Stitch Studio

Orah makes Video-Stitch Studio, which is a similar product but with a slightly different feature set and approach. One limitation I couldn’t find a way around is that the program expects the various fisheye source images to be in separate files, and unlike AVP I couldn’t get the source cropping tool to work without rendering the dual fisheye images into separate square video source files. There should be a way to avoid that step, but I couldn’t find one. (You can use the crop effect to remove 1920 pixels on one side or the other to make the conversions in Media Encoder relatively quickly.) Splitting the source file and rendering separate fisheye spheres adds a workflow step and render time, and my one-minute clip took 11 minutes to export. This is a slower option, which might be significant if you have hours of footage to process instead of minutes.

Clearly, there are a variety of ways to get your raw footage stitched for editing. The results vary greatly between the different programs, so I made video to compare the different stitching options on the same source clip. My first attempt was with a locked-off shot in the park, but that shot was too simple to see the differences, and it didn’t allow for comparison of the stabilization options available in some of the programs. I shot some footage from a moving vehicle to see how well the motion and shake would be handled by the various programs. The result is now available on YouTube, fading between each of the five labeled options over the course of the minute long clip. I would categorize this as testing how well the various applications can handle non-ideal source footage, which happens a lot in the real world.

I didn’t feel that any of the stitching options were perfect solutions, so hopefully we will see further developments in that regard in the future. You may want to explore them yourself to determine which one best meets your needs. Once your footage is correctly mapped to equirectangular projection, ideally in a 2:1 aspect ratio, and the projects are rendered and exported (I recommend Cineform or DNxHR), you are ready to edit your processed footage.

Launch Premiere Pro and import your footage as you normally would. If you are using the Skybox Player plugin, turn on Adobe Transmit with the HMD selected as the only dedicated output (in the Skybox VR configuration window, I recommend setting the hot corner to top left, to avoid accidentally hitting the start menu, desktop hide or application close buttons during preview). In the playback monitor, you may want to right click the wrench icon and select Enable VR to preview a pan-able perspective of the video, instead of the entire distorted equirectangular source frame. You can cut, trim and stack your footage as usual, and apply color corrections and other non-geometry-based effects.

In version 11.1.2 of Premiere, there is basically one VR effect (VR Projection), which allows you to rotate the video sphere along all three axis. If you have the Skybox Suite for Premiere installed, you will have some extra VR effects. The Skybox Rotate Sphere effect is basically the same. You can add titles and graphics and use the Skybox Project 2D effect to project them into the sphere where you want. Skybox also includes other effects for blurring and sharpening the spherical video, as well as denoise and glow. If you have Kolor AVP installed that adds two new effects as well. GoPro VR Horizon is similar to the other sphere rotation ones, but allows you to drag the image around in the monitor window to rotate it, instead of manually adjusting the axis values, so it is faster and more intuitive. The GoPro VR Reframe effect is applied to equirectangular footage, to extract a flat perspective from within it. The field of view can be adjusted and rotated around all three axis.

Most of the effects are pretty easy to figure out, but Skybox Project 2D may require some experimentation to get the desired results. Avoid placing objects near the edges of the 2D frame that you apply it to, to keep them facing toward the viewer. The rotate projection values control where the object is placed relative to the viewer. The rotate source values rotate the object at the location it is projected to. Personally, I think they should be placed in the reverse order in the effects panel.

Encoding the final output is not difficult, just send it to Adobe Media Encoder using either H.264 or H.265 formats. Make sure the “Video is VR” box is checked at the bottom of the Video Settings pane, and in this case that the frame layout is set to monoscopic. There are presets for some of the common framesizes, but I would recommend lowering the bitrates, at least if you are using Gear 360 footage. Also, if you have ambisonic audio set channels to 4.0 in the audio pane.

Once the video is encoded, you can upload it directly to Facebook. If you want to upload to YouTube, exports from AME with the VR box checked should work fine, but for videos from other sources you will need to modify the metadata with this app here.  Once your video is uploaded to YouTube, you can embed it on any webpage that supports 2D web videos. And YouTube videos can be streamed directly to your Rift headset using the free DeoVR video player.

That should give you a 360-video production workflow from start to finish. I will post more updated articles as new software tools are developed, and as I get new 360 cameras with which to test and experiment.


Mike McCarthy is an online editor/workflow consultant with 10 years of experience on feature films and commercials. He has been involved in pioneering new solutions for tapeless workflows, DSLR filmmaking and multi-screen and surround video experiences. Check out his site.

VR Audio: Crytek goes to new heights for VR game ‘The Climb’

By Jennifer Walden

Dealing with locomotion, such as walking and especially running, is a challenge for VR content developers — but what hasn’t been a challenge in creating VR content? Climbing, on the other hand, has proved to be a simple, yet interesting, locomotion that independent game developer Crytek found to be sustainable for the duration of a full-length game.

Crytek, known for the Crysis game series, recently released their first VR game title, The Climb, a rock climbing adventure exclusively for the Oculus Rift. Players climb, swing and jump their way up increasingly difficult rock faces modeled after popular climbing destinations in places like Indonesia, the Grand Canyon and The Alps.

Crytek’s director of audio, Simon Pressey, says their game engine, CryEngine, is capable of UltraHD resolutions higher than 8K. They could have taken GPS data of anywhere in the world and turned that into a level on The Climb. “But to make the climbing interesting and compelling, we found that real geography wasn’t the way to go. Still, we liked the idea of representing different areas of the world,” he says. While the locations Crytek designed aren’t perfect geographical imitations, geologically they’re pretty accurate. “The details of how the rocks look up close — the color, the graininess and texture — they are as close to photorealistic as we can get in the Oculus Rift. We are running at a resolution that the Rift can handle. So how detailed it looks depends on the Rift’s capabilities.”

Keep in mind that this is first-generation VR technology. “It’s going to get better,” promises Pressey. “By the third-generation of this, I’m sure we’ll have visuals you can’t tell apart from reality.”

Simon Pressey

Simon Pressey

The Sound Experience
Since the visuals aren’t perfect imitations of reality, the audio is vital for maintaining immersion and supporting the game play. Details in the audio actually help the brain process the visuals faster. Even still, flaws and all, first-gen VR headsets give the player a stronger connection to his/her actions in-game than was previously possible with traditional 2D (flat screen) games. “You can look away from the screen in a traditional game, but you can’t in VR. When you turn around in The Climb, you can see a thousand feet below you. You can see that it’s a long way down, and it feels like a long way down.”

One key feature of the Oculus Rift is the integrated audio — it comes equipped with headphones. For Pressey, that meant knowing the exact sound playback system of the end user, a real advantage from a design and mix standpoint. “We were designing for a known playback variable. We knew that it would be a binaural experience. Early on we started working with the Oculus-provided 3D encoder plug-in for Audiokinetic’s Wwise, which Oculus includes with their audio SDK. That plug-in provides HRTF binaural encoding, adding the z-axis that you don’t normally experience even with surround sound,” says Pressey.

He explains that the sounds start as mono source-points, positioned in a 3D space using middleware like Wwise. Then, using the Oculus audio SDK via the middleware, those audio signals are being downmixed to binaural stereo, which gets HRTF (head related transfer function) processing, adding a spatialized effect to the sounds. So even though the player is listening through two speakers, he/she perceives sounds as coming from the left, the right, in front, behind, above and below.

Since most VR is experienced with headphones, Pressey feels there is an opportunity to improve the binaural presentation of the audio [i.e., better headphones or in-ear monitors], and to improve 3D positional audio with personalized HRTFs and Ambisonics. “While the visuals are still very apparently a representation of reality, the audio is perceived as realistic, even if it is a totally manufactured reality. The headphone environment is very intimate and allows greater use of dynamic range, so subtle mixes and more realistic recordings and rendering are sort of mandatory.”

Realistic Sound
Pressey leads the Crytek audio team, and together they collaborated on The Climb’s audio design, which includes many different close-up hand movements and grabs that signify the quality of the player’s grip. There are sweaty, wet sounding hand grabs. There are drier, firmer hand grabs for when a player’s hands are freshly chalked. There are rock crumbles for when holds crumble away.

At times a player needs to wipe dirt away from a hold, or brush aside vegetation. These are very subtle details that in most games wouldn’t be sounded, says Pressey. “But in VR, we are going into very subtle detail. Like, when you rub your hands over plants searching for grips, we are following your movement speed to control how much sound it makes as you ruffle the leaves.” It’s that level of detail that makes the immersion work. Even though in real life a sound so small would probably be masked by other environmental sounds, in the intimacy of VR, those sounds engage the player in the action of climbing.

Crytek_TheClimb_Asia_Screenshot4

Breathing and heartbeat elements also pull a player into the game experience. After moving through several holds, a player’s hands get sweaty, and the breathing sound becomes more labored. If the hold crumbles or if a player is losing his/her grip, the audio design employs a heartbeat sound. “It is not like your usual game situation where you hear a heartbeat if you have low health. In The Climb you actually think, “I’ve got to jump!” Your heart is racing, and after you make the jump and chalk your hands, then your heartbeat and your breathing slow down, and you physically relax,” he says.

Crytek’s aim was to make The Climb believable, to have realistic qualities, dynamic environments and a focused sound to mimic the intensity of focus felt when concentrating on important life or death decisions. They wanted the environment sounds to change, such as the wind changing as a player moves around a corner. But, they didn’t want to intentionally draw the player’s attention away from climbing.

For example, there’s a waterfall near one of the climbs, and the sound for it plays subtly in the background. If the player turns to look at it, then the waterfall sound fades up. They are able to focus the player’s attention by attenuating non-immediate sounds. “You don’t want to hear that waterfall as the focus of your attention and so we steer the sound. But, if that is what you’re focusing on, then we want to be more obvious,” explains Pressey.

The Crytek audio team

The Crytek audio team

The Crytek audio team records, designs and edits sounds in Steinberg’s Nuendo 7, which works directly with Audiokinetic’s Wwise middleware that connects directly to the CryEngine. The audio team, which has been working this way for the past two years, feels the workflow is very iterative, with the audio flowing easily in that pipeline from Nuendo 7 to Wwise to CryEngine and back again. They are often able to verify the audio in-game without needing to request code support. If a sound isn’t working in-game, it can be tweaked in Wwise or completely reworked in Nuendo. All aspects of the pipeline are version controlled and built for sharing work across the audio team.

“It’s a really tight workflow and we can do things quickly. In the game world, speed is everything,” says Pressey. “The faster you get your game to market the sooner you recoup on your very heavy R&D.”

Two factors that propelled this workflow are the collaboration between Crytek, Audiokinetic and Steinberg in designing software tailored to the specific needs of game audio pros, and Crytek’s overhaul of CryEngine where they removed the integrated FMOD-based audio engine in favor of using an external audio engine. Running the audio engine separate from the game engine not only improves the game engine efficiency, it also allows updates to the audio engine as needed without fear of breaking the game engine.

Within hours of Wwise releasing an update, for example, Pressey says their system can be up to date. “Previously, it could’ve been a long and complicated process to incorporate the latest updates. There was always the risk of crashing the whole system by making a change because the code was so mixed up with the rest of the system. By separating them we can always be running the latest versions of things without risking anything.”

Having that adaptability is essential for VR content creation since the industry is changing all the time. For example, Sony’s PS4 VR headset release is slated for this fall, so they’re releasing a new SDK about every week or so, according to Pressey.

CryEngine is freely available for anyone to use. VR games developed with CryEngine will work for any VR platform. CryEngine is also audio middleware agnostic, meaning it can talk to any audio middleware, be it Wwise, FMOD or proprietary middleware. Users can choose a workflow that best suits the needs of their game.

Pressey finds creating for VR to be an intensely experimental process, for every discipline involved in game development. While most members on the Crytek team have solved problems relating to a new IP or a new console, Pressey says, “We were not prepared for this amount of new. We were all used to knowing what we were doing, and now we are experimenting with no net to fall back on. The experience is surprisingly different; the interaction using your eye and head tracking is much more physical. It is more intimate. There is an undeniable and inescapable immersion, in that you can’t look away as the game world is all around you. You can’t switch off your ears.” The first time Pressey put on a VR headset, he knew there was no going back. “Before that, I had no real idea. It is the difference between reading about a country and visiting it.”

Upcoming Release
Crytek will be presenting a new VR release titled Robinson — The Journey at E3 this month, and Pressey gives us a few hints as to what the game experience might be like. He says that VR offers new ways of storytelling, such as nonlinear storytelling. “Crytek and the CryEngine team have developed a radically new Dynamic Response System to allow the game to be intelligent in what dialog gets presented to the player at what time. Aspects of a story can be sewn together and presented based on the player’s approach to the game. This technology takes the idea of RPG-like branching storylines to a new level, and allows narrative progression in what I hope will be new and exciting territory for VR.”

The Climb uses this Dynamic Response System in a limited capacity during the tutorial where the instructor is responsive to the player’s actions. “Previously, to be that responsive, a narrative designer or level designer would have to write pages of logic to do what our new system does very simply,” concludes Pressey.

Jennifer Walden is an audio engineer and writer based in New Jersey.

Dell embraces VR via Precision Towers

It’s going to be hard to walk the floor at NAB this year without being invited to demo some sort of virtual reality experience. More and more companies are diving in and offering technology that optimizes the creation and viewing of VR content. Dell is one of the latest to jump in.

Dell has been working closely on this topic with their hardware and software partners, and are formalizing their commitment to the future of VR by offering solutions that are optimized for VR consumption and creation alongside the mainstream professional ISV apps used by industry pros.

Dell has introduced new, recommended minimum system hardware configurations to support an optimal VR experience for pro users with HTC Vive or Oculus Rift VR solutions. The VR-ready solutions feature a set of three criteria, whether users are consuming or creating VR content; minimum CPU, memory and graphics requirements to support VR viewing experiences; graphics drivers that are qualified to work with these solutions; and pass performance tests conducted by the company using test criteria based on HMD (head-mounted display) suppliers, ISVs or third-party benchmarks.

Dell has also made upgrades to their Dell Precision Tower, including increased performance, graphics and memory for VR content creation. The refreshed Dell Precision Tower 5810, 7810 and 7910 workstations and rack 7910 have been upgraded with new Intel Broadwell EP processors that have more cores and performance for multi-threaded applications that support professional modeling, analysis and calculations.

Additional upgrades include the latest pro graphics technology from AMD and Nvidia, Dell Precision Ultra-Speed PCle drives with up to 4x faster performance than traditional SATA SSD storage, and up to 1TB of DDR4 Memory running at 2400MHz speed.

The Foundry updates Nuke Studio, tackles VR/AR content at NAB

The Foundry has updated Nuke Studio, its node-based VFX, editorial and finishing studio for creative individuals and teams. New features include multiple overlay tracks and blending modes in the timeline; enhanced audio editorial tools that let users view an audio track’s waveform within the timeline and apply cross-fades and fade in/out to audio tracks using handles; and XML and AAF support for grades, nonlinear retimes, transforms and cropping, which enables automatic creation of external editorial effects within the application.

The Foundry has also added a burn-in soft effect to the application’s realtime in-timeline effects, allowing users to add review information directly in the timeline for use during review. Finally, Nuke Studio and the Nuke and NukeX Flipbook now have native support for stereo playback — both in the application and through SDI-out.

At the same time, the company is developing technologies to address specific challenges in virtual and augmented reality (VR/AR) content creation. These technologies are based on internal R&D, technology partnerships, and collaboration with more than 30 media and entertainment companies.

Specifically, The Foundry has worked with its VFX clients on experimental new tools and workflows to manipulate and composite live-action VR using Nuke. Developments include calibration and stitching of live-action 360-degree footage from multicamera rigs; live connection to Oculus Rift to review stitching, grading, and depth; support for compositing with ray-trace rendering for CG placement and projections; and spherical-aware operations and viewing with equirectangular images.

The Foundry will preview these experimental new VR/AR content creation technologies at the 2015 NAB Show as part of a live stream on Tuesday, April 14, from 10 a.m. to 2 p.m. PDT. The company will also demonstrate Nuke Studio’s new features during that streaming session.

Scratch Play V.8.2 offers support for Oculus Rift, new formats, cameras

Assimilate’s Scratch Play V.8.2, a free media player for pro and web formats, has added support for native realtime playback of cinematic content created for Oculus Rift – 360 virtual reality viewing.

Scratch Play v8.2, which is available immediately as a free download at www.assimilateinc.com, also supports several new cameras and formats, including the Blackmagic Pocket Camera, Magic Lantern support for Canon DSLRs, Panasonic RAW (vrw) and a performance boost for XAVC 4K.

Scratch Play also supports Cylindrical, Equirectangular and Cube formats for output to a secondary monitor or an Oculus Rift DKI or DKII. With a Rift connected, these formats are mapped in realtime to Rift output, with full head-tracking support.

Scratch Play will output in 360 mode with just a normal secondary display/monitor acting as a window into the virtual world, with realtime ability to pan, track and tilt around the sphere. This means that developers creating cinematic content for the Rift now have a playback tool that allows them to instantly review VR shots in realtime — whether or not they have a Rift headset.

“Scratch Play has increased my productivity when comparing footage and speeding up iterations for Cinematic VR,” says Greg Downing, CTO of xRez Studio in Santa Monica. “Being able to do A/B comparisons quickly, live in the Rift headset, is an effective way to rapidly evaluate your work and decide if you are going in the right direction.

“Another feature I really like that speeds up the workflow is the ability to read frame sequences rather than having to go through the additional steps of encoding a video,” he says. “Short-cutting the video encoding step when working with CGI, at this resolution, is a big time saver.”

Scratch Play Premium, the ad-free version of Play, is available with annual subscription of $5 per year.

Animation house Reel FX dives into VR with new division

With a groundswell moving toward more immersive content, animation studio Reel FX, with offices in Santa Monica and Dallas, has opened a virtual reality division, Reel FX VR. This new entity will create content for virtual reality platforms, including the Samsung Gear VR Innovator Edition. The VR division is based in the studio’s Dallas office.

The division is focused on client work for major studios and commercial agencies. The idea for Reel FX VR was born after the release of multiple high-profile VR projects in early 2014 ranging from content for Chuck E. Cheese to a piece for Legendary Picture’s Pacific Rim, which debuted at 2014 ComicCon. More content debuted at IFA in Berlin in early September, including a mobile version of the Pacific Rim Jaeger Pilot experience and a stereo trailer for Reel FX’s own The Book of Life.

Reel FX’s history and relationship with Oculus Rift dates back to Oculus’ 2012 Kickstarter campaign. Founder and ECD Dale Carman was an early adopter of the technology and one of the first to begin developing for the platform.

“As a studio, we are passionate about finding, using, and pushing technology,” said Dale Carman, founder/executive creative director of Reel FX. “I knew the moment I encountered Oculus [Rift technology] on Kickstarter that it was magic. Creating these experiences for clients like Legendary and our own films and working closely with Samsung and Oculus has been a great journey so far, and we are excited about the future.”

ReelFX_VR

The relationship with Oculus and expertise in the technology has made Reel FX VR one of Oculus Rift’s top developers. “Legendary is always seeking out new entertainment frontiers and technology that can offer our fans a greater level of immersion,” added Emily Castel, chief marketing officer for Legendary Pictures. “Reel FX was a fantastic partner and their VR team did a terrific job at bringing the transportive universe of Pacific Rim to life.”

In preparation of the Reel FX VR launch, Reel FX has brought on multiple team members over the past year that will focus on and grow the VR business. Reel FX VR will be overseen by Carman, GM Keith McCabe and executive producer Gary Banks. Veteran interactive executive Dan Ferguson joined the team in late 2013 as director of digital interactive. Rounding out the unit is interactive account manager Tina Ghezzi. Artists from Reel FX’s feature and commercial crews will lead the creative teams.

Main Image:  Guillermo del Toro takes in Pacific Rim’s Oculus experience at Comic Con.