Tag Archives: Doug Liman

VR Production: A roadmap for stereo 360, AR, VR and beyond

By Beth Marchant

It may still be the Wild West in the emerging virtual reality market, but adapting new and existing tools to recreate production workflows is nothing new for the curious and innovative filmmakers hungry for expanding ways to tell stories.

We asked directors at a large VR studio and at a nimble startup how they are navigating the formats, gear and new pipelines that come with the territory.

Patrick Meegan

Jaunt
Patrick Meegan was the first VR-centric filmmaker hired by Jaunt, a prolific producer of immersive content based in Los Angeles. Now a creative director and director of key content for the company, he will also be helping Jaunt refine and revamp its virtual reality app in the coming months. “I came straight from my MFA at USC’s interactive media program to Jaunt, so I’ve been doing VR since day one there. The nice thing about USC is it has a very robust research lab associated with the film school. I worked with a lot of prototype VR technology while completing my degree and shooting my thesis. I pretty much had a hacker mentality in graduate school but I wanted to work with an engineering and content company that was streamlining the VR process, and I found it here.”

Meegan shot with a custom camera system built with GoPro cameras on those first Jaunt shoots. “They had developed a really nice automated VR stitching and post workflow early on,” he says, “but I’d built my own 360 camera from 16 GoPros in grad school, so it wasn’t so dissimilar from what I was used to.” He’s since been shooting with the company’s purpose-built Jaunt One camera, a ground-up, modular design that includes a set of individual modules optimized with desirable features like global shutter, gunlock for frame sync and improved dynamic range.

Focusing primarily on live-action 3D spherical video but publishing across platforms, Jaunt has produced a range of VR experiences to date that include Doug Limon’s longer-form cinematic serial Invisible, (see VR Post) and short documentaries like Greenpeace’s A Journey to the Arctic and Camp4 Collective’s Home Turf: Iceland. The content is stored in the cloud, mostly to take advantage of scalable cloud-based rendering. “We’re always supporting every platform that’s out there but within the last year, to an increasing degree, we’re focusing more on the more fully immersive Oculus, HTC Vive, Gear VR and Google Daydream experiences,” says Meegan. “We’re increasingly looking at the specs and capabilities of those more robust headsets and will do more of that in 2017. But right now, we’re focused on the core market, which is 360 video.”

invisible

Invisible

When out on the VR directing jobs he bids on through Jaunt’s studios, Meegan typically shoots with a Jaunt One as his primary tool and rotates in other bespoke camera arrays as needed. “We’re still in a place where there is no one camera but many terrific options,” he says. “Jaunt One is a great baseline. But if you want to shoot at night or do aerial, you’ll need to consider any number of custom rigs that blend off-the-shelf cameras and components in different types of arrays. Volumetric and light field video are also on the horizon, as the headsets enable more interaction with the audience. So we’ll continue to work with a range of camera systems here at Jaunt to achieve those things.”

Meegan recently took the Jaunt One and a GoPro drone array to the Amazon Rain Forest to shoot a 10-minute 360-degree film for Conservation International, a non-profit organization with a trifold community, corporate partnership and research approach to saving our planet’s natural resources. An early version of the film screened this November in Marrakech during the UN’s Climate Change Conference and will be in wide release through the Jaunt app in January. “I’ve been impressed that there are real budgets out there for cause-based VR documentaries,” he says. “It’s a wonderful thing to infuse in the medium early on, as many did with HD and then 4K. Escaping into a nature-based experience is not an isolating thing — it’s very therapeutic, and most people will never have the means or inclination to go to these places in the first place.”

Pitched as a six-minute documentary, the piece showcases a number of difficult VR camera moves that ended up extending its run. “When we submitted 10-minute cuts to the clients, no one complained about length,” says Meegan. “They just wanted more. Probably half the piece is in motion. We did a lot of cable cams through the jungle, as if you are walking with the indigenous people who live there and encountering wildlife, and also a number of VR firsts, like vertical ascending and descending shots up along these massive trees.”

Tree climbing veterans from shows like Planet Earth were on hand to help set the rigs on high. “These were shots that would take three days to rig into a tree so we could deliver that magical float down through the layers of the forest with the camera. Plus, everything we had to bring into the jungle for the shoot had to fit on tiny planes and canoes. Due to weight limits, we had to cut back on some of the grip equipment we’d originally planned on bringing, like custom cases and riggings to carry and protect the gear from wildlife and the elements. We had to measure everything out to the gram.” Jaunt also customized the cable cam motors to slow down the action of the rigs. “In VR you want to move a lot slower than with a traditional camera so you get a comfortable feel of movement,” says Meegan. “Because people are looking around within the environment, you want to give them time to soak it all in.”

An example of the Jaunt camera at work – Let’s Go Mets!

The isolated nature of the shoot posed an additional challenge: how to keep the cameras rolling, with charging stations, for eight hours at a time. “We did a lot on the front end to determine the best batteries and data storage systems to make that happen,” he says. “And there were many lessons learned that we will start to apply to upcoming work. The post production was more about rig removal and compositing and less about stitching, so for these kinds of documentary shoots, that helps us put more of our resources into the production.”

The future of narrative VR, on the other hand, may have an even steeper learning curve. “What ‘Invisible’ starts to delve into,” explains Meegan, “is how do we tell a more elaborate, longer-form story in VR? Flash back to a year or so ago, when all we thought people could handle in the headset at one time was five or six minutes. At least as headsets get more comfortable — and eventually become untethered — people will become more engaged.” That wire, he believes, is one of VR’s biggest current drawbacks. “Once it goes away, and viewers are no longer reminded they are actually wearing technology, we can finally start to envision longer-form stories.”

As VR production technology matures, Meegan also sees an opening for less tech-savvy filmmakers to join the party. “This field still requires healthy hybrids of creative and technical people, but I think we are starting to see a shift in priorities more toward defining the grammar of storytelling in VR, not just the workflows. These questions are every bit as challenging as the technology, but we need all kinds of filmmakers to engage with them. Coming from a game-design program where you do a lot of iterations, like in visual effects and animation, I think now we can begin to similarly iterate with content.”

The clues to the future may already be in plain sight. “In VR, you can’t cut around performances the way you do when shooting traditional cinema,” says Meegan. “But there’s a lot we can learn from ambient performances in theater, like what the folks at Punchdrunk are doing in Sleep No More immersive live theater experience in New York.” The same goes for the students he worked with recently at USC’s new VR lab, which officially opened this semester.

“I’m really impressed by how young people are able to think around stories in new ways, especially when they come to it without any preconceived notions about the traditional structure of filmmaker-driven perspectives. If we can engage the existing community of cinematic and video game storytellers and get them talking to these new voices, we’ll get the best of both worlds. Our Amazon project reflected that; it was a true blend of veteran nature filmmakers and young kid VR hackers all coming together to tell this beautiful story. That’s when you start to get a really nice dialog of what’s possible in the space.”

Wairua
A former pro skateboarder, director of photography and post pro Jim Geduldick thrives on high-stakes obstacles on the course and on set. He combined both passions as the marketing manager of GoPro’s professional division until this summer, when he returned to his filmmaking roots and co-founded the creative production and technology company Wairua. “In the Maori tradition, the term wairua means a spirit not bound to one body or vessel,” he explains. “It fits the company perfectly because we want to pivot and shape shift. While we’re doing traditional 2D, mixed reality and full-on immersive production, we didn’t want to be called just another VR studio or just a technology studio. If we want to work on robotics and AI for a project, we’ll do that. If we’re doing VR or camera tech, it gives us leeway to do that without being pegged as a service, post or editorial house. We didn’t want to get pigeonholed into a single vertical.”

With his twinned background in camera development and post, Geduldick takes a big-picture approach to every job. “My partner and I both come from working for camera manufacturers, so we know the process that it takes to create the right builds,” he says. “A lot of times we have to build custom solutions for different jobs, whether that be high-speed Phantom set-ups or spherical multicam capture. It leaves us open to experiment with a blend of all the new technology out there, from VR to AR to mixed reality to AI to robotics. But we’re not just one piece of the puzzle; knowing capture through the post pipeline to delivery, we can scale to fit whatever the project needs. And it’s inevitable — the way people are telling stories and will want to tell them will drastically change in the next 10 years.”

Jim Geduldick with a spherical GoPro rig.

Early clients like Ford Motors are already fans of Wairua’s process. One of the new company’s first projects was to bring rally cross racer Ken Block of the Hoonigan Racing Division and his viral Gymkhana video series to VR. The series features Block driving against the clock the Ford Focus RS RX rallycross car he helped design and drove in the 2016 FIA World Rallycross Championship on a racing obstacle course, explaining how he performs extreme stunts like the “insane” and the “train drift” along the way. Part one of Gymkhana Nine VR is now available via the Ford VR app for iOS and Android.

“Those brands that are focused on a younger market are a little more willing to take risks with new content like VR,” Geduldick says. ‘We’re doing our own projects to test our theories and own internal pipelines, and some of those we will pitch to our partners in the future. But the clients who are already reaching out to us are doing so through word of mouth, partly because of our technical reputations but mostly because they’ve seen some of our successful VR work.” Guiding clients during the transition to VR begins with the concept, he says. “Often they are not sure what they want and often you have to consult with them and say, ‘This is what’s available. Are they going for a social reach? Or do you want to push the technology as far as it will go?’ Budgets, of course, determine most of that. If it’s not for a headset experience, it’s usually going to a platform or a custom app.”

Wairua’s kit, as you might expect, is a mix of custom tools and off-the-shelf camera gear and software. “We’re using GoPro cameras and the GoPro Odyssey, which is a Google Jump-ready rig, as well as the Nokia Ozo and other cameras and making different rigs,” he says. “If you’re shooting an interview, maybe you can get away with shooting it single camera on a panohead with one Red Epic with a fisheye lens or a Sony A7s ii. I choose camera systems based on what is the best for the project I’m working on at that moment.”

His advice for seasoned producers and directors — and even film students — transitioning to VR is try before you buy. “Go ahead and purchase the prosumer-level cameras, but don’t worry about buying the bigger spherical capture stuff. Go rent them or borrow them, and test, test, test. So many of the rental houses have great education nights to get you started.”

The shot of NYC was captured by a spherical array shoot on the top of the Empire State Building.

Once you know where your VR business is headed, he suggests, it’s time to invest. “Because of the level that we’re at, we’ve purchased a number of different camera systems, such as Red Epic, Phantom, tons of GoPros and even a Ricoh Theta S camera, which is the perfect small spherical camera for scouting locations. That one is with me in my backpack every time I’m out.”

Geduldick is also using The Foundry’s Cara VR plug-in with Nuke, Kolor’s Autopan Video Pro and Chris Bobotis’s Mettle plug-in for Adobe After Effects. “If you’re serious about VR post and doing your own stitching, and you already use After Effects, Mettle is a terrific thing to have,” he says. A few custom tetrahedral and ambisonic microphones made by the company’s sound design partners and used in elaborate audio arrays, as well as the more affordable Sennheiser Ambeo VR mic, are among Wairua’s go-to audio recording gear. “The latter is one of the more cost-effective tools for spatial audio capture,” says Geduldick.

The idea of always mastering to the best high-resolution archival format available to you still holds true for VR production, he adds. “Do you shoot in 4K just to future-proof it, even if it’s more expensive? That’s still the case for 360 VR and immersive today. Your baseline should always be 4K and you should avoid shooting any resolution less than that. The headsets may not be at 4K resolution per eye yet, but it’s coming soon enough.”

Geduldick does not believe any one segment of expanded reality with take the ultimate prize. “I think it’s silly to create a horse race between augmented reality and virtual reality,” he says. “It’s all going to eventually meld together into immersive storytelling and immersive technology. The headsets are a stopgap. 360 video is a stopgap. They are gateways into what will be and can come in the next five to 10 years, even two years. Yes, some companies will disappear and others will be leaders. Facebook and Google have a lot of money behind it, and the game engine companies also have an advantage. But there is no king yet. There is no one camera or or no single software that will solve all of our problems, and in my opinion, it’s way too soon to be labeling this a movement at all.”

Jim with a GoPro Omni on the Mantis Rover for Gymkhana.

That doesn’t mean that Wairua isn’t already looking well beyond the traditional entertainment marketing and social media space at the VR apps of tomorrow. “We are very excited about industrial, education and health applications,” Geduldick says. “Those are going to be huge, but the money is in advertising and entertainment right now, and the marketing dollars are paying for these new VR experiences. We’re using that income to go right back into R&D and to build these other projects that have the potential to really help people — like cancer patients, veterans and burn victims — and not just dazzle them.”

Geduldick’s advice for early adopters? Embrace failure, absorb everything and get on with it. “The takeaway for every single production you do, whether it be for VR or SD, you should be learning something new and taking that lesson with you to your next project,” he says. “With VR, there’s so much to learn — how the technology can benefit you, how it can hurt you, how it can slow you down as a storyteller and a filmmaker? Don’t listen to everybody; just go out and find out for yourself what works. What works for me won’t necessarily work for someone like Ridley Scott. Just get out there and experiment, learn and collaborate.”

Main Image: A Ford project via Wairua.


Beth Marchant has been covering the production and post industry for 21 years. She was the founding editor-in-chief of Studio/monthly magazine and the co-editor of StudioDaily.com. She continues to write about the industry.

Creating VR audio workflows for ‘Mars 2030’ and beyond

Source Sound is collaborating with others and capturing 360 sound for VR environments.

By Jennifer Walden

Everyone wants it, but not everyone can make it. No, I’m not talking about money. I’m talking about virtual reality content.

Let’s say you want to shoot a short VR film. You’ve got a solid script, a cast of known actors, you’ve got a 360-degree camera and a pretty good idea of how to use it, but what about the sound? The camera has a built-in mic, but will that be enough coverage? Should the cast be mic’d as they would be for a traditional production? How will the production sound be handled in post?

Tim Gedemer, owner/sound supervisor at Source Sound in Woodland Hills, California, can help answer these questions. “In VR, we are audio directors,” he says. “Our services include advising clients at the script level on how they should be shooting their visuals to be optimal for sound.”

Tim Gedemer

As audio directors, Source Sound walks their clients through every step of the process, from production to distribution. Starting with the recording on set, they manage all of the technical aspects of sound file management through production, and then guide their clients through the post sound process, both creatively and technically.

They recommend what technology should be used, how clients should be using it and what deals they need to make to sort out their distribution. “It really is a point-to-point service,” says Gedemer. “We decided early on that we needed to influence the entire process, so that is what we do.”

Two years ago, Dolby Labs referred Jaunt Studio to Source Sound to for their first VR film gig. Gedemer explains that because of Source Sound’s experience with games and feature films, Dolby felt they would be a good match to handle Jaunt’s creative sound needs while Dolby worked with Jaunt on the technical challenges.

Jaunt’s Kaiju Fury! premiered at the 2015 Sundance Film Festival. The experience puts the viewer in the middle of an epic Godzilla-like monster battle. “They realized their film needed cinematic sound, so Dolby called us up and asked if we’d like to get involved. We said, ‘We’re really busy with projects, but show us the tech and maybe we’ll help.’ We were disinterested at first, figuring it was going to be gimmicky, but I went to San Francisco and I looked at their first test, and I was just shocked. I had never seen anything like that before in my life. I realized, in that first moment of putting on those goggles, that we needed to do this.”

Paul McCartney on the "Out There" tour 2014.

Paul McCartney on the “Out There” tour 2014.

Kaiju Fury! was just the start. Source Sound completed three more VR projects for Jaunt, all within a week. There was the horror VR short film called Black Mass, a battle sequence called The Mission and the Atmos VR mastering of Paul McCartney’s Live and Let Die in concert.

Gedemer admits, “It was just insane. No one had ever done anything like this and no one knew how to do it. We just said, ‘Okay, we’ll just stay up for a week, figure all of that out and get it done.’”

Adjusting The Workflow
At first, their Pro Tools-based post sound workflow was similar to a traditional production, says Gedemer, “because we didn’t know what we didn’t know. It was only when we got into creating the final mix that we realized we didn’t have the tools to do this.”

Specifically, how could they experience the full immersion of the 360-degree video and concurrently make adjustments to the mix? On that first project, there was no way to slave the VR picture playing back through the Oculus headgear to the sound playing back via Pro Tools. “We had to manually synchronize,” explains Gedemer. “Literally, I would watch the equi-rectangular video that we were working with in Pro Tools, and at the precise moment I would just press play on the laptop, playing back the VR video through the Oculus HMD to try and synchronize it that way. I admit I got pretty good at that, but it’s not really the way you want to be working!”

Since that time, Dolby has implemented timecode synchronization and a video player that will playback the VR video through the Oculus headset. Now the Source Sound team can pick up the Oculus and it will be synchronized to the Pro Tools session.

Working Together For VR
Over the last few years, Source Sound has been collaborating with tech companies like Dolby, Avid, Oculus, Google, YouTube and Nokia on developing audio-related VR tools, workflow solutions and spec standards that will eventually become available to the wider audio post industry.

“We have this holistic approach to how we want to work, both in virtual and augmented reality audio,” says Gedemer. “We’re working with many different companies, beta testing technology and advising on what they should be thinking about regarding VR sound — with a keen eye toward new product development.”

Kaiju Fury

Kaiju Fury!

Since Kaiju Fury, Source Sound has continued to create VR experiences with Jaunt. They have worked with other VR content creators, including the Emblematic Group (founded by “the godmother of VR,” Nonny de la Peña), 30 Ninjas (founded by director Doug Liman, The Bourne Identity and Edge of Tomorrow), Fusion Media, Mirada, Disney, Google, YouTube and many others.

Mars 2030
Currently, Source Sound is working with Fusion Media on a project with NASA called Mars 2030, which takes a player to Mars as an astronaut and allows him/her to experience what life might be like while living in a Mars habitat. NASA feels that human exploration of Mars may be possible in the year 2030, so why not let people see and feel what it’s like.

The project has given Source Sound unprecedented access to the NASA facilities and engineers. One directive for Mars 2030 is to be as accurate as possible, with information on Mars coming directly from NASA’s Mars missions. For example, NASA collected information about the surface of Mars, such as the layout of all the rocks and the type of sand covering the surface. All of that data was loaded into the Unreal Engine, so when a player steps out of the habitat in the Mars 2030 experience and walks around, that surface is going to be the exact surface that is on Mars. “It’s not a facsimile,” says Gedemer. “That rock is actually there on Mars. So in order for us to be accurate from an audio perspective, there’s a lot that we have to do.”

In the experience the player gets to drive the Mars Rover. At NASA in Houston, there are multiple iterations of the rover that are being developed for this mission. They also have a special area that is set up like the Mars surface with a few craters and rocks.

For audio capture, Gedemer and sound effects recordist John Fasal headed to Houston with Sound Devices recorders and a slew of mic options. While the rover is too slow to do burnouts and donuts, Gedemer and Fasal were able to direct a certified astronaut driver and record the rover from every relevant angle. They captured sounds and ambiences from the various habitats on site. “There is a new prototype space suit that is designed for operation on Mars, and as such we will need to capture all the relevant sound associated with it,” says Gedemer. “We’ll be looking into helmet shape and size, communication systems, life support air flow, etc. when recreating this in the Unreal Engine.”

Another question the sound team nSS_NASA_2USEeeds to address is, “What does it sound like out on the surface of Mars?” It has an atmosphere, but the tricky thing is that a human can never actually walk around on the surface of Mars without wearing a suit. Sounds traveling through the Mars atmosphere will sound different than sounds traveling through Earth’s atmosphere, and additional special considerations need to be made for how the suit will impact sound getting to the astronaut’s ears.

“Only certain sounds and/or frequencies will penetrate the suit, and if it is loud enough to penetrate the suit, what is it going to sound like to the astronaut?” asks Gedemer. “So we are trying to figure out some of these technical things along the way. We hope to present a paper on this at the upcoming AES Conference on Audio for Virtual and Augmented Reality.”

Going Live
Another interesting project at Source Sound is the work they’re doing with Nokia to develop specialized audio technology for live broadcasts in VR. “We are currently the sole creative provider of spatial audio for Nokia’s VR broadcasting initiative,” reveals Gedemer. Source Sound has been embedded with the Nokia Ozo Live team at events where they have been demonstrating their technology. They were part of the official Ozo Camera Launches in Los Angeles and London. They captured and spatialized a Los Angeles Lakers basketball game at the Staples Center. And once again they teamed up with Nokia at their NAB event this past spring.

“We’ve been working with them very closely on the technology that they are developing for live capture and distribution of stereoscopic visual and spatial audio in VR. I can’t elaborate on any details, but we have some very cool things going on there.”

However, Gedemer does break down one of the different requirements of live VR broadcast versus a cinematic VR experience — an example being the multi-episode VR series called Invisible, which Source Sound and Doug Liman of 30 Ninjas are currently collaborating on.

For a live broadcast you want an accurate representation of the event, but for a cinematic experience the opposite is true. Accuracy is not the objective. A cinematic experience needs a highly curated soundtrack in order to tell the story.

Gedemer elaborates, “The basic premise is that, for VR broadcasts you need to have an accurate audio representation of camera location. There is the matter of proper perspective to attend to. If you have a multi-camera shoot, every time you change camera angles to the viewer, you change perspective, and the sound needs to follow. Unlike a traditional live environment, which has a stereo or 5.1 mix that stays the same no matter the camera angle, our opinion is that approach is not adequate for true VR. We think Nokia is on the right track, and we are helping them perfect the finer points. To us that is truly exciting.”

Jennifer Walden is a New Jersey based writer and audio engineer.