Tag Archives: Light Field

NAB 2016: VR/AR/MR and light field technology impressed

By Greg Ciaccio

The NAB 2016 schedule included its usual share of evolutionary developments, which are truly exciting (HDR, cloud hosting/rendering, etc.). One, however, was a game changer with reach far beyond media and entertainment.

This year’s NAB floor plan featured a Virtual Reality Pavilion in the North Hall. In addition, the ETC (USC’s Entertainment Technology Center) held a Virtual Reality Summit that featured many great panel discussions and opened quite a few minds. At least that’s what I gathered by the standing room only crowds that filled the suite. The ETC’s Ken Williams and Erik Weaver, among others, should be credited for delivering quite a program. While VR itself is not a new development, the availability of relatively inexpensive viewers (with Google Cardboard the most accessible) will put VR in the hands of practically everyone.

Programs included discussions on where VR/AR (Augmented Reality) and now MR (Mixed Reality) are heading, business cases and, not to be forgotten, audio. Keep in mind that with headset VR experiences, multi-channel directional sound must be perceivable with just our two ears.

The panels included experts in the field, including Dolby, DTS, Nokia, NextVR, Fox and CNN. In fact, Juan Santillian from Vantage.tv mentioned that Coachella is streaming live in VR. Often, concerts and other live events have a fixed audience size, and many can’t attend due to financial or sell-out situations. VR can allow a much more intimate and immersive experience than being almost anywhere but onstage.

One example, from Fox Sports’ Michael Davies, involved two friends in different cities virtually attending a football game in a third city. They sat next to each other and chatted during the game, with their audio correctly mapped to their seats. There are no limits to applications for VR/AR/MR, and, by all accounts, once you experience it, there is no doubt that this tech is here to stay.

I’ve heard many times this year that mobile will be the monetary driver for wide adoption of VR. Halsey Minor with Voxelus estimates that 85 percent of VR usage will be via a mobile device. Given that more photos and videos are shot on our phones (by far) than on dedicated cameras, this is not surprising. Some of the latest crop of mobile phones are not only fast and contain high dynamic range and wide color gamut, they feature high-end audio processing from Dolby and others. Plus, our reliance on our mobiles ensures that you’ll never forget to bring it with you.

Light Field Imaging
On both Sunday and Tuesday of NAB 2016, programs were devoted to light field imaging. I was already familiar with this truly revolutionary tech, and learned about Lytro, Inc. a few years ago from Internet ads for an early consumer camera. I was intrigued with the idea of controlling focus after shooting. I visited www.lytro.com and was impressed, but the resolution was low, so, for me, this was mainly a proof of concept. Fast forward three years, and Lytro now has a cinema camera!

Jon Karafin (pictured right), Lytro’s head of Light Field Imaging, not only unveiled the camera onstage, but debuted their short Life, produced in association with The Virtual Reality Company (VRC). Life takes us through a man’s life and is told with no dialog, letting us take in the moving images without distraction. Jon then took us through all the picture aspects using Nuke plug-ins, and minds started blowing. The short is directed by Academy Award-winner Robert Stromberg, and shot by veteran cinematographer David Stump, who is chief imaging scientist at VRC.

Many of us are familiar with camera raw capture and know that ISO, color temperature and other picture aspects can be changed post-shooting. This has proven to be very valuable. However, things like focus, f-stop, shutter angle and many other parameters can now be changed, thanks to light field technology — think of it as an X-ray compared to an MRI. In the interests of trying to keep a complicated technology relatively simple, sensors in the camera capture light fields in not only in X and Y space, but two more “angular” directions, forming what Lytro calls 4D space. The result is accurate depth mapping which opens up so many options for filmmakers.

Lytro_Cinema_2

Lytro Cinema Camera

For those who may think that this opens up too many options in post, all parameters can be locked so only those who are granted access can make edits. Some of the parameters that can be changed in post include: Focus, F-Stop, Depth of Field, Shutter Speed, Camera Position, Shutter Angle, Shutter Blade Count, Aperture Aspect Ratio and Fine Control of Depth (for mattes/comps).

Yes, this camera generates a lot of data. The good news is that you can make changes anywhere with an Internet connection, thanks to proxy mode in Nuke and processing rendered in the cloud. Jon demoed this, and images were quickly processed using Google’s cloud.

The camera itself is very large, but Lytro knows that they’ll need to reduce the size (from around seven feet long) to a more maneuverable form factor. However, this is a huge step in proving that a light field cinema camera and a powerful, manageable workflow is not only possible, but will no doubt prove valuable to filmmakers wanting the power and control offered by light field cinematography.

Greg Ciaccio is a technologist focused primarily on finding new technology and workflow solutions for Motion Picture and Television clients. Ciaccio served in technical management roles for the respective Creative Services divisions for both Deluxe and Technicolor.

Lytro camera allows capture of massive light field data on all frames

Imagine if your camera could capture the entire light field of a scene in 3D, turning every frame into a three-dimensional model? That is the idea behind the Lytro Cinema system, which uses Light Field technology to capture massive amounts of information per frame, allowing you to control the depth of field, hence, creating more flexibility in post. Oh, and it captures 300 frames per second, adding a level of speed control, including adjustable motion blur, that was previously limited to the live-action process.

In a video released by the company, Brendan Bevensee, lead engineer for Light Field Cideo/Lytro, said Light Field cinematography allows “the ability to capture everything about a scene — from different perspectives,different focal planes and different apertures. Every pixel now has color properties and directional properties, as well as exact placement in 3D space. Essentially we have a virtual camera that can be controlled in post production.”

Lytro says their capture system enables “the complete virtualization of the live-action camera —transforming creative camera controls from fixed, on-set decisions to computational post processes.”

In the aforementioned video, the head of Light Field Video/Lytro Jon Karafin said, “Lytro Cinema offers an infinite ability to focus anywhere in your scene. You have the infinite ability to focus and create any aperture or any depth of field. You can shift your camera to the left or to the right, as if you made that exact decision on set. It can even move your camera in and out. Automated camera tracking removes that tedious task of integration and matching. It has all of the volume, all of that depth information that easily allows you to composite and matte your CG objects. With Depth Screen it’s as if you have a greenscreen for every object, but it’s not limited to any one object, it’s anywhere in space.”

The rich dataset captured by the system produces a Light Field master that can be rendered in any format in post, allowing for a range of creative possibilities. The Light Field Master enables creators to render content in multiple formats —including IMAX, RealD and traditional cinema and broadcast at variable frame rates and shutter angles.

“Lytro has always been a company thinking about what the future of imaging will be,” said Ted Schilowitz, futurist at Fox Studios. “There are a lot of companies that have been applying new technologies and finding better ways to create cinematic content, and they are all looking for better ways and better tools to achieve live-action, highly immersive content. Lytro is focusing on getting a much bigger, better and more sophisticated cinematography-level dataset that can then flow through the VFX pipeline and modernize that world.”

Lytro Cinema offers:
— A sensor that offers 755 RAW megapixels at up to 300fps.
—Up to 16 stops of dynamic range and wide color gamut.
—Integrated high-resolution active scanning.

The Lytro Cinema package includes a camera, a server array for storage and processing — which can also be done in the cloud — and software to edit Light Field data. The entire system integrates into existing production and post workflows, working in tandem with popular industry standard tools.

Life the first short produced with Lytro Cinema in association with The Virtual Reality Company (VRC) will premiere at NAB on April 19 at 4pm PST in Room S222. Life was directed by Academy Award-winner Robert Stromberg, CCO at VRC (The Virtual Reality Company), and shot by David Stump, ASC, chief imaging scientist at VRC.

Senior finishing artist at Light Iron New York Katie Hinsen sees the possibilities. “The coolest thing about Lytro’s tech is that it captures the whole light field coming in to it, rather than a flat representation of the scene. So you can change focus in post, where you could pull stuff out that isn’t there. Basically, once you take a picture it’s still alive. Imagine you take a photo (or a video, now), and it’s got issues. With Lytro you’re capturing all the light information of the scene, not the image. So it’s all there and you can change it.”

Lytro Cinema will be available for production in Q3 of 2016 to exclusive partners on a subscription basis.