Category Archives: Light Field

Digging Deeper: Fraunhofer’s Dr. Siegfried Foessel

By Randi Altman

If you’ve been to NAB, IBC, AES or regional conferences involving media and entertainment technology, you have likely seen Fraunhofer exhibiting or heard one of their representatives speaking on a panel.

Fraunhofer first showed up on my radar years ago at an AES show in New York City when they were touting the new MP3 format, which they created. From that moment on, I’ve made it a point to keep up on what Fraunhofer has been doing in other areas of the industry, but for some, what Fraunhofer is and does is a mystery.

We decided to help with that mystery by throwing some questions at Dr. Siegfried Foessel, Fraunhofer IIS Department Moving Picture Technologies.

Can you describe Fraunhofer?
Fraunhofer-Gesellschaft is an organization for applied research that has 67 institutes and research units at locations throughout Germany. At present, there are around 24,000 people. The majority are qualified scientists and engineers who work with an annual research budget of more than 2.1 billion euros.

More than 70 percent of the Fraunhofer-Gesellschaft’s research revenue is derived from contracts with industry and from publicly financed research projects. Almost 30 percent is contributed by the German federal and Länder governments in the form of base funding. This enables the institutes to work ahead on solutions to problems that will become relevant to industry and society within the next five or ten years from now.

How did it all begin? Is it a think tank of sorts? Tell us about Fraunhofer’s business model.
The Fraunhofer-Gesellschaft was founded in 1949 and is a recognized non-profit organization that takes its name from Joseph von Fraunhofer (1787–1826), the illustrious Munich researcher, inventor and entrepreneur. Its focus was clearly defined to do application-oriented research and to develop future-relevant key technologies. Through their research and development work, the Fraunhofer Institutes help to reinforce the competitive strength of the economy. They do so by promoting innovation, strengthening the technological base, improving the acceptance of new technologies and helping to train the urgently needed future generation of scientists and engineers.

What is Fraunhofer IIS?
The Fraunhofer Institute for Integrated Circuits IIS is an application-oriented research institution for microelectronic and IT system solutions and services. With the creation of MP3 and the co-development of AAC, Fraunhofer IIS has reached worldwide recognition. In close cooperation with partners and clients, the ISS institute provides research and development services in the following areas: audio and multimedia, imaging systems, energy management, IC design and design automation, communication systems, positioning, medical technology, sensor systems, safety and security technology, supply chain management and non-destructive testing. About 880 employees conduct contract research for industry, the service sector and public authorities.

Fraunhofer IIS partners with companies as well as public institutions?
We develop, implement and optimize processes, products and equipment until they are ready for use in the market. Flexible interlinking of expertise and capacities enables us to meet extremely broad project requirements and complex system solutions. We do contracted research for companies of all sizes. We license our technologies and developments. We work together with partners in publicly funded research projects or carry out commercial and technical feasibility studies.

IMF transcoding.

What is the focus of Fraunhofer IIS’ Department of Moving Picture Technologies?
For more than 15 years, our Department Moving Picture Technologies has driven developments for digital cinema and broadcast solutions focused on imaging systems, post production tools, formats and workflow solutions. The Department Moving Picture Technologies was chosen by the Digital Cinema Initiatives (DCI) to develop and implement the first certification test plan for digital cinema as the main reference for all systems in this area. As a leader in the ISO standardization committee for digital cinema within JPEG, my team and I are driving standardization for JPEG 2000 and formats, such as DCP and the Interoperable Master Format (IMF.)

We also are working together with SMPTE and other standardization bodies worldwide. Renowned developments for the department that are highly respected are the Arri D20/D21 camera, the easyDCP post production suite for DCP and IMF creation and playback, as well as the latest developments and results of multi-camera/light-field technology.

What are some of the things you are working on and how does that work find its way to post houses and post pros?
The engineers and scientists of the Department Moving Picture Technologies are working on tools and workflow solutions for new media file formats like IMF to enable smooth integration and use in existing workflows and to optimize performance and quality. As an example, we always enhance and augment the features available through the post production easyDCP suite. The team discusses and collaborates with customers, industry partners and professionals in the post production and digital cinema industries to identify the “most wanted and needed” requirements.

easyDCP

We preview new technologies and present developments that meet these requirements or facilitate process steps. Examples of this include the acceleration process of IMF or DCP creation by using an approach based on a hybrid JPEG 2000 functionality or introducing a media asset management tool for DCP/IMF or dailies. We present our ideas, developments and results at exhibitions such as NAB, the HPA Tech Retreat and IBC, as well as SMPTE conferences and plugfests all around the world.

Together with distribution partners who are selling the products like easyDCP, Fraunhofer IIS licenses those developments and puts them into the market. Therefore, the team always looks for customer feedback for their developments that is supported by a very active community.

Who are some of your current customers and partners?
We have more than 1,500 post houses as customers, managed by our licensing partner easyDCP GmbH. Nearly all of the Hollywood studios and post houses on all continents are our customers. We also work together with integration partners like Blackmagic and Quantel. Most of the names of our partners in the contract research area are confidential, but to name some partners from the past and present: Arri, DCI, IHSE GmbH.

Which technologies are available for license now?
• Tools for creation and playback of DCPs and IMPs, as standalone tools and for integration into third party tools
• Tools for quality control of DCPs and IMPs
• Tools for media asset management of DCPs and IMPs
• Plug-ins for light-field-processing and depth map generation
• Codecs for mezzanine compression of images

Lightfield tech

What are you working on now that people should know about?
We are developing new tools and plug-ins for bringing lightfield technology to the movie industry to enhance creativity opportunities. This includes system aspects in combination with existing post tools. We are chairing and actively participating on adhoc groups for lightfield-related standardization efforts in the JPEG/MPEG Joint Adhoc Group for digital representations of light/sound fields for immersive media applications (see https://jpeg.org/items/20160603_pleno_report.html).

We are also working together with DIN on a proposal to standardize digital long-term archive formats for movies. Basic work is done with German archives and service providers at DIN NVBF3 and together with CST from France at SMPTE with IMF App#4. Furthermore, we are developing mezzanine image compression formats for the transmission of video over IP in professional broadcast environments and GPU accelerated tools for creation and playback of JPEG 2000 code streams.

How do you pick what you will work on?
The employees at Fraunhofer IIS are very creative people. By observation of the market, research in joint projects and cooperation with universities, ideas are created and evaluated. Employees and our student scientists are discussing with industry partners what might be possible in the near future and which ideas have the greatest potential. Selected ideas will then be evaluated with respect to the business opportunities and transformed into internal projects or proposed as research projects. Our employees are tasked with working much like our eponym Joseph von Fraunhofer, as researchers, inventors and entrepreneurs — all at the same time.

What other “hats” do you wear in the industry?
As mentioned earlier, Fraunhofer is involved in standardization bodies and industry associations. For example, I chair the Systems Group within ISO SC29WG1 (JPEG) and the post production group within ISO TC36 (Cinematography). I am also a SMPTE governor (EMEA and Central and South America region) and a SMPTE fellow, along with supporting SMPTE conferences as a program committee member.

Currently, I am president of the German Society Fernseh- und Kinotechnische Gesellschaft (FKTG) and am involved in associations like EDCF and ISDCF. Additionally, I’m a speaker for the German VDE/ITG society in the area of media technology. Last, but not least, I chair the German standardization body at DIN for NVBF3 and consult the German federal film board in questions related to new technical challenges in the film industry.

What does Fraunhofer Digital Media Alliance do? A lot!

By Jonathan Abrams

While the vast majority of the companies with exhibit space at NAB are for-profit, there is one non-profit that stands out. With a history of providing ubiquitous technology to the masses since 1949, Fraunhofer focuses on applied research and developments that end up — at some point in the near future — as practical products or ready-for-market technology.

In terms of their revenue, one-third of their funding is for basic research, with the remaining two-thirds applied toward industry projects and coming directly from private companies. Their business model is focused on contract research and licensing of technologies. They have sold first prototypes and work with distributors, though Fraunhofer always keeps the rights to continue development.

What projects were they showcasing at NAB 2106 that have real-world applications in the near future? You may have heard about the Lytro camera. Fraunhofer Digital Media Alliance member Fraunhofer IIS has been taking a camera agnostic approach to their work with light-field technology. Their goal is to make this technology available for many different camera set-ups, and they were proving it with a demo of their multi-cam light-field plug-in for The Foundry’s Nuke. After capturing a light-field, users can perform framing correction and relighting, including changes to angles, depth and the creation of point clouds.

The Nuke plug-in (see our main image) allows the user to create virtual lighting (relighting) and interactive lighting. Light-field data also allows for depth estimation (called depth maps) and is useful for mattes and secondary color correction. Similar to Lytro, focus pulling can be performed with this light-field plug-in. Why Nuke? That is what their users requested. Even though Nuke is an OFX host, the Fraunhofer IIS light field plug-in only works within Nuke. As for using this light-field plug-in outside of Nuke, I was told that “porting to Mac should be an easy task.” Hopefully that is an accurate statement, though we will have to wait to find out.

DCP
Fraunhofer IIS has its hand in other parts of production and post as well. The last two steps of most projects are the creation of deliverables and their delivery. If you need to create and deliver a DCP (Digital Cinema Package), then easyDCP may be for you.easydcp1

This project began in 2008, when creating a DCP was not as familiar as it is today to most users, and a deep expertise of the specifications for correctly making a DCP was very complex. Small- to medium-sized post companies, in particular, profit from the easy-to-use easyDCP suite. The engineers of Fraunhofer IIS were also working on behalf of the DCI specifications for Digital Cinema, therefore they are experienced in integrating all important features in this software for DCPs.

The demo I saw indicated that the JPEG2000 encode was as fast as 108fps! In 2013, Fraunhofer partnered with both Blackmagic and Quantel to make this software available to the users of those respective finishing suites. The demo I saw was using a Final Cut Pro X project file and it was with the Creator+ version since it had support for encryption. Avid Media Composer users will have to export their sequence and import it into Resolve to use easyDCP Creator. Amazingly, this software works as far back as Mac OS X Leopard. IMF creation and playback can also be done with the easyDCP software suite.

VR/360
VR and 360-degree video were prominent at NAB, and the institutes of the Fraunhofer Digital Media Alliance are involved in this as well, having worked on live streaming and surround sound as part of a project with the Berlin Symphony Orchestra.

Fraunhofer had a VR demo pod at the ATSC 3.0 Consumer Experience (in South Hall Upper) — I tried it and the sound did track with my head movement. Speaking of ATSC 3.0, it calls for an immersive audio codec. Each country or geographic region that adopts ATSC 3.0 can choose to implement either Dolby AC-4 or MPEG-H, the latter of which is the result of research and development by Fraunhofer, Technicolor and Qualcomm. South Korea announced earlier this year that they will begin ATSC 3.0 (UHDTV) broadcasting in February 2017 using the MPEG-H audio codec.

From what you see to what you hear, from post to delivery, the Fraunhofer Digital Media Alliance has been involved in the process.

Jonathan S. Abrams is the Chief Technical Engineer at Nutmeg, a creative marketing, production and post resource.

G-Tech 6-15

NAB 2016: VR/AR/MR and light field technology impressed

By Greg Ciaccio

The NAB 2016 schedule included its usual share of evolutionary developments, which are truly exciting (HDR, cloud hosting/rendering, etc.). One, however, was a game changer with reach far beyond media and entertainment.

This year’s NAB floor plan featured a Virtual Reality Pavilion in the North Hall. In addition, the ETC (USC’s Entertainment Technology Center) held a Virtual Reality Summit that featured many great panel discussions and opened quite a few minds. At least that’s what I gathered by the standing room only crowds that filled the suite. The ETC’s Ken Williams and Erik Weaver, among others, should be credited for delivering quite a program. While VR itself is not a new development, the availability of relatively inexpensive viewers (with Google Cardboard the most accessible) will put VR in the hands of practically everyone.

Programs included discussions on where VR/AR (Augmented Reality) and now MR (Mixed Reality) are heading, business cases and, not to be forgotten, audio. Keep in mind that with headset VR experiences, multi-channel directional sound must be perceivable with just our two ears.

The panels included experts in the field, including Dolby, DTS, Nokia, NextVR, Fox and CNN. In fact, Juan Santillian from Vantage.tv mentioned that Coachella is streaming live in VR. Often, concerts and other live events have a fixed audience size, and many can’t attend due to financial or sell-out situations. VR can allow a much more intimate and immersive experience than being almost anywhere but onstage.

One example, from Fox Sports’ Michael Davies, involved two friends in different cities virtually attending a football game in a third city. They sat next to each other and chatted during the game, with their audio correctly mapped to their seats. There are no limits to applications for VR/AR/MR, and, by all accounts, once you experience it, there is no doubt that this tech is here to stay.

I’ve heard many times this year that mobile will be the monetary driver for wide adoption of VR. Halsey Minor with Voxelus estimates that 85 percent of VR usage will be via a mobile device. Given that more photos and videos are shot on our phones (by far) than on dedicated cameras, this is not surprising. Some of the latest crop of mobile phones are not only fast and contain high dynamic range and wide color gamut, they feature high-end audio processing from Dolby and others. Plus, our reliance on our mobiles ensures that you’ll never forget to bring it with you.

Light Field Imaging
On both Sunday and Tuesday of NAB 2016, programs were devoted to light field imaging. I was already familiar with this truly revolutionary tech, and learned about Lytro, Inc. a few years ago from Internet ads for an early consumer camera. I was intrigued with the idea of controlling focus after shooting. I visited www.lytro.com and was impressed, but the resolution was low, so, for me, this was mainly a proof of concept. Fast forward three years, and Lytro now has a cinema camera!

Jon Karafin (pictured right), Lytro’s head of Light Field Imaging, not only unveiled the camera onstage, but debuted their short Life, produced in association with The Virtual Reality Company (VRC). Life takes us through a man’s life and is told with no dialog, letting us take in the moving images without distraction. Jon then took us through all the picture aspects using Nuke plug-ins, and minds started blowing. The short is directed by Academy Award-winner Robert Stromberg, and shot by veteran cinematographer David Stump, who is chief imaging scientist at VRC.

Many of us are familiar with camera raw capture and know that ISO, color temperature and other picture aspects can be changed post-shooting. This has proven to be very valuable. However, things like focus, f-stop, shutter angle and many other parameters can now be changed, thanks to light field technology — think of it as an X-ray compared to an MRI. In the interests of trying to keep a complicated technology relatively simple, sensors in the camera capture light fields in not only in X and Y space, but two more “angular” directions, forming what Lytro calls 4D space. The result is accurate depth mapping which opens up so many options for filmmakers.

Lytro_Cinema_2

Lytro Cinema Camera

For those who may think that this opens up too many options in post, all parameters can be locked so only those who are granted access can make edits. Some of the parameters that can be changed in post include: Focus, F-Stop, Depth of Field, Shutter Speed, Camera Position, Shutter Angle, Shutter Blade Count, Aperture Aspect Ratio and Fine Control of Depth (for mattes/comps).

Yes, this camera generates a lot of data. The good news is that you can make changes anywhere with an Internet connection, thanks to proxy mode in Nuke and processing rendered in the cloud. Jon demoed this, and images were quickly processed using Google’s cloud.

The camera itself is very large, but Lytro knows that they’ll need to reduce the size (from around seven feet long) to a more maneuverable form factor. However, this is a huge step in proving that a light field cinema camera and a powerful, manageable workflow is not only possible, but will no doubt prove valuable to filmmakers wanting the power and control offered by light field cinematography.

Greg Ciaccio is a technologist focused primarily on finding new technology and workflow solutions for Motion Picture and Television clients. Ciaccio served in technical management roles for the respective Creative Services divisions for both Deluxe and Technicolor.