Tag Archives: DigitalFilm Tree

Storage in the Studio: Post Houses

By Karen Maierhofer

There are many pieces that go into post production, from conform, color, dubbing and editing to dailies and more. Depending on the project, a post house can be charged with one or two pieces of this complex puzzle, or even the entire workload. No matter the job, the tasks must be done on time and on budget. Unforeseen downtime is unacceptable.

That is why when it comes to choosing a storage solution, post houses are very particular. They need a setup that is secure, reliable and can scale. For them, one size simply does not fit all. They all want a solution that fits their particular needs and the needs of their clients.

Here, we look at three post facilities of various sizes and range of services, and the storage solutions that are a good fit for their business.

Liam Ford

Sim International
The New York City location of Sim has been in existence for over 20 years, operating under the former name of Post Factory NY up until about a month ago when Sim rebranded it and its seven other founding post companies as Sim International. Whether called by its new moniker or its previous one, the facility has grown to become a premier space in the city for offline editorial teams as well as one of the top high-end finishing studios in town, as the list of feature films and episodic shows that have been cut and finished at Sim is quite lengthy. And starting this past year, Sim has launched a boutique commercial finishing division.

According to senior VP of post engineering Liam Ford, the vast majority of the projects at the NYC facility are 4K, much of which is episodic work. “So, the need is for very high-capacity, very high-bandwidth storage,” Ford says. And because the studio is located in New York, where space is limited, that same storage must be as dense as possible.

For its finishing work, Sim New York is using a Quantum Xcellis SAN, a StorNext-based appliance system that can be specifically tuned for 4K media workflow. The system, which was installed approximately two years ago, runs on a 16Gb Fibre Channel network. Almost half a petabyte of storage fits into just a dozen rack units. Meanwhile, an Avid Nexis handles the facility’s offline work.

The Sim SAN serves as the primary playback system for all the editing rooms. While there are SSDs in some of the workstations for caching purposes, the scheduling demands of clients do not leave much time for staging material back and forth between volumes, according to Ford. So, everything gets loaded back to the SAN, and everything is played back from the SAN.

As Ford explains, content comes into the studio from a variety of sources, whether drives, tapes or Internet transfers, and all of that is loaded directly onto the SAN. An online editor then soft-imports all that material into his or her conform application and creates an edited, high-resolution sequence that is rendered back to the SAN. Once at the SAN, that edited sequence is available for a supervised playback session with the in-house colorists, finishing VFX artists and so forth.

“The point is, our SAN is the central hub through which all content at all stages of the finishing process flows,” Ford adds.

Before installing the Xcellis system, the facility had been using local workstation storage only, but the huge growth in the finishing division prompted the transition to the shared SAN file system. “There’s no way we could do the amount of work we now have, and with the flexibility our clients demand, using a local storage workflow,” says Ford.

When it became necessary for the change, there were not a lot of options that met Sim’s demands for high bandwidth and reliable streaming, Ford points out, as Quantum’s StorNext and SGI’s CXFS were the main shared file systems for the M&E space. Sim decided to go with Quantum because of the work the vendor has done in recent years toward improving the M&E experience as well as the ease of installing the new system.

Nevertheless, with the advent of 25Gb and 100Gb Ethernet, Sim has been closely monitoring the high-performance NAS space. “There are a couple of really good options out there right now, and I can see us seriously looking at those products in the near future as, at the very least, an augmentation to our existing Fibre Channel-based storage,” Ford says.

At Sim, editors deal with a significant amount of Camera Raw, DPX and OpenEXR data. “Depending on the project, we could find ourselves needing 1.5GB/sec or more of bandwidth for a single playback session, and that’s just for one show,” says Ford. “We typically have three or four [shows] playing off the SAN at any one time, so the bandwidth needs are huge!”

Master of None

And the editors’ needs continue to evolve, as does their need for storage. “We keep needing more storage, and we need it to be faster and faster. Just when storage technology finally got to the point that doing 10-bit 2K shows was pretty painless, everyone started asking for 16-bit 4K,” Ford points out.

Recently, Sim completed work on the feature American Made and the Netflix show Master of None, in addition to a number of other episodic projects. For these and others shows, the SAN acts as the central hub around which the color correction, online editing, visual effects and deliverables are created.

“The finishing portion of the post pipeline deals exclusively with the highest-quality content available. It used to be that we’d do our work directly from a film reel on a telecine, but those days are long past,” says Ford. “You simply can’t run an efficient finishing pipeline anymore without a lot of storage.”

DigitalFilm Tree
DigitalFilm Tree (DFT) opened its doors in 1999 and now occupies a 10,000-square-foot space in Universal City, California, offering full round-trip post services, including traditional color grading, conform, dailies and VFX, as well as post system rentals and consulting services.

While Universal City may be DFT’s primary location, it has dozens of remote satellite systems — mini post houses for production companies and studios – around the world. Those remote post systems, along with the increase in camera resolution (Alexa, Raw, 4K), have multiplied DFT’s storage needs. Both have resulted in a sea change in the facility’s storage solution.

According to CEO Ramy Katrib, most companies in the media and entertainment industry historically have used block storage, and DFT was no different. But four years ago, the company began looking at object storage, which is used by Silicon Valley companies, like Dropbox and AWS, to store large assets. After significant research, Katrib felt it was a good fit for DFT as well, believing it to be a more economical way to build petabytes of storage, compared to using proprietary block storage.

Ramy Katrib

“We were unique from most of the post houses in that respect,” says Katrib. “We were different from many of the other companies using object storage — they were tech, financial institutions, government agencies, health care; we were the rare one from M&E – but our need for extremely large, scalable and resilient storage was the same as theirs.”

DFT’s primary work centers around scripted television — an industry segment that continues to grow. “We do 15-plus television shows at any given time, and we encourage them to shoot whatever they like, at whatever resolution they desire,” says Katrib. “Most of the industry relies on LTO to back up camera raw materials. We do that too, but we also encourage productions to take advantage of our object storage, and we will store everything they shoot and not punish them for it. It is a rather Utopian workflow. We now give producers access to all their camera raw material. It is extremely effective for our clients.”

Over four years ago, DFT began using a cloud-based platform called OpenStack, which is open-source software that controls large pools of data, to build and design its own object storage system. “We have our own software developers and people who built our hardware, and we are able to adjust to the needs of our clients and the needs of our own workflow,” says Katrib.

DFT designs its custom PC- and Linux-based post systems, including chassis from Super Micro, CPUs from Intel and graphic cards from Nvidia. Storage is provided from a number of companies, including spinning-disc and SSD solutions from Seagate Technology and Western Digital.

DFT then deploys remote dailies systems worldwide, in proximity to where productions are shooting. Each day clients plug their production hard drives (containing all camera raw files) into DFT’s remote dailies system. From DFT’s facility, dailies technicians remotely produce editorial, viewing and promo dailies files, and transfer them to their destinations worldwide. All the while, the camera raw files are transported from the production location to DFT’s ProStack “massively scalable object storage.” In this case, “private cloud storage” consists of servers DFT designed that house all the camera raw materials, with management from DFT post professionals who support clients with access to and management of their files.

DFT provides color grading for Great News.

Recently, storage vendors such as Quantum and Avid have begun building and branding their own object storage solutions not unlike what DFT has constructed at its Universal City locale. And the reason is simple: Object storage provides a clear advantage because of reliability and the low cost. “We looked at it because the storage we were paying for, proprietary block storage, was too expensive to house all the data our clients were generating. And resolutions are only going up. So, every year we needed more storage,” Katrib explains. “We needed a solution that could scale with the practical reality we were living.”

Then, about four years ago when DFT started becoming a software company, one of the developers brought OpenStack to Katrib’s attention. “The open-source platform provided several storage solutions, networking capabilities and cloud compute capabilities for free,” he points out. Of course, the solution is not a panacea, as it requires a company to customize the offering for its own needs and even contribute back to the OpenStack community. But then again, that requirement enables DFT to evolve to the changing needs of its clients without waiting for a manufacturer to do it.

“It does not work out of the box like a solution from IBM, for instance. You have to develop around it,” Katrib says. “You have to have a lab mentality, designing your own hardware and software based on pain points in your own environment. And, sometimes it fails. But when you do it correctly, you realize it is an elegant solution.” However, there are vibrant communities, user groups and tech summits of those leveraging the technology who are willing to assist and collaborate.

DFT has evolved its object storage solution, extending its capabilities from an initial hundreds of terabytes – which is nothing to sneeze at — to hundreds of petabytes of storage. DFT also designs remote post systems and storage solutions for customers in remote locations around the world. And those remote locations can be as simple as a workstation running applications such as Blackmagic’s Resolve or Adobe After Effects and connected to object storage housing all the client’s camera raw material.

The key, Katrib notes, is to have great post and IT pros managing the projects and the system. “I can now place a remote post system with a calibrated 4K monitor and object storage housing the camera raw material, and I can bring the post process to you wherever you are, securely,” he adds. “From wherever you are, you can view the conform, color and effects, and sign off on the final timeline, as if you were at DFT.”

DFT posts American Housewife

In addition to the object storage, DFT is also using Facilis TerraBlock and Avid Nexis systems locally and on remote installs. The company uses those commercial solutions because they provide benefits, including storage performance and feature sets that optimize certain software applications. As Katrib points out, storage is not one flavor fits all, and different solutions work better for certain use cases. In DFT’s case, the commercial storage products provide performance for the playback of multiple 4K streams across the company’s color, VFX and conform departments, while its ProStack high-capacity object storage comes into play for storing the entirety of all files produced by our clients.

“Rather than retrieve files from an LTO tape, as most do when working on a TV series, with object storage, the files are readily available, saving hours in retrieval time,” says Katrib.

Currently, DFT is working on a number of television series, including Great News (color correction only) and Good Behavior (dailies only). For other shows, such as the Roseanne revival, NCIS: Los Angeles, American Housewife and more, it is performing full services such as visual effects, conform, color, dailies and dubbing. And in some instances, even equipment rental.

As the work expands, DFT is looking to extend upon its storage and remote post systems. “We want to have more remote systems where you can do color, conform, VFX, editorial, wherever you are, so the DP or producer can have a monitor in their office and partake in the post process that’s particular to them,” says Katrib. “That is what we are scaling as we speak.”

Broadway Video
Broadway Video is a global media and entertainment company that is primarily engaged in post-production services for television, film, music, digital and commercial projects for the past four decades. Located in New York and Los Angeles, the facility offers one-stop tools and talent for editorial, audio, design, color grading, finishing and screening, as well as digital file storage, preparation, aggregation and delivery of digital content across multiple platforms.

Since its founding in 1979, Broadway Video has grown into an independent studio. During this timeframe, content has evolved greatly, especially in terms of resolution, to where 4K and HD content — including HDR and Atmos sound — is becoming the norm. “Staying current and dealing with those data speeds are necessary in order to work fluidly on a 4K project at 60p,” says Stacey Foster, president and managing director, Broadway Video Digital and Production. “The data requirements are pretty staggering for throughput and in terms of storage.”

Stacey Foster

This led Broadway Video to begin searching a year ago for a storage system that would meet its needs now as well as in the foreseeable future — in short, it also needed a system that is scalable. Their solution: an all-Flash Hitachi Vantara Virtual Storage Platform (VSP) G series. Although quite expensive, a flash-based system is “ridiculously powerful,” says Foster. “Technology is always marching forward, and Flash-based systems are going to become the norm; they are already the norm at the high end.”

Foster has had a long-standing relationship with Hitachi for more than a decade and has witnessed the company’s growth into M&E from the medical and financial worlds where it has been firmly ensconced. According to Foster, Hitachi’s VSP series will enhance Broadway Video’s 4K offerings and transform internal operations by allowing quick turnaround, efficient and cost-effective production, post production and delivery of television shows and commercials. And, the system offers workload scalability, allowing the company to expand and meet the changing needs of the digital media production industry.

“The systems we had were really not that capable of handling DPX files that were up to 50TB, and Hitachi’s VSP product has been handling them effortlessly,” says Foster. “I don’t think other [storage] manufacturers can say that.”

Foster explains that as Broadway Video continued to expand its support of the latest 4K content and technologies, it became clear that a more robust, optimized storage solution was needed as the company moved in this new direction. “It allows us to look at the future and create a foundation to build our post production and digital distribution services on,” Foster says.

Broadway Video’s with Netflix projects sparked the need for a more robust system. Recently, Comedians in Cars Getting Coffee, an Embassy Row production, transitioned to Netflix, and one of the requirements by its new home was the move from 2K to 4K. “It was the perfect reason for us to put together a 4K end-to-end workflow that satisfies this client’s requirements for technical delivery,” Foster points out. “The bottleneck in color and DPX file delivery is completely lifted, and the post staff is able to work quickly and sometimes even faster than in real time when necessary to deliver the final product, with its very large files. And that is a real convenience for them.”

Broadway Video’s Hitachi Vantara Virtual Storage Platform G series.

As a full-service post company, Broadway Video in New York operates 10 production suites of Avids running Adobe Premiere and Blackmagic Resolve, as well as three full mixing suites. “We can have all our workstations simultaneously hit the [storage] system hard and not have the system slow down. That is where Hitachi’s VSP product has set itself apart,” Foster says.

For Comedians in Cars Getting Coffee, like many projects Broadway Video encounters, the cut is in a lower-resolution Avid file. The 4K media is then imported into the Resolve platform, so it is colored in its original material and format. In terms of storage, once the material is past the cutting stage, it is all stored on the Hitachi system. Once the project is completed, it is handed off on spinning disc for archival, though Foster foresees a limited future for spinning discs due to their inherent nature for a limited life span — “anything that spins breaks down,” he adds.

All the suites are fully HD-capable and are tied with shared SAN and ISIS storage; because work on most projects is shared between editing suites, there is little need to use local storage. Currently Broadway Video is still using its previous Avid ISIS products but is slowly transitioning to the Hitachi system only. Foster estimates that at this time next year, the transition will be complete, and the staff will no longer have to support the multiple systems. “The way the systems are set up right now, it’s just easier to cut on ISIS using the Avid workstations. But that will soon change,” he says.

Currently, Broadway Video is still using its Avid ISIS products but is slowly transitioning to the Hitachi system. Foster estimates that at this time next year, the transition will be complete, and the staff will no longer have to support the multiple systems. “The way the systems are set up right now, it’s just easier to cut on ISIS using the Avid workstations. But that will soon change,” he says.

Other advantages the Hitachi system provides is stability and uptime, which Foster maintains is “pretty much 100 percent guaranteed.” As he points out, there is no such thing as downtime in banking and medical, where Hitachi earned its mettle, and bringing that stability to the M&E industry “has been terrific.”

Of course, that is in addition to bandwidth and storage capacity, which is expandable. “There is no limit to the number of petabytes you can have attached,” notes Foster.

Considering that the majority of calls received by Broadway Video center on post work for 4K-based workflows, the new storage solution is a necessary technical addition to the facility’s other state-of-the-art equipment. “In the environment we work in, we spend more and more time on the creative side in terms of the picture cutting and sound mixing, and then it is a rush to get it out the door. If it takes you days to import, color correct, export and deliver — especially with the file sizes we are talking about – then having a fast system with the kind of throughput and bandwidth that is necessary really lifts the burden for the finishing team,” Foster says.

He continues: “The other day the engineers were telling me we were delivering 20 times faster using the Hitachi technology in the final cutting and coloring of a Jerry Seinfeld stand-up special we had done in 4K” resulting in a DPX file that was about 50TB. “And that is pretty significant,” Foster adds.

Main Image: DigitalFilm Tree’s senior colorist Patrick Woodard.

NAB: Critique upped to version 4, using AWS for cloud

From the minds at LA-based post house DigitalFilm Tree comes a new version of Critique, its cloud-collaboration software. Critique, which is now in v.4, is already used on shows such as Modern Family, The Simpsons and NCIS: Los Angeles. In addition to many new features and security controls in Critique 4, this is the first time the app has been deployed on AWS.

Critique’s new relationship with AWS is key to version 4, says Guillaume Aubuchon, CIO of Critique. “AWS is not only the largest cloud provider, but they are the cloud provider of choice in the M&E space. Our infrastructure shift to AWS afforded us the ability to architect the software to leverage the range of services in the AWS cloud platform. It allowed us to build Critique 4 from scratch in a matter of mere months.”

Critique 4 is a secure digital media asset management (MAM) platform with extensive features to support creative processes and production workflow for both the media and entertainment space as well as enterprise. Built to be extremely easy to use, Critique facilitates collaboration through realtime chat, live annotations, and secure sharing over the Internet to deliver productions on time and on budget. Realtime chat and drawing annotations are viewable across the Web and iOS — they also work with the new Apple Pencil for iPad Pro.

Designed to improve workflow, the software facilitates every step from protected dailies screening to VFX workflows to post to distribution while capitalizing on enterprise-level security to protect valuable assets.

Critique 4 was born of the minds of its executive team of Aubuchon, a veteran in the production space having worked on such projects such as Her, NCIS:LA and Angie Tribeca, and Chris Chen, an expert in the production streaming space and the former CTO of DAX. With its ability to use its own DigitalFilm Tree as a beta test site, Critique is built to ensure it works in real-world media environments.

One of the new exciting features of Critique 4 is its ability to index Amazon Simple Storage Service (Amazon S3) to allow companies to manage their own content inside of Critique’s award-winning interface. It also offers high-performance cloud MAM for simultaneous video and document management: Users can collaborate with Critique’s review, approval and annotation workflows not only for video but also for production documents including scripts, graphics and still images.

“Digital Rights Management (DRM) protection is rarely used, if at all, for unreleased content, which is arguably where it is needed the most,” notes Chen. “Critique was designed to leverage DRM invisibly throughout its video distribution system on desktop, web, and mobile environments. This allows Critique to break through the legacy walled-garden approach, allowing a new level of flexibility in collaboration while maintaining security. But we do it in such a way that the users don’t even know it’s there.”

The ability to share assets in this way expands its mobility and Critique is available via web, phones, tablets and Apple TV. The video service is backed by a true CDN running multi-bit-rate video to prevent glitches on any platform. “Users can take advantage of Critique anywhere — in their office, living room, the subway or even on a plane,” explains Chen. “And it will be true to the original media.

Other highlights of Critique 4 include: storage, archiving and management of Raw material; automatic transcoding of Raw material into a proxy format for viewing; granular permissions on files, folders, and projects; easy-to-manage sharing functions for users outside the system with the ability to time-limit and revoke/extend individual permissions; customizable watermarking on videos.

While Critique was born in the creative and operations side of the media and entertainment market, it is extending to enterprise, small to medium-size businesses, publishing, education and government/military sectors.

This latest version of Critique is available now for a free 30-day trial (AWS usage fees apply). Pricing is extremely competitive with 10, 20, 50 and 100 user levels starting as low as $39 per user. Enterprise level contracts are available for larger projects and companies with multiple projects. The fee includes unlimited streaming of current content and 24/7 white-glove tech support. AppleTV, Apple iPad and iPhone apps are also included. For a nominal fee, users can add DRM, high-resolution cloud transcode and storage for camera raw and mezzanine files.

HPA offers up winners of Engineering Excellence, Judges awards

With the HPA Awards ceremony a month away, the Hollywood Post Alliance has released the names of companies and organizations that will be receiving honors this year for their technology and innovations.

The 2014 Engineering Excellence Award and the HPA Judges Award for Creativity and Innovation in Post Production awards recognize the technical excellence and creative innovation that continually drive the post production industry in the advancement and support of content creation.

The awards will be handed out on November 6 during the 9th Annual HPA Awards event at the Skirball Cultural Center in Los Angeles.

According to the HPA, the Engineering Excellence Award spotlights companies and individuals who draw upon technical and creative ingenuity and apply it to real-world post demands while raising the profile of breakthrough technologies.

The winners of the 2014 HPA Engineering Excellence Awards are:

Macom: 12G-SDI Chipset
12G-SDI is the next data rate in the evolution of SDI, and Macom has introduced the industry’s first complete chipset to enable next-generation 4K video production applications.  Our newest family of SDI equalizers, reclockers and cable drivers supports 4K video resolutions at 60 frames per second over a single link and complements our industry-leading crosspoint switch portfolio.

Nvidia’s VCA

Nvidia: Nvidia VCA
Accelerate design and VFX workflows with Nvidia Visual Computing Appliance (VCA), the “fastest way” to interactive photorealistic digital 3D models and scenes. This network-attached appliance easily integrates into the design workflow and scales to multiple VCAs, each decreasing the time to noiseless, physically based global illumination.

Wohler and Cinnafilm Joint Venture: Tachyon Wormhole 

Tachyon Wormhole is a file-based Retiming and Standards Transcoding solution which leverages commodity enterprise hardware to deliver content based on time or standards requirements. Fully automated with audio pitch correction and caption retiming, Tachyon Wormhole processes two files faster than real time, simultaneously, on a single small footprint server.

“The HPA Engineering Excellence Award is a tremendous honor and true recognition of the technology innovations achieved with Tachyon Wormhole.  I am excited that this technology also offers our post production customers a truly file-based solution to other more expensive and less versatile alternatives,” said Craig Newbury, VP of  sales, Wohler.

The HPA Judges Award for Creativity and Innovation in Post Production was conceived to recognize companies and individuals who have demonstrated excellence, whether in the development of workflow and process to support creative storytelling and/or technical innovation. A jury of peers and industry experts determines the HPA Judges Award winners.

The 2014 winners of the HPA Judges Award for Creativity and Innovation in Post Production are:

American Society of Cinematographers: Color Decision List (ASC CDL)
The ASC CDL helps maintain and communicate creative intent, from production, dailies, post, VFX and editorial, in a multi-facility, multi-vendor environment. The ASC CDL enables creatives to set the look of a shot via system-independent primary color corrections communicated as metadata from on-set through dailies and post. Developed by the ASC Technology Committee — a broad group of industry members working to benefit the industry — the ASC CDL is freely available, saving time and money, improving the artistic result and audience experience in the vast majority of motion pictures, TV shows and VFX.

“The ASC CDL was developed to help all participants in the production and post pipeline most effectively produce the best result and deliver the creative vision of filmmakers to audiences,” said David Reisner, D-Cinema Consulting. “The ASC CDL could only have been created by filmmakers, the post community, and vendors’ active collaboration.”

DigitalFilm Tree Cloud Post Workflow Initiative (ProStack)
ProStack enables instantly accessible, widely deployable post production. ProStack packs the capacity and bandwidth needed to manage the entire post process- high-end color correction, VFX and DI workflows- into a scalable solution. Flexible design and implementation incorporates the best tools for every job, connecting all users via cloud storage. ProStack, in conjunction with Critique, integrates the entire spectrum of file-based workflows into one hub, while simultaneously embracing the evolving role of production and the post facility.

Digitalfilm Tree’s Guillaume Aubuchon

“We have been focused on developing a system that is truly cloud based and easily useable for our clients, an OpenStack environment with secure and instantaneous access,” said Guillaume Aubuchon, DigitalFilm Tree CTO. “It is an honor to be recognized by the HPA for our work on ProStack, and we are honored to receive our second C&I Award.”

Tickets for the HPA Awards are on sale now.

DigitalFilm Tree embraces OpenStack and cloud-based post workflows

The studio is running OpenStack private clouds for TNT’s Perception and ABC’s Mistresses.

By Randi Altman
Los Angeles — Ramy Katrib and the team over at DigitalFilm Tree (www.digitalfilmtree.com) have always set their own path. I first met Ramy at NAB in 2001. He was there looking at tools that would allow him to embrace a data-based workflow, something he thought was the future of post. He thought right.

He has spent over 14 years successfully creating data-based workflows for TV series like Scrubs, Cougar Town, NCIS: Los Angeles and feature films like Her.

Continue reading