Category Archives: Storage Update

Virtual Roundtable: Storage

By Randi Altman

The world of storage is ever changing and complicated. There are many flavors that are meant to match up to specific workflow needs. What matters most to users? In addition to easily-installed and easy-to-use systems that let them focus on the creative and not the tech? Scalability, speed, data protection, the cloud and the need to handle higher and higher frame rates with higher resolutions — meaning larger and larger files. The good news is the tools are growing to meet these needs. New technologies and software enhancements around NVMe are providing extremely low-latency connectivity that supports higher performance workflows. Time will tell how that plays a part in day-to-day workflows.

For this virtual roundtable, we reached out to makers of storage and users of storage. Their questions differ a bit, but their answers often overlap. Enjoy.

Company 3 NY and Deluxe NY Data/IO Supervisor Hollie Grant

Company 3 specializes in DI, finishing and color correction, and Deluxe is an end-to-end post house working on projects from dailies through finishing.

Hollie Grant

How much data did you use/backup this year? How much more was that than the previous year? How much more data do you expect to use next year?
Over the past year, as a rough estimate, my team dealt with around 1.5 petabytes of data. The latter half of this year really ramped up storage-wise. We were cruising along with a normal increase in data per show until the last few months where we had an influx of UHD, 4K and even 6K jobs, which take up to quadruple the space of a “normal” HD or 2K project.

I don’t think we’ll see a decrease in this trend with the take off of 4K televisions as the baseline for consumers and with streaming becoming more popular than ever. OTT films and television have raised the bar for post production, expecting 4K source and native deliveries. Even smaller indie films that we would normally not think twice about space-wise are shooting and finishing 4K in the hopes that Netflix or Amazon will buy their film. This means that even for the projects that once were not a burden on our storage will have to be factored in differently going forward.

Have you ever lost important data due to a hardware failure?Have you ever lost data due to an operator error? (Accidental overwrite, etc.)
Triple knock on wood! In my time here we have not lost any data due to an operator error. We follow strict procedures and create redundancy in our data, so if there is a hardware failure we don’t lose anything permanently. We have received hard drives or tapes that failed, but this far along in the digital age most people have more than one copy of their work, and if they don’t, a backup is the first thing I recommend.

Do you find access speed to be a limiting factor with you current storage solution?
We can reach read and write speeds of 1GB on our SAN. We have a pretty fast configuration of disks. Of course, the more sessions you have trying to read or write on a volume, the harder it can be to get playback. That’s why we have around 2.5PB of storage across many volumes so I can organize projects based on the bandwidth they will need and their schedules so we don’t have trouble with speed. This is one of the more challenging aspects of my day-to-day as the size of projects and their demand for larger frame playback increase.

Showtime’s Escape From Dannemora – Co3 provided color grading and conform.

What percentage of your data’s value do you budget toward storage and data security?
I can’t speak to exact percentages, but storage upgrades are a large part of our yearly budget. There is always an ask for new disks in the funding for the year because every year we’re growing along with the size of the data for productions. Our production network infrastructure is designed around security regulations set forth by many studios and the MPAA. A lot of work goes into maintaining that and one of the most important things to us is keeping our clients’ data safe behind multiple “locks and keys.”

What trends do you see in storage?
I see the obvious trends in physical storage size decreasing while bandwidth and data size increases. Along those lines I’m sure we’ll see more movies being post produced with everything needed in “the cloud.” The frontrunners of cloud storage have larger, more secure and redundant forms of storing data, so I think it’s inevitable that we’ll move in that direction. It will also make collaboration much easier. You could have all camera-original material stored there, as well as any transcoded files that editorial and VFX will be working with. Using the cloud as a sort of near-line storage would free up the disks in post facilities to focus on only having online what the artists need while still being able to quickly access anything else. Some companies are already working in a manner similar to this, but I think it will start to be a more common solution moving forward.

creative.space‘s Nick Anderson

What is the biggest trend you’ve seen in the past year in terms of storage?
The biggest trend is NVMe storage. SSDs are finally entering a range where they are forcing storage vendors to re-evaluate their architectures to take advantage of its performance benefits.

Nick Anderson

Can you talk more about NVMe?
When it comes to NVMe, speed, price and form factor are three key things users need to understand. When it comes to speed, it blasts past the limitations of hard drives speeds to deliver 3GB/s per drive, which requires a faster connector (PCIe) to take advantage of. With parallel access and higher IOPS (input/output operations per second), NVMe drives can handle operations that would bring an HDD to its knees. When it comes to price, it is cheaper per GB than past iterations of SSD, making it a feasible alternative for tier one storage in many workflows. Finally, when it comes to form factor, it is smaller and requires less hardware bulk in a purpose-built system so you can get more drives in a smaller amount of space at a lower cost. People I talk to are surprised to hear that they have been paying a premium to put fast SSDs into HDD form factors that choke their performance.

Users want more flexible workflows — storage in the cloud, on-premises, etc. Are your offerings reflective of that?
This is something we have been thinking a lot about and we have some exciting stuff in the works that addresses this need that I can’t go into at this time. For now, we are working with our early adopters to solve these needs in ways that are practical to them, integrating custom software as needed. Moving forward we hope to bring an intuitive and seamless storage experience to the larger industry.

With AI, VR and machine learning, etc., our industry is even more dependent on storage. How are you addressing this?
This gets down to a shift in what kind of data is being processed and how it can be accessed. When it comes to video, big media files and image sequences have driven the push for better performance. 360° video pushes the performance storage further past 4K into 8K, 12K, 16K and beyond. On the other hand, as CGI continues to become more photorealistic and we emerge from the “uncanny valley,” the performance need shifts from big data to small data in many cases as render engines are used instead of video or image files. Moving lots of small data is what these systems were originally designed for, so it will be a welcome shift for users.

When it comes to AI, our file system architectures and NVMe technology are making data easily accessible with less impact on performance. Apart from performance, we monitor thousands of metrics on the system that can be easily connected to your machine learning system of choice. We are still in the early days of this technology and its application to media production, so we are excited to see how customers take advantage of it.

What do you do in your products to help safeguard your users’ data?
From a data integrity perspective, every bit of data gets checksumed on copy and can be restored from that checksum if it gets corrupted. This means that that storage is self-healing with 100% data integrity once it is written to the disk.

As far as safeguarding data from external threats, this is a complicated issue. There are many methods of securing a system, but for post production, performance can’t be compromised. For companies following MPAA recommendations, putting the storage behind physical security is often considered enough. Unfortunately, for many companies without an IT staff, this is where the security stops and the system is left open once you get access to the network. To solve this problem, we developed an LDAP user management system that is built-in to our units that provides that extra layer of software security at no additional charge. Storage access becomes user-based, so system activity can be monitored. As far as administering support, we designed an API gatekeeper to manage data to and from the database that is auditable and secure.

AlphaDogs‘ Terence Curren

Alpha Dogs is a full-service post house in Burbank, California. They provide color correction, graphic design, VFX, sound design and audio mixing.

How much data did you go use/backup this year? How much more was that than the previous year? How much more data do you expect to use next year?
We are primarily a finishing house, so we use hundreds of TBs per year on our SAN. We work at higher resolutions, which means larger file sizes. When we have finished a job and delivered the master files, we archive to LTO and clear the project off the SAN. When we handle the offline on a project, obviously our storage needs rise exponentially. We do foresee those requirements rising substantially this year.

Terence Curren

Have you ever lost important data due to a hardware failure? Have you ever lost data due to an operator error? (Accidental overwrite, etc.)
We’ve been lucky in that area (knocking on wood) as our SANs are RAID-protected and we maintain a degree of redundancy. We have had clients’ transfer drives fail. We always recommend they deliver a copy of their media. In the early days of our SAN, which is the Facilis TerraBlock, one of our editors accidentally deleted a volume containing an ongoing project. Fortunately, Facilis engineers were able to recover the lost partition as it hadn’t been overwritten yet. That’s one of the things I really have appreciated about working with Facilis over the years — they have great technical support which is essential in our industry.

Do you find access speed to be a limiting factor with you current storage solution?
Not yet, As we get forced into heavily marketed but unnecessary formats like the coming 8K, we will have to scale to handle the bandwidth overload. I am sure the storage companies are all very excited about that prospect.

What percentage of your data’s value do you budget toward storage and data security?
Again, we don’t maintain long-term storage on projects so it’s not a large consideration in budgeting. Security is very important and one of the reasons our SANs are isolated from the outside world. Hopefully, this is an area in which easily accessible tools for network security become commoditized. Much like deadbolts and burglar alarms in housing, it is now a necessary evil.

What trends do you see in storage?
More storage and higher bandwidths, some of which is being aided by solid state storage, which is very expensive on our level of usage. The prices keep coming down on storage, yet it seems that the increased demand has caused our spending to remain fairly constant over the years.

Cinesite London‘s Chris Perschky

Perschky ensures that Cinesite’s constantly evolving infrastructure provides the technical backbone required for a visual effects facility. His team plans, installs and implements all manner of technology, in addition to providing technical support to the entire company.

Chris Perschky

How much data did you go use/backup this year? How much more was that than the previous year? How much more data do you expect to use next year?
Depending on the demands of the project that we are working on we can generate terabytes of data every single day. We have become increasingly adept at separating out data we need to keep long-term from what we only require for a limited time, and our cleanup tends to be aggressive. This allows us to run pretty lean data sets when necessary.

I expect more 4K work to creep in next year and, as such, expect storage demands to increase accordingly.

Have you ever lost important data due to a hardware failure? Have you ever lost data due to an operator error? (Accidental overwrite, etc.)
Our thorough backup procedures mean that we have an offsite copy of all production data within a couple of hours of it being written. As such, when an artist has accidentally overwritten a file we are able to retrieve it from backup swiftly.

Do you find access speed to be a limiting factor with your current storage solution?
Only remotely, thereby requiring a caching solution.

What percentage of your data’s value do you budget toward storage and data security?
Due to the requirements of our clients, we do whatever is necessary to ensure the security of their IP and our work.

Cinesite also worked on Iron Spider for Avengers Infinity War ©2018 Marvel Studios

What trends do you see in storage?
The trendy answer is to move all storage to the cloud, but it is just too expensive. That said, the benefits of cloud storage are well documented, so we need some way of leveraging it. I see more hybrid on-prem and cloud solutions. providing the best of both worlds as demand requires. Full SSD solutions are still way too expensive for most of us, but multi-tier storage solutions will have a larger SSD cache tier as prices drop.

Panasas‘ RW Hawkins

What is the biggest trend you’ve seen in the past year in terms of storage?
The demand for more capacity certainly isn’t slowing down! New formats like ProRes RAW, HDR and stereoscopic images required for VR continue to push the need to scale storage capacity and performance. New Flash technologies address the speed, but not the capacity. As post production houses scale, they see that complexity increases dramatically. Trying to scale to petabytes with individual and limited file servers is a big part of the problem. Parallel file systems are playing a more important role, even in medium-sized shops.

RW Hawkins

With AI, VR and machine learning, etc., our industry is even more dependent on storage. How are you addressing this?
VR (and, more generally, interactive content creation) is particularly interesting as it takes many of the aspects of VFX and interactive gaming and combines them with post. The VFX industry, for many years, has built batch-oriented pipelines running on multiple Linux boxes to solve many of their production problems. This same approach works well for interactive content production where the footage often needs to be pre-processed (stitched, warped, etc.) before editing. High speed, parallel filesystems are particularly well suited for this type of batch-based work.

The AI/ML space is red hot, and the applications seem boundless. Right now, much of the work is being done at a small scale where direct-attach, all-Flash storage boxes serve the need. As this technology is used on a larger scale, it will put demands on storage that can’t be met by direct-attached storage, so meeting those high IOP needs at scale is certainly something Panasas is looking at.

Can you talk about NVMe?
NVMe is an exciting technology, but not a panacea for all storage problems. While being very fast, and excellent at small operations, it is still very expensive, has small capacity and is difficult to scale to petabyte sizes. The next-generation Panasas ActiveStor Ultra platform uses NVMe for metadata while still leveraging spinning disk and SATA SSD. This hybrid approach, using each storage medium for what it does best, is something we have been doing for more than 10 years.

What do you do in your products to help safeguard your users’ data?
Panasas uses object-based data protection with RAID- 6+. This software-based erasure code protection, at the file level, provides the best scalable data protection. Only files affected by a particular hardware failure need to be rebuilt, and increasing the number of drives doesn’t increase the likelihood of losing data. In a sense, every file is individually protected. On the hardware side, all Panasas hardware provides non-volatile components, including cutting-edge NVDIMM technology to protect our customers’ data. The file system has been proven in the field. We wouldn’t have the high-profile customers we do if we didn’t provide superior performance as well as superior data protection.

Users want more flexible workflows — storage in the cloud, on-premises, etc. How are your offerings reflective of that?
While Panasas leverages an object storage backend, we provide our POSIX-compliant file system client called DirectFlow to allow standard file access to the namespace. Files and directories are the “lingua franca” of the storage world, allowing ultimate compatibility. It is very easy to interface between on-premises storage, remote DR storage and public cloud/REST storage using DirectFlow. Data flows freely and at high speed using standard tools, which makes the Panasas system an ideal scalable repository for data that will be used in a variety of pipelines.

Alkemy X‘s Dave Zeevalk

With studios in Philly, NYC, LA and Amsterdam, Alkemy X provides live-action, design, post, VFX and original content for spots, branded content and more.

Dave Zeevalk

How much data did you go use/backup this year? How much more was that than the previous year? How much more data do you expect to use next year?
Each year, our VFX department generates nearly a petabyte of data, from simulation caches to rendered frames. This year, we have seen a significant increase in data usage as client expectations continue to grow and 4K resolution becomes more prominent in episodic television and feature film projects.

In order to use our 200TB server responsibly, we have created a solid system for preserving necessary data and clearing unnecessary files on a regular basis. Additionally, we are diligent in archiving finale projects to our LTO tape systems, and removing them from our production server.

Have you ever lost important data due to a hardware failure? Have you ever lost data due to an operator error? (Accidental overwrite, etc.)

Because of our data redundancy, through hourly snapshots and daily backups, we have avoided any data loss even with hardware failure. Although hardware does fail, with these snapshots and backups on a secondary server, we are able to bring data back online extremely quickly in the case of hardware failure on our production server. Years ago, while migrating to Linux, a software issue completely wiped out our production server. Within two hours, we were able to migrate all data back from our snapshots and backups to our production server with no data loss.

Do you find access speed to be a limiting factor with your current storage solution?
There are a few scenarios where we do experience some issues with access speed to the production server. We do a good amount of heavy simulation work, at times writing dozens of terabytes per hour. While at our peak, we have experienced some throttled speeds due to the amount of data being written to the server. Our VFX team also has a checkpoint system for simulation where raw data is saved to the server in parallel to the simulation cache. This allows us to restart a simulation mid-way through the process if a render node drops or fails the job. This raw data is extremely heavy, so while using checkpoints on heavy simulations, we also experience some slower than normal speeds.

What percentage of your data’s value do you budget toward storage and data security?
Our active production server houses 200TB of storage space. We have a secondary backup server, with equivalent storage space that we store hourly snapshots and daily back-ups to.

What trends do you see in storage?
With client expectations continuing to rise, and 4K (and higher at times) becoming more and more regular on jobs, the need for more storage space is ever increasing.

Quantum‘s Jamie Lerner

What is the biggest trend you’ve seen in the past year in terms of storage?
Although the digital transformation to higher resolution content in M&E has been taking place over the past several years, the interesting aspect is that the pace of change over the past 12 months is accelerating. Driving this trend is the mainstream adoption of 4K and high dynamic range (HDR) video, and the strong uptick in applications requiring 8K formats.

Jamie Lerner

Virtual reality and augmented reality applications are booming across the media and entertainment landscape; everywhere from broadcast news and gaming to episodic television. These high-resolution formats add data to streams that must be ingested at a much higher rate, consume more capacity once stored and require significantly more bandwidth when doing realtime editing. All of this translates into a significantly more demanding environment, which must be supported by the storage solution.

With AI, VR and machine learning, etc., our industry is even more dependent on storage. How are you addressing this?
New technologies for producing stunning visual content are opening tremendous opportunities for studios, post houses, distributors, and other media organizations. Sophisticated next-generation cameras and multi-camera arrays enable organizations to capture more visual information, in greater detail than ever before. At the same time, innovative technologies for consuming media are enabling people to view and interact with visual content in a variety of new ways.

To capitalize on new opportunities and meet consumer expectations, many media organizations will need to bolster their storage infrastructure. They need storage solutions that offer scalable capacity to support new ingest sources that capture huge amounts of data, with the performance to edit and add value to this rich media.

Can you talk about NVMe?
The main benefit of NVMe storage is that it provides extremely low latency — therefore allowing users to seek content at very high speed — which is ideal for high stream counts and compressed 4K content workflows.

However, NVMe resources are expensive. Quantum addresses this issue head-on by leveraging NVMe over fabrics (NVMeoF) technology. With NVMeoF, multiple clients can use pooled NVMe storage devices across a network at local speeds and latencies. And when combined with our StorNext, all data is accessible by multiple clients in a global namespace, making this high-performance tier of storage much more cost-effective. Finally, Quantum is in early field trials of a new advancement that will allow customers to benefit even more from NVMe-enabled storage.

What do you do in your products to help safeguard your users’ data?
A storage system must be able to accommodate policies ranging from “throw it out when the job is done” to “keep it forever” and everything in between. The cost of storage demands control over where data lives and when, how many copies of the data exist and where those copies reside over time.

Xcellis scale-out storage powered by StorNext incorporates a broad range of features for data protection. This includes integrated features such as RAID, automated copying, versioning and data replication functionality, all included within our latest release of StorNext.

Users want more flexible workflows — storage in the cloud, on-premises, etc. Are your offerings reflective of that?
Given the differences in size and scope of organizations across the media industry, production workflows are incredibly varied and often geographically dispersed. Within this context, flexibility becomes a paramount feature of any modern storage architecture.

We provide flexibility in a number of important ways for our customers. From the perspective of system architecture, and recognizing there is no one-size fits all solution, StorNext allows customers to configure storage with multiple media types that balance performance and capacity requirements across an entire end-to-end workflow. Second, and equally important for those companies that have a global workforce, is that our data replication software FlexSync allows for content to be rapidly distributed to production staff around the globe. And no matter what tier of storage the data resides on, FlexTier provides coordinated and unified access to the content within a single global namespace.

EditShare‘s Bill Thompson

What is the biggest trend you’ve seen in the past year in terms of storage?
In no particular order, the biggest trends for storage in the media and entertainment space are:
1. The need to handle higher and higher data rates associated with higher resolution and higher frame rate content. Across the industry, this is being address with Flash-based storage and the use of emerging technology like NVMe over “X” and 25/50/100G networking.

Bill Thompson

2. The ever-increasing concern about content security and content protection, backup and restoration solutions.

3. The request for more powerful analytics solutions to better manage storage resources.

4. The movement away from proprietary hardware/software storage solutions toward ones that are compatible with commodity hardware and/or virtual environments.

Can you talk about NVMe?
NVMe technology is very interesting and will clearly change the M&E landscape going forward. One of the challenges is that we are in the midst of changing standards and we expect current PCIe-based NVMe components to be replaced by U2/M2 implementations. This migration will require important changes to storage platforms.

In the meantime, we offer non-NVMe Flash-based storage solutions whose performance and price points are equivalent to those claimed by early NVMe implementations.

What do you do in your products to help safeguard your users’ data?
EditShare has been in the forefront of user data protection for many years beginning with our introduction of disk-based and tape-based automated backup and restoration solutions.

We expanded the types of data protection schemes and provided easy-to-use management tools that allow users to tailor the type of redundant protection applied to directories and files. Similarly, we now provide ACL Media Spaces, which allow user privileges to be precisely tailored to their tasks at hand; providing only the rights needed to accomplish their tasks, nothing more, nothing less.

Most recently, we introduced EFS File Auditing, a content security solution that enables system administrators to understand “who did what to my content” and “when and how did they did it.”

Users want more flexible workflows — storage in the cloud, on-premises, etc. Are your offerings reflective of that?
The EditShare file system is now available in variants that support EditShare hardware-based solutions and hybrid on-premise/cloud solutions. Our Flow automation platform enables users to migrate from on-premise high-speed EFS solutions to cloud-based solutions, such as Amazon S3 and Microsoft Azure, offering the best of both worlds.

Rohde & Schwarz‘s Dirk Thometzek

What is the biggest trend you’ve seen in the past year in terms of storage?
Consumer behavior is the most substantial change that the broadcast and media industry has experienced over the past years. Content is consumed on-demand. In order to stay competitive, content providers need to produce more content. Furthermore, to make the content more desirable, technologies such as UHD and HDR need to be adopted. This obviously has an impact on the amount of data being produced and stored.

Dirk Thometzek

With AI, VR and machine learning, etc., our industry is even more dependent on storage. How are you addressing this?
In media and entertainment there has always been a remarkable growth of data over time, from the very first simple SCSI hard drives to huge network environments. Nowadays, however, there is a tremendous growth approximating an exponential function. Considering all media will be preserved for a very long time, the M&E storage market segment will keep on growing and innovating.

Looking at the amount of footage being produced, a big challenge is to find the appropriate data. Taking it a step further, there might be content that a producer wouldn’t even think of looking for, but has a relative significance to the original metadata queried. That is where machine learning and AI come into the play. We are looking into automated content indexing with the minimum amount of human interaction where the artificial intelligence learns autonomously and shares information with other databases. The real challenge here is to protect these intelligences from being compromised by unintentional access to the information.

What do you do to help safeguard your users’ data?
In collaboration with our Rohde & Schwarz Cybersecurity division, we are offering complete and protected packages to our customers. It begins with access restrictions to server rooms up to encrypted data transfers. Cyber attacks are complex and opaque, but the security layer must be transparent and usable. In media though, latency is just as critical, which is usually introduced with every security layer.

Can you talk about NVMe?
In order to bring the best value to the customer, we are constantly looking for improvements. The direct PCI communication of NVMe certainly brings a huge improvement in terms of latency since it completely eliminates the SCSI communication layer, so no protocol translation is necessary anymore. This results in much higher bandwidth and more IOPS.

For internal data processing and databases, R&S SpycerNode NVMe is used, which really boosts its performance. Unfortunately, the economic aspects of using this technology for media data storage is currently not considered to be efficient. We are dedicated to getting the best performance-to-cost ratio for the market, and since we have been developing video workstations and servers besides storage for decades now, we know how to get the best performance out of a drive — spinning or solid state.

Economically, it doesn’t seem to be acceptable to a build system with the latest and greatest technology for a workflow when standards will do, just because it is possible. The real art of storage technology lies in a highly customized configuration according to the technical requirements of an application or workflow. R&S SpycerNode will evolve over time and technologies will be added to the family.

Users want more flexible workflows — storage in the cloud, on-premises, etc. Are your offerings reflective of that?
Although hybrid workflows are highly desirable, it is quite important to understand the advantages and limits of this technology. High-bandwidth and low-latency wide-area network connections involve certain economical aspects. Without the suitable connection, an uncompressed 4K production does not seem feasible from a remote location — uploading several terabytes to a co-location can take hours or even days to be transferred, even if protocol acceleration is used. However, there are workflows, such as supplemental rendering or proxy editing, that do make sense to offload to a datacenter. R&S SpycerNode is ready to be an integral part of geographically scattered networks and the Spycer Storage family will grow.

Dell EMC‘s Tom Burns

What is the biggest trend you’ve seen in the past year in terms of storage?
The most important storage trend we’ve seen is an increasing need for access to shared content libraries accommodating global production teams. This is becoming an essential part of the production chain for feature films, episodic television, sports broadcasting and now e-sports. For example, teams in the UK and in California can share asset libraries for their file-based workflow via a common object store, whether on-prem or hybrid cloud. This means they don’t have to synchronize workflows using point-to-point transmissions from California to the UK, which can get expensive.

Tom Burns

Achieving this requires seamless integration of on-premises file storage for the high-throughput, low-latency workloads with object storage. The object storage can be in the public cloud or you can have a hybrid private cloud for your media assets. A private or hybrid cloud allows production teams to distribute assets more efficiently and saves money, versus using the public cloud for sharing content. If the production needs it to be there right now, they can still fire up Aspera, Signiant, File Catalyst or other point-to-point solutions and have prioritized content immediately available, while allowing your on-premise cloud to take care of the shared content libraries.

Users want more flexible workflows — storage in the cloud, on-premises, etc. Are your offerings reflective of that?
Dell Technologies offers end-to-end storage solutions where customers can position the needle anywhere they want. Are you working purely in the cloud? Are you working purely on-prem? Or, like most people, are you working somewhere in the middle? We have a continuous spectrum of storage between high-throughput low-latency workloads and cloud-based object storage, plus distributed services to support the mix that meets your needs.

The most important thing that we’ve learned is that data is expensive to store, granted, but it’s even more expensive to move. Storing your assets in one place and having that path name never change, that’s been a hallmark of Isilon for 15 years. Now we’re extending that seamless file-to-object spectrum to a global scale, deploying Isilon in the cloud in addition to our ECS object store on premises.

With AI, VR and machine learning, etc., our industry is even more dependent on storage. How are you addressing this?
AR, VR, AI and other emerging technologies offer new opportunities for media companies to change the way they tell and monetize their stories. However, due to the large amounts of data involved, many media organizations are challenged when they rely on storage systems that lack either scalability or performance to meet the needs of these new workflows.

Dell EMC’s file and object storage solutions help media companies cost effectively tier their content based upon access. This allows media organizations to use emerging technologies to improve how stories are told and monetize their content with the assistance of AI-generated metadata, without the challenges inherent in many traditional storage systems.

With artificial intelligence, for example, where it was once the job of interns to categorize content in projects that could span years, AI gives media companies the ability to analyze content in near-realtime and create large, easily searchable content libraries as the content is being migrated from existing tape libraries to object-based storage, or ingested for current projects. The metadata involved in this process includes brand recognition and player/actor identification, as well as speech-to-text, making it easy to determine logo placement for advertising analytics and to find footage for use in future movies or advertisements.

With Dell EMC storage, AI technologies can be brought to the data, removing the need to migrate or replicate data to direct-attach storage for analysis. Our solutions also offer the scalability to store the content for years using affordable archive nodes in Isilon or ECS object storage.

In terms of AR and VR, we are seeing video game companies using this technology to change the way players interact with their environments. Not only have they created a completely new genre with games such as Pokemon Go, they have figured out that audiences want nonlinear narratives told through realtime storytelling. Although AR and VR adoption has been slower for movies and TV compared to the video game industry, we can learn a lot from the successes of video game production and apply similar methodologies to movie and episodic productions in the future.

Can you talk about NVMe?
NVMe solutions are a small but exciting part of a much larger trend: workflows that fully exploit the levels of parallelism possible in modern converged architectures. As we look forward to 8K, 60fps and realtime production, the usage of PCIe bus bandwidth by compute, networking and storage resources will need to be much more balanced than it is today.

When we get into realtime productions, these “next-generation” architectures will involve new production methodologies such as realtime animation using game engines rather than camera-based acquisition of physically staged images. These realtime processes will take a lot of cooperation between hardware, software and networks to fully leverage the highly parallel, low-latency nature of converged infrastructure.

Dell Technologies is heavily invested in next-generation technologies that include NVMe cache drives, software-defined networking, virtualization and containerization that will allow our customers to continuously innovate together with the media industry’s leading ISVs.

What do you do in your products to help safeguard your users’ data?
Your content is your most precious capital asset and should be protected and maintained. If you invest in archiving and backing up your content with enterprise-quality tools, then your assets will continue to be available to generate revenue for you. However, archive and backup are just two pieces of data security that media organizations need to consider. They must also take active measures to deter data breaches and unauthorized access to data.

Protecting data at the edge, especially at the scale required for global collaboration can be challenging. We simplify this process through services such as SecureWorks, which includes offerings like security management and orchestration, vulnerability management, security monitoring, advanced threat services and threat intelligence services.

Our storage products are packed with technologies to keep data safe from unexpected outages and unauthorized access, and to meet industry standards such as alignment to MPAA and TPN best practices for content security. For example, Isilon’s OneFS operating system includes SyncIQ snapshots, providing point-in-time backup that updates automatically and generates a list of restore points.

Isilon also supports role-based access control and integration with Active Directory, MIT Kerberos and LDAP, making it easy to manage account access. For production houses working on multiple customer projects, our storage also supports multi-tenancy and access zones, which means that clients requiring quarantined storage don’t have to share storage space with potential competitors.

Our on-prem object store, ECS, provides long-term, cost-effective object storage with support for globally distributed active archives. This helps our customers with global collaboration, but also provides inherent redundancy. The multi-site redundancy creates an excellent backup mechanism as the system will maintain consistency across all sites, plus automatic failure detection and self-recovery options built into the platform.

Scale Logic‘s Bob Herzan

What is the biggest trend you’ve seen in the past year in terms of storage?
There is and has been a considerable buzz around cloud storage, object storage, AI and NVMe. Scale Logic recently took a private survey to its customer base to help determine the answer to this question. What we found is none of those buzzwords can be considered a trend. We also found that our customers were migrating away from SAN and focusing on building infrastructure around high-performance and scalable NAS.

Bob Herzan

They felt on-premises LTO was still the most viable option for archiving, and finding a more efficient and cost-effective way to manage their data was their highest priority for the next couple of years. There are plenty of early adopters testing out the buzzwords in the industry, but the trend — in my opinion — is to maximize a stable platform with the best overall return on the investment.

End users are not focused so much on storage, but on how a company like ours can help them solve problems within their workflows where storage is an important component.

Can you talk more about NVMe?
NVMe provides an any-K solution and superior metadata low-latency performance and works with our scale-out file system. All of our products have had 100GbE drivers for almost two years, enabling mesh technologies with NVMe for networks as well. As cost comes down, NVMe should start to become more mainstream this year — our team is well versed in supporting NVMe and ready to help facilities research the price-to-performance of NVMe to see if it makes sense for their Genesis and HyperFS Scale Out system.

With AI, VR and machine learning, our industry is even more dependent on storage. How are you addressing this?
We are continually refining and testing our best practices. Our focus on broadcast automation workflows over the years has already enabled our products for AI and machine learning. We are keeping up with the latest technologies, constantly testing in our lab with the latest in software and workflow tools and bringing in other hardware to work within the Genesis Platform.

What do you do in your products to help safeguard your users’ data?
This is a broad question that has different answers depending on which aspect of the Genesis Platform you may be talking about. Simply speaking, we can craft any number of data safeguard strategies and practices based on our customer needs, the current technology they are using and, most importantly, where they see their growth of capacity and data protection needs moving forward. Our safeguards start as simple as enterprise-quality components, mirrored sets, RAID -6, RAID-7.3 and RAID N+M, asynchronous data sync to a second instance, full HA with synchronous data sync to a second instance, virtual IP failover between multiple sites, multi-tier DR and business continuity solutions.

In addition, the Genesis Platform’s 24×7 health monitoring service (HMS) communicates directly with installed products at customer sites, using the equipment serial number to track service outages, system temperature, power supply failure, data storage drive failure and dozens of other mission-critical status updates. This service is available to Scale Logic end users in all regions of the world and complies with enterprise-level security protocols by relying only on outgoing communication via a single port.

Users want more flexible workflows — storage in the cloud, on-premises. Are your offerings reflective of that?
Absolutely. This question defines our go-to-market strategy — it’s in our name and part of our day-to-day culture. Scale Logic takes a consultative role with its clients. We take our 30-plus years of experience and ask many questions. Based on the answers, we can give the customer several options. First off, many customers feel pressured to refresh their storage infrastructure before they’re ready. Scale Logic offers customized extended warranty coverage that takes the pressure off the client and allows them to review their options and then slowly implement the migration and process of taking new technology into production.

Also, our Genesis Platform has been designed to scale, meaning clients can start small and grow as their facility grows. We are not trying to force a single solution to our customers. We educate them on the various options to solve their workflow needs and allow them the luxury of choosing the solution that best meets both their short-term and long-term needs as well as their budget.

Facilis‘ Jim McKenna

What is the biggest trend you’ve seen in the past year in terms of storage?
Recently, I’ve found that conversations around storage inevitably end up highlighting some non-storage aspects of the product. Sort of the “storage and…” discussion where the technology behind the storage is secondary to targeted add-on functionality. Encoding, asset management and ingest are some of the ways that storage manufacturers are offering value-add to their customers.

Jim McKenna

It’s great that customers can now expect more from a shared storage product, but as infrastructure providers we should be most concerned with advancing the technology of the storage system. I’m all for added value — we offer tools ourselves that assist our customers in managing their workflow — but that can’t be the primary differentiator. A premium shared storage system will provide years of service through the deployment of many supporting products from various manufacturers, so I advise people to avoid being caught-up in the value-add marketing from a storage vendor.

With AI, VR and machine learning, etc., our industry is even more dependent on storage. How are you addressing this?
Our industry has always been dependent upon storage in the workflow, but now facilities need to manage large quantities of data efficiently, so it’s becoming more about scaled networks. In the traditional SAN environment, hard-wired Fibre Channel clients are the exclusive members of the production workgroup.

With scalable shared-storage through multiple connection options, everyone in the facility can be included in the collaboration on a project. This includes offload machines for encoding and rendering large HDR and VR content, and MAM systems with localized and cloud analysis of data. User accounts commonly grow into the triple digits when producers, schedulers and assistants all require secure access to the storage network.

Can you talk about NVMe?
Like any new technology, the outlook for NVMe is promising. Solid state architecture solves a lot of problems inherent in HDD-based systems — seek times, read speeds, noise and cooling, form factor, etc. If I had to guess a couple years ago, I would have thought that SATA SSDs would be included in the majority of systems sold by now; instead they’ve barely made a dent in the HDD-based unit sales in this market. Our customers are aware of new technology, but they also prioritize tried-and-true, field-tested product designs and value high capacity at a lower cost per GB.

Spinning HDD will still be the primary storage method in this market for years to come, although solid state has advantages as a helper technology for caching and direct access for high-bandwidth requirements.

What do you do in your products to help safeguard your users’ data?
Integrity and security are priority features in a shared storage system. We go about our security differently than most, and because of this our customers have more confidence in their solution. By using a system of permissions that emanate from the volume-level, and are obscured from the complexities of network ownership attributes, network security training is not required. Because of the simplicity of securing data to only the necessary people, data integrity and privacy is increased.

In the case of data integrity during hardware failure, our software-defined data protection has been guarding our customers assets for over 13 years, and is continually improved. With increasing drive sizes, time to completion of drive recovery is an important factor, as well as system usability during the process.

Users want more flexible workflows — storage in the cloud, on-premises, etc. Are your offerings reflective of that?
When data lifecycle is a concern of our customers, we consult on methods of building a storage hierarchy. There is no one-size-fits-all approach here, as every workflow, facility and engineering scope is different.

Tier 1 storage is our core product line, but we also have solutions for nearline (tier 2) and archive (tier 3). When the discussion turns to the cloud as a replacement for some of the traditional on-premises storage offerings, the complexity of the pricing structure, access model and interface becomes a gating factor. There are a lot of ways to effectively use the cloud, such as compute (AI, encoding, etc.), business continuity, workflow (WAN collaboration) or simple cold storage. These tools, when combined with a strong on-premises storage network, will enhance productivity and ensure on-time delivery of product.

mLogic’s co-founder/CEO Roger Mabon

What is the biggest trend you’ve seen in the past year in terms of storage?
In the M&E industry, high-resolution 4K/8K multi-camera shoots,
stereoscopic VR and HDR video are commonplace and are contributing to the unprecedented amounts of data being generated in today’s media productions. This trend will continue as frame rates and resolutions increase and video professionals move to shoot in these new formats to future-proof their content.

Roger Mabon

With AI, VR and machine learning, etc., our industry is even more dependent on storage. Can you talk about that?
Absolutely. In this environment, content creators must deploy storage solutions that are high capacity, high-performance and fault-tolerant. Furthermore, all of this content must be properly archived so it can be accessed well in to the future. mLogic’s mission is to provide affordable RAID and LTO tape storage solutions that fit this critical need.

How are you addressing this?
The tsunami of data being produced in today’s shoots must be properly managed. First and foremost is the need to protect the original camera files (OCF). Our high-performance mSpeed Thunderbolt 3 RAID solutions are being deployed on-set to protect these OCF. mSpeed is a desktop RAID that features plug-and-play Thunderbolt connectivity, capacities up to 168TB and RAID-6 data protection. Once the OCF is transferred to mSpeed, camera cards can be wiped and put back into production

The next step involves moving the OCF from the on-set RAID to LTO tape. Our portable mTape Thunderbolt 3 LTO solutions are used extensively by media pros to transfer OCF to LTO tape. LTO tape cartridges are shelf stable for 30+ years and cost around $10 per TB. That said, I find that many productions skip the LTO transfer and rely solely on single hard drives to store the OCF. This is a recipe for disaster as hard drives sitting on a shelf have a lifespan of only three to five years. Companies working with the likes of Netflix are required to use LTO for this very reason. Completed projects should also be offloaded from hard drives and RAIDs to LTO tape. These hard drives systems can then be put back into action for the tasks that they are designed for… editing, color correction, VFX, etc.

Can you talk about NVMe?
mLogic does not currently offer storage solutions that incorporate NVMe technology, but we do recognize numerous use cases for content creation applications. Intel is currently shipping an 8TB SSD with PCIe NVMe 3.1 x4 interface that can read/write data at 3000+ MB/second! Imagine a crazy fast and ruggedized NVMe shuttle drive for on-set dailies…

What do you do in your products to help safeguard your users data?

Our 8- and12-drive mSpeed solutions feature hardware RAID data protection. mSpeed can be configured in multiple RAID levels including RAID-6, which will protect the content stored on the unit even if two drives should fail. Our mTape solutions are specifically designed to make it easy to offload media from spinning drives and archive the content to LTO tape for long term data preservation.

Users want more flexible workflows — storage in the cloud, on premise, etc. Are your offerings reflective of that?
We recommend that you make two LTO archives of your content that are geographically separated in secure locations such as the post facility and the production facility. Our mTape Thunderbolt solutions accomplish this task.

In regards to the cloud, transferring terabytes upon terabytes of data takes an enormous amount of time and can be prohibitively expensive, especially when you need to retrieve the content. For now, cloud storage is reserved for productions with big pipes and big budgets.

OWC president Jennifer Soulé 


With AI, VR and machine learning, etc., our industry is even more dependent on storage. How are you addressing this?

We’re constantly working to provide more capacity and faster performance.  For spinning disk solutions, we’re making sure that we’re offering the latest sizes in ever-increasing bays. Our ThunderBay line started as a four-bay, went to a six-bay and will grow to eight-bay in 2019. With 12TB drives, that’s 96TB in a pretty workable form factor. Of course, you also need performance, and that is where our SSD solutions come in as well as integrating the latest interfaces like Thunderbolt 3. For those with greater graphics needs, we also have our Helios FX external GPU box.

Can you talk about NVME?
With our Aura Pro X, Envoy Pro EX, Express 4M2 and ThunderBlade, we’re already into NVMe and don’t see that stopping. By the end of 2019 we expect virtually all of our external Flash-based solutions will be NVMe-based rather than SATA. As the cost of Flash goes down and performance and capacity go up, we expect broader adoption as both primary storage and in secondary cache setups. 2TB drive supply will stabilize and we should see 4TB  and PCIe Gen 4 will double bandwidth.  Bigger, faster and cheaper is a pretty awesome combination.

What do you do in your products to help safeguard your users data?
We focus more on providing products that are compatible with different encryption schemas rather than building something in. As far as overall data protection, we’re always focused on providing the most reliable storage we can. We make sure our power supplies are over what is required to make sure insufficient power is never a factor. We test a multitude of drives in our enclosures to ensure we’re providing the best performing drives.

For our RAID solutions, we do burn-in testing to make sure all the drives are solid. Our SoftRAID technology also provides in-depth drive health monitoring so you know well in advance if a drive is failing.  This is critical because many other SMART-based systems fail to detect bad drives leading to subpar system performance and corrupted data. Of course, all the hardware and software technology we put into our drives don’t do much if people don’t back up their data — so we also work with our customers to find the right solution for their use case or workflow.

Users want more flexible workflows — storage in the cloud, on premise, etc. Are your offerings reflective of that?
I definitely think we hit on flexibility within the on prem-space by offering a full range of single and multi-drive solutions, spinning disk and SSD options, portable to rack mounted that can be fully setup solutions or DIY where you can use drives you might already have. You’ll have to stay tuned on the cloud part, but we do have plans to use the cloud to expand on the data protection our drives already offer.

A Technologist’s Data Storage Primer

By Mike McCarthy

Storage is the concept of keeping all of the files for a particular project or workflow, but they may not all be stored in the same place — different types of data have different requirements and different storage solutions have different strengths and features.

At a fundamental level, most digital data is stored on HDDs or SSDs. HDDs, or hard disk drives, are mechanical devices that store the data on a spinning magnetic surface and move read/write heads over that surface to access the data. They currently max out around 200MB/s and 5ms latency.

SSDs, or solid-state drives, involve no moving parts. SSDs can be built with a number of different architectures and interfaces, but most are based on the same basic Flash memory technology as the CF or SD card in your camera. Some SSDs are SATA drives that use the same interface and form factor as a spinning disk for easy replacement in existing HDD-compatible devices. These devices are limited to SATA’s bandwidth of 600MB/s. Other SSDs use the PCIe interface, either in full-sized PCIe cards or the smaller M.2 form factor. These have much higher potential bandwidths, up to 3000MB/s.

Currently, HDDs are much cheaper for storing large quantities of data but require some level of redundancy for security. SSDs are also capable of failure, but it is a much more rare occurrence. Data recovery for either is very expensive. SSDs are usually cheaper for achieving high bandwidth, unless large capacities are also needed.

RAIDs
Traditionally, hard drives used in professional contexts are grouped together for higher speeds and better data security. These are called RAIDs, which stands for redundant array of independent disks. There are a variety of different approaches to RAID design that are very different from one another.

RAID-0 or striping is technically not redundant, but every file is split across each disk, so each disk only has to retrieve its portion of a requested file. Since these happen in parallel, the result is usually faster than if a single disk had read the entire file, especially for larger files. But if one disk fails, every one of your files will be missing a part of its data, making the remaining partial information pretty useless. The more disks in the array, the higher the chances of one failing, so I rarely see striped arrays composed of more than four disks. It used to be popular to create striped arrays for high-speed access to restorable data, like backed-up source footage, or temp files, but now a single PCIe SSD is far faster, cheaper, smaller and more efficient in most cases.

Sugar

RAID-1 or mirroring is when all of the data is written to more than one drive. This limits the array’s capacity to the size of the smallest source volume, but the data is very secure. There is no speed benefit to writes since each drive must write all of the data, but reads can be distributed across the identical drives with similar performance as RAID-0.

RAID-3, -5 and -6 try to achieve a balance between those benefits for larger arrays with more disks (minimum three). They all require more complicated controllers, so they are more expensive for the same levels of performance. RAID-3 stripes data across all but one drive and then calculates and stores parity (odd/even) data across the data drives and stores it on the last drive. This allows the data from any single failed drive to be restored, based on the parity data. RAID-5 is similar, but the parity volume is alternated depending on the block, allowing the reads to be shared across all disks, not just the “data drives.”

So the capacity of a RAID-3 or RAID-5 array will be the minimum individual disk capacity times the number of disks minus one. RAID-6 is similar but stores two drives worth of parity data, which via some more advanced math than odd/even, allows it to restore the data even if two drives fail at the same time. RAID-6 capacity will be the minimum individual disk capacity times the number of disks minus two, and is usually only used on arrays with many disks. RAID-5 is the most popular option for most media storage arrays, although RAID-6 becomes more popular as the value of the data stored increases and the price of extra drives decreases over time.

Storage Bandwidth
Digital data is stored as a series of ones and zeros, each of which is a bit. One byte is 8 bits, which frequently represents one letter of text, or one pixel of an image (8-bit single channel). Bits are frequently referenced in large quantities to measure data rates, while bytes are usually referenced when measuring stored data. I prefer to use bytes for both purposes, but it is important to know the difference. A Megabit (Mb) is one million bits, while a Megabyte (MB) is one million bytes, or 8 million bits. Similar to metric, Kilo is thousand, Mega is million, Giga is billion, and Tera is trillion. Anything beyond that you can learn as you go.

Networking speeds are measured in bits (Gigabits), but with headers and everything else, it is safer to divide by 10 when converting speed into bytes per second. Estimate 100MB/s for Gigabit, up to 1000MB/s on 10GB, and around 500MB/s for the new N-BaseT standard. Similarly, when transferring files over a 30Mb Internet connection, expect around 3MB/s, then multiple by 60 or 3,600 to get to minutes or hours (180MB/min or 9600MB/hr in this case). So if you have to download a 10GB file on that connection, come back to check on it in an hour.

Magnopus

Because networking standards are measured in bits, and because networking is so important for sharing video files, many video file types are measured in bits as well. An 8Mb H.264 stream is 1MB per second. DNxHD36 is 36Mb/s (or 4.5MB/s when divided by eight), DV and HDV are 25Mb, DVCProHD is 100Mb, etc. Other compression types have variable bit rates depending on the content, but there are still average rates we can make calculations from. Any file’s size divided by its duration will reveal its average data rate. It is important to make sure that your storage has the bandwidth to handle as many streams of video as you need, which will be that average data rate times the number of streams. So 10 streams of DNxHD36 will be 360Mb or 45MB/s.

The other issue to account for is IO requests and drive latency. Lots of small requests require not just high total transfer rates, but high IO performance as well. Hard drives can only fulfill around 100 individual requests per second, regardless of how big those requests are. So while a single drive can easily sustain a 45MB/s stream, satisfying 10 different sets of requests may keep it so busy bouncing between the demands that it can’t keep up. You may need a larger array, with a higher number of (potentially) smaller disks to keep up with the IO demands of multiple streams of data. Audio is worse in this regard in that you are dealing with lots of smaller individual files as your track count increases, even though the data rate is relatively low. SSDs are much better at handling larger numbers of individual requests, usually measured in the thousands or tens of thousands per second per drive.

Storage Capacity
Capacity on the other hand is simpler. Megabytes are usually the smallest increments of data that we have to worry about calculating. A media type’s data rate (in MB/sec) times its duration (in seconds) will give you its expected file size. If you are planning to edit a feature film with 100 hours of offline content in DNxHD36, that is 3600×100 seconds, times 4.5MB/s, equaling 1620000MB, 1620GB, or simply about 1.6TB. But you should add some headroom for unexpected needs, and then a 2TB disk is about 1.8TB when formatted, so it will just barely fit. It is probably worth sizing up to at least 3TB if you are planning to store your renders and exports on there as well.

Once you have a storage solution of the required capacity there is still the issue of connecting it to your system. The most expensive options connect through the network to make them easier to share (although more is required for true shared storage), but that isn’t actually the fastest option or the cheapest. A large array can be connected over USB3 or Thunderbolt, or via the SATA or SAS protocol directly to an internal controller.

There are also options for Fibre Channel, which can allow sharing over a SAN, but this is becoming less popular as 10GbE becomes more affordable. Gigabit Ethernet and USB3 won’t be fast enough for high-bandwidth files to playback, but 10GbE, multichannel SAS, Fibre Channel and Thunderbolt can all handle almost anything up to uncompressed 4K.

Direct attached storage will always have the highest bandwidth and lowest latency, as it has the fewest steps between the stored files and the user. Using Thunderbolt or USB adds another controller and hop, Ethernet even more so.

Different Types of Project Data
Now that we know the options for storage, let’s look at the data we anticipate needing to store. First off we will have lots of video footage of source media (either camera original files, transcoded editing dailies, or both). This is usually in the Terabytes, but the data rates vary dramatically — from 1Mb H.264 files to 200Mb ProRes files to 2400Mb Red files. The data rate for the files you are playing back, combined with the number of playback streams you expect to use, will determine the bandwidth you need from your storage system. These files are usually static in that they don’t get edited or written to in any way after creation.

The exceptions would be sidecar files like RMD and XML files, which will require write access to the media volume. If a certain set of files is static, as long as a backup of the source data exists, they don’t need to be backed up on a regular basis and don’t even necessarily need redundancy. Although if the cost of restoring that data would be high, in regards to lost time during that process, some level of redundancy is still recommended.

Another important set of files we will have is our project files, which actually record the “work” we do in our application. They contain instructions for manipulating our media files during playback or export. The files are usually relatively small, and are constantly changing as we use them. That means they need to be backed up on a regular basis. The more frequent the backups, the less work you lose when something goes wrong.

We will also have a variety of exports and intermediate renders over the course of the project. Whether they are flattened exports for upload and review, VFX files or other renders, these are a more dynamic set of files than our original source footage. And they are generated on our systems instead of being imported from somewhere else. These can usually be regenerated from their source projects, if necessary, but the time and effort required usually makes it worth it to invest in protecting or backing them up. In most workflows, these files don’t change once they are created, which makes it easier to back them up if desired.

There will also be a variety of temp files generated by most editing or VFX programs. Some of these files need high-speed access for best application performance, but they rarely need to be protected or backed up because they can be automatically regenerated by the source applications on the fly if needed.

Choosing the Right Storage for Your Needs
Ok, so we have source footage, project files, exports and temp files that we need to find a place for. If you have a system or laptop with a single data volume, the answer is simple: It all goes on the C drive. But we can achieve far better performance if we have the storage infrastructure to break those files up onto different devices. Newer laptops frequently have both a small SSD and a larger hard disk. In that case we would want our source footage on the (larger) HDD, while the project files should go on the (safer) SSD.

Usually your temp file directories should be located on the SSD as well since it is faster, and your exports can go either place, preferably the SSD if they fit. If you have an external drive of source footage connected, you can back all files up there, but you should probably work from projects stored on the local system, playing back media from the external drive.

A professional workstation can have a variety of different storage options available. I have a system with two SSDs and two RAIDs, so I store my OS and software on one SSD, my projects and temp files on the other SSD, my source footage on one RAID and my exports on the other. I also back up my project folder to the exports RAID on a daily basis, since the SSDs have no redundancy.

Individual Store Solution Case Study Examples
If you are natively editing a short film project shot on Red, then R3Ds can be 300MB/s. That is 1080GB/hour, so five hours of footage will be just over 5TB. It could be stored on a single 6TB external drive, but that won’t give you the bandwidth to play back in real-time (hard drives usually top out around 200MB/s).

Striping your data across two drives in one of those larger external drives would probably provide the needed performance, but with that much data you are unlikely to have a backup elsewhere. So data security becomes more of a concern, leading us toward a RAID-5-based solution. A four-disk array of 2TB drives provides 6TB of usable storage at RAID-5 (4x2TB = 8TB raw capacity, minus 2TB of parity data equals 6TB of usable storage capacity). Using an array of 8 1TB drives would provide higher performance, and 7TB of space before formatting (8x1TB = 8TB raw capacity, minus 1TB of parity, because a single drive failure would only lose 1TB of data in this configuration) but will cost more. (eight-port RAID controller, eight-bay enclosure, and two 1TB drives are usually more expensive than one 2TB drive.)

Larger projects deal with much higher numbers. Another project has 200TB of Red footage that needs to be accessible on a single volume. A 24-bay enclosure with 12TB drives provides 288TB of space, minus two drives worth of data for RAID-6 redundancy (288TB raw-[2x12TB for parity]=264TB usable capacity), which will be more like 240TB of space available in Windows once it is formatted.

Sharing Storage and Files With Others
As Ethernet networking technology has improved, the benefits of expensive SAN (storage area network) solutions over NAS (network attached storage) solutions has diminished. 10Gigabit Ethernet (10GbE) transfers over 1GB of data a second and is relatively cheap to implement. NAS has the benefit of a single host system controlling the writes, usually with software included in the OS. This prevents data corruption and also isolates the client devices from the file system allowing PC, Mac and Linux devices to all access the same files. This comes at the cost of slightly increased latency and occasionally lower total bandwidth, but the prices and complexity of installation are far lower.

So now all but the largest facilities and most demanding workflows are being deployed with NAS-based shared storage solutions. This can be as simple as a main editing system with a large direct attached array sharing its media with an assistant station, over a direct 10GbE link, for about $50. This can be scaled up by adding a switch and connecting more users to it, but the more users sharing the data, the greater the impact on the host system, and the lower the overall performance. Over 3-4 users, it becomes prudent to have a dedicated host system for the storage, for both performance and stability. Once you are buying a dedicated system, there are a variety of other functionalities offered by different vendors to improve performance and collaboration.

Bin Locking and Simultaneous Access
The main step to improve collaboration is to implement what is usually referred to as a “bin locking system.” Even with a top-end SAN solution and strict permissions controls there is still the possibility of users overwriting each other’s work, or at the very least branching the project into two versions that can’t easily be reconciled.

If two people are working on the same sequence at the same time, only one of their sets of changes is going to make it to the master copy of the file without some way of combining the changes (and solutions are being developed). But usually the way to avoid that is to break projects down into smaller pieces and make sure that no two people are ever working on the exact same part. This is accomplished by locking the part (or bin) of the project that a user is editing so that no one else may edit it at the same time. This usually requires some level of server functionality because it involves changes that are not always happening at the local machine.

Avid requires specific support for that from the storage host in order for it to enable that feature. Adobe on the other hand has implemented a simpler storage-based solution, which is effective but not infallible, that works on any shared storage device that offers users write access.

A Note on iSCSI
iSCSI arrays offer some interesting possibilities for read-only data, like source footage, as iSCSI gives block-level access for maximum performance and runs on any network without expensive software. The only limit is that only one system can copy new media to the volume, and there must be a secure way to ensure the remaining systems have read-only access. Projects and exports must be stored elsewhere, but those files require much less capacity and bandwidth than source media. I have not had the opportunity to test out this hybrid SAN theory since I don’t have iSCSI appliances to test with.

A Note on Top-End Ethernet Options
40Gb Ethernet products have been available for a while and we are now seeing 25GB and 100Gb Ethernet products as well. 40Gb cards can be gotten quite cheaply, and I was tempted to use them for direct connect, hoping to see 4GB/s to share fast SSDs between systems. But 40Gb Ethernet is actually a trunk of four parallel 10Gb links and each individual connection is limited to 10Gb. It is easy to share the 40Gb of aggregate bandwidth across 10 systems accessing a 40Gb storage host, but very challenging to get more than 10Gb to a single client system. Having extra lanes on the highway doesn’t get you to work any faster if there are no other cars on the road, it only helps when there is lots of competing traffic.

25Gb Ethernet on the other hand will give you access to nearly 3GB/s for single connections, but as that is newer technology, the prices haven’t come down yet ($500 instead of $50 for a 10GbE direct link). 100Gb Ethernet is four 25Gb links trunked together, and subject to the same aggregate limitations as 40Gb.

Main Image: Courtesy of Sugar Studios LA


Mike McCarthy is an online editor/workflow consultant with 10 years of experience on feature films and commercials. He has been involved in pioneering new solutions for tapeless workflows, DSLR filmmaking and multi-screen and surround video experiences. Check out his site.

DigitalGlue 12.3

Storage for Post Studios

By Karen Moltenbrey

The post industry relies heavily on storage solutions, without question. Facilities are jugging a variety of tasks and multiple projects all at once. And deadlines are always looming. Thus, these studios need a storage solution that is fast and reliable. Each studio has different needs and searches to find the right system to fit their particular workflow. Luckily, there are many storage choices for pros to choose from.

For this article, we spoke with two post houses about their storage solutions and why they are a good fit for each of their needs.

Sugar Studios LA
Sugar Studios LA is one-stop shop playground for filmmakers that offers a full range of post production services, including editorial, color, VFX, audio, production and finishing, with each department led by seasoned professionals. Its office suites in the Wiltern Theater Tower, in the center of LA, serve an impressive list of clients, from numerous independent film producers and distributors to Disney, Marvel, Sony, MGM, Universal, Showtime, Netflix, AMC, Mercedes-Benz, Ferrari and others.

Jijo Reed and Sting in one of their post suites.

With so much important data in play at one time, Sugar needs a robust, secure and reliable storage system. However, with diverse offerings come diverse requirements. For its online and color projects, Sugar uses a Symply SAN with 200TB of usable storage. The color workstations are connected via 10Gb Ethernet over Fibre with a 40Gb uplink to the network. For mass storage and offline work, the studio uses a MacOS server acting as a NAS, with 530TB of usable storage connected via a 40Gb network uplink. For Avid offline jobs, the facility has an Avid Nexis Pro with 40TB of storage, and for Avid Pro Tools collaboration, a Facilis TerraBlock with 40TB of usable storage.

“We can collaborate with any and all client stations working on the same or different media and sharing projects across multiple software platforms,” says Jijo Reed, owner/executive producer of Sugar. “No station is limited to what it can do, since every station has access to all media. Centralized storage is so important because not only does it allow collaboration, we always have access to all media and don’t have to fumble through drives. It is also RAID-protected, so we don’t have to be concerned with losing data.”

Prior to employing the centralized storage, Sugar had been using G-Technology’s G-RAID drives, changing over in late 2016. “Once our technical service advisor, Zach Moller, came on board, he began immediately to institute a storage network solution that was tailored to our workflow,” says Reed.

Reed, an award-winning director/producer, founded the company in 2012, using a laptop (running Final Cut Pro 7) and an external hard drive he had purchased on sale at Fry’s. His target base at the time was producers and writers needing sizzle trailers to pitch their projects — at a time when the term “sizzle trailer” was not part of the common vernacular. “I attended festivals to pitch my wares, producing over 15 sizzles the first year,” he says, “and it grew from there.”

Since Reed was creating sizzles for yet-to-be-made features, he was in “pole position” to handle the post for some of these independent films when they got funded. In 2015, he, along with his senior editor, Paul Buhl, turned their focus to feature post work, which was “more lucrative and less exhausting, but mostly, we wanted to tell stories – the whole story.” He rebranded and changed the name of the company from Sizzlepitch to Sugar Studios, and brought on a feature post producer, Chris Harrington. Reed invested heavily in the company, purchasing equipment and acquiring space. Soon, one bay became two, then three and so on. Currently, the company spans three full floors, including the penthouse of the Wiltern Theater Tower.

As Reed proudly points out, the studio space features 21 bays and workstations, two screening theaters, including a 25-seat color and mix DI stage with a Barco DP4K projector and Dolby Atmos configuration. “We are fully staffed, all under one roof, with editorial, full audio services, color correction/grading, VFX and a greenscreen cyclorama stage with on-site 4K cameras, grip and lighting,” he details. “But, it’s the people who make this work. Our passion is obvious to our clients.”

While Sugar was growing and expanding, so, too, was its mass storage solution. According to Zach Moller, it started with the NAS due to its low price and fast (10Gb) connection to every client machine. “The Symply SAN solution was needed because we required a high-bandwidth system for online and color playback that used Fibre Channel technology for the low latency and local drive configuration,” he says.

Moreover, the facility wanted flexibility with its SAN solution; it was very expensive to have every machine connected via Fibre Channel, “and frankly, we didn’t need that bandwidth,” Reed says. “Symply allowed us to have client machines choose whether they connected via Fibre Channel or 10Gb. If this wasn’t the case, we would have been in a pickle, having to purchase expansion chassis for every machine to open up additional PCI slots.” (The bulk of the machines at Sugar connect using the pre-existing 10Gb Ethernet over Fibre network, thus negating the need to use another PCI slot on a Fibre Channel card.)

American Dreamer

At Sugar, the camera masters and production audio are loaded directly to the NAS for mass storage. Then, the group archives the camera masters to LTO for deep archival, for an additional backup. During LTO archival, the studio creates the dailies for the offline edit on either Avid Media Composer (where the MXFs are migrated to the Avid Nexis server) or Adobe Premiere (where the ProRes dailies continue to live on the NAS).

When adding visual effects, the artists render to the Symply SAN when preparing for the online, color and finishing.

The studio works with a wide range of codecs, some of which are extremely taxing on the systems. And, the SAN is ideal, especially for the raster image files (EXRs), since each frame has such a high density — and there can be 100,000 frames per folder. “This can only be accomplished with a premium storage solution: our SAN,” Reed says.

When the studio moved to the EXR codec for the VFX on the American Dreamer feature film, for example, its original NAS solution over 10Gb didn’t have enough bandwidth for playback on its systems (1.2GB/sec). Once it upgraded the SAN solution with dual 16Gb Fibre Channel, they were able to play back uncompressed 4K EXR footage without the headache or frustration of stuttering.

“We have created an environment that caters to the creative process with a technical infrastructure that is superfast and solid. Filmmakers love us, and I couldn’t be prouder of my team for making this happen,” says Reed.

Mike Seabrooke

Postal
Established in 2015, Postal is a boutique creative studio that produces motion graphics, visual effects, animation, live action and editorial, with the vision of transcending all mediums — whether it’s short animations for social media or big-budget visual effects for broadcast. “As a studio, we love to experiment with different techniques. We feel strongly that the idea should always come first,” says Mike Seabrooke, producer at New York’s Postal.

To ensure that these ideas make it to the final stage of a project, the company uses a mixture of hard drives, LTO tapes and servers that house the content while the artists are working on projects, as well as for archival purposes. Specifically, the studio employs the EditShare Storage v.7 shared storage platform and EditShare Ark Tape for managing the LTO tape libraries that serve as nearline and offline backup. This is the system setup that Postal deployed initially when it started up a few years ago, and since then Postal has been continuously updating and expanding it based on its growth as a studio.

Let’s face it, hard drives always have the possibility of failing. But, failure is not something that Postal — or any other post house — can afford. That is why the studio keeps two instances per job on archive drives: a master and a backup. “Organized hard drives give us quick access to previous jobs if need be, which sometimes can be quite the lifesaver,” says Seabrooke.

 

Postal’s Nordstrom project.

LTO tapes, meanwhile, are used to back up the facility’s servers running EditShare v7 – which house Postal’s editorial jobs — on the off chance that something happens to that precious piece of hardware. “The recovery process isn’t the fastest, but the system is compact, self-contained and gives us peace of mind in case anything does go wrong,” Seabrooke explains.

In addition, the studio uses Retrospect backup and restore software for its working projects server. Seabrooke says, “We chose it because it offers a backup service that does not require much oversight.”

When Postal began shopping for a solution for its studio three years ago, reliability was at the top of its list. The facility needed a system it could rely on to back up its data, which would comprise the facility’s entire scope of work. Ease of use was also a concern, as was access. This decision prompted questions such as: Would we have to monitor it constantly? In what timeframe would we be able to access the data? Moreover, cost was yet another factor: Would the solution be effective without breaking our budget?

Postal’s solution indeed enabled them to check off every one of those boxes. “Our projects demand a system that we can count on, with the added benefit of quick retrieval,” Seabrooke says.

Throughout the studio’s production process, the artists are accessing project data on the servers. Then, once they complete the project, the data is transferred to the archival drives for backup. This frees up space on the company servers for new jobs, while providing access to the stored data if needed.

“Storage is so important in our work because it is our work. Starting over on a project is an outcome we cannot allow, so responsible storage is a necessity,” concludes Seabrooke.


Karen Moltenbrey is a long-time VFX and post production writer.


A VFX pro on avoiding the storage ‘space trap’

By Adam Stern

Twenty years is an eternity in any technology-dependent industry. Over the course of two-plus decades of visual effects facility ownership, changing standards, demands, capability upgrades and staff expansion have seen my company, Vancouver-based Artifex Studios, usher in several distinct eras of storage, each with its own challenges. As we’ve migrated to bigger and better systems, one lesson we’ve learned has proven critical to all aspects of our workflow.

Adam Stern

In the early days, Artifex used off-the-shelf hard drives and primitive RAIDs for our storage needs, which brought with it slow transfer speeds and far too much downtime when loading gigabytes of data on and off the system. We barely had any centralized storage, and depended on what was essentially a shared network of workstations — which our then-small VFX house could get away with. Even considering where we were then, which was sub-terabyte, this was a messy problem that needed solving.

We took our first steps into multi-TB NAS using off-the-shelf solutions from companies like Buffalo. This helped our looming storage space crunch but brought new issues, including frequent breakdowns that cost us vital time and lost iterations — even with plenty of space. I recall a particular feature film project we had to deliver right before Christmas. It almost killed us. Our NAS crashed and wouldn’t allow us to pull final shots, while throwing endless error messages our way. I found myself frantically hot-wiring spare drives to enable us to deliver to our client. We made it, but barely.

At that point it was clear a change was needed. We started using a solution that Annex Pro — a Vancouver-based VAR we’d been working with for years — helped put into place. That company was bought and then went away completely.

Our senior FX TD, Stanislav Enilenis, who was also handling IT for us back then, worked with Annex to install the new system. According to Stan, “the switch allowed bandwidth for expansion. However, when we would be in high-production mode, bandwidth became an issue. While the system was an overall improvement from our first multi-terabyte NAS, we had issues. The company was bought out, so getting drives became problematic, parts became harder to source and there were system failures. When we hit top capacity with the-then 20-plus staff all grinding, the system would slow to a crawl and our artists spent more time waiting than working.”

Artifex machine room.

As we transitioned from SD to HD, and then to 4K, our delivery requirements increased along with our rendering demands, causing severe bottlenecks in the established setup. We needed a better solution but options were limited. We were potentially looking at a six-figure investment, in a system not geared to M&E.

In 2014, Artifex was working on the TV series Continuum, which had fairly massive 3D requirements on an incredibly tight turnaround. It was time to make a change. After a number of discussions with Annex, we made the decision to move to an offering from a new company called Qumulo, which provided above-and-beyond service, training and setup. When we expanded into our new facility, Qumulo helped properly move the tech. Our new 48TB pipeline flowed freely and offered features we didn’t previously have, and Qumulo were constantly adding new and requested updates.

Laila Arshid, our current IS manager, has found this to be particularly valuable. “In Qumulo’s dashboard I can see realtime analytics of everything in the system. If we have a slowdown, I can track it to specific workstations and address any issues. We can shut that workstation or render-node down or reroute files so the system stays fast.”

The main lesson we’ve learned throughout every storage system change or upgrade is this: It isn’t just about having a lot of space. That’s an easy trap to fall into, especially today when we’re seeing skyrocketing demands from 4K+ workflows. You can have unlimited storage, but If you can’t utilize it efficiently and at speed, your storage space becomes irrelevant.

In our industry, the number of iterations we can produce has a dramatic impact on the quality of work we’re able to provide, especially with today’s accelerated schedules. One less pass can mean work with less polish, which isn’t acceptable.

Artifex provided VFX for Faster Than Light

Looking forward, we’re researching extended storage on the cloud: an ever-expanding storage pool with the advantages of fast local infrastructure. We currently use GCP for burst rendering with Zync, along with nearline storage, which has been fantastic — but the next step will be integrating these services with our daily production processes. That brings a number of new challenges, including how to combine local and cloud-based rendering and storage in ways that are seamless to our team.

Constantly expanding storage requirements, along with maintaining the best possible speed and efficiency to allow for artist iterations, are the principal drivers for every infrastructure decision at our company — and should be a prime consideration for everyone in our industry.


Adam Stern is the founder of Vancouver, British Columbia’s Artifex. He says the studio’s main goal is to heighten the VFX experience, both artistically and technically, and collaborate globally with filmmakers to tell great stories.


Storage Roundtable

Manufacturers weigh in on trends, needs.

By Randi Altman

Storage is the backbone of today’s workflows, from set to post to archive. There are many types of storage offerings from many different companies, so how do you know what’s right for your needs?

In an effort to educate, we gathered questions from users in the field. “If you were sitting across a table from makers of storage, what would you ask?”

The following is a virtual roundtable featuring a diverse set of storage makers answering a variety of questions. We hope it’s helpful. If you have a question that you would like to ask of these companies, feel free to email me directly at randi@postPerspective.com and I will get them answered.

SCALE LOGIC’S BOB HERZAN
What are the top three requests you get from your post clients?
A post client’s primary concern is reliability. They want to be assured that the storage solution they are buying supports all of their applications and will provide the performance each application will need when they need it. The solution needs the ability to interact with MAM or PAM solutions and they need to be able to search and retrieve their assets and to future proof, scale and manage the storage in a tiered infrastructure.

Secondly, the client wants to be able to use their content in a way that makes sense. Assets need to be accessible to the stakeholders of a project, no matter how big or complex the storage ecosystem.

Finally, the client wants to see the options available to develop a long-term archiving process that can assure the long-term preservation of their finished assets. All three of these areas can be very daunting to our customers, and being able to wade through all of the technology options and make the right choices for each business is our specialty.

How should post users decide between SAN, NAS and object storage?
There are a number of factors to consider, including overall bandwidth, individual client bandwidth, project lifespan and overall storage requirements. Because high-speed online storage typically has the highest infrastructure costs, a tiered approach makes the most sense for many facilities, where SAN, NAS, cloud or object storage may all be used at the same time. In this case, the speed with which a user will need access to a project is directly related to the type of storage the project is stored on.

Scale Logic uses a consultative approach with our customers to architect a solution that will fit both their workflow and budget requirements. We look at the time it takes to accomplish a task, what risks, if any, are acceptable, the size of the assets and the obvious, but nonetheless, vital budgetary considerations. One of the best tools in our toolbox is our HyperFS file system, which allows customers the ability to choose any one of four tiers of storage solutions while allowing full scalability to incorporate SAN, NAS, cloud and object storage as they grow.

How many data streams of 4K 10-bit DPX at 24fps can your storage provide?
Above everything else we want to tailor a solution to the needs of the clients. With our consultative approach we take a look not only at the requirements to build the best solution for today, but also  the ability to grow and scale up to the needs of tomorrow. We look at scalability not just from the perspective of having more ability to do things, but in doing the most with what we have. While even our entry level system is capable of doing 10 streams of 4K, it’s equally, if not more, important to make sure that those streams are directed to the people who need them most while allowing other users access at lower resolutions.

GENESIS Unlimited

Our Advanced QoS can learn the I/O patterns/behavior for an application while admins can give those applications a “realtime” or “non-realtime” status. This means “non-realtime” applications auto-throttle down to allow realtime apps the bandwidth. Many popular applications come pre-learned, like Finder, Resolve, Premiere or Flame. In addition, admins can add their own apps.

What do you expect to see as the next trend relating to storage?
Storage always evolves. Whatever is next in post production storage is already in use elsewhere as we are a pretty risk-averse group, for obvious reasons. With that said, the adoption of Unified Storage Platforms and hybrid cloud workflows will be the next big thing for big media producers like post facilities. The need for local online and nearline storage must remain for realtime, resolution-intense processes and data movement between tiers, but the decision-making process and asset management is better served globally by increased shared access and creative input.

The entertainment industry has pushed the limits of storage for over 30 years with no end in sight. In addition, the ability to manage storage tiers and collaborate both on-prem and off will dictate the type of storage solutions our customers will need to invest in. The evolution of storage needs continues to be driven by the consumer: TVs and displays have moved to demanding 4K content from the producers. The increased success of the small professional cameras allows more access to multi-camera shoots. However, as performance and capacity continues to grow for our customers, it brings the complexity down to managing large data farms effectively, efficiently and affordably. That is on the horizon in our future solution designs. Expensive, proprietary hardware will be a thing of the past and open, affordable storage will be the norm, with user-friendly and intuitive software developed to automate, simplify, and monetize our customer assets while maintaining industry compatibility.

SMALL TREE‘S CORKY SEEBER
How do your solutions work with clients’ existing storage? And who is your typical client?
There are many ways to have multiple storage solutions co-exist within the post house, most of these choices are driven by the intended use of the content and the size and budget of the customer. The ability to migrate content from one storage medium to another is key to allowing customers to take full advantage of our shared storage solutions.

Our goal is to provide simple solutions for the small to medium facilities, using Ethernet connectivity from clients to the server to keep costs down and make support of the storage less complicated. Ethernet connectivity also enables the ability to provide access to existing storage pools via Ethernet switches.

What steps have you taken to work with technologies outside of your own?
Today’s storage providers need to actively design their products to allow the post house to maximize the investment in their shared storage choice. Our custom software is open-sourced based, which allows greater flexibility to integrate with a wider range of technologies seamlessly.

Additionally, the actual communication between products from different companies can be a problem. Storage designs that allow the ability to use copper or optical Ethernet and Fibre Channel connectivity provide a wide range of options to ensure all aspects of the workflow can be supported from ingest to archive.

What challenges, if any, do larger drives represent?
Today’s denser drives, while providing more storage space within the same physical footprint, do have some characteristics that need to be factored in when making your storage solution decisions. Larger drives will take longer to configure and rebuild data sets once a failed disk occurs, and in some cases may be slightly slower than less dense disk drives. You may want to consider using different RAID protocols or even using software RAID protection rather than hardware RAID protection to minimize some of the challenges that the new, larger disk drives present.

When do you recommend NAS over SAN deployments?
This is an age-old question as both deployments have advantages. Typically, NAS deployments make more sense for smaller customers as they may require less networking infrastructure. If you can direct connect all of your clients to the storage and save the cost of a switch, why not do that?

SAN deployments make sense for larger customers who have such a large number of clients that making direct connections to the server is impractical or impossible: these require additional software to keep everything straight.

In the past, SAN deployments were viewed as the superior option, mostly due to Fibre Channel being faster than Ethernet. With the wide acceptance of 10GbE, there is a convergence of sorts, and NAS performance is no longer considered a weakness compared to SAN. Performance aside, a SAN deployment makes more sense for very large customers with hundreds of clients and multiple large storage pools that need to support universal access.

QUANTUM‘S JANET LAFLEUR
What are the top three requests that you get from post users?
1) Shared storage with both SAN and NAS access to collaborate more broadly acrossJanet Lafleur groups. For streaming high-resolution content to editorial workstations, there’s nothing that can match the performance of shared SAN storage, but not all production team members need the power of SAN.

For example, animation and editorial workflows often share content. While editorial operations stream content from a SAN connection, a NAS gateway using a higher-speed IP protocol optimized for video (such as our StorNext DLC) can be used for rendering. By working with NAS, producers and other staff who primarily access proxies, images, scripts and other text documents can more easily access this content directly from their desktops. Our Xcellis workflow storage offers NAS access out of the box, so content can be shared over IP and over Fibre Channel SAN.

2) A starting point for smaller shops that scales smoothly. For a small shop with a handful of workstations, it can be hard to find a storage solution that fits into the budget now but doesn’t require a forklift upgrade later when the business grows. That’s one reason we built Xcellis workflow storage with a converged architecture that combines metadata storage and content storage. Xcellis provides a tighter footprint for smaller sites, but still can scale up for hundreds of users and multiple petabytes of content.

3) Simple setup and management of storage. No one wants to spend time deploying, managing and upgrading complex storage infrastructure, especially not post users who just want storage that supports their workflow. That’s why we are continuing to enhance StorNext Connect, which can not only identify problems before they affect users but also reduce the risk of downtime or degraded performance by eliminating error-prone manual tasks. We want our customers to be able to focus on content creation, not on managing storage.

How should post users decide between SAN, NAS and object storage?
Media workflows are complex, with unique requirements at each step. SAN, NAS and object storage all have qualities that make them ideal for specific workflow functions.

SAN: High-resolution, high-image-quality content production requires low-latency, high-performance storage that can stream 4K or greater — plus HDR, HFR content — to multiple workstations without dropping frames. Fibre Channel SANs are the only way to ensure performance for multi-streaming this content.

Object storage: For content libraries that are being actively monetized, object storage delivers the disk-level of performance needed for transcoding and reuse. Object storage also scales beyond the petabyte level, and the self-balancing nature of its erasure code algorithms make replacing aging disks with next-generation ones much simpler and faster than is possible with RAID systems.

Quantum XcellisNAS: High-performance IP-based connections are ideal for enabling render server farms to access content from shared storage. The simplicity of deploying NAS is also recommended for low-bandwidth functions such as review and approval, plus DVD authoring, closed captioning and subtitling.

With an integrated, complete storage infrastructure, such as those built with our StorNext platform, users can work with any or all of these technologies — as well as digital tape and cloud — and target the right storage for the right task.

How many data streams of 4K 10-bit DPX at 24fps can your storage provide?
So much depends on the configuration: how many spindles, how many controllers, etc. At NAB 2016, our StorNext Pro 4K demo system delivered eight to 10 streams of 4K 10-bit DPX with headroom to stream more. The solution included four RAID-6 arrays of 24 drives each with redundant Xcellis Workflow Directors for an 84TB usable capacity in a neat 10U rack.

The StorNext platform allows users to scale performance and capacity independently. The need for more capacity can be addressed with the simple addition of Xcellis storage expansion arrays. The need for more performance can be met with an upgrade of the Xcellis Workflow Director to support more concurrent file systems.

PANASAS‘ DAVID SALLAK
What are the top three storage-related requests/needs that you get from your post clients or potential post clients?
They want native support for Mac, high performance and a system that is easier to grow and manage than SAN.

When comparing shared storage product choices, what are the advantages of NAS over SAN? Does the easier administration of NAS compared to SAN factor into your choice of storage?
NAS is easier to manage than SAN. Scale-out NAS is easier to grow thPanasasan SAN, and is designed for high availability. If scale-out NAS could be as fast as SAN, then SAN buyers would be very attracted to scale-out NAS.

How many data streams of 4K 10-bit DPX at 24fps can your storage provide?
As many streams as possible. Post users always need more performance for future projects and media formats, so storage should support a lot of streams of ProRes HD or DNxHD and be capable of handling uncompressed DPX formats that come from graphics departments.

What do you expect to see as the next trend relating to storage? The thing that’s going to push storage systems even further?
Large post production facilities need greater scalability, higher performance, easier use, and affordable pricing.

HGST‘s JEFF GREENWALD
What are the top three requests you get from your post clients or potential post clients?
They’re looking for better ways to develop cost efficiencies of their workflows. Secondly, they’re looking for ways to improve the performance of those workflows. Finally, they’re looking for ways to improve and enhance data delivery and availability.

How should post users decide between SAN, NAS and object storage?
There are four criteria that customers must evaluate in order to make trade-offs between the various storage technologies as well as storage tiers. Customers must evaluate the media quantity of data, and they must also evaluate the frequency of acceptability. They must evaluate the latency requirements of data delivery, and, finally they must balance these three evaluations across their financial budgets.

Active ArchiverHow many data streams of 4K 10-bit DPX at 24fps can your storage provide?
In order to calculate quantity of video streams you must balance available bandwidth as well as file sizes and data delivery requirements toward the desired capacity. Also, jitter and data loss continue to shrink available bandwidth for retries and resends.

What do you expect to see as the next trend relating to storage, and what will push storage even further?
There are two trends that will dramatically transform the storage industry. The first is storage analytics, and the second is new and innovative usage of automatic meta-tagging of file data.

New technologies like SMR, optical and DNA-based object storage have not yet proven to be technology disruptors in storage, therefore it is likely that storage technology advancements will be evolutionary as opposed to revolutionary in the next 10 years.

G-TECH‘S MICHAEL WILLIAMS
Who is using your gear in the post world? What types of pros?
Filmmakers, digital imaging technicians, editors, audio technicians and photographers all use our solutions. These are the pros that capture, store, transfer and edit motion pictures, indie films, TV shows, music, photography and more. We offer everything from rugged standalone portable drives to high-performance RAID solutions to high-capacity network storage for editing and collaboration.

You recently entered the world of NAS storage. Can you talk about the types of pros taking advantage of that tech?
Our NAS customers run the gamut from DITs to production coordinators to video editors and beyond. With camera technology advancing so rapidly, they are looking for storage solutions that can fit within the demanding workflows they encounter every day.

With respect to episodic, feature film, commercials or in-house video production storage, needs are rising faster than ever before and many IT staffs are shrinking, so we introduced the G-Rack 12 NAS platform. We are able to use HGST’s new 10TB enterprise-class hard drives to deliver 120TB of raw storage in a 2RU platform, providing the required collaboration and performance.

We have also made sure that our NAS OS on the G-Rack 12 is designed to be easily administered by the DIT, video editor or someone else on the production staff and not necessarily a Linux IT tech.

Production teams need to work smarter — DITs, video editors, DPs and the like can do the video shoot, get the video ingested into a device and get the post team working on it much faster than in days past. We all know that time is money; this is why we entered the NAS market.

Any other new tech on the horizon that might affect how you make storage or a certain technology that might drive your storage in other directions?
The integration of G-Technology — along with SanDisk and HGST — into Western Digital is opening up doors in terms of new technologies. In addition to our current high-capacity, enterprise-class HDD-based offerings, SSD devices are now available to give us the opportunity to expand our offerings to a broader range of solutions.

G-RACK 12This, in addition to new external device interfaces, is paving the way for higher-performance storage solutions. At NAB this year, we demonstrated Thunderbolt 3 and USB-C solutions with higher-performance storage media and network connectivity. We are currently shipping the USB solutions and the technology demos we gave provide a glimpse into future solutions. In addition, we’re always on the lookout for new form factors and technologies that will make our storage solutions faster, more powerful, more reliable and affordable.

What kind of connections do your drives have, and if it’s Thunderbolt 2 or Thunderbolt 3, can they be daisy chained?
When we look at interfaces, as noted above, there’s a USB Type-C for the consumer market as well as Thunderbolt and 10Gb Ethernet for the professional market.

As far as daisy-chaining, yes. Thunderbolt is a very flexible interface, supporting up to six devices in a daisy chain, on a single port. Thunderbolt 3 is a very new interface that is gaining momentum, one that will not only support extremely high data transfer speeds (up to 2.7GB/s) but also supports up to two 4K displays. We should also not forget that there are still more than 200M devices supporting Thunderbolt 1 and 2 connections.

LACIE‘S GASPARD PLANTROU
How do your solutions work with clients existing storage? And who are your typical M&E users?
With M&E workflows, it’s rare that users work with a single machine and storage solution. From capture to edit to final delivery, our customers’ data interacts with multiple machines, storage solutions and users. Many of our storage solutions feature multiple interfaces such as Thunderbolt, USB 3.0 or FireWire so they can be easily integrated into existing workflows and work seamlessly across the entire video production process.

Our Rugged features Thunderbolt and USB 3.0. That means it’s guaranteed to work with any standard computer or storage scenario on the market. Plus it’s shock, dust and moisture-resistant, allowing it to handle being passed around set or shipped to a client. Lacie 12bigLaCie’s typical M&E users are mid-size post production studios and independent filmmakers and editors looking for RAID solutions.

How many data streams of 4K 10-bit DPX at 24fps can your storage provide?
The new LaCie 12big Thunderbolt 3 pushes up to 2600MB/s and can handle three streams of 4K 10-bit DPX at 24fps (assuming one stream is 864MB/s). In addition, the storage solution features 96TB to edit and hold tons of 4K footage.

What steps have you taken to work with technologies outside of your own?
With video file sizes growing exponentially, it is more important than ever for us to deliver fast, high-capacity solutions. Recent examples of this include bringing the latest technologies from Intel — Thunderbolt 3 — into our line. We work with engineers from our parent company, Seagate, to incorporate the latest enterprise class core technology for speed and reliability. Plus, we always ensure our solutions are certified to work seamlessly on Mac and Windows.

NETAPP‘S JASON DANIELSON
What are the top three requests that you get from post users?Jason Danielson
As a storage vendor, the first three requests we’re likely to get are around application integration, bandwidth and cost. Our storage systems support well over 100 different applications across a variety of workflows (VFX, HD broadcast post, uncompressed 4K finishing) in post houses of all sizes, from boutiques in Paris to behemoths in Hollywood.

Bandwidth is not an issue, but the bandwidth per dollar is always top of mind for post. So working with the post house to design a solution with suitable bandwidth at an acceptable price point is what we spend much of our time doing.

How should post users decide between SAN, NAS and object storage?
The decision to go with SAN versus NAS depends on the facility’s existing connectivity to the workstations. Our E-Series storage arrays support quite a few file systems. For SAN, our systems integrators usually use Quantum StorNext, but we also see Scale Logic’s HyperFS and Tiger Technology’s metaSAN being used.

For NAS, our systems integrators tend to use EditShare XStream EFS and IBM GPFS. While there are rumblings of a transition away from Fibre Channel-based SAN to Ethernet-based NAS, there are complexities and costs associated with tweaking a 10GigE client network.

The object storage question is a bit more nuanced. Object stores have been so heavily promoted by storage vendors that thE5624ere are many misconceptions about their value. For most of the post houses we talk to, object storage isn’t the answer today. While we have one of the most feature-rich and mature object stores out there, even we say that object stores aren’t for everyone. The questions we ask are:

1) Do you have 10 million files or more? 2) Do you store over a petabyte? 3) Do you have a need for long-term retention? 4) Does your infrastructure need to support multisite production?

If the answer to any of those questions is “yes,” then you should at least investigate object storage. A high-end boutique with six editors is probably not in this realm. It is true that an object store represents a slightly lower-cost bucket for an active archive (content repository), but it comes at a workflow cost of introducing a second tier to the architecture, which needs to be managed by either archive management or media asset management software. Unless such a software system is already in place, then the cost of adding one will drive up the complexity and cost of the implementation. I don’t mean to sound negative about object stores. I am not. I think object stores will play a major role in active-archive content storage in the future. They are just not a good option for a high-bandwidth production tier today or, possibly, ever.

How many data streams of 4K 10-bit DPX at 24fps can your storage provide?
In order to answer that question, we would ask the post house: “How many streams do you want to play back?” Let’s say we’re talking about 4K (4096×2160), versus the several resolutions that are called 4K). At 4:4:4, that works out to 33MB per frame or 792MB per second. We would typically use flash (SSDs) for 4K playback. Our  2RU 24-SSD storage array, the EF560, can do a little over 9GB per second. That amounts to 11 streams.

But that is only half the answer. This storage array is usually deployed under a parallel file system, which will aggregate the bandwidth of several arrays for shared editing purposes. A larger installation might have eight storage arrays — each with 18 SSDs (to balance bandwidth and cost) — and provide sustained video playback for 70 streams.

What do you expect to see as the next trend relating to storage? What’s going to push storage systems even further?
The introduction of larger, more cost-effective flash drives (SSDs) will have a drastic effect on storage architectures over the next three years. We are now shipping 15TB SSDs. That is a petabyte of extremely fast storage in six rack units. We think the future is flash production tiers in front of object-store active-archive tiers. This will eliminate the need for archive managers and tape libraries in most environments.

HARMONIC‘S ANDY WARMAN
What are the top three requests that you hear from your post clients or potential post clients?andy warman
The most common request is for sustained performance. This is an important aspect since you do not want performance to degrade due to the number of concurrent users, the quantity of content, how full the storage is, or the amount of time the storage has been in service.

Another aspect related to this is the ability to support high-write and -read bandwidth. Being able to offer equal amounts of read and write bandwidth can be very beneficial for editing and transcode workflows, versus solutions that have high-read bandwidth, but relatively low-write performance. Customers are also looking for good value for money. Generally, we would point to value coming from the aforementioned performance as well as cost-effective expansion.

You guys have a “media-aware” solution for post. Can you explain what that is and why you opted to go this way?
Media-aware storage refers to the ability to store different media types in the most effective manner for the file system. A MediaGrid storage system supports multiple different block sizes, rather than a single block size for all media types. In this way, video assets, graphics and audio and project files can use different block sizes that make reading and writing data more efficient. This type of file I/O “tuning” provides some additional performance gains for media access, meaning that video could use, say, 2MB blocks, graphics and audio 512KB, and projects and other files 128KB. Not only can different block sizes be used by different media types, but they are also configurable so UHD files could, say, use 8MB block sizes.

How many data streams of 4K 10-bit DPX at 24fps can your storage provide?
The storage has no practical storage capacity or bandwidth limit, so we can build a storage system that suits the customer needs. To size a system it becomes a case of balancing the bandwidth and storage capacity by selecting the appropriate number of drives and drive size(s) to match specific needs. The system is built on SAS drives; multiple, fully redundant 10 Gigabit Ethernet connections to client workstations and attached devices and 12 Gigabit redundant SAS interconnects between storage expansion nodes. This means we have high-speed connectivity within the storage as well as out to clients.

Harmonic MediaGrid 4K Content ServerAs needs change, the system can be expanded online with all users maintaining full access. Bandwidth scales in a linear fashion, and because there is a single name space in MediaGrid, the entire storage system can be treated as a single drive, or divided up and granted user level rights to folders within the file system.

Performance is further enhanced by the use of parallel access to data throughout the storage system. The file system provides a map to where all media is stored or is to be stored on disk. Data is strategically placed across the whole storage system to provide the best throughput. Clients simultaneously read and write data through the 10 Gigabit network to all network attached storage nodes rather than data being funneled through a single node or data connection. The result is that performance is the same whether the storage system is 5% or 95% full.

What do you expect to see as the next trend relating to storage? What’s going to push storage systems even further?
The advent of UHD has driven demands on storage further as codecs and therefore data throughput and storage requirements have increased significantly. Faster and more readily accessible storage will continue to grow in importance as delivery platforms continue to expand and expectations for throughput of storage systems continue to grow. We will use whatever performance and storage capacity is available, so offering more of both is inevitable to feed our needs for creativity and storytelling.

JMR’s STEVE KATZ
What are the top three storage-related requests you get from post users?
The most requested is ease of installation and operation. The JMR Share is delivered with euroNAS OS on mirrored SSD boot disks, with enough processing power and memory to Steve Headshot 6.27.16support efficient, high-volume workflows and a perpetual license to support the amount of storage requested, from 20TB minimum to the “unlimited” maximum. It’s intuitive to use and comfortable for anyone familiar with using popular browsers.

Compatibility and interoperability with clients using various hardware, operating systems and applications.

How many data streams of 4K 10-bit DPX at 24fps can your storage provide?
This can all be calculated by usable bandwidth and data transfer rates, which as with any networked storage can be limited by the network itself. For those using a good 10GbE switch, the network limits data rates to 1250MB/s maximum, which can support more than 270 streams of DNxHD 36, but only one stream of 4K 10-bit “film” resolution. Our product can support ~1800MB/s in a single 16-disk appliance, but without a very robust network this can’t be achieved.

When comparing shared storage product choices, what are the advantages of NAS over SAN, for example?
SAN actually has some advantages over NAS, but unless the user has Fibre Channel hardware installed, it might be a very costly option. The real advantage of NAS is that everyone already has an Ethernet network available that may be sufficient for video file server use. If not, it may be upgraded fairly inexpensively.

JMR Share comes standard with both GbE and 10GbE networking capability right out of the box, and has performance that will saturate 10GbE links; high-availability active/active failover is available as well as SAN Cluster (an extra cost option). The SAN Cluster is equipped with specialized SAN software as well as with 8Gb or 16Gb fibre channel host adapters installed, so it’s ready to go.

What do you expect to see as the next trend relating to storage? The thing that’s going to push storage systems even further?
Faster and lower cost, always! Going to higher speed network adapters, 12Gb SAS internal storage and even SSDs or NVMe drives, it seems the sky is the limit — or, actually, the networking is the limit. We already offer SAS SSDs in the Share as an option, and our higher-end dual-processor/dual-controller Share models (a bit higher cost) using NVMe drives can provide internal data transfer speeds exceeding what any network can support (even multiple 40Gb InfiniBand links). We are seeing a bit of a trend toward SSDs now that higher-capacity models at more reasonable cost, with reasonable endurance, are becoming available.


The State of Storage

Significant trends are afoot in media and entertainment storage.

By Tom Coughlin

Digital storage plays a significant role in the media and entertainment industry, and our specific demands are often very different from typical IT storage. We are dealing with performance requirements of realtime video in capture, editing and post, as well as distribution. On the other hand, the ever-growing archive of long-tail digital content and digitized historical analog content is swelling the demand for archives (both cold and warm) using tape, optical discs and hard drive arrays.

My company, Coughlin Associates, has conducted surveys of digital storage use by media and entertainment professionals since 2009. These results are used in our annual Digital Storage in Media and Entertainment Report. This article presents results from the 2016 survey and some material from the 222-page report to discuss the status of digital storage for professional media and entertainment.

Content Creation and Capture
Pro video cameras are undergoing rapid evolution, driven by higher-resolution content as well as multi-camera content capture, including stereoscopic and virtual reality. In addition, the physical storage media for professional cameras is undergoing rapid evolution as film and magnetic digital tape is impacted by the rapid file access convenience of hard disk drives, optical discs, and the ruggedness of flash-based solid-state storage.

The table below compares the results from the 2009, 2010, 2012, 2013, 2014 and 2015 surveys with those from 2016. Flash memory is the clear leader in pro video camera media, increasing from 19% in 2009 to 66% in 2015 and then down to 54% in 2016, while magnetic tape shows a consistent decline over the same period.

Optical disc use between 2009 and 2016 bounced around between 7% and 17%. Film shows a general decline from 15% usage in 2009 to 2% in 2016. The trend with declining film use follows the trend toward completely digital workflows.

Note that about 60% of survey participants said that they used external storage devices to capture content from their cameras in 2016 (perhaps this is why the HDD percentages are so high). In 2016, 83% said that over 80% of their content is created in a digital format.

In 2016, 93.2% of the survey respondents said they reuse their recording media (compared to 89.9% in 2015, 93.3% in 2014, 84.5% in 2013, 86% in 2012, 79% in 2010 and 75% in 2009). In 2016, 75% of respondents said they archive their camera recording media (compared to 73.6% in 2015, 74.2% in 2014, 81.4% in 2013, 85% in 2012 and 77% in 2010).

Archiving the original recording media may be a practice in decline — especially with expensive reusable media such as flash memory cards. Digital storage on tape, hard disk drives or flash storage allows the reuse of media.

Post Production
The size of content — and amount — has put strains on post network bandwidth and storage. This includes editing and other important operations. As much of this work may take place in smaller facilities, these companies may be doing much of their work on direct attached storage devices and they may share or archive this media in the cloud in order to avoid the infrastructure costs of running a data center.

The graph below shows that for the 2016 survey participants there was a general increase in the use of shared network storage (such as SAN or NAS), and a decrease in DAS storage as the number of people working in a post facility increases. The DAS storage in the larger facilities may be different than that used in smaller facilities.

DAS vs. shared storage by number of people in a post facility.

When participants were asked about their use of direct attached and network storage in digital editing and post, the survey showed the following summary statistics in 2016 (compared to earlier surveys):

– 74.5% had DAS
– 89.8% of these had more than 1 TB of DAS
– 10 to 50 TB was the most popular DAS size (27.5%)
– 17.4% of these had more than 50 TB of DAS storage
– 2.9% had more than 500 TB of DAS storage
– 68.1% had NAS or SAN
– 57.4% had 50 TB or more of network storage in 2016
– About 15% had more than 500 TB of NAS/SAN storage in 2016
– Many survey participants had considerable storage capacities in both DAS and NAS/SAN.

We asked whether survey participants used cloud-based storage for editing and post. In 2016 23.0% of responding participants said yes. The respondents, 20.9% of them, said that they had 1TB or more of their storage capacity in the cloud.

Content Distribution
Distribution of professional video content has many channels. It can use physical media for getting content to digital cinemas or to consumers, the distribution can be done electronically using broadcast, cable or satellite transmission, or through the Internet or mobile phone networks.

The table below gives responses for the percentage of physical media used by the survey respondents for content distribution in 2016, 2015, 2014, 2013, 2012 and 2010. Note that these are the average for the survey population giving their percentage for each physical media and do not and should not be expected to add to 100%. Digital tape, DVD discs, HDDs and Flash Memory are the most popular distribution formats.

Average percentage content on physical media for professional content distribution.

Following are survey observations for electronic content distribution, such as video on demand.

– The average number of hours on a central content delivery system was 2,174 hours in 2016.
– There was an average of 427 hours ingested monthly in 2016.
– In 2016, 38% of respondents had more than 5% of their content on edge servers.
– About 31% used flash memory on their edge servers in 2016.

Archiving and Preservation
Today, most new entertainment and media content is born digital, so it is natural that this content should be preserved in digital form. This requirement places new demands on format preservation for long-term digital archives as well as management and systematic format refreshes during the expected life of a digital archive.

In addition, the cost of analog content digitization and preservation in a digital format has gone down considerably, and many digitization projects are proceeding apace. The growth of digital content archiving will swell the amount of content available for repurposing and long-tail distribution. It will also swell the amount of storage and storage facilities required to store these long-term professional content archives.

Following are some observations from our 2016 survey on trends in digital archiving and content preservation.

– 41% had less than 2,000 hours of content in a long-term archive
– 56.9% archived all the content captured from their cameras
– 54.0% archived copies of content in all of their distribution formats
– 35.9% digitally archived all content captured from their dailies
– 31.3% digitally archived all content captured from rough cuts
– 36.5% digitally archived all content captured from their intermediaries
– 50.9% of the respondents said that their annual archive growth rate was less than 6% in 2016
– About 28.6% had less than 2,000 hours of unconverted analog content
– 16.7% of participants had over 5,000 hours of unconverted analog content
– About 52.5% of the survey respondents have an annual analog conversion rate of 2% or less
– The average rate of conversion is about 3.4% in 2016

Professional media and entertainment content was traditionally archived on film or analog videotapes. Today, the options available for archive media to store digital content depend upon the preferences and existing infrastructure of digital archive facilities. Figure 6 gives the percentage distribution of archive media used by the survey participants.

Percentage of digital long-term archives on various media

Some other observations from the archive and preservation section of the survey:

– About 42.6% never update their digital archives.
– About 76.2% used different storage for archiving and working storage.
– About 49.2% copied and replaced their digital long-term archives every 10 years or less.
– 38.1% said they would use a private or public cloud for archiving in 2016.

Conclusions
Larger content files, driven by higher resolution, higher frame rates, higher dynamic range and stereoscopic and virtual reality video are creating larger video files. This is driving the need for high-performance storage to work on this content and to provide fast delivery, which could drive more creative work to use solid-state storage.

At the same time, cost -effective storage and management of completed work is driving the increased use of hard disk drives, magnetic tape and even optical storage for low-cost storage.

The price of storing content in the cloud has gone down so much that there are magnetic tape-based cloud storage offerings that are less expensive than building one’s own storage data center, at least for small- and moderate-sized facilities.

This trend is expected to grow the use of cloud storage in media and entertainment, especially for archiving, as shown in the figure below.

Growth of cloud storage in media and entertainment.


Dr. Tom Coughlin, president of Coughlin & Associates, is a storage analyst and consultant with over 30 years in the data storage industry. He is the founder and organizer of the Annual Storage Visions Conference as well as the Creative Storage Conference.