Significant trends are afoot in media and entertainment storage.
By Tom Coughlin
Digital storage plays a significant role in the media and entertainment industry, and our specific demands are often very different from typical IT storage. We are dealing with performance requirements of realtime video in capture, editing and post, as well as distribution. On the other hand, the ever-growing archive of long-tail digital content and digitized historical analog content is swelling the demand for archives (both cold and warm) using tape, optical discs and hard drive arrays.
My company, Coughlin Associates, has conducted surveys of digital storage use by media and entertainment professionals since 2009. These results are used in our annual Digital Storage in Media and Entertainment Report. This article presents results from the 2016 survey and some material from the 222-page report to discuss the status of digital storage for professional media and entertainment.
Content Creation and Capture
Pro video cameras are undergoing rapid evolution, driven by higher-resolution content as well as multi-camera content capture, including stereoscopic and virtual reality. In addition, the physical storage media for professional cameras is undergoing rapid evolution as film and magnetic digital tape is impacted by the rapid file access convenience of hard disk drives, optical discs, and the ruggedness of flash-based solid-state storage.
The table below compares the results from the 2009, 2010, 2012, 2013, 2014 and 2015 surveys with those from 2016. Flash memory is the clear leader in pro video camera media, increasing from 19% in 2009 to 66% in 2015 and then down to 54% in 2016, while magnetic tape shows a consistent decline over the same period.
Optical disc use between 2009 and 2016 bounced around between 7% and 17%. Film shows a general decline from 15% usage in 2009 to 2% in 2016. The trend with declining film use follows the trend toward completely digital workflows.
Note that about 60% of survey participants said that they used external storage devices to capture content from their cameras in 2016 (perhaps this is why the HDD percentages are so high). In 2016, 83% said that over 80% of their content is created in a digital format.
In 2016, 93.2% of the survey respondents said they reuse their recording media (compared to 89.9% in 2015, 93.3% in 2014, 84.5% in 2013, 86% in 2012, 79% in 2010 and 75% in 2009). In 2016, 75% of respondents said they archive their camera recording media (compared to 73.6% in 2015, 74.2% in 2014, 81.4% in 2013, 85% in 2012 and 77% in 2010).
Archiving the original recording media may be a practice in decline — especially with expensive reusable media such as flash memory cards. Digital storage on tape, hard disk drives or flash storage allows the reuse of media.
The size of content — and amount — has put strains on post network bandwidth and storage. This includes editing and other important operations. As much of this work may take place in smaller facilities, these companies may be doing much of their work on direct attached storage devices and they may share or archive this media in the cloud in order to avoid the infrastructure costs of running a data center.
The graph below shows that for the 2016 survey participants there was a general increase in the use of shared network storage (such as SAN or NAS), and a decrease in DAS storage as the number of people working in a post facility increases. The DAS storage in the larger facilities may be different than that used in smaller facilities.
DAS vs. shared storage by number of people in a post facility.
When participants were asked about their use of direct attached and network storage in digital editing and post, the survey showed the following summary statistics in 2016 (compared to earlier surveys):
– 74.5% had DAS
– 89.8% of these had more than 1 TB of DAS
– 10 to 50 TB was the most popular DAS size (27.5%)
– 17.4% of these had more than 50 TB of DAS storage
– 2.9% had more than 500 TB of DAS storage
– 68.1% had NAS or SAN
– 57.4% had 50 TB or more of network storage in 2016
– About 15% had more than 500 TB of NAS/SAN storage in 2016
– Many survey participants had considerable storage capacities in both DAS and NAS/SAN.
We asked whether survey participants used cloud-based storage for editing and post. In 2016 23.0% of responding participants said yes. The respondents, 20.9% of them, said that they had 1TB or more of their storage capacity in the cloud.
Distribution of professional video content has many channels. It can use physical media for getting content to digital cinemas or to consumers, the distribution can be done electronically using broadcast, cable or satellite transmission, or through the Internet or mobile phone networks.
The table below gives responses for the percentage of physical media used by the survey respondents for content distribution in 2016, 2015, 2014, 2013, 2012 and 2010. Note that these are the average for the survey population giving their percentage for each physical media and do not and should not be expected to add to 100%. Digital tape, DVD discs, HDDs and Flash Memory are the most popular distribution formats.
Average percentage content on physical media for professional content distribution.
Following are survey observations for electronic content distribution, such as video on demand.
– The average number of hours on a central content delivery system was 2,174 hours in 2016.
– There was an average of 427 hours ingested monthly in 2016.
– In 2016, 38% of respondents had more than 5% of their content on edge servers.
– About 31% used flash memory on their edge servers in 2016.
Archiving and Preservation
Today, most new entertainment and media content is born digital, so it is natural that this content should be preserved in digital form. This requirement places new demands on format preservation for long-term digital archives as well as management and systematic format refreshes during the expected life of a digital archive.
In addition, the cost of analog content digitization and preservation in a digital format has gone down considerably, and many digitization projects are proceeding apace. The growth of digital content archiving will swell the amount of content available for repurposing and long-tail distribution. It will also swell the amount of storage and storage facilities required to store these long-term professional content archives.
Following are some observations from our 2016 survey on trends in digital archiving and content preservation.
– 41% had less than 2,000 hours of content in a long-term archive
– 56.9% archived all the content captured from their cameras
– 54.0% archived copies of content in all of their distribution formats
– 35.9% digitally archived all content captured from their dailies
– 31.3% digitally archived all content captured from rough cuts
– 36.5% digitally archived all content captured from their intermediaries
– 50.9% of the respondents said that their annual archive growth rate was less than 6% in 2016
– About 28.6% had less than 2,000 hours of unconverted analog content
– 16.7% of participants had over 5,000 hours of unconverted analog content
– About 52.5% of the survey respondents have an annual analog conversion rate of 2% or less
– The average rate of conversion is about 3.4% in 2016
Professional media and entertainment content was traditionally archived on film or analog videotapes. Today, the options available for archive media to store digital content depend upon the preferences and existing infrastructure of digital archive facilities. Figure 6 gives the percentage distribution of archive media used by the survey participants.
Percentage of digital long-term archives on various media
Some other observations from the archive and preservation section of the survey:
– About 42.6% never update their digital archives.
– About 76.2% used different storage for archiving and working storage.
– About 49.2% copied and replaced their digital long-term archives every 10 years or less.
– 38.1% said they would use a private or public cloud for archiving in 2016.
Larger content files, driven by higher resolution, higher frame rates, higher dynamic range and stereoscopic and virtual reality video are creating larger video files. This is driving the need for high-performance storage to work on this content and to provide fast delivery, which could drive more creative work to use solid-state storage.
At the same time, cost -effective storage and management of completed work is driving the increased use of hard disk drives, magnetic tape and even optical storage for low-cost storage.
The price of storing content in the cloud has gone down so much that there are magnetic tape-based cloud storage offerings that are less expensive than building one’s own storage data center, at least for small- and moderate-sized facilities.
This trend is expected to grow the use of cloud storage in media and entertainment, especially for archiving, as shown in the figure below.
Growth of cloud storage in media and entertainment.
Dr. Tom Coughlin, president of Coughlin & Associates, is a storage analyst and consultant with over 30 years in the data storage industry. He is the founder and organizer of the Annual Storage Visions Conference as well as the Creative Storage Conference.