Tag Archives: Tom Coughlin

NAB 2019: Storage for M&E workflows

By Tom Coughlin

Storage is a vital element in modern post production, since that’s where the video content lives. Let’s look at trends in media post production storage and products shown at the 2019 NAB show. First let’s look at general post production storage architectures and storage trends.

My company produces the yearly “Digital Storage in Media and Entertainment Report,” so we are keeping an eye on storage all year round. The image to the right is a schematic from our 2018 report — it’s a nonlinear editing station showing  optional connections to shared online (or realtime) storage via a SAN or NAS (or even a cloud-based object storage system) and a host bus adapter (HBA or xGbE card). I hope this gives you some good background for what’s to come.

Our 2018 report also includes data from our annual Digital Storage in Media and Entertainment Professional Survey. The report shows that storage capacity annual demand is expected to be over 110 Exabytes of storage by 2023. In 2018 48% of responding survey participants said that they used cloud-based storage for editing and post production. And 56% also said that they have 1TB or more storage capacity in the cloud. In 2018, Internet distribution was the most popular way to view proxies.

All of this proves that M&E pros will continue to use multiple types of digital storage to enable their workflows, with significant growth in the use of cloud storage for collaborative and field projects. With that in mind, let’s dig into some of the storage offerings that were on display at NAB 2019.

Workflow Storage
Dell Technologies said that significant developments in its work with VMware unlock the value of virtualization for applications and tools to automate many critical M&E workflows and operations. Dell EMC and VMware said that they are about to unveil the recipe book for making virtualization a reality for the M&E industry.

Qumulo announced an expansion of its cloud-native file storage offerings. The company introduced two new products —CloudStudio and CloudContinuity — as well as support for Qumulo’s cloud-native, distributed hybrid file system on the Google Cloud Platform (GCP). Qumulo has partnered with Google to support Qumulo’s hybrid cloud file system on GCP and on the Google Cloud Platform Marketplace. Enterprises will be able to take advantage of the elastic compute resources, operational agility, and advanced services that Google’s public cloud offers. With the addition of the Google Cloud Platform, Qumulo is able to provide multi-cloud platform support, making it easy for users to store, manage and access their data, workloads and applications in both Amazon Web Services (AWS) and GCP. Qumulo also enables data replication between clouds for migration or multi-copy requirements.

M&E companies of any size can scale production into the public cloud with CloudStudio, which securely moves traditionally on-prem workspaces, including desktops, applications and data, to the public cloud on both the AWS and GCP platforms. Qumulo’s file storage software is the same whether on-prem or in the cloud, making the transition seamless and easy and eliminating the need to reconfigure applications or retrain users.

CloudContinuity enables users to automatically replicate their data from an on-prem Qumulo cluster to a Qumulo instance running in the cloud. Should a primary on-prem storage system experience a catastrophic failure, customers can redirect users and applications to the Qumulo cloud, where they will have access to all of their data immediately. CloudContinuity also enables quick, automated fail-back to an on-prem cluster in disaster recovery scenarios.

Quantum announced its VS-Series, designed for surveillance and industrial IoT applications. The VS-Series is available in a broad range of server choices, suitable for deployments with fewer than 10 cameras up to the largest environments with thousands of cameras. Using the VS-Series, security pros can efficiently record and store surveillance footage and run an entire security infrastructure on a single platform.

Quantum’s VS-Series architecture is based on the Quantum Cloud Storage Platform (CSP), a new software-defined storage platform specifically designed for storing machine and sensor-generated data. Like storage technologies used in the cloud, the Quantum CSP is software-defined and can be deployed on bare metal, as a virtual machine, or as part of a hyperconverged infrastructure.Unlike other software-defined storage technologies, the Quantum CSP was designed specifically for video and other forms of high-resolution content — engineered for extremely low latency, maximizing the streaming performance of large files to storage.

The Quantum Cloud Storage Platform allows high-speed video recording with optimal camera density and can host and run certified VMS management applications, recording servers and other building control servers on a single platform.

Quantum say that the VS-Series product line is being offered in a variety of deployment options, including software-only, mini-tower and 1U, 2U and 4U hyperconverged servers.

Key VS-Series attributes:
– Supports high camera density and software architecture that enables users to run their entire security infrastructure on a single hyperconverged platform.
– Offers a software-defined platform with the broadest range of deployment options. Many appliances can scale out for more cameras or scale up for increased retention.
– Comes pre-installed with certified VMS applications and can be installed and configured in minutes.
– Offers a fault-tolerant design to minimize hardware and software issues, which is meant to virtually eliminate downtime

Quantum was also showing its R-3000 at NAB. This box was designed for in-vehicle data capture for developing driver assistance and autonomous driving systems. This NAS box includes storage modules of 60TB with HDDs and 23TB or 46TB using SSDs. It works off 12 volt power and features two 10 GbE ports.

Arrow Distribution bundled NetApp storage appliances with Axle AI software. The three solutions offered are the VM100, VM200 and VM400 with 100TB, 200TB and 400TB, respectively, with 10GbE network interfaces and NetApp’s FAS architecture. Each configuration also includes an Intel-based application server running a five-user version of Axle AI 2019. The software includes a browser front-end that allows multiple users to tag, catalog and search their media files, as well as a range of AI-driven options for automatically cataloging and discovering specific visual and audio attributes within those files.

Avid Nexis|Cloudspaces

Avid Nexis|Cloudspaces is a storage as a service (SaaS) offering for post, news and sports teams, enabling them to store and park media and projects not currently in production in the cloud, leveraging Microsoft Azure. This frees up local Nexis storage space for production work. The company is offering all Avid Nexis users a limited-time free offer of 2TB of Microsoft Azure storage that is auto-provisioned for easy setup and can scale as needed. Avid Nexis manages these Cloudspaces alongside local workspaces, allowing unified content management.

DDP was showing a rack with hybrid SSD/HDD storage that the company says provides 24/7 365 days of reliable operation with zero interruptions and a transparent failover setup. DDP has redesigned its GUI to provide faster operation and easier use.

Facilis displayed its new Hub shared storage line developed specifically for media production workflows. Built as an entirely new platform, Facilis Hub represents the evolution of the Facilis shared file system with the block-level virtualization and multi-connectivity performance required in shared creative environments. This solution offers both block-mode Fibre Channel and Ethernet connectivity simultaneously, allowing connection through either method with the same permissions, user accounts and desktop appearance.

Facilis’ Object Cloud is an integrated disk-caching system for cloud and LTO backup and archive that includes up to 100TB of cloud storage for one low yearly cost. A native Facilis virtual volume can display cloud, tape and spinning disk data in the same directory structure, on the client desktop. Every Facilis Hub shared storage server comes with unlimited seats of the Facilis FastTracker asset tracking application. The Object Cloud software and storage package is available for most Facilis servers running version 7.2 or higher.

Facilis also had particular product updates. The Facilis 8 has 1GB/s data rates through standard dual-port 10GbE and options for 40GbE and Fibre Channel connectivity with 32TB, 48TB and 64TB capacities. The Facilis Hub 16 model offers 2GB/s speed with 16 HDDs with 64TB, 96TB and 128TB capacities. The company’s Hub Hybrid 16 model and SSD offers SSDs in an integrated high-capacity HDD-based storage system offering performance of 3GB/s and 4GB/s. With two or more Hub 16 or Hub 32 servers attached through 32Gb Fibre Channel controllers, Facilis Hub One configurations can be fully redundant, with multi-server bandwidth aggregated into a single point of network connectivity. The Hub One starts at 128GB and scales to 1PB.

Pixit Media announced the launch of PixStor 5, the latest version of its leading scale-out data-driven storage platform. According to the company, “PixStor 5 is an enterprise-class scale-out NAS platform delivering guaranteed 99% performance for all types of workflow and a single global namespace across multiple storage tiers — from on-prem for the cloud.”

New PixStor 5 highlights include:

PixStor 5

– Secure container services – This new feature offers multi-tenancy from a single storage fabric. PixStor 5 enables creative studios to deploy secure media environments without crippling productivity and creativity and aligns with TPN security accreditation standards to attract A-list clients.
– Cloud workflow flexibility — PixStor 5 expands your workflows cost-effectively into the cloud with fully automated seamless deployment to cloud marketplaces, enabling hybrid workflows for burst render and cloud-first workflows for global collaboration. PixStor 5 will soon be available in the Google Cloud Platform Marketplace, followed shortly by AWS and Azure.
– Enhanced search capabilities — Using machine learning and artificial intelligence cloud-based tools to drive powerful media indexing and search capabilities, users can perform fast, easy and accurate content searches across their entire global namespace.
– Deep granular analytics – With single-pane-of-glass management and user-friendly dashboards, PixStor 5 allows a holistic view of the entire filesystem and delivers business-relevant metrics to reinforce storage strategies.

GB Labs launched new software, features and updates to its FastNAS and Space, Echo and Vault ranges at NAB. The Space, Echo and Vault ranges got intelligent new software features, including the Mosaic asset organizer and the latest Analytics Center, along with brand-new Core.4 and Core.4 Lite software. The new Core software is also now included in the FastNAS product range.

GB Labs

Mosaic software, which already features on the FastNAS, range, could be compared to a MAM.  It is an asset organizer that can automatically scour all in-built metadata and integrate with AI tagging systems to give users the power to find what they’re looking for without having to manually enter any metadata.

Analytics Center will give users the visibility into their network so that they can see how they’re using their data, giving them a better understanding of individual or system-wide use, with suggestions on how to optimize their systems much more quickly and at a lower cost.

The new Core.4 software for both ranges builds on GB Labs’ current Core.3 OS offering a high-performance custom OS that is specifically used to serve media files. It allows a stable performance for every user and the best from the least amount of disk, which saves power.

EditShare’s flagship EditShare EFS scale-out storage enterprise scale-out storage solution was on display. It was developed for large-scale media organizations and supports hundreds of users simultaneously, with embedded tools for sharing media and collaborating across departments, across sites and around the world.

EditShare was showcasing advancements in its EFS File Auditing technology, the industry’s only realtime auditing platform designed to manage, monitor and secure your media from inception to delivery. EFS File Auditing keeps track of all digital assets and captures every digital footprint that a file takes throughout its life cycle, including copying, modifying and deleting of any content within a project.

Storbyte introduced its eco-friendly SBJ-496 at the 2019 NAB show. According to the company, this product is a new design in high-capacity disk systems for long-term management of digital media content with enterprise-class availability and data services. Ideal for large archive libraries, the SBJ-496 requires little to no electricity to maintain data, and its environmentally friendly green design allows unrestricted air flow, generates minimal heat and saves on cooling expenses.

Echo Flash

The new EcoFlash SBS-448, for digital content creation and streaming, is an efficient solid-state storage array that can deliver over 20GB of data per second. EcoFlash SBS-448 consumes less than half the electrical power and produces a lot less heat. Its patented design extends its lifespan significantly, resulting in a total operating cost per terabyte that is 300-500% lower.

NGD Systems was showing its computational storage product with several system partners at NAB, including at the Echostreams booth for its 1U platforms. NGD said that its M.2 and upcoming EDSFF form factors can be used in dense and performance-optimized solutions within the EchoStreams 1U server and canister system. In addition to providing data analytics and realtime analysis capture, the combination of NGD Systems products and EchoStreams 1U platforms allow for deployment at the extreme edge for use in onsite video acquisition and post processing at the edge.

OpenDrives was showcasing its Atlas software platform and product family of shared storage solutions. Its NAB demo was built on a single Summit system, including the OmniDrive media accelerator, powered by NVMe, to significantly boosts editorial, transcoding, color grading and visual effects shared workflows. OpenDrives is moving to a 2U form factor in its manufacturing, streamlining systems without sacrificing performance.

iX Systems said that their TrueNAS enterprise storage appliances deliver a perfect range of features and scalability for next-gen M&E workflows. AIC had an exhibit showing several enterprise storage systems, including some with NGD Systems computational storage SSDs. Promise Technology said that its VTrak NAS has been optimized for video application environments. Sony was offering PCIe SSD data storage servers. Other companies showing workflow storage products included Asustor, elements, PAC Storage and Rocstor.

Conclusions
The media and entertainment industry has unique requirements for storage to support modern digital workflows. A number of large and small companies have come up with a variety of local and cloud-based approaches to provide storage for post production applications. The NAB show is one of the world’s largest forums for such products and a great place to learn about what the digital storage and memory industry has to offer media and entertainment professionals.


Tom Coughlin, president of Coughlin Associates, is a digital storage analyst and business/technology consultant. He is active with SMPTE, SNIA, the IEEE — he is president of IEEE-USA and active in the CES, where he is chairman of the Future Directions Committee) and other pro organizations. 

High-performance flash storage at NAB 2018

By Tom Coughlin

After years of watching the development of flash memory-based storage for media and entertainment applications, especially for post, it finally appears that these products are getting some traction. This is driven by the decreasing cost of flash memory and also the increase in 4K up to 16K workflows with high frame rates and multi-camera video projects. The performance demanded for working storage to support multiple UHD raw video streams makes high performance storage attractive. Examples of 8K workflows were everywhere at the 2018 NAB show.

Flash memory is the clear leader in professional video camera media, increasing from 19% in 2009 to 66% in 2015, 54% in 2016 and 59% in 2017. The 2017 media and entertainment professional survey results are shown below.

Flash memory capacity used in M&E applications is believed to have been about 3.1% in 2016, but will be larger in coming years. Overall, revenues for flash memory in M&E should increase by more than 50% in the next few years as flash prices go down and it becomes a more standard primary storage for many applications.

At the 2018 NAB Show, and the NAB ShowStoppers, there were several products geared for this market and in discussion with vendors it appears that there is some real traction for solid state memory for some post applications, in addition to cameras and content distribution. This includes solid-state storage systems built with SAS, SATA and the newer NVMe interface. Let’s look at some of these products and developments.

Flash-Based Storage Systems
Excelero reports that its NVMe software-defined block storage solution with its low-latency and high-bandwidth improves the interactive editing process and enables customers to stream high-resolution video without dropping frames. Technicolor has said that it achieved 99.8% of local NVMe storage server performance across the network in an initial use of Excelero’s NVMesh. Below is the layout of the Pixit Media Excelero demonstration for 8K+ workflows at the NAB show.

“The IT infrastructure required to feed dozens of workstations of 4K files at 24ps is mindboggling — and that doesn’t even consider what storage demands we’ll face with 8K or even 16K formats,” says Amir Bemanian, engineering director at Technicolor. “It’s imperative that we can scale to future film standards today. Now, with innovations like the shared NVMe storage such as Excelero provides, Technicolor can enjoy a hardware-agnostic approach, enabling flexibility for tomorrow while not sacrificing performance.”

Excelero was showcasing 16K post production workflows with the Quantum StorNext storage and data management platform and Intel on the Technicolor project and at Mellanox with its 100Gb Ethernet switch.

Storbyte, a company based in Washington, DC, was showing its Eco Flash servers at the NAB show. Their product featured hot-swappable and accessible flash storage bays and redundant hot-swappable server controllers. The product features the company’s Hydra Dispersed Algorithmic Modeling (HDAM) that allows them to avoid having a flash transition layer, garbage collection, as well as dirty block management resulting in less performance overhead. Their Data Remapping Accelerator Core (DRACO) is said to offer up to a 4X performance increase over conventional flash architectures that can maintain peak performance even at 100% drive capacity and life and thus eliminates a write cliff and other problems that flash memory is subject to.

DDN was showing its ExaScaler DGX solution that combined a DDN ExaScaler ES14KX high-performance all-flash array integrated with a single Nvidia DGX-1 GPU server (initially announced at the 2018 GPU Technology Conference). Performance of the combination achieved up to 33GB/s of throughput. The company was touting this combination to accelerate machine learning, reducing the load times of large datasets to seconds for faster training. According to DDN, the combination also allows massive ingest rates and cost-effective capacity scaling and achieved more than 250,000 random read 4K IOPS. In addition to HDD-based storage, DDN offers hybrid HDD/SSD as well as all-flash array products. The new DDN SFA200NV all-flash platform product was on display at the 2018 NAB show

Dell EMC was showing its Isilon F800 all-flash scale-out NAS for creative applications. According to the company, the Isilon all-flash array gives visual effects artists and editors the power to work with multiple streams of uncompressed, full-aperture 4K material, enabling collaborative, global post and VFX pipelines for episodic and feature projects.

 

Dell EMC said this allows a true scale-out architecture with high concurrency and super-fast all-flash network-attached storage with low latency for high-throughput and random-access workloads. The company was demonstrating 4K editing of uncompressed DPX files with Adobe Premiere using a shared Isilon F800 all-flash array. They were also showing 4K and UHD workflows with Blackmagic’s DaVinci Resolve.

NetApp had a focus on solid-state storage for media workflows in their “Lunch and Learn sessions,” co-hosted by Advanced Systerms Group (ASG). The sessions discussed how NVMe and Storage Class Memory (SCM) are reshaping the storage industry. NetApp provides SSD-based E-series products that are used in the media and entertainment industry.

Promise Technology had its own NVMe SSD-based products. The company had data sheets on two NVMe fabric products. One was an HA storage appliance in a 2RU form factor (NVF-9000) with 24 NVMe drive slots and 100GbE ports offering up to 15M IOPS and 40GB/s throughout and many other enterprise features. The company said that its fabric allows servers to connect to a pool of storage nodes as if they had local NVMe SSDs. Promise’s NVMe Intelligent Storage is a 1U appliance (NVF-7000) with multiple 100 GbE connectors offering up to 5M IOPS and 20GB/s throughput. Both products offer RAID redundancy and end-to-end RDMA memory access.

Qumulo was showing its Qumulo P-Series NVMe all-flash solution. The P-series combines Qumulo File Fabric (QF2) software with high-speed NVMe, Intel Skylake SP processors and high-bandwidth Intel SSDs and 100GbE networking. It offers 16GB/s in a minimum four-node configuration (4GB/s per node). The P-series nodes come in 23 and 92TB size. According to Qumulo, QF2 provides realtime visibility and control regardless of the size of the file system, realtime capacity quotas, continuous replication, support for both SMB and NFS protocols, complete programmability with REST API and fast rebuild times. Qumulo says the P-series can run on-premise or in the cloud and can create a data fabric that interconnects every QF2 cluster whether it is all-flash, hybrid SSD/HDD or running on EC2 instances in AWS.

AIC was at the show with its J2024-04 2U 24-bay NVMe all-flash array using a Broadcom PCIe switch. The product includes dual hot-swap redundant 1.3 KW power supplies. AIC was also showing this AFA product providing a Storage Software Fabric platform with EXTEN smart NICs using Broadcom chips to create a storage software fabric platform, as well as an NVMe JBOF.

Companies such as Luma Forge were showing various hierarchical storage options, including flash memory, as shown in the image below.

Some other solid-state products included the use of two SATA SSDs for performance improvements for the SoftIron HyperDrive Ceph-based object storage appliance. Scale Logic has a hybrid SSD SAN/NAS product called Genesis Unlimited, which can support multiple 4K streams with a combination of HDDs and SSDs. Another NVMe offering was the RAIDIX NVMEXP software RAID engine for building NVMe-based arrays offering 4M IOPS and 30GB/s per 1U and offering RAID levels 5, 6 and 7.3. Nexsan has all-flash versions of its Unity storage products. Pure Storage had a small booth in the back of the South Hall lower showing their flash array products. Spectra Logic was showing new developments in its flash-based Black Pearl product, but we will cover that in another blog.

External Flash Storage Products
Other World Computing (OWC) was showing its solid-state and HDD-based products. They had a line-up of Thunderbolt 3 storage products, including the ThunderBlade and the Envoy Pro EX (VE) with Thunderbolt 3. The ThunderBlade uses a combination of M.2 SSDs to achieve transfer speeds up to 2.8 GB/s read and 2.45 GB/s write (pretty symmetrical R/W) with 1TB to 8TB storage capacity. It is fanless and has a dimmable LED so it won’t interfere with production work. OWC’s mobile bus-powered SSD product, Envoy Pro EX (VE) with Thunderbolt 3 provides sustained data rates up to 2.6 GB/s read and 1.6 GB/s write. This small 1TB to 2TB drive can be carried in a backpack or coat pocket.

Western Digital and Seagate had external SSD drives they were showing. Below is shown the G-Drive Mobile SSD-R, introduced late in 2017.

Memory Cards and SSDs
Samsung was at the NAB showing their EVO 860 2.5-inch. These SATA SSDs provide up to 4TB capacity and 550MB/s sequential read and 520MB/s sequential write speeds for media workstation applications. However, there were also showings of the product used in all-flash arrays as shown below.

ProGrade was showing its line of professional memory cards for high-end digital cameras. These included their SFExpress 1.0 memory card with 1TB capacity and 1.4GB/s read data transfer speed as well as burst write speed greater than 1GB/s. This new Compact Flash standard is a successor to both the C FAST and XQD formats. The product uses two lanes of PCIe and includes NVMe support. The product is interoperable with the XQD form factor. They also announced their V90 premium line of SDXC UHS-II memory cards with sustained read speeds of up to 250MB/s and sustained write speeds up to 200MB/s.

2018 Creative Storage Conference
For those who love storage, the 12th Annual Creative Storage Conference (CS 2018) will be held on June 7 at the Double Tree Hotel West Los Angeles in Culver City. This event brings together digital storage providers, equipment and software manufacturers and professional media and entertainment end users to explore the conference theme: “Enabling Immersive Content: Storage Takes Off.”

Also, my company Coughlin Associate is conducting a survey of digital storage requirements and practices for media and entertainment professionals with results presented at the 2018 Creative Storage Conference. M&E professionals can participate in the survey through this link. Those who complete the survey, with their contact information, will receive a free full pass to the conference.

Our main image: Seagate products in an editing session, including products in a Pelican case for field work. 


Tom Coughlin is president of Coughlin Associates, a digital storage analyst and  technology consultant. He has over 35 years in the data storage industry. He is also the founder of the Annual Storage Visions Conference and the Creative Storage Conference.

 

Riding the digital storage bus at the HPA Tech Retreat

By Tom Coughlin

At the 2018 HPA Tech Retreat in Palm Desert there were many panels that spoke to the changing requirements for digital storage to support today’s diverse video workflows. While at the show, I happened to snap a picture of the Maxx Digital bus — these guys supply video storage and RAID. I liked this picture because it had the logos of a number of companies with digital storage products serving the media and entertainment industry. So, this blog will ride the storage bus to see where digital storage in M&E is going.

Director of photography Bill Bennett, ASC, and senior scientist for RealD Tony Davis gave an interesting talk about why it can be beneficial to capture content at high frame rates, even if it will ultimately be shown at much lower frame rate. They also offered some interesting statics about Ang Lee’s 2016 technically groundbreaking movie, Billy Lynn’s Long Halftime Walk, which was shot in in 3D at 4K resolution and 120 frames per second.

The image above is a slide from the talk describing the size of the data generated in creating this movie. Single Sony F65 frames with 6:1 compression were 5.2MB in size with 7.5TB of average footage per day over 49 days. They reported that 104-512GB cards were used to capture and transfer the content and the total raw negative size (including test materials) was 404TB. This was stored on 1.5PB of hard disk storage. The actual size of the racks used for storage and processing wasn’t all that big. The photo below shows the setup in Ang Lee’s apartment.

Bennett and Davis went on to describe the advantages of shooting at high frame rates. Shooting at high frame rates gives greater on-set flexibility since no motion data is lost during shooting, so things can be fixed in post more easily. Even when shown at lower resolution in order to get conventional cinematic aesthetics, a synthetic shutter can be created with different motion sense in different parts of the frame to create effective cinematic effects using models for particle motion, rotary motion and speed ramps.

During Gary Demos’s talk on Parametric Appearance Compensation he discussed the Academy Color Encoding System (ACES) implementation and testing. He presented an interesting slide on a single master HDR architecture shown below. A master will be an important element in an overall video workflow that can be part of an archival package, probably using the SMPTE (and now ISO) Archive eXchange Format (AXF) standard and also used in a SMPTE Interoperable Mastering Format (IMF) delivery package.

The Demo Area
At the HPA Retreat exhibits area we found several interesting storage items. Microsoft had on exhibit one of it’s Data Boxes, that allow shipping up to 100 TB of data to its Azure cloud. The Microsoft Azure Data Box joins Amazon’s Snowball and Google’s similar bulk ingest box. Like the AWS Snowball, the Azure Data Box includes an e-paper display that also functions as a shipping label. Microsoft did early testing of their Data Box with Oceaneering International, which performs offline sub-sea oil industry inspection and uploaded their data to Azure using Data Box.

ATTO was showing its Direct2GPU technology that allowed direct transfer from storage to GPU memory for video processing without needing to pass through a system CPU. ATTO is a manufacturer of HBA and other connectivity solutions for moving data, and developing smarter connectors that can reduce overall system overhead.

Henry Gu’s GIC company was showing its digital video processor with automatic QC, and IMF tool set enabling conversion of any file type to IMF and transcoding to any file format and playback of all file types including 4K/UHD. He was doing his demonstration using a DDN storage array (right).

Digital storage is a crucial element in modern professional media workflows. Digital storage enables higher frame rate, HDR video recording and processing to create a variety of display formats. Digital storage also enables uploading bulk content to the cloud and implementing QC and IMF processes. Even SMPTE standards for AXF, IMF and others are dependent upon digital storage and memory technology in order to make them useful. In a very real sense, in the M&E industry, we are all riding the digital storage bus.


Dr. Tom Coughlin, president of Coughlin Associates, is a storage analyst and consultant. Coughlin has six patents to his credit and is active with SNIA, SMPTE, IEEE and other pro organizations. Additionally, Coughlin is the founder and organizer of the annual Storage Visions Conference as well as the Creative Storage Conference.
.

SMPTE ST 2110 enables IP workflows

By Tom Coughlin

At IBC2017 and this year’s SMPTE Conference there were significant demonstrations of IP-based workflows with interoperability demonstrations and conference sessions. Clearly proprietary media networking will be supplanted by IP-based workflows. This will enable new equipment economies and open up new opportunities for using and repurposing media. IP workflows will also impact the way we store and use digital content and thus the storage systems where they live.

SMPTE has just ratified ST 2110 standards for IP transport in media workflows. The standard puts video, audio and ancillary data into separate routable streams as shown in the figure below. PCM Audio streams are covered by SMPTE ST 2110-30, uncompressed video streams are covered by ST 2110-20 and ancillary data is covered by ST 2110-40. Some other parts of the standards cover traffic shaping of uncompressed video (ST 2110-21), AES3 transparent transport (ST 2110-31) and ST 2110-50 allows integration with older specification ST 2022-6 that covers legacy SDI over IP.

The separate streams have timestamps that allow proper alignment of the different streams when they are combined together — this timestamp is provided by ST 2059. Each stream contains metadata that tells the receiver how to interpret what is inside of the stream. The uncompressed video stream supports up to 32k X 32k images, HDR and all common color systems and formats.

The important thing about these IP standards is that they allow using conventional Ethernet cabling rather than special proprietary cables. This saves a lot of money on hardware. In addition, having an IP-based workflow allows easy ingest into a core IP network and distribution of content using IP-based broadcast, telco, cable and broadband technologies as well as satellite channels. As most consumers have IP content access, these IP networks connect directly to consumer equipment. The image below from an Avid presentation by Shailendra Mathur at SMPTE 2017 illustrates the workflow below.

At IBC and the SMPTE 2017 Conference there were interoperability demonstrations. Although the IBC interop demo had many more participants the SMPTE demo was pretty extensive. The photo below shows the SMPTE interoperability demonstration setup.

As many modern network storage systems, whether file or object based, use Ethernet connectivity, having the rest of the workflow using an IP network makes movement of data through the workflow to and from digital storage easier. Since access to cloud-based assets is also though IP-based networks and these can feed CDNs and other distribution networks, on-premise and cloud storage interact through IP networks and can be used to support working storage, archives as well as content distribution libraries.

IP workflows combined with IP-based digital storage provide end-to-end processing and storage of data. This provides hardware economics and access to a lot of software built to manage and monitor IP flows to help optimize a media production and distribution system. By avoiding the overhead of converting from one type of network to another the overall system complexity and efficiency will be improved, resulting in faster projects and easier repair of problems when they arise.


Tom Coughlin is president of Coughlin Associates. He is the founder and organizer of the annual Storage Visions Conference as well as the Creative Storage Conference. He has also been the general chairman of the annual Flash Memory Summit.

Origins: The Creative Storage Conference

By Tom Coughlin

I was recently asked how the Creative Storage Conference came to be. So here I am to give you some background.

In 2006, the Storage Visions Conference that my colleagues and I had been organizing just before the CES show in January was in its fifth year. I had been doing more work on digital storage for professional media and entertainment, including a report on this important topic. In order to increase my connections and interaction with both media and entertainment professionals, and the digital storage and service companies that support them, it seems that a conference focusing on digital storage for media and entertainment would be in order.

That same year, my partner Ron Dennison and I participated in the MediaTech Conference in the LA area, working with Bryan Eckus, the director of the group at the time. In 2007, we held the first Creative Storage Conference in conjunction with the MediaTech Conference in Long Beach, California. It featured a dynamite line-up of storage companies and end users.

The conference has grown in size over the years, and we have had a stream of great companies showing their stuff, media and entertainment professional attendees and speakers, informative sessions and insightful keynote talks on numerous topics related to M&E digital storage.

The 2017 Creative Storage Conference
This year, the Creative Storage Conference is taking place on May 24 in Culver City. Attendees can learn more about the use of Flash memory in M&E as well as the growth in VR content in professional video, and how this will drive new digital storage demand and technologies to support the high data rates needed for captured content and cloud-based VR services. This is the 11th year of the conference and we look forward to having you join us.

We are planning for six sessions and four keynotes during the day and a possible reception in the evening on May 24.

Here is a list of the planned sessions:
• Impact of 4K/HDR/VR on Storage Requirements From Capture to Studio
• Collaboration in the Clouds: Storing and Delivering Content Where it is Needed
• Content on the Move: Delivering Storage Content When and Where it is Needed
• Preserving Digital Content — the Challenges, Needs and Options
• Accelerating Workflows: Solid State Storage in Media and Entertainment
• Independent Panel — Protecting the Life of Content

Don’t miss this opportunity to meet giants in the field of VR content capture and post production and meet the storage and service companies to help you make sure your next professional projects are a big success.

• Hear how major media equipment suppliers and entertainment industry customers use digital storage technology in all aspects of content creation and distribution.
• Find out the role that digital storage plays in new content distribution and marketing opportunities for a rapidly evolving market.
• See presentations on digital storage in digital acquisition and capture, nonlinear editing and special effects.
• Find out how to convert and preserve content digitally and protect it in long-term dependable archives.
• Learn about new ways to create and use content metadata, making it easier to find and use.
• Discover how to combine and leverage hard disk drives, flash memory, magnetic tape and optical storage technology with new opportunities in the digital media market.
• Be at the juncture of digital storage and the next generation of storage for the professional media market.

Online registration is open until May 23, 2017. As a media and entertainment professional you can register now with a $100 discount using this link:

—–
Thomas Coughlin, president of Coughlin Associates is a storage analyst and consultant with over 30 years in the data storage industry. He is active with SNIA, SMPTE, IEEE, and other professional organizations.

HPA Tech Retreat takes on realities of virtual reality

By Tom Coughlin

The HPA Tech Retreat, run by the Hollywood Professional Association in association with SMPTE, began with an insightful one-day VR seminar— Integrating Virtual Reality/Augmented Reality into Entertainment Applications. Lucas Wilson from SuperSphere kicked off the sessions and helped with much of the organization of the seminar.

The seminar addressed virtual reality (VR), augmented reality (AR) and mixed reality (MR, a subset of AR where the real world and the digital world interact, like Pokeman Go). As in traditional planar video, 360-degree video still requires a director to tell a story and direct the eye to see what is meant to be seen. Successful VR requires understanding how people look at things, how they perceive reality, and using that understanding to help tell a story. Some things that may help with this are reinforcement of the viewer’s gaze with color and sound that may vary with the viewer — e.g. these may be different for the “good guy” and the “bad guy.”

VR workflows are quite different from traditional ones, with many elements changing with multiple-camera content. For instance, it is much more difficult to keep a camera crew out of the image, and providing proper illumination for all the cameras can be a challenge. The image below from Jaunt shows their 360-degree workflow, including the use of their cloud-based computational image service to stitch the images from the multiple cameras.
Snapchat is the biggest MR application, said Wilson. Snapchat’s Snapchat-stories could be the basis of future post tools.

Because stand-alone headsets (head-mounted displays, or HMDs) are expensive, most users of VR rely on smart phone-based displays. There are also some places that allow one or more people to experience VR, such as the IMAX center in Los Angeles. Activities such as VR viewing will be one of the big drivers for higher-resolution mobile device displays.

Tools that allow artists and directors to get fast feedback on their shots are still in development. But progress is being made, and today over 50 percent of VR is used for video viewing rather than games. Participants in a VR/AR market session, moderated by the Hollywood Reporter’s Carolyn Giardina and including Marcie Jastrow, David Moretti, Catherine Day and Phil Lelyveld, seemed to agree that the biggest immediate opportunity is probably with AR.

Koji Gardiner from Jaunt gave a great talk on their approach to VR. He discussed the various ways that 360-degree video can be captured and the processing required to create finished stitched video. For an array of cameras with some separation between the cameras (no common axis point for the imaging cameras), there will be area that needs to be stitched together between camera images using common reference points between the different camera images as well as blind spots near to the cameras where they are not capturing images.

If there is a single axis for all of the cameras then there are effectively no blind spots and no stitching possible as shown in the image below. Covering all the space to get a 360-degree video requires additional cameras located on that axis to cover all the space.

The Fraunhofer Institute, in Germany, has been showing a 360-degree video camera with an effective single axis for several cameras for several years, as shown below. They do this using mirrors to reflect images to the individual cameras.

As the number of cameras is increased, the mathematical work to stitch the 360-degree images together is reduced.

Stitching
There are two approaches commonly used in VR stitching of multiple camera videos. The easiest to implement is a geometric approach that uses known geometries and distances to objects. It requires limited computational resources but results in unavoidable ghosting artifacts at seams from the separate images.

The Optical Flow approach synthesizes every pixel by computing correspondences between neighboring cameras. This approach eliminates the ghosting artifacts at the seams but has its own more subtle artifacts and requires significantly more processing capability. The Optical Flow approach requires computational capabilities far beyond those normally available to content creators. This has led to a growing market to upload multi-camera video streams to cloud services that process the stitching to create finished 360-degree videos.

Files from the Jaunt One camera system are first downloaded and organized on a laptop computer and then uploaded to Jaunt’s cloud server to be processed and create the stitching to make a 360 video. Omni-directionally captured audio can also be uploaded and mixed ambisonically, resulting in advanced directionality in the audio tied to the VR video experience.

Google and Facebook also have cloud-based resources for computational photography used for this sort of image stitching.

The Jaunt One 360-degree camera has a 1-inch 20MP rolling shutter sensor with frame rates up to 60fps with 3200 ISO max, 29dB SNR at ISO800. It has a 10 stops per camera module, with 130-degree diagonal FOV, 4/2.9 optics and with up to 16K resolution (8K per eye). Jaunt One at 60fps provides 200GB/minute uncompressed. This can fill a 1TB SSD in five minutes. They are forced to use compression to be able to use currently affordable storage devices. This compression creates 11GB per minute, which can fill a 1TB SSD in 90 minutes.

The actual stitched image, laid out flat, looks like a distorted projection. But when viewed in a stereoscopic viewer it appears to look like a natural image of the world around the viewer, giving an immersive experience. At one point in time the viewer does not see all of the image but only the image in a restricted space that they are looking directly at as shown in the red box in the figure below.

The full 360-degree image can be pretty high resolution, but unless the resolution is high enough, the resolution inside the scene being viewed at any point in time will be much less that the resolution of the overall scene, unless special steps are taken.

The image below shows that for a 4k 360-degree video the resolution in the field of view (FOV) may be only 1K, much less resolution and quite perceptible to the human eye.

In order to provide a better viewing experience in the FOV, either the resolution of the entire view must be better (e.g. the Jaunt One high-resolution version has 8K per eye and thus 16K total displayed resolution) or there must be a way to increase the resolution in the most significant FOV in a video, so at least in that FOV, the resolution leads to a greater feeling of reality.

Virtual reality, augmented reality and mixed reality create new ways of interacting with the world around us and will drive consumer technologies and the need for 360-degree video. New tools and stitching software, much of this cloud-based, will enable these workflows for folks who want to participate in this revolution in content. The role of a director is as important as ever as new methods are needed to tell stories and guide the viewer to engage in this story.

2017 Creative Storage Conference
You can learn more about the growth in VR content in professional video and how this will drive new digital storage demand and technologies to support the high data rates needed for captured content and cloud-based VR services at the 2017 Creative Storage Conference — taking place May 24, 2017 in Culver City.


Thomas M. Coughlin of Coughlin Associates is a storage analyst and consultant. He has over 30 years in the data storage industry and is the author of Digital Storage in Consumer Electronics: The Essential Guide.

IBC 2016: VR and 8K will drive M&E storage demand

By Tom Coughlin

While attending the 2016 IBC show, I noticed some interesting trends, cool demos and new offerings. For example, while flying drones were missing, VR goggles were everywhere; IBM was showing 8K video editing using flash memory and magnetic tape; the IBC itself featured a fully IP-based video studio showing the path to future media production using lower-cost commodity hardware with software management; and, it became clear that digital technology is driving new entertainment experiences and will dictate the next generation of content distribution, including the growing trend to OTT channels.

In general, IBC 2016 featured the move to higher resolution and more immersive content. On display throughout the show was 360-degree video for virtual reality, as well as 4K and 8K workflows. Virtual reality and 8K are driving new levels of performance and storage demand, and these are just some of the ways that media and entertainment pros are future-zone-2increasing the size of video files. Nokia’s Ozo was just one of several multi-camera content capture devices on display for 360-degree video.

Besides multi-camera capture technology and VR editing, the Future Tech Zone at IBC included even larger 360-degree video display spheres than at the 2015 event. These were from Puffer Fish (pictured right). The smaller-sized spherical display was touch-sensitive so you could move your hand across the surface and cause the display to move (sadly, I didn’t get to try the big sphere).

IBM had a demonstration of a 4K/8K video editing workflow using the IBM FlashSystem and IBM Enterprise tape storage technology, which was a collaboration between the IBM Tokyo Laboratory and IBM’s Storage Systems division. This work was done to support the move to 4K/8K broadcasts in Japan by 2018, with a broadcast satellite and delivery of 8K video streams of the 2020 Tokyo Olympic Games. The combination of flash memory storage for working content and tape for inactive content is referred to as FLAPE (flash and tAPE).

The graphic below shows a schematic of the 8K video workflow demonstration.

The argument for FLAPE appears to be that flash performance is needed for editing 8K content and the magnetic tape provides low-cost storage the 8K content, which may require greater than 18TB for an hour of raw content (depending upon the sampling and frame rate). Note that magnetic tape is often used for archiving of video content, so this is a rather unusual application. The IBM demonstration, plus discussions with media and entertainment professionals at IBC indicate that with the declining costs of flash memory and the performance demands of 8K, 8K workflows may finally drive increased demand for flash memory for post production.

Avid was promoting their Nexis file system, the successor to ISIS. The company uses SSDs for metadata, but generally flash isn’t used for actual editing yet. They agreed that as flash costs drop, flash could find a role for higher resolution and richer media. Avid has embraced open source for their code and provides free APIs for their storage. The company sees a hybrid of on-site and cloud storage for many media and entertainment applications.

EditShare announced a significant update to its XStream EFS Shared Storage Platform (our main image). The update provides non-disruptive scaling to over 5PB with millions of assets in a single namespace. The system provides a distributed file system with multiple levels of hardware redundancy and reduced downtime. An EFS cluster can be configured with a mix of capacity and performance with SSDs for high data rate content and SATA HDD for cost-efficient higher-performance storage — 8TB HDDs have been qualified for the system. The latest release expands optimization support for file-per-frame media.

The IBC IP Interoperability Zone was showing a complete IP-based studio (pictured right) was done with the cooperation of AIMS and the IABM. The zone brings to life the work of the JT-NM (the Joint Task Force on Networked Media, a combined initiative of AMWA, EBU, SMPTE and VSF) and the AES on a common roadmap for IP interoperability. Central to the IBC Feature Area was a live production studio, based on the technologies of the JT-NM roadmap that Belgian broadcaster VRT has been using daily on-air all this summer as part of the LiveIP Project, which is a collaboration between VRT, the European Broadcasting Union (EBU) and LiveIP’s 12 technology partners.

Summing Up
IBC 2016 showed some clear trends to more immersive, richer content with the numerous displays of 360-degree and VR content and many demonstrations of 4K and even 8K workflows. Clearly, the trend is for higher-capacity, higher-performance workflows and storage systems that support this workflow. This is leading to a gradual move to use flash memory to support these workflows as the costs for flash go down. At the same time, the move to IP-based equipment will lead to lower-cost commodity hardware with software control.

Storage analyst Tom Coughlin is president of Coughlin Associates. He has over 30 years in the data storage industry and is the author of Digital Storage in Consumer Electronics: The Essential Guide. He also  publishes the Digital Storage Technology Newsletter, the Digital Storage in Media and Entertainment Report.

The State of Storage

Significant trends are afoot in media and entertainment storage.

By Tom Coughlin

Digital storage plays a significant role in the media and entertainment industry, and our specific demands are often very different from typical IT storage. We are dealing with performance requirements of realtime video in capture, editing and post, as well as distribution. On the other hand, the ever-growing archive of long-tail digital content and digitized historical analog content is swelling the demand for archives (both cold and warm) using tape, optical discs and hard drive arrays.

My company, Coughlin Associates, has conducted surveys of digital storage use by media and entertainment professionals since 2009. These results are used in our annual Digital Storage in Media and Entertainment Report. This article presents results from the 2016 survey and some material from the 222-page report to discuss the status of digital storage for professional media and entertainment.

Content Creation and Capture
Pro video cameras are undergoing rapid evolution, driven by higher-resolution content as well as multi-camera content capture, including stereoscopic and virtual reality. In addition, the physical storage media for professional cameras is undergoing rapid evolution as film and magnetic digital tape is impacted by the rapid file access convenience of hard disk drives, optical discs, and the ruggedness of flash-based solid-state storage.

The table below compares the results from the 2009, 2010, 2012, 2013, 2014 and 2015 surveys with those from 2016. Flash memory is the clear leader in pro video camera media, increasing from 19% in 2009 to 66% in 2015 and then down to 54% in 2016, while magnetic tape shows a consistent decline over the same period.

Optical disc use between 2009 and 2016 bounced around between 7% and 17%. Film shows a general decline from 15% usage in 2009 to 2% in 2016. The trend with declining film use follows the trend toward completely digital workflows.

Note that about 60% of survey participants said that they used external storage devices to capture content from their cameras in 2016 (perhaps this is why the HDD percentages are so high). In 2016, 83% said that over 80% of their content is created in a digital format.

In 2016, 93.2% of the survey respondents said they reuse their recording media (compared to 89.9% in 2015, 93.3% in 2014, 84.5% in 2013, 86% in 2012, 79% in 2010 and 75% in 2009). In 2016, 75% of respondents said they archive their camera recording media (compared to 73.6% in 2015, 74.2% in 2014, 81.4% in 2013, 85% in 2012 and 77% in 2010).

Archiving the original recording media may be a practice in decline — especially with expensive reusable media such as flash memory cards. Digital storage on tape, hard disk drives or flash storage allows the reuse of media.

Post Production
The size of content — and amount — has put strains on post network bandwidth and storage. This includes editing and other important operations. As much of this work may take place in smaller facilities, these companies may be doing much of their work on direct attached storage devices and they may share or archive this media in the cloud in order to avoid the infrastructure costs of running a data center.

The graph below shows that for the 2016 survey participants there was a general increase in the use of shared network storage (such as SAN or NAS), and a decrease in DAS storage as the number of people working in a post facility increases. The DAS storage in the larger facilities may be different than that used in smaller facilities.

DAS vs. shared storage by number of people in a post facility.

When participants were asked about their use of direct attached and network storage in digital editing and post, the survey showed the following summary statistics in 2016 (compared to earlier surveys):

– 74.5% had DAS
– 89.8% of these had more than 1 TB of DAS
– 10 to 50 TB was the most popular DAS size (27.5%)
– 17.4% of these had more than 50 TB of DAS storage
– 2.9% had more than 500 TB of DAS storage
– 68.1% had NAS or SAN
– 57.4% had 50 TB or more of network storage in 2016
– About 15% had more than 500 TB of NAS/SAN storage in 2016
– Many survey participants had considerable storage capacities in both DAS and NAS/SAN.

We asked whether survey participants used cloud-based storage for editing and post. In 2016 23.0% of responding participants said yes. The respondents, 20.9% of them, said that they had 1TB or more of their storage capacity in the cloud.

Content Distribution
Distribution of professional video content has many channels. It can use physical media for getting content to digital cinemas or to consumers, the distribution can be done electronically using broadcast, cable or satellite transmission, or through the Internet or mobile phone networks.

The table below gives responses for the percentage of physical media used by the survey respondents for content distribution in 2016, 2015, 2014, 2013, 2012 and 2010. Note that these are the average for the survey population giving their percentage for each physical media and do not and should not be expected to add to 100%. Digital tape, DVD discs, HDDs and Flash Memory are the most popular distribution formats.

Average percentage content on physical media for professional content distribution.

Following are survey observations for electronic content distribution, such as video on demand.

– The average number of hours on a central content delivery system was 2,174 hours in 2016.
– There was an average of 427 hours ingested monthly in 2016.
– In 2016, 38% of respondents had more than 5% of their content on edge servers.
– About 31% used flash memory on their edge servers in 2016.

Archiving and Preservation
Today, most new entertainment and media content is born digital, so it is natural that this content should be preserved in digital form. This requirement places new demands on format preservation for long-term digital archives as well as management and systematic format refreshes during the expected life of a digital archive.

In addition, the cost of analog content digitization and preservation in a digital format has gone down considerably, and many digitization projects are proceeding apace. The growth of digital content archiving will swell the amount of content available for repurposing and long-tail distribution. It will also swell the amount of storage and storage facilities required to store these long-term professional content archives.

Following are some observations from our 2016 survey on trends in digital archiving and content preservation.

– 41% had less than 2,000 hours of content in a long-term archive
– 56.9% archived all the content captured from their cameras
– 54.0% archived copies of content in all of their distribution formats
– 35.9% digitally archived all content captured from their dailies
– 31.3% digitally archived all content captured from rough cuts
– 36.5% digitally archived all content captured from their intermediaries
– 50.9% of the respondents said that their annual archive growth rate was less than 6% in 2016
– About 28.6% had less than 2,000 hours of unconverted analog content
– 16.7% of participants had over 5,000 hours of unconverted analog content
– About 52.5% of the survey respondents have an annual analog conversion rate of 2% or less
– The average rate of conversion is about 3.4% in 2016

Professional media and entertainment content was traditionally archived on film or analog videotapes. Today, the options available for archive media to store digital content depend upon the preferences and existing infrastructure of digital archive facilities. Figure 6 gives the percentage distribution of archive media used by the survey participants.

Percentage of digital long-term archives on various media

Some other observations from the archive and preservation section of the survey:

– About 42.6% never update their digital archives.
– About 76.2% used different storage for archiving and working storage.
– About 49.2% copied and replaced their digital long-term archives every 10 years or less.
– 38.1% said they would use a private or public cloud for archiving in 2016.

Conclusions
Larger content files, driven by higher resolution, higher frame rates, higher dynamic range and stereoscopic and virtual reality video are creating larger video files. This is driving the need for high-performance storage to work on this content and to provide fast delivery, which could drive more creative work to use solid-state storage.

At the same time, cost -effective storage and management of completed work is driving the increased use of hard disk drives, magnetic tape and even optical storage for low-cost storage.

The price of storing content in the cloud has gone down so much that there are magnetic tape-based cloud storage offerings that are less expensive than building one’s own storage data center, at least for small- and moderate-sized facilities.

This trend is expected to grow the use of cloud storage in media and entertainment, especially for archiving, as shown in the figure below.

Growth of cloud storage in media and entertainment.


Dr. Tom Coughlin, president of Coughlin & Associates, is a storage analyst and consultant with over 30 years in the data storage industry. He is the founder and organizer of the Annual Storage Visions Conference as well as the Creative Storage Conference.

Creative Storage Conference is now soliciting presentations

No one can deny how important storage is in the day-to-day lives of post pros. From big studios and facilities to indie editors working with external drives, storage is an indispensable part of the way people work today.  So if you want to talk storage and learn about the latest trends and product offerings, you might want to attend the Creative Storage Conference (CS 2016), celebrating its 10th anniversary this year.

CS 2016 is accepting submissions for presentations, speakers and panels now through May 1, 2016 at www.creativestorage.org. CS 2016 will bring together digital storage providers, equipment and software manufacturers, and professional media and entertainment end users to explore this year’s conference theme, “The Art of Storage.”

The 2016 event will be held Thursday, June 23 at the DoubleTree Hotel West Los Angeles in Culver City. “We have a great agenda planned with six sessions throughout the day and four keynote talks during the conference,” said Tom Coughlin, who is both the chairman and organizer of the event.

The Creative Storage Conference is put on by the Entertainment Storage Alliance and Coughlin Associates.

IBC 2015: Adventures in archiving

By Tom Coughlin

Once you have your content and have completed that award-winning new project, Oscar-nominated film or brilliant and effective commercial, where does your data go? Today, video content can be reused and repurposed a number of times, producing a continuing revenue stream by providing viewing for many generations of people. That makes video archives valuable and also requires changes from in-active to more active archives. This article looks at some of the archiving products on display at the 2015 IBC.

The figure to the right shows our estimate of revenue spent on various media and entertainment storage markets in 2014 (from the Digital Storage in Media and Entertainment Report from Coughlin Associates). Note that although almost 96 percent of all M&E storage capacity is used for archiving about 45 percent of the spending is for archiving.

Quantum showcased its StorNext 5 shared storage architecture, which includes high-performance online storage, extended online storage and tape- and cloud-based archives. The company also highlighted the StorNext Connect, a management and monitoring console that provides an at-a-glance dashboard of the entire StorNext environment. At IBC, Quantum introduced their Q-Cloud Archive that extends StorNext workflow capabilities to the cloud, allowing end-to-end StorNext environments to leverage cloud storage fully with no additional hardware, separate applications or programming while maintaining full compatibility with existing software applications.

The Quantum Storage Manager migrates data from online storage to its object-based Lattus, allowing secure, long-term storage with greater durability than RAID and extremely high scalability. Content can be migrated from Lattus to tape archives or Q-Cloud archives automatically. In addition Quantum’s Artico intelligent NAS archive appliance was on display, offering low cost scale-out storage for active media archives that can scale to PBs of content across HDDs, extended online storage, tape and cloud storage.

Also during IBC, the LTO Program Technology Provider Companies — HP, IBM and Quantum —announced the LTO-7 tape format that will be available in late 2015. The native capacity of this drive is 6TB, while 2.5:1 compression provides 15TB of storage with up to 750MB/s data rates. This product will provide over twice the capacity of the LTO-6 drive generation. The LTO roadmap goes out to a generation 10 product with up to 120TB of compressed content and about 48TB native capacity.

LTO proponents said that tape has some advantages over hard disk drives for archiving, despite the difference in latency to access content. In particular, they said tape has and error rate two orders of magnitude lower than HDDs, providing more accurate recording and reading of content. Among the interesting LTO developments at IBC were the M-Logic Thunderbolt interface tape drives.

Tape can also be combined with capacity SATA HDDs to provide storage systems with performance approaching hard disk drive arrays and costs approaching magnetic tape libraries. Crossroads has teamed up with Fujifilm to provide NAS systems combining HDDs and tape and including cloud storage combining tape and HDDs. In fact archiving is becoming one of the biggest growing applications in the media and entertainment industry, according to the 2015 Digital Storage in Media and Entertainment Report from Coughlin Associates.

Oracle was also showing its tape storage systems with 8TB native storage capacity in a half-inch tape form factor. Oracle now includes Front Porch Digital with its cloud archiving platform as well as digital ingest solutions for older analog and digital format media.

Some companies also use flash memory as a content cache in order to match the high speeds of data transfers to and from a tape library system. Companies such as Xendata provide LTO tape and optical disc libraries for media and entertainment customers. Spectra Logic has made a big push into HDD-based archiving, using shingled magnetic recording (SMR) 3.5-inch HDDs in their DPE storage system to provide unstructured storage costs as low as 9 cents/GB. This system can provide up to 7.4PB of raw capacity in a single rack with 1GB/s data rates. This sort of system is best for data that is seldom or never overwritten because of the use of SMR HDDS.

Sony was showing its 300GB Blu-ray optical WORM discs, although it was not clear if the product is being shipped in storage cartridges in 2015. Archiving is a significant driver of M&E storage demand. This is because all video eventually ends up in an archive. Because of more frequent access of archived content, the performance requirements of many archives are more demanding than in the past. This has led to the use of HDD-based archives and archives combining HDDs and magnetic tape. Even flash memory can play a role as a write and read cache in a tape based system.

Dr. Tom Coughlin, president of Coughlin Associates, has over 35 years in the data storage industry. Coughlin is also the founder and organizer of the annual Storage Visions Conference, a partner to the International Consumer Electronics Show, as well as the Creative Storage Conference