Tag Archives: Flash Storage

High-performance flash storage at NAB 2018

By Tom Coughlin

After years of watching the development of flash memory-based storage for media and entertainment applications, especially for post, it finally appears that these products are getting some traction. This is driven by the decreasing cost of flash memory and also the increase in 4K up to 16K workflows with high frame rates and multi-camera video projects. The performance demanded for working storage to support multiple UHD raw video streams makes high performance storage attractive. Examples of 8K workflows were everywhere at the 2018 NAB show.

Flash memory is the clear leader in professional video camera media, increasing from 19% in 2009 to 66% in 2015, 54% in 2016 and 59% in 2017. The 2017 media and entertainment professional survey results are shown below.

Flash memory capacity used in M&E applications is believed to have been about 3.1% in 2016, but will be larger in coming years. Overall, revenues for flash memory in M&E should increase by more than 50% in the next few years as flash prices go down and it becomes a more standard primary storage for many applications.

At the 2018 NAB Show, and the NAB ShowStoppers, there were several products geared for this market and in discussion with vendors it appears that there is some real traction for solid state memory for some post applications, in addition to cameras and content distribution. This includes solid-state storage systems built with SAS, SATA and the newer NVMe interface. Let’s look at some of these products and developments.

Flash-Based Storage Systems
Excelero reports that its NVMe software-defined block storage solution with its low-latency and high-bandwidth improves the interactive editing process and enables customers to stream high-resolution video without dropping frames. Technicolor has said that it achieved 99.8% of local NVMe storage server performance across the network in an initial use of Excelero’s NVMesh. Below is the layout of the Pixit Media Excelero demonstration for 8K+ workflows at the NAB show.

“The IT infrastructure required to feed dozens of workstations of 4K files at 24ps is mindboggling — and that doesn’t even consider what storage demands we’ll face with 8K or even 16K formats,” says Amir Bemanian, engineering director at Technicolor. “It’s imperative that we can scale to future film standards today. Now, with innovations like the shared NVMe storage such as Excelero provides, Technicolor can enjoy a hardware-agnostic approach, enabling flexibility for tomorrow while not sacrificing performance.”

Excelero was showcasing 16K post production workflows with the Quantum StorNext storage and data management platform and Intel on the Technicolor project and at Mellanox with its 100Gb Ethernet switch.

Storbyte, a company based in Washington, DC, was showing its Eco Flash servers at the NAB show. Their product featured hot-swappable and accessible flash storage bays and redundant hot-swappable server controllers. The product features the company’s Hydra Dispersed Algorithmic Modeling (HDAM) that allows them to avoid having a flash transition layer, garbage collection, as well as dirty block management resulting in less performance overhead. Their Data Remapping Accelerator Core (DRACO) is said to offer up to a 4X performance increase over conventional flash architectures that can maintain peak performance even at 100% drive capacity and life and thus eliminates a write cliff and other problems that flash memory is subject to.

DDN was showing its ExaScaler DGX solution that combined a DDN ExaScaler ES14KX high-performance all-flash array integrated with a single Nvidia DGX-1 GPU server (initially announced at the 2018 GPU Technology Conference). Performance of the combination achieved up to 33GB/s of throughput. The company was touting this combination to accelerate machine learning, reducing the load times of large datasets to seconds for faster training. According to DDN, the combination also allows massive ingest rates and cost-effective capacity scaling and achieved more than 250,000 random read 4K IOPS. In addition to HDD-based storage, DDN offers hybrid HDD/SSD as well as all-flash array products. The new DDN SFA200NV all-flash platform product was on display at the 2018 NAB show

Dell EMC was showing its Isilon F800 all-flash scale-out NAS for creative applications. According to the company, the Isilon all-flash array gives visual effects artists and editors the power to work with multiple streams of uncompressed, full-aperture 4K material, enabling collaborative, global post and VFX pipelines for episodic and feature projects.

 

Dell EMC said this allows a true scale-out architecture with high concurrency and super-fast all-flash network-attached storage with low latency for high-throughput and random-access workloads. The company was demonstrating 4K editing of uncompressed DPX files with Adobe Premiere using a shared Isilon F800 all-flash array. They were also showing 4K and UHD workflows with Blackmagic’s DaVinci Resolve.

NetApp had a focus on solid-state storage for media workflows in their “Lunch and Learn sessions,” co-hosted by Advanced Systerms Group (ASG). The sessions discussed how NVMe and Storage Class Memory (SCM) are reshaping the storage industry. NetApp provides SSD-based E-series products that are used in the media and entertainment industry.

Promise Technology had its own NVMe SSD-based products. The company had data sheets on two NVMe fabric products. One was an HA storage appliance in a 2RU form factor (NVF-9000) with 24 NVMe drive slots and 100GbE ports offering up to 15M IOPS and 40GB/s throughout and many other enterprise features. The company said that its fabric allows servers to connect to a pool of storage nodes as if they had local NVMe SSDs. Promise’s NVMe Intelligent Storage is a 1U appliance (NVF-7000) with multiple 100 GbE connectors offering up to 5M IOPS and 20GB/s throughput. Both products offer RAID redundancy and end-to-end RDMA memory access.

Qumulo was showing its Qumulo P-Series NVMe all-flash solution. The P-series combines Qumulo File Fabric (QF2) software with high-speed NVMe, Intel Skylake SP processors and high-bandwidth Intel SSDs and 100GbE networking. It offers 16GB/s in a minimum four-node configuration (4GB/s per node). The P-series nodes come in 23 and 92TB size. According to Qumulo, QF2 provides realtime visibility and control regardless of the size of the file system, realtime capacity quotas, continuous replication, support for both SMB and NFS protocols, complete programmability with REST API and fast rebuild times. Qumulo says the P-series can run on-premise or in the cloud and can create a data fabric that interconnects every QF2 cluster whether it is all-flash, hybrid SSD/HDD or running on EC2 instances in AWS.

AIC was at the show with its J2024-04 2U 24-bay NVMe all-flash array using a Broadcom PCIe switch. The product includes dual hot-swap redundant 1.3 KW power supplies. AIC was also showing this AFA product providing a Storage Software Fabric platform with EXTEN smart NICs using Broadcom chips to create a storage software fabric platform, as well as an NVMe JBOF.

Companies such as Luma Forge were showing various hierarchical storage options, including flash memory, as shown in the image below.

Some other solid-state products included the use of two SATA SSDs for performance improvements for the SoftIron HyperDrive Ceph-based object storage appliance. Scale Logic has a hybrid SSD SAN/NAS product called Genesis Unlimited, which can support multiple 4K streams with a combination of HDDs and SSDs. Another NVMe offering was the RAIDIX NVMEXP software RAID engine for building NVMe-based arrays offering 4M IOPS and 30GB/s per 1U and offering RAID levels 5, 6 and 7.3. Nexsan has all-flash versions of its Unity storage products. Pure Storage had a small booth in the back of the South Hall lower showing their flash array products. Spectra Logic was showing new developments in its flash-based Black Pearl product, but we will cover that in another blog.

External Flash Storage Products
Other World Computing (OWC) was showing its solid-state and HDD-based products. They had a line-up of Thunderbolt 3 storage products, including the ThunderBlade and the Envoy Pro EX (VE) with Thunderbolt 3. The ThunderBlade uses a combination of M.2 SSDs to achieve transfer speeds up to 2.8 GB/s read and 2.45 GB/s write (pretty symmetrical R/W) with 1TB to 8TB storage capacity. It is fanless and has a dimmable LED so it won’t interfere with production work. OWC’s mobile bus-powered SSD product, Envoy Pro EX (VE) with Thunderbolt 3 provides sustained data rates up to 2.6 GB/s read and 1.6 GB/s write. This small 1TB to 2TB drive can be carried in a backpack or coat pocket.

Western Digital and Seagate had external SSD drives they were showing. Below is shown the G-Drive Mobile SSD-R, introduced late in 2017.

Memory Cards and SSDs
Samsung was at the NAB showing their EVO 860 2.5-inch. These SATA SSDs provide up to 4TB capacity and 550MB/s sequential read and 520MB/s sequential write speeds for media workstation applications. However, there were also showings of the product used in all-flash arrays as shown below.

ProGrade was showing its line of professional memory cards for high-end digital cameras. These included their SFExpress 1.0 memory card with 1TB capacity and 1.4GB/s read data transfer speed as well as burst write speed greater than 1GB/s. This new Compact Flash standard is a successor to both the C FAST and XQD formats. The product uses two lanes of PCIe and includes NVMe support. The product is interoperable with the XQD form factor. They also announced their V90 premium line of SDXC UHS-II memory cards with sustained read speeds of up to 250MB/s and sustained write speeds up to 200MB/s.

2018 Creative Storage Conference
For those who love storage, the 12th Annual Creative Storage Conference (CS 2018) will be held on June 7 at the Double Tree Hotel West Los Angeles in Culver City. This event brings together digital storage providers, equipment and software manufacturers and professional media and entertainment end users to explore the conference theme: “Enabling Immersive Content: Storage Takes Off.”

Also, my company Coughlin Associate is conducting a survey of digital storage requirements and practices for media and entertainment professionals with results presented at the 2018 Creative Storage Conference. M&E professionals can participate in the survey through this link. Those who complete the survey, with their contact information, will receive a free full pass to the conference.

Our main image: Seagate products in an editing session, including products in a Pelican case for field work. 


Tom Coughlin is president of Coughlin Associates, a digital storage analyst and  technology consultant. He has over 35 years in the data storage industry. He is also the founder of the Annual Storage Visions Conference and the Creative Storage Conference.

 

IBC 2016: VR and 8K will drive M&E storage demand

By Tom Coughlin

While attending the 2016 IBC show, I noticed some interesting trends, cool demos and new offerings. For example, while flying drones were missing, VR goggles were everywhere; IBM was showing 8K video editing using flash memory and magnetic tape; the IBC itself featured a fully IP-based video studio showing the path to future media production using lower-cost commodity hardware with software management; and, it became clear that digital technology is driving new entertainment experiences and will dictate the next generation of content distribution, including the growing trend to OTT channels.

In general, IBC 2016 featured the move to higher resolution and more immersive content. On display throughout the show was 360-degree video for virtual reality, as well as 4K and 8K workflows. Virtual reality and 8K are driving new levels of performance and storage demand, and these are just some of the ways that media and entertainment pros are future-zone-2increasing the size of video files. Nokia’s Ozo was just one of several multi-camera content capture devices on display for 360-degree video.

Besides multi-camera capture technology and VR editing, the Future Tech Zone at IBC included even larger 360-degree video display spheres than at the 2015 event. These were from Puffer Fish (pictured right). The smaller-sized spherical display was touch-sensitive so you could move your hand across the surface and cause the display to move (sadly, I didn’t get to try the big sphere).

IBM had a demonstration of a 4K/8K video editing workflow using the IBM FlashSystem and IBM Enterprise tape storage technology, which was a collaboration between the IBM Tokyo Laboratory and IBM’s Storage Systems division. This work was done to support the move to 4K/8K broadcasts in Japan by 2018, with a broadcast satellite and delivery of 8K video streams of the 2020 Tokyo Olympic Games. The combination of flash memory storage for working content and tape for inactive content is referred to as FLAPE (flash and tAPE).

The graphic below shows a schematic of the 8K video workflow demonstration.

The argument for FLAPE appears to be that flash performance is needed for editing 8K content and the magnetic tape provides low-cost storage the 8K content, which may require greater than 18TB for an hour of raw content (depending upon the sampling and frame rate). Note that magnetic tape is often used for archiving of video content, so this is a rather unusual application. The IBM demonstration, plus discussions with media and entertainment professionals at IBC indicate that with the declining costs of flash memory and the performance demands of 8K, 8K workflows may finally drive increased demand for flash memory for post production.

Avid was promoting their Nexis file system, the successor to ISIS. The company uses SSDs for metadata, but generally flash isn’t used for actual editing yet. They agreed that as flash costs drop, flash could find a role for higher resolution and richer media. Avid has embraced open source for their code and provides free APIs for their storage. The company sees a hybrid of on-site and cloud storage for many media and entertainment applications.

EditShare announced a significant update to its XStream EFS Shared Storage Platform (our main image). The update provides non-disruptive scaling to over 5PB with millions of assets in a single namespace. The system provides a distributed file system with multiple levels of hardware redundancy and reduced downtime. An EFS cluster can be configured with a mix of capacity and performance with SSDs for high data rate content and SATA HDD for cost-efficient higher-performance storage — 8TB HDDs have been qualified for the system. The latest release expands optimization support for file-per-frame media.

The IBC IP Interoperability Zone was showing a complete IP-based studio (pictured right) was done with the cooperation of AIMS and the IABM. The zone brings to life the work of the JT-NM (the Joint Task Force on Networked Media, a combined initiative of AMWA, EBU, SMPTE and VSF) and the AES on a common roadmap for IP interoperability. Central to the IBC Feature Area was a live production studio, based on the technologies of the JT-NM roadmap that Belgian broadcaster VRT has been using daily on-air all this summer as part of the LiveIP Project, which is a collaboration between VRT, the European Broadcasting Union (EBU) and LiveIP’s 12 technology partners.

Summing Up
IBC 2016 showed some clear trends to more immersive, richer content with the numerous displays of 360-degree and VR content and many demonstrations of 4K and even 8K workflows. Clearly, the trend is for higher-capacity, higher-performance workflows and storage systems that support this workflow. This is leading to a gradual move to use flash memory to support these workflows as the costs for flash go down. At the same time, the move to IP-based equipment will lead to lower-cost commodity hardware with software control.

Storage analyst Tom Coughlin is president of Coughlin Associates. He has over 30 years in the data storage industry and is the author of Digital Storage in Consumer Electronics: The Essential Guide. He also  publishes the Digital Storage Technology Newsletter, the Digital Storage in Media and Entertainment Report.