Tag Archives: Tom Coughlin

SMPTE ST 2110 enables IP workflows

By Tom Coughlin

At IBC2017 and this year’s SMPTE Conference there were significant demonstrations of IP-based workflows with interoperability demonstrations and conference sessions. Clearly proprietary media networking will be supplanted by IP-based workflows. This will enable new equipment economies and open up new opportunities for using and repurposing media. IP workflows will also impact the way we store and use digital content and thus the storage systems where they live.

SMPTE has just ratified ST 2110 standards for IP transport in media workflows. The standard puts video, audio and ancillary data into separate routable streams as shown in the figure below. PCM Audio streams are covered by SMPTE ST 2110-30, uncompressed video streams are covered by ST 2110-20 and ancillary data is covered by ST 2110-40. Some other parts of the standards cover traffic shaping of uncompressed video (ST 2110-21), AES3 transparent transport (ST 2110-31) and ST 2110-50 allows integration with older specification ST 2022-6 that covers legacy SDI over IP.

The separate streams have timestamps that allow proper alignment of the different streams when they are combined together — this timestamp is provided by ST 2059. Each stream contains metadata that tells the receiver how to interpret what is inside of the stream. The uncompressed video stream supports up to 32k X 32k images, HDR and all common color systems and formats.

The important thing about these IP standards is that they allow using conventional Ethernet cabling rather than special proprietary cables. This saves a lot of money on hardware. In addition, having an IP-based workflow allows easy ingest into a core IP network and distribution of content using IP-based broadcast, telco, cable and broadband technologies as well as satellite channels. As most consumers have IP content access, these IP networks connect directly to consumer equipment. The image below from an Avid presentation by Shailendra Mathur at SMPTE 2017 illustrates the workflow below.

At IBC and the SMPTE 2017 Conference there were interoperability demonstrations. Although the IBC interop demo had many more participants the SMPTE demo was pretty extensive. The photo below shows the SMPTE interoperability demonstration setup.

As many modern network storage systems, whether file or object based, use Ethernet connectivity, having the rest of the workflow using an IP network makes movement of data through the workflow to and from digital storage easier. Since access to cloud-based assets is also though IP-based networks and these can feed CDNs and other distribution networks, on-premise and cloud storage interact through IP networks and can be used to support working storage, archives as well as content distribution libraries.

IP workflows combined with IP-based digital storage provide end-to-end processing and storage of data. This provides hardware economics and access to a lot of software built to manage and monitor IP flows to help optimize a media production and distribution system. By avoiding the overhead of converting from one type of network to another the overall system complexity and efficiency will be improved, resulting in faster projects and easier repair of problems when they arise.


Tom Coughlin is president of Coughlin Associates. He is the founder and organizer of the annual Storage Visions Conference as well as the Creative Storage Conference. He has also been the general chairman of the annual Flash Memory Summit.

Origins: The Creative Storage Conference

By Tom Coughlin

I was recently asked how the Creative Storage Conference came to be. So here I am to give you some background.

In 2006, the Storage Visions Conference that my colleagues and I had been organizing just before the CES show in January was in its fifth year. I had been doing more work on digital storage for professional media and entertainment, including a report on this important topic. In order to increase my connections and interaction with both media and entertainment professionals, and the digital storage and service companies that support them, it seems that a conference focusing on digital storage for media and entertainment would be in order.

That same year, my partner Ron Dennison and I participated in the MediaTech Conference in the LA area, working with Bryan Eckus, the director of the group at the time. In 2007, we held the first Creative Storage Conference in conjunction with the MediaTech Conference in Long Beach, California. It featured a dynamite line-up of storage companies and end users.

The conference has grown in size over the years, and we have had a stream of great companies showing their stuff, media and entertainment professional attendees and speakers, informative sessions and insightful keynote talks on numerous topics related to M&E digital storage.

The 2017 Creative Storage Conference
This year, the Creative Storage Conference is taking place on May 24 in Culver City. Attendees can learn more about the use of Flash memory in M&E as well as the growth in VR content in professional video, and how this will drive new digital storage demand and technologies to support the high data rates needed for captured content and cloud-based VR services. This is the 11th year of the conference and we look forward to having you join us.

We are planning for six sessions and four keynotes during the day and a possible reception in the evening on May 24.

Here is a list of the planned sessions:
• Impact of 4K/HDR/VR on Storage Requirements From Capture to Studio
• Collaboration in the Clouds: Storing and Delivering Content Where it is Needed
• Content on the Move: Delivering Storage Content When and Where it is Needed
• Preserving Digital Content — the Challenges, Needs and Options
• Accelerating Workflows: Solid State Storage in Media and Entertainment
• Independent Panel — Protecting the Life of Content

Don’t miss this opportunity to meet giants in the field of VR content capture and post production and meet the storage and service companies to help you make sure your next professional projects are a big success.

• Hear how major media equipment suppliers and entertainment industry customers use digital storage technology in all aspects of content creation and distribution.
• Find out the role that digital storage plays in new content distribution and marketing opportunities for a rapidly evolving market.
• See presentations on digital storage in digital acquisition and capture, nonlinear editing and special effects.
• Find out how to convert and preserve content digitally and protect it in long-term dependable archives.
• Learn about new ways to create and use content metadata, making it easier to find and use.
• Discover how to combine and leverage hard disk drives, flash memory, magnetic tape and optical storage technology with new opportunities in the digital media market.
• Be at the juncture of digital storage and the next generation of storage for the professional media market.

Online registration is open until May 23, 2017. As a media and entertainment professional you can register now with a $100 discount using this link:

—–
Thomas Coughlin, president of Coughlin Associates is a storage analyst and consultant with over 30 years in the data storage industry. He is active with SNIA, SMPTE, IEEE, and other professional organizations.

HPA Tech Retreat takes on realities of virtual reality

By Tom Coughlin

The HPA Tech Retreat, run by the Hollywood Professional Association in association with SMPTE, began with an insightful one-day VR seminar— Integrating Virtual Reality/Augmented Reality into Entertainment Applications. Lucas Wilson from SuperSphere kicked off the sessions and helped with much of the organization of the seminar.

The seminar addressed virtual reality (VR), augmented reality (AR) and mixed reality (MR, a subset of AR where the real world and the digital world interact, like Pokeman Go). As in traditional planar video, 360-degree video still requires a director to tell a story and direct the eye to see what is meant to be seen. Successful VR requires understanding how people look at things, how they perceive reality, and using that understanding to help tell a story. Some things that may help with this are reinforcement of the viewer’s gaze with color and sound that may vary with the viewer — e.g. these may be different for the “good guy” and the “bad guy.”

VR workflows are quite different from traditional ones, with many elements changing with multiple-camera content. For instance, it is much more difficult to keep a camera crew out of the image, and providing proper illumination for all the cameras can be a challenge. The image below from Jaunt shows their 360-degree workflow, including the use of their cloud-based computational image service to stitch the images from the multiple cameras.
Snapchat is the biggest MR application, said Wilson. Snapchat’s Snapchat-stories could be the basis of future post tools.

Because stand-alone headsets (head-mounted displays, or HMDs) are expensive, most users of VR rely on smart phone-based displays. There are also some places that allow one or more people to experience VR, such as the IMAX center in Los Angeles. Activities such as VR viewing will be one of the big drivers for higher-resolution mobile device displays.

Tools that allow artists and directors to get fast feedback on their shots are still in development. But progress is being made, and today over 50 percent of VR is used for video viewing rather than games. Participants in a VR/AR market session, moderated by the Hollywood Reporter’s Carolyn Giardina and including Marcie Jastrow, David Moretti, Catherine Day and Phil Lelyveld, seemed to agree that the biggest immediate opportunity is probably with AR.

Koji Gardiner from Jaunt gave a great talk on their approach to VR. He discussed the various ways that 360-degree video can be captured and the processing required to create finished stitched video. For an array of cameras with some separation between the cameras (no common axis point for the imaging cameras), there will be area that needs to be stitched together between camera images using common reference points between the different camera images as well as blind spots near to the cameras where they are not capturing images.

If there is a single axis for all of the cameras then there are effectively no blind spots and no stitching possible as shown in the image below. Covering all the space to get a 360-degree video requires additional cameras located on that axis to cover all the space.

The Fraunhofer Institute, in Germany, has been showing a 360-degree video camera with an effective single axis for several cameras for several years, as shown below. They do this using mirrors to reflect images to the individual cameras.

As the number of cameras is increased, the mathematical work to stitch the 360-degree images together is reduced.

Stitching
There are two approaches commonly used in VR stitching of multiple camera videos. The easiest to implement is a geometric approach that uses known geometries and distances to objects. It requires limited computational resources but results in unavoidable ghosting artifacts at seams from the separate images.

The Optical Flow approach synthesizes every pixel by computing correspondences between neighboring cameras. This approach eliminates the ghosting artifacts at the seams but has its own more subtle artifacts and requires significantly more processing capability. The Optical Flow approach requires computational capabilities far beyond those normally available to content creators. This has led to a growing market to upload multi-camera video streams to cloud services that process the stitching to create finished 360-degree videos.

Files from the Jaunt One camera system are first downloaded and organized on a laptop computer and then uploaded to Jaunt’s cloud server to be processed and create the stitching to make a 360 video. Omni-directionally captured audio can also be uploaded and mixed ambisonically, resulting in advanced directionality in the audio tied to the VR video experience.

Google and Facebook also have cloud-based resources for computational photography used for this sort of image stitching.

The Jaunt One 360-degree camera has a 1-inch 20MP rolling shutter sensor with frame rates up to 60fps with 3200 ISO max, 29dB SNR at ISO800. It has a 10 stops per camera module, with 130-degree diagonal FOV, 4/2.9 optics and with up to 16K resolution (8K per eye). Jaunt One at 60fps provides 200GB/minute uncompressed. This can fill a 1TB SSD in five minutes. They are forced to use compression to be able to use currently affordable storage devices. This compression creates 11GB per minute, which can fill a 1TB SSD in 90 minutes.

The actual stitched image, laid out flat, looks like a distorted projection. But when viewed in a stereoscopic viewer it appears to look like a natural image of the world around the viewer, giving an immersive experience. At one point in time the viewer does not see all of the image but only the image in a restricted space that they are looking directly at as shown in the red box in the figure below.

The full 360-degree image can be pretty high resolution, but unless the resolution is high enough, the resolution inside the scene being viewed at any point in time will be much less that the resolution of the overall scene, unless special steps are taken.

The image below shows that for a 4k 360-degree video the resolution in the field of view (FOV) may be only 1K, much less resolution and quite perceptible to the human eye.

In order to provide a better viewing experience in the FOV, either the resolution of the entire view must be better (e.g. the Jaunt One high-resolution version has 8K per eye and thus 16K total displayed resolution) or there must be a way to increase the resolution in the most significant FOV in a video, so at least in that FOV, the resolution leads to a greater feeling of reality.

Virtual reality, augmented reality and mixed reality create new ways of interacting with the world around us and will drive consumer technologies and the need for 360-degree video. New tools and stitching software, much of this cloud-based, will enable these workflows for folks who want to participate in this revolution in content. The role of a director is as important as ever as new methods are needed to tell stories and guide the viewer to engage in this story.

2017 Creative Storage Conference
You can learn more about the growth in VR content in professional video and how this will drive new digital storage demand and technologies to support the high data rates needed for captured content and cloud-based VR services at the 2017 Creative Storage Conference — taking place May 24, 2017 in Culver City.


Thomas M. Coughlin of Coughlin Associates is a storage analyst and consultant. He has over 30 years in the data storage industry and is the author of Digital Storage in Consumer Electronics: The Essential Guide.

IBC 2016: VR and 8K will drive M&E storage demand

By Tom Coughlin

While attending the 2016 IBC show, I noticed some interesting trends, cool demos and new offerings. For example, while flying drones were missing, VR goggles were everywhere; IBM was showing 8K video editing using flash memory and magnetic tape; the IBC itself featured a fully IP-based video studio showing the path to future media production using lower-cost commodity hardware with software management; and, it became clear that digital technology is driving new entertainment experiences and will dictate the next generation of content distribution, including the growing trend to OTT channels.

In general, IBC 2016 featured the move to higher resolution and more immersive content. On display throughout the show was 360-degree video for virtual reality, as well as 4K and 8K workflows. Virtual reality and 8K are driving new levels of performance and storage demand, and these are just some of the ways that media and entertainment pros are future-zone-2increasing the size of video files. Nokia’s Ozo was just one of several multi-camera content capture devices on display for 360-degree video.

Besides multi-camera capture technology and VR editing, the Future Tech Zone at IBC included even larger 360-degree video display spheres than at the 2015 event. These were from Puffer Fish (pictured right). The smaller-sized spherical display was touch-sensitive so you could move your hand across the surface and cause the display to move (sadly, I didn’t get to try the big sphere).

IBM had a demonstration of a 4K/8K video editing workflow using the IBM FlashSystem and IBM Enterprise tape storage technology, which was a collaboration between the IBM Tokyo Laboratory and IBM’s Storage Systems division. This work was done to support the move to 4K/8K broadcasts in Japan by 2018, with a broadcast satellite and delivery of 8K video streams of the 2020 Tokyo Olympic Games. The combination of flash memory storage for working content and tape for inactive content is referred to as FLAPE (flash and tAPE).

The graphic below shows a schematic of the 8K video workflow demonstration.

The argument for FLAPE appears to be that flash performance is needed for editing 8K content and the magnetic tape provides low-cost storage the 8K content, which may require greater than 18TB for an hour of raw content (depending upon the sampling and frame rate). Note that magnetic tape is often used for archiving of video content, so this is a rather unusual application. The IBM demonstration, plus discussions with media and entertainment professionals at IBC indicate that with the declining costs of flash memory and the performance demands of 8K, 8K workflows may finally drive increased demand for flash memory for post production.

Avid was promoting their Nexis file system, the successor to ISIS. The company uses SSDs for metadata, but generally flash isn’t used for actual editing yet. They agreed that as flash costs drop, flash could find a role for higher resolution and richer media. Avid has embraced open source for their code and provides free APIs for their storage. The company sees a hybrid of on-site and cloud storage for many media and entertainment applications.

EditShare announced a significant update to its XStream EFS Shared Storage Platform (our main image). The update provides non-disruptive scaling to over 5PB with millions of assets in a single namespace. The system provides a distributed file system with multiple levels of hardware redundancy and reduced downtime. An EFS cluster can be configured with a mix of capacity and performance with SSDs for high data rate content and SATA HDD for cost-efficient higher-performance storage — 8TB HDDs have been qualified for the system. The latest release expands optimization support for file-per-frame media.

The IBC IP Interoperability Zone was showing a complete IP-based studio (pictured right) was done with the cooperation of AIMS and the IABM. The zone brings to life the work of the JT-NM (the Joint Task Force on Networked Media, a combined initiative of AMWA, EBU, SMPTE and VSF) and the AES on a common roadmap for IP interoperability. Central to the IBC Feature Area was a live production studio, based on the technologies of the JT-NM roadmap that Belgian broadcaster VRT has been using daily on-air all this summer as part of the LiveIP Project, which is a collaboration between VRT, the European Broadcasting Union (EBU) and LiveIP’s 12 technology partners.

Summing Up
IBC 2016 showed some clear trends to more immersive, richer content with the numerous displays of 360-degree and VR content and many demonstrations of 4K and even 8K workflows. Clearly, the trend is for higher-capacity, higher-performance workflows and storage systems that support this workflow. This is leading to a gradual move to use flash memory to support these workflows as the costs for flash go down. At the same time, the move to IP-based equipment will lead to lower-cost commodity hardware with software control.

Storage analyst Tom Coughlin is president of Coughlin Associates. He has over 30 years in the data storage industry and is the author of Digital Storage in Consumer Electronics: The Essential Guide. He also  publishes the Digital Storage Technology Newsletter, the Digital Storage in Media and Entertainment Report.

The State of Storage

Significant trends are afoot in media and entertainment storage.

By Tom Coughlin

Digital storage plays a significant role in the media and entertainment industry, and our specific demands are often very different from typical IT storage. We are dealing with performance requirements of realtime video in capture, editing and post, as well as distribution. On the other hand, the ever-growing archive of long-tail digital content and digitized historical analog content is swelling the demand for archives (both cold and warm) using tape, optical discs and hard drive arrays.

My company, Coughlin Associates, has conducted surveys of digital storage use by media and entertainment professionals since 2009. These results are used in our annual Digital Storage in Media and Entertainment Report. This article presents results from the 2016 survey and some material from the 222-page report to discuss the status of digital storage for professional media and entertainment.

Content Creation and Capture
Pro video cameras are undergoing rapid evolution, driven by higher-resolution content as well as multi-camera content capture, including stereoscopic and virtual reality. In addition, the physical storage media for professional cameras is undergoing rapid evolution as film and magnetic digital tape is impacted by the rapid file access convenience of hard disk drives, optical discs, and the ruggedness of flash-based solid-state storage.

The table below compares the results from the 2009, 2010, 2012, 2013, 2014 and 2015 surveys with those from 2016. Flash memory is the clear leader in pro video camera media, increasing from 19% in 2009 to 66% in 2015 and then down to 54% in 2016, while magnetic tape shows a consistent decline over the same period.

Optical disc use between 2009 and 2016 bounced around between 7% and 17%. Film shows a general decline from 15% usage in 2009 to 2% in 2016. The trend with declining film use follows the trend toward completely digital workflows.

Note that about 60% of survey participants said that they used external storage devices to capture content from their cameras in 2016 (perhaps this is why the HDD percentages are so high). In 2016, 83% said that over 80% of their content is created in a digital format.

In 2016, 93.2% of the survey respondents said they reuse their recording media (compared to 89.9% in 2015, 93.3% in 2014, 84.5% in 2013, 86% in 2012, 79% in 2010 and 75% in 2009). In 2016, 75% of respondents said they archive their camera recording media (compared to 73.6% in 2015, 74.2% in 2014, 81.4% in 2013, 85% in 2012 and 77% in 2010).

Archiving the original recording media may be a practice in decline — especially with expensive reusable media such as flash memory cards. Digital storage on tape, hard disk drives or flash storage allows the reuse of media.

Post Production
The size of content — and amount — has put strains on post network bandwidth and storage. This includes editing and other important operations. As much of this work may take place in smaller facilities, these companies may be doing much of their work on direct attached storage devices and they may share or archive this media in the cloud in order to avoid the infrastructure costs of running a data center.

The graph below shows that for the 2016 survey participants there was a general increase in the use of shared network storage (such as SAN or NAS), and a decrease in DAS storage as the number of people working in a post facility increases. The DAS storage in the larger facilities may be different than that used in smaller facilities.

DAS vs. shared storage by number of people in a post facility.

When participants were asked about their use of direct attached and network storage in digital editing and post, the survey showed the following summary statistics in 2016 (compared to earlier surveys):

– 74.5% had DAS
– 89.8% of these had more than 1 TB of DAS
– 10 to 50 TB was the most popular DAS size (27.5%)
– 17.4% of these had more than 50 TB of DAS storage
– 2.9% had more than 500 TB of DAS storage
– 68.1% had NAS or SAN
– 57.4% had 50 TB or more of network storage in 2016
– About 15% had more than 500 TB of NAS/SAN storage in 2016
– Many survey participants had considerable storage capacities in both DAS and NAS/SAN.

We asked whether survey participants used cloud-based storage for editing and post. In 2016 23.0% of responding participants said yes. The respondents, 20.9% of them, said that they had 1TB or more of their storage capacity in the cloud.

Content Distribution
Distribution of professional video content has many channels. It can use physical media for getting content to digital cinemas or to consumers, the distribution can be done electronically using broadcast, cable or satellite transmission, or through the Internet or mobile phone networks.

The table below gives responses for the percentage of physical media used by the survey respondents for content distribution in 2016, 2015, 2014, 2013, 2012 and 2010. Note that these are the average for the survey population giving their percentage for each physical media and do not and should not be expected to add to 100%. Digital tape, DVD discs, HDDs and Flash Memory are the most popular distribution formats.

Average percentage content on physical media for professional content distribution.

Following are survey observations for electronic content distribution, such as video on demand.

– The average number of hours on a central content delivery system was 2,174 hours in 2016.
– There was an average of 427 hours ingested monthly in 2016.
– In 2016, 38% of respondents had more than 5% of their content on edge servers.
– About 31% used flash memory on their edge servers in 2016.

Archiving and Preservation
Today, most new entertainment and media content is born digital, so it is natural that this content should be preserved in digital form. This requirement places new demands on format preservation for long-term digital archives as well as management and systematic format refreshes during the expected life of a digital archive.

In addition, the cost of analog content digitization and preservation in a digital format has gone down considerably, and many digitization projects are proceeding apace. The growth of digital content archiving will swell the amount of content available for repurposing and long-tail distribution. It will also swell the amount of storage and storage facilities required to store these long-term professional content archives.

Following are some observations from our 2016 survey on trends in digital archiving and content preservation.

– 41% had less than 2,000 hours of content in a long-term archive
– 56.9% archived all the content captured from their cameras
– 54.0% archived copies of content in all of their distribution formats
– 35.9% digitally archived all content captured from their dailies
– 31.3% digitally archived all content captured from rough cuts
– 36.5% digitally archived all content captured from their intermediaries
– 50.9% of the respondents said that their annual archive growth rate was less than 6% in 2016
– About 28.6% had less than 2,000 hours of unconverted analog content
– 16.7% of participants had over 5,000 hours of unconverted analog content
– About 52.5% of the survey respondents have an annual analog conversion rate of 2% or less
– The average rate of conversion is about 3.4% in 2016

Professional media and entertainment content was traditionally archived on film or analog videotapes. Today, the options available for archive media to store digital content depend upon the preferences and existing infrastructure of digital archive facilities. Figure 6 gives the percentage distribution of archive media used by the survey participants.

Percentage of digital long-term archives on various media

Some other observations from the archive and preservation section of the survey:

– About 42.6% never update their digital archives.
– About 76.2% used different storage for archiving and working storage.
– About 49.2% copied and replaced their digital long-term archives every 10 years or less.
– 38.1% said they would use a private or public cloud for archiving in 2016.

Conclusions
Larger content files, driven by higher resolution, higher frame rates, higher dynamic range and stereoscopic and virtual reality video are creating larger video files. This is driving the need for high-performance storage to work on this content and to provide fast delivery, which could drive more creative work to use solid-state storage.

At the same time, cost -effective storage and management of completed work is driving the increased use of hard disk drives, magnetic tape and even optical storage for low-cost storage.

The price of storing content in the cloud has gone down so much that there are magnetic tape-based cloud storage offerings that are less expensive than building one’s own storage data center, at least for small- and moderate-sized facilities.

This trend is expected to grow the use of cloud storage in media and entertainment, especially for archiving, as shown in the figure below.

Growth of cloud storage in media and entertainment.


Dr. Tom Coughlin, president of Coughlin & Associates, is a storage analyst and consultant with over 30 years in the data storage industry. He is the founder and organizer of the Annual Storage Visions Conference as well as the Creative Storage Conference.

Creative Storage Conference is now soliciting presentations

No one can deny how important storage is in the day-to-day lives of post pros. From big studios and facilities to indie editors working with external drives, storage is an indispensable part of the way people work today.  So if you want to talk storage and learn about the latest trends and product offerings, you might want to attend the Creative Storage Conference (CS 2016), celebrating its 10th anniversary this year.

CS 2016 is accepting submissions for presentations, speakers and panels now through May 1, 2016 at www.creativestorage.org. CS 2016 will bring together digital storage providers, equipment and software manufacturers, and professional media and entertainment end users to explore this year’s conference theme, “The Art of Storage.”

The 2016 event will be held Thursday, June 23 at the DoubleTree Hotel West Los Angeles in Culver City. “We have a great agenda planned with six sessions throughout the day and four keynote talks during the conference,” said Tom Coughlin, who is both the chairman and organizer of the event.

The Creative Storage Conference is put on by the Entertainment Storage Alliance and Coughlin Associates.

IBC 2015: Adventures in archiving

By Tom Coughlin

Once you have your content and have completed that award-winning new project, Oscar-nominated film or brilliant and effective commercial, where does your data go? Today, video content can be reused and repurposed a number of times, producing a continuing revenue stream by providing viewing for many generations of people. That makes video archives valuable and also requires changes from in-active to more active archives. This article looks at some of the archiving products on display at the 2015 IBC.

The figure to the right shows our estimate of revenue spent on various media and entertainment storage markets in 2014 (from the Digital Storage in Media and Entertainment Report from Coughlin Associates). Note that although almost 96 percent of all M&E storage capacity is used for archiving about 45 percent of the spending is for archiving.

Quantum showcased its StorNext 5 shared storage architecture, which includes high-performance online storage, extended online storage and tape- and cloud-based archives. The company also highlighted the StorNext Connect, a management and monitoring console that provides an at-a-glance dashboard of the entire StorNext environment. At IBC, Quantum introduced their Q-Cloud Archive that extends StorNext workflow capabilities to the cloud, allowing end-to-end StorNext environments to leverage cloud storage fully with no additional hardware, separate applications or programming while maintaining full compatibility with existing software applications.

The Quantum Storage Manager migrates data from online storage to its object-based Lattus, allowing secure, long-term storage with greater durability than RAID and extremely high scalability. Content can be migrated from Lattus to tape archives or Q-Cloud archives automatically. In addition Quantum’s Artico intelligent NAS archive appliance was on display, offering low cost scale-out storage for active media archives that can scale to PBs of content across HDDs, extended online storage, tape and cloud storage.

Also during IBC, the LTO Program Technology Provider Companies — HP, IBM and Quantum —announced the LTO-7 tape format that will be available in late 2015. The native capacity of this drive is 6TB, while 2.5:1 compression provides 15TB of storage with up to 750MB/s data rates. This product will provide over twice the capacity of the LTO-6 drive generation. The LTO roadmap goes out to a generation 10 product with up to 120TB of compressed content and about 48TB native capacity.

LTO proponents said that tape has some advantages over hard disk drives for archiving, despite the difference in latency to access content. In particular, they said tape has and error rate two orders of magnitude lower than HDDs, providing more accurate recording and reading of content. Among the interesting LTO developments at IBC were the M-Logic Thunderbolt interface tape drives.

Tape can also be combined with capacity SATA HDDs to provide storage systems with performance approaching hard disk drive arrays and costs approaching magnetic tape libraries. Crossroads has teamed up with Fujifilm to provide NAS systems combining HDDs and tape and including cloud storage combining tape and HDDs. In fact archiving is becoming one of the biggest growing applications in the media and entertainment industry, according to the 2015 Digital Storage in Media and Entertainment Report from Coughlin Associates.

Oracle was also showing its tape storage systems with 8TB native storage capacity in a half-inch tape form factor. Oracle now includes Front Porch Digital with its cloud archiving platform as well as digital ingest solutions for older analog and digital format media.

Some companies also use flash memory as a content cache in order to match the high speeds of data transfers to and from a tape library system. Companies such as Xendata provide LTO tape and optical disc libraries for media and entertainment customers. Spectra Logic has made a big push into HDD-based archiving, using shingled magnetic recording (SMR) 3.5-inch HDDs in their DPE storage system to provide unstructured storage costs as low as 9 cents/GB. This system can provide up to 7.4PB of raw capacity in a single rack with 1GB/s data rates. This sort of system is best for data that is seldom or never overwritten because of the use of SMR HDDS.

Sony was showing its 300GB Blu-ray optical WORM discs, although it was not clear if the product is being shipped in storage cartridges in 2015. Archiving is a significant driver of M&E storage demand. This is because all video eventually ends up in an archive. Because of more frequent access of archived content, the performance requirements of many archives are more demanding than in the past. This has led to the use of HDD-based archives and archives combining HDDs and magnetic tape. Even flash memory can play a role as a write and read cache in a tape based system.

Dr. Tom Coughlin, president of Coughlin Associates, has over 35 years in the data storage industry. Coughlin is also the founder and organizer of the annual Storage Visions Conference, a partner to the International Consumer Electronics Show, as well as the Creative Storage Conference

Presentation submission deadline for Storage Visions 2016 nears

The Storage Visions Conference presentation submission deadline is September 14, 2015, so the event’s creators are asking you to submit ideas here.  The conference theme is Storage for the Next 5,000 Years. You can check out the conference’s preliminary agenda —it will be adjusted as speakers are placed in the agenda after the deadline — here or below…

Session A1: Epic Proportions: Storage for High Resolution Content Capture and Production
Session B1: Drive Trust: Protecting Our Stuff
Session C1: Analyst Perspective: How will we use Ubiquitous Storage?
Session D1: One for All: The Convergence of Memory and Storage
Session E1: Next Millennium Storage: Developments Propel Storage System Options
Session F1: Storage End Users on The Long Term
Session A2: Storage Visions 2016 Rising Stars Panel
Session B2: Pumping Big Data: Putting Data where it needs to be
Session C2: Saving Data Forever: Long Term Content Protection and Archiving
Session D2: Consumer Cloud Services: What will we do in the Cloud?
Session E2: Keeping it Now: Emerging Storage Technologies
Session F2: We are the Champions: Storage for Consumers

If you are looking to submit something for the Storage Visions Visionary awards you have a bit more time — deadline for that is November 14, 2015. Submissions can be made at http://storagevisions.com/2016Awards.htm.

Storage Visions 2016 will focus on how to drive a new wave of business tied to higher resolution content, big data, security requirements, consumer electronics, digital distribution, new storage architectures and applications as well as new digital storage technologies.

NAB: Media archives drive storage innovation

By Tom Coughlin

Higher resolution, frame rate, pixel depth and the number of cameras for pro video projects is increasing the storage capacity requirements for professional media. The growing size of new projects, combined with the vast collection of media and entertainment content already in archives will drive the size of media archives.

In addition, the desire for ready access to this content to support reuse and other purposes increases the need for faster more active archives. The 2014 Digital Storage for Media and Entertainment Report from my own Coughlin Associates estimates that over 96% of all digital storage in the media and entertainment industry is used in archiving and preservation.

The 2015 NAB show was a showcase for many digital media archive solutions. Magnetic tape is a big player in the traditional media archive market. The most popular tape format in media and entertainment is the LTO tape. The LTO Consortium says that tape storage costs are now less than $0.008 per GB ($8/TB). The LTFS file system used in LTO and other tapes is now supported by SNIA standards. Magnetic tape is now being used in object storage environments and Swift now supports LTFS. The group says that they have seen roughly a 6% growth in magnetic tape in the media and entertainment market.

Fujifilm had a booth with Crossroads Systems where they were showing the Dternity tape-based cloud storage system with the Crossroads Strongbox product. Oracle and IBM were also showing their tape based storage systems that they are targeting for media and entertainment (IBM currently offers up to 10TB of capacity on a half-inch tape cartridge). Several companies were showing the USB interface tape drive made by IBM in their NAB booths. Several companies were showing the USB interface tape drive made by IBM in their NAB booths. mLogic was showing a Thunderbolt LTO tape drive.

ProMax acquired Cache-A in July of 2014, At the 2015 NAB show the company released a significant update to its archiving appliance lineup. The Pro-Cache and Power-Cache products have been redesigned from the ground up to improve the speed and reliability of archiving to LTO tape. The Pro-Cache’s expanded network connectivity includes 4 x 1GbE network controllers accompanied by a 10GbE network performance boost that makes it up to 2x faster than previous systems. The Power-Cache (our main photo) onboard RAID is now up to 16TB with room for an optional onboard LTO-6 or LTO-5 drive.

ProMAX Pro-Cache and Power-Cache copy

Spectra Logic announced updates and enhancements to their BlackPearl Deep Storage Gateway. The newly enhanced Spectra Logic BlackPearl Deep Storage Gateway sits in front of deep storage tape libraries and allows users to move data anywhere within their network using simple HTTP commands. An interesting feature of the BlackPearl hardware is that they use flash memory for caching write and read information to a magnetic tape library. Because of the very fast data rates for mounted tapes flash memory avoids dropping of data or needing to rerun tapes to complete data transfers. Spectra Logic also announced that they were providing an archive tier to private clouds running on the NetApp StorageGrid using the BlackPearl Deep Storage Gateway.

Sony has been driving cartridge archive system using 12 Blu-ray write-once optical media in their Optical Disc Archive (ODA). These cartridges are scheduled for a significant increase in capacity in 2015 with the planned introduction of 300GB 5.25-inch optical write-one archive discs. In addition to the Sony booth companies such as XenData and Qualstar were showing both disc and tape archive libraries in their booth. MassTech also announced support for Sony’s ODA by their asset management system. There is a significant segment of the media and entertainment market that prefers optical discs for cold archiving rather magnetic tape.

backup progress with proxies copy PRP Back Up Screen Shot

Imagine Products and Sony have created a new ODA archiving solution for Mac users. The company’s PreRoll Post LTFS archiving application (pictured above) creates thumbs, proxies and rich metadata while backing up files and folders to multiple locations at once. It also uses checksum technology to ensure that the files and folders are copied accurately. Users simply drag and drop files and folders into the queue for copying to the disc.

Software to support backup also has an important role in M&E. SGL was showing a full archive demonstration with Grass Valley. The company was also launching an Asset Migration Service (FlashNet Migration Service) to move assets between one tape platform or generation to another. They were also showing partial file restore via Avid Web Services.

For more active archives there were several HDD systems geared to active archiving on display at the NAB. HGST (a division of Western Digital) was showing their HDD-based Active Archive System. This was HGST’s first foray into large storage system design and it was leveraging its internal drive costs to offer a cost effective object based storage system cheaper than many white-box solutions. HGST’s Active Archive System is an object storage system that delivers 4.7 petabytes (PB) of raw data storage in a single rack. The company says this product will provide long term retention of media data with fast access when needed.

Dr. Tom Coughlin, president of Coughlin Associates, is a storage analyst and consultant with over 30 years in the data storage industry. He has six patents to his credit and is the author of Digital Storage in Consumer Electronics: The Essential Guide. He also helps put together the Storage Visions Conference. You can we reach him at tom@tomcoughlin.com.

.

Talking future workflows and future archives

By Tom Coughlin

This year’s HPA Tech Retreat, which took place in February in Indian Wells, California, had some interesting presentations and displays, pointing the way to the future of media and entertainment. But before I dig into some general observations and an update on the future workflows and archive solutions that were on display, I will share this: You likely have already heard that the HPA is now part of SMPTE, but the more recent bit of news is the organization is changing its name from the Hollywood Post Alliance to the Hollywood Professional Alliance. Ok, now let’s get to the tech talk…

In the CES Review at the HPA Retreat, Peter Putnum pointed out that there weren’t as many TVs on display as Continue reading