Tag Archives: LTO tape

The benefits of LTO

By Mike McCarthy

LTO stands for Linear Tape Open, and was initially developed nearly 20 years ago as an “open” format technology that allows manufacturing by any vendor that wishes to license the technology. It records any digital files onto half-inch magnetic tapes, stored in square single reel cartridges. The capacity started at 100GB and has increased by a factor of two nearly every generation; the most recent LTO-8 cartridges store 12TB of uncompressed data.

If you want to find out more about LTO, you should check out the LTO Consortium, which is made up of Hewlett Packard Enterprises, IBM and Quantum, although there are other companies that make LTO drives and tape cartridges. You might be familiar with their LTO Ultrium logo.

‘Tapeless’ Workflows
While initially targeting server markets, with the introduction of “tapeless workflows” in the media and entertainment industry, there became a need for long-term media storage. Since the first P2 cards and SxS sticks were too expensive for single write operations, they were designed to be reused repeatedly once their contents had been offloaded to hard drives. But hard drives are not ideal for long-term data storage, and insurance and bonding companies wanted their clients to have alternate data archiving solutions.

So, by the time the Red One and Canon 5D were flooding post facilities with CF cards, LTO had become the default archive solution for most high-budget productions. But this approach was not without limitations and pitfalls. The LTO archiving solutions being marketed at the time were designed around the Linux-based Tar system of storing files, while most media work is done on Windows and Mac OS X. Various approaches were taken by different storage vendors to provide LTO capabilities to M&E customers. Some were network appliances running Linux under the hood, while others wrote drivers and software to access the media from OS X or, in one case, Windows. Then there was the issue that Tar isn’t a self-describing file system, so you needed a separate application to keep track of what was on each tape in your library. All of these aspects cost lots of money, so the initial investment was steep, even though the margin cost of tape cartridges was the cheapest way to store data per GB.

Linear Tape File System (LTFS) was first introduced with LTO-5 and was intended to make LTO tapes easier to use and interchange between systems. A separate partition on the tape stores the index of data in XML and other associated metadata. It was intended to be platform independent, although it took a while for reliable drivers and software to be developed for use in Windows and OS X.

At this point, LTFS-formatted tapes in LTO tape drives operate very similarly to old 3.5-inch floppy drives. You insert a cartridge, it makes some funny noises, and then after a minute it asks you to format a new tape, or it displays the current contents of the tape as a new drive letter. If you drag files into that drive, it will start copying the data to the tape, and you can hear it grinding away. The biggest difference is when you hit eject it will take the computer a minute or two to rewind the tape, write the updated index to the first partition and then eject the cartridge for you. Otherwise it is a seamless drag and drop, just like any other removable data storage device.

LTO Drives
All you need in order to use LTO in your media workflow — for archive or data transfer — is an LTO drive. I bought one last year on Amazon for $1,600, which was a bit of a risk considering that I didn’t know if I was going to be able to get it to work on my Windows 7 desktop. As far as I know, all tape drives are SAS devices, although you can buy ones that have adapted the SAS interface to Thunderbolt or Fibre Channel.

Most professional workstations have integrated SAS controllers, so internal LTO drives fit into a 5.25-inch bay and can just connect to those, or any SAS card. External LTO drives usually use Small Form Factor cables (SFF-8088) to connect to the host device. Internal SAS ports can be easily adapted to SFF-8088 ports, or a dedicated eSAS PCIe card can be installed in the system.

Capacity & Compression
How much data do LTO tapes hold? This depends on the generation… and the compression options. The higher capacity advertised on any LTO product assumes a significant level of data compression, which may be achievable with uncompressed media files (DPX, TIFF, ARRI, etc.) The lower value advertised is the uncompressed data capacity, which is the more accurate estimate of how much data it will store. This level of compression is achieved using two different approaches, eliminating redundant data segments and eliminating the space between files. LTO was originally designed for backing up lots of tiny files on data servers, like credit card transactions or text data, and those compression approaches don’t always apply well to large continuous blocks of unique data found in encoded video.

Using data compression on media files which are already stored in a compressed codec doesn’t save much space (there is little redundancy in the data, and few gaps between individual files).

Uncompressed frame sequences, on the other hand, can definitely benefit from LTO’s hardware data compression. Regardless of compression, I wouldn’t count on using the full capacity of each cartridge. Due to the way the drives are formatted, and the way storage vendors measure data, I have only been able to copy 2.2TB of data from Windows onto my 2.5TB LTO-6 cartridges. So keep that in mind when estimating real-world capacity, like with any other data storage medium.

Choosing the ‘Right’ Version to Use
So which generation of LTO is the best option? That depends on how much data you are trying to store. Since most media files that need to be archived these days are compressed, either as camera source footage or final deliverables, I will be calculating based on the uncompressed capacities. VFX houses using DPX frames, or vendors using DCDMs might benefit from calculating based on the compressed capacities.

Prices are always changing, especially for the drives, but these are the numbers as of summer 2018. On the lowest end, we have LTO-5 drives available online for $600-$800, which will probably store 1-1.2TB of data on a $15 tape. So if you have less than 10TB of data to backup at a time, that might be a cost-effective option. Any version lower than LTO-5 doesn’t support the partitioning required for LTFS, and is too small to be useful in modern workflows anyway.

As I mentioned earlier, I spent $1,600 on an LTO-6 drive last year, and while that price is still about the same, LTO-7 and LTO-8 drives have come down in cost since then. My LTO-6 drive stores about 2.2TB of data per $23 tape. That allowed me to backup 40TB of Red footage onto 20 tapes in 90 hours, or an entire week. Now I am looking at using the same drive to ingest 250TB of footage from a production in China, but that would take well over a month, so LTO-6 is not the right solution for that project. But the finished deliverables will probably be a similar 10TB set of DPX and TIFF files, so LTO-6 will still be relevant for that application.

I see prices as low as $2,200 for LTO-7 drives, so they aren’t much more expensive than LTO-6 drives at this point, but the 6TB tapes are. LTO-7 switched to a different tape material, which increased the price of the media. At $63 they are just over $10 per TB, but that is higher than the two previous generations.

LTO-8 drives are available for as low as $2,600, and store up to 12TB on a single $160 tape. LTO-8 drives can also write up to 9TB onto a properly formatted LTO-7 tape in a system called “LTO-7 Type M” This is probably the cheapest cost per TB approach at the moment, since 9TB on a $63 tape is $7/TB.

Compatibility Between Generations
One other consideration is backwards compatibility. What will it take to read your tapes back in the future? The standard for LTO has been that drives can write the previous generation tapes and read tapes from two generations back.

So if you invested in an LTO-2 drive and have tons of tapes, they will still work when you upgrade to an LTO-4 drive. You can then copy them to newer cartridges with the same hardware at a 4:1 ratio since the capacity will have doubled twice. The designers probably figured that after two generations (about five years) most data will have been restored at some point, or be irrelevant (the difference between backups and archives).

If you need your media archived longer than that, it would probably be wise to transfer it to fresh media of a newer generation to ensure it is readable in the future. The other issue is transfer if you are using LTO cartridges to move data from one place to another. You must use the same generation of tape and be within one generation to go both ways. If I want to send data to someone who has an LTO-5 drive, I have to use an LTO-5 tape, but I can copy the data to the tape with my LTO-6 drive (and be subject to the LTO-5 capacity and performance limits). If they then sent that LTO-5 tape to someone with an LTO-7 drive, they would be able to read the data, but wouldn’t be able to write to the tape. The only exception to this is that the LTO-8 drives won’t read LTO-6 tapes (of course, because I have a bunch of LTO-6 tapes now, right?).

So for my next 250TB project, I have to choose between a new LTO-7 drive with backwards compatibility to my existing gear or an LTO-8 drive that can fit 50% more data on a $63 cartridge, and use the more expensive 12TB ones as well. Owning both LTO-6 and LTO-8 drives would allow me to read or write to any LTFS cartridge (until LTO-9 is released), but the two drives couldn’t exchange tapes with each other.

Automated Backup Software & Media Management
I have just been using HPE’s free StoreOpen Utility to operate my internal LTO drive and track what files I copy to which tapes. There are obviously much more expensive LTO-based products, both in hardware with robotic tape libraries and in software with media and asset management programs and automated file backup solutions.

I am really just exploring the minimum investment that needs to be made to take advantage of the benefits of LTO tape, for manually archiving your media files and backing up your projects. The possibilities are endless, but the threshold to start using LTO is much lower than it used to be, especially with the release of LTFS support.

Mike McCarthy is an online editor/workflow consultant with 10 years of experience on feature films and commercials. He has been involved in pioneering new solutions for tapeless workflows, DSLR filmmaking and multi-screen and surround video experiences. Check out his site.

NAB: Media archives drive storage innovation

By Tom Coughlin

Higher resolution, frame rate, pixel depth and the number of cameras for pro video projects is increasing the storage capacity requirements for professional media. The growing size of new projects, combined with the vast collection of media and entertainment content already in archives will drive the size of media archives.

In addition, the desire for ready access to this content to support reuse and other purposes increases the need for faster more active archives. The 2014 Digital Storage for Media and Entertainment Report from my own Coughlin Associates estimates that over 96% of all digital storage in the media and entertainment industry is used in archiving and preservation.

The 2015 NAB show was a showcase for many digital media archive solutions. Magnetic tape is a big player in the traditional media archive market. The most popular tape format in media and entertainment is the LTO tape. The LTO Consortium says that tape storage costs are now less than $0.008 per GB ($8/TB). The LTFS file system used in LTO and other tapes is now supported by SNIA standards. Magnetic tape is now being used in object storage environments and Swift now supports LTFS. The group says that they have seen roughly a 6% growth in magnetic tape in the media and entertainment market.

Fujifilm had a booth with Crossroads Systems where they were showing the Dternity tape-based cloud storage system with the Crossroads Strongbox product. Oracle and IBM were also showing their tape based storage systems that they are targeting for media and entertainment (IBM currently offers up to 10TB of capacity on a half-inch tape cartridge). Several companies were showing the USB interface tape drive made by IBM in their NAB booths. Several companies were showing the USB interface tape drive made by IBM in their NAB booths. mLogic was showing a Thunderbolt LTO tape drive.

ProMax acquired Cache-A in July of 2014, At the 2015 NAB show the company released a significant update to its archiving appliance lineup. The Pro-Cache and Power-Cache products have been redesigned from the ground up to improve the speed and reliability of archiving to LTO tape. The Pro-Cache’s expanded network connectivity includes 4 x 1GbE network controllers accompanied by a 10GbE network performance boost that makes it up to 2x faster than previous systems. The Power-Cache (our main photo) onboard RAID is now up to 16TB with room for an optional onboard LTO-6 or LTO-5 drive.

ProMAX Pro-Cache and Power-Cache copy

Spectra Logic announced updates and enhancements to their BlackPearl Deep Storage Gateway. The newly enhanced Spectra Logic BlackPearl Deep Storage Gateway sits in front of deep storage tape libraries and allows users to move data anywhere within their network using simple HTTP commands. An interesting feature of the BlackPearl hardware is that they use flash memory for caching write and read information to a magnetic tape library. Because of the very fast data rates for mounted tapes flash memory avoids dropping of data or needing to rerun tapes to complete data transfers. Spectra Logic also announced that they were providing an archive tier to private clouds running on the NetApp StorageGrid using the BlackPearl Deep Storage Gateway.

Sony has been driving cartridge archive system using 12 Blu-ray write-once optical media in their Optical Disc Archive (ODA). These cartridges are scheduled for a significant increase in capacity in 2015 with the planned introduction of 300GB 5.25-inch optical write-one archive discs. In addition to the Sony booth companies such as XenData and Qualstar were showing both disc and tape archive libraries in their booth. MassTech also announced support for Sony’s ODA by their asset management system. There is a significant segment of the media and entertainment market that prefers optical discs for cold archiving rather magnetic tape.

backup progress with proxies copy PRP Back Up Screen Shot

Imagine Products and Sony have created a new ODA archiving solution for Mac users. The company’s PreRoll Post LTFS archiving application (pictured above) creates thumbs, proxies and rich metadata while backing up files and folders to multiple locations at once. It also uses checksum technology to ensure that the files and folders are copied accurately. Users simply drag and drop files and folders into the queue for copying to the disc.

Software to support backup also has an important role in M&E. SGL was showing a full archive demonstration with Grass Valley. The company was also launching an Asset Migration Service (FlashNet Migration Service) to move assets between one tape platform or generation to another. They were also showing partial file restore via Avid Web Services.

For more active archives there were several HDD systems geared to active archiving on display at the NAB. HGST (a division of Western Digital) was showing their HDD-based Active Archive System. This was HGST’s first foray into large storage system design and it was leveraging its internal drive costs to offer a cost effective object based storage system cheaper than many white-box solutions. HGST’s Active Archive System is an object storage system that delivers 4.7 petabytes (PB) of raw data storage in a single rack. The company says this product will provide long term retention of media data with fast access when needed.

Dr. Tom Coughlin, president of Coughlin Associates, is a storage analyst and consultant with over 30 years in the data storage industry. He has six patents to his credit and is the author of Digital Storage in Consumer Electronics: The Essential Guide. He also helps put together the Storage Visions Conference. You can we reach him at tom@tomcoughlin.com.