Tag Archives: LTO Storage

The benefits of LTO

By Mike McCarthy

LTO stands for Linear Tape Open, and was initially developed nearly 20 years ago as an “open” format technology that allows manufacturing by any vendor that wishes to license the technology. It records any digital files onto half-inch magnetic tapes, stored in square single reel cartridges. The capacity started at 100GB and has increased by a factor of two nearly every generation; the most recent LTO-8 cartridges store 12TB of uncompressed data.

If you want to find out more about LTO, you should check out the LTO Consortium, which is made up of Hewlett Packard Enterprises, IBM and Quantum, although there are other companies that make LTO drives and tape cartridges. You might be familiar with their LTO Ultrium logo.

‘Tapeless’ Workflows
While initially targeting server markets, with the introduction of “tapeless workflows” in the media and entertainment industry, there became a need for long-term media storage. Since the first P2 cards and SxS sticks were too expensive for single write operations, they were designed to be reused repeatedly once their contents had been offloaded to hard drives. But hard drives are not ideal for long-term data storage, and insurance and bonding companies wanted their clients to have alternate data archiving solutions.

So, by the time the Red One and Canon 5D were flooding post facilities with CF cards, LTO had become the default archive solution for most high-budget productions. But this approach was not without limitations and pitfalls. The LTO archiving solutions being marketed at the time were designed around the Linux-based Tar system of storing files, while most media work is done on Windows and Mac OS X. Various approaches were taken by different storage vendors to provide LTO capabilities to M&E customers. Some were network appliances running Linux under the hood, while others wrote drivers and software to access the media from OS X or, in one case, Windows. Then there was the issue that Tar isn’t a self-describing file system, so you needed a separate application to keep track of what was on each tape in your library. All of these aspects cost lots of money, so the initial investment was steep, even though the margin cost of tape cartridges was the cheapest way to store data per GB.

LTFS
Linear Tape File System (LTFS) was first introduced with LTO-5 and was intended to make LTO tapes easier to use and interchange between systems. A separate partition on the tape stores the index of data in XML and other associated metadata. It was intended to be platform independent, although it took a while for reliable drivers and software to be developed for use in Windows and OS X.

At this point, LTFS-formatted tapes in LTO tape drives operate very similarly to old 3.5-inch floppy drives. You insert a cartridge, it makes some funny noises, and then after a minute it asks you to format a new tape, or it displays the current contents of the tape as a new drive letter. If you drag files into that drive, it will start copying the data to the tape, and you can hear it grinding away. The biggest difference is when you hit eject it will take the computer a minute or two to rewind the tape, write the updated index to the first partition and then eject the cartridge for you. Otherwise it is a seamless drag and drop, just like any other removable data storage device.

LTO Drives
All you need in order to use LTO in your media workflow — for archive or data transfer — is an LTO drive. I bought one last year on Amazon for $1,600, which was a bit of a risk considering that I didn’t know if I was going to be able to get it to work on my Windows 7 desktop. As far as I know, all tape drives are SAS devices, although you can buy ones that have adapted the SAS interface to Thunderbolt or Fibre Channel.

Most professional workstations have integrated SAS controllers, so internal LTO drives fit into a 5.25-inch bay and can just connect to those, or any SAS card. External LTO drives usually use Small Form Factor cables (SFF-8088) to connect to the host device. Internal SAS ports can be easily adapted to SFF-8088 ports, or a dedicated eSAS PCIe card can be installed in the system.

Capacity & Compression
How much data do LTO tapes hold? This depends on the generation… and the compression options. The higher capacity advertised on any LTO product assumes a significant level of data compression, which may be achievable with uncompressed media files (DPX, TIFF, ARRI, etc.) The lower value advertised is the uncompressed data capacity, which is the more accurate estimate of how much data it will store. This level of compression is achieved using two different approaches, eliminating redundant data segments and eliminating the space between files. LTO was originally designed for backing up lots of tiny files on data servers, like credit card transactions or text data, and those compression approaches don’t always apply well to large continuous blocks of unique data found in encoded video.

Using data compression on media files which are already stored in a compressed codec doesn’t save much space (there is little redundancy in the data, and few gaps between individual files).

Uncompressed frame sequences, on the other hand, can definitely benefit from LTO’s hardware data compression. Regardless of compression, I wouldn’t count on using the full capacity of each cartridge. Due to the way the drives are formatted, and the way storage vendors measure data, I have only been able to copy 2.2TB of data from Windows onto my 2.5TB LTO-6 cartridges. So keep that in mind when estimating real-world capacity, like with any other data storage medium.

Choosing the ‘Right’ Version to Use
So which generation of LTO is the best option? That depends on how much data you are trying to store. Since most media files that need to be archived these days are compressed, either as camera source footage or final deliverables, I will be calculating based on the uncompressed capacities. VFX houses using DPX frames, or vendors using DCDMs might benefit from calculating based on the compressed capacities.

Prices are always changing, especially for the drives, but these are the numbers as of summer 2018. On the lowest end, we have LTO-5 drives available online for $600-$800, which will probably store 1-1.2TB of data on a $15 tape. So if you have less than 10TB of data to backup at a time, that might be a cost-effective option. Any version lower than LTO-5 doesn’t support the partitioning required for LTFS, and is too small to be useful in modern workflows anyway.

As I mentioned earlier, I spent $1,600 on an LTO-6 drive last year, and while that price is still about the same, LTO-7 and LTO-8 drives have come down in cost since then. My LTO-6 drive stores about 2.2TB of data per $23 tape. That allowed me to backup 40TB of Red footage onto 20 tapes in 90 hours, or an entire week. Now I am looking at using the same drive to ingest 250TB of footage from a production in China, but that would take well over a month, so LTO-6 is not the right solution for that project. But the finished deliverables will probably be a similar 10TB set of DPX and TIFF files, so LTO-6 will still be relevant for that application.

I see prices as low as $2,200 for LTO-7 drives, so they aren’t much more expensive than LTO-6 drives at this point, but the 6TB tapes are. LTO-7 switched to a different tape material, which increased the price of the media. At $63 they are just over $10 per TB, but that is higher than the two previous generations.

LTO-8 drives are available for as low as $2,600, and store up to 12TB on a single $160 tape. LTO-8 drives can also write up to 9TB onto a properly formatted LTO-7 tape in a system called “LTO-7 Type M” This is probably the cheapest cost per TB approach at the moment, since 9TB on a $63 tape is $7/TB.

Compatibility Between Generations
One other consideration is backwards compatibility. What will it take to read your tapes back in the future? The standard for LTO has been that drives can write the previous generation tapes and read tapes from two generations back.

So if you invested in an LTO-2 drive and have tons of tapes, they will still work when you upgrade to an LTO-4 drive. You can then copy them to newer cartridges with the same hardware at a 4:1 ratio since the capacity will have doubled twice. The designers probably figured that after two generations (about five years) most data will have been restored at some point, or be irrelevant (the difference between backups and archives).

If you need your media archived longer than that, it would probably be wise to transfer it to fresh media of a newer generation to ensure it is readable in the future. The other issue is transfer if you are using LTO cartridges to move data from one place to another. You must use the same generation of tape and be within one generation to go both ways. If I want to send data to someone who has an LTO-5 drive, I have to use an LTO-5 tape, but I can copy the data to the tape with my LTO-6 drive (and be subject to the LTO-5 capacity and performance limits). If they then sent that LTO-5 tape to someone with an LTO-7 drive, they would be able to read the data, but wouldn’t be able to write to the tape. The only exception to this is that the LTO-8 drives won’t read LTO-6 tapes (of course, because I have a bunch of LTO-6 tapes now, right?).

So for my next 250TB project, I have to choose between a new LTO-7 drive with backwards compatibility to my existing gear or an LTO-8 drive that can fit 50% more data on a $63 cartridge, and use the more expensive 12TB ones as well. Owning both LTO-6 and LTO-8 drives would allow me to read or write to any LTFS cartridge (until LTO-9 is released), but the two drives couldn’t exchange tapes with each other.

Automated Backup Software & Media Management
I have just been using HPE’s free StoreOpen Utility to operate my internal LTO drive and track what files I copy to which tapes. There are obviously much more expensive LTO-based products, both in hardware with robotic tape libraries and in software with media and asset management programs and automated file backup solutions.

I am really just exploring the minimum investment that needs to be made to take advantage of the benefits of LTO tape, for manually archiving your media files and backing up your projects. The possibilities are endless, but the threshold to start using LTO is much lower than it used to be, especially with the release of LTFS support.


Mike McCarthy is an online editor/workflow consultant with 10 years of experience on feature films and commercials. He has been involved in pioneering new solutions for tapeless workflows, DSLR filmmaking and multi-screen and surround video experiences. Check out his site.

NAB: Imagine Products and StorageDNA enhance LTO and LTFS

By Jonathan S. Abrams

That’s right. We are still taking NAB. There was a lot to cover!

So, the first appointment I booked for NAB Show 2018, both in terms of my show schedule (10am Monday) and the vendors I was in contact with, was with StorageDNA’s Jeff Krueger, VP of worldwide sales. Weeks later, I found out that StorageDNA was collaborating with Imagine Products on myLTOdna, so I extended my appointment. Doug Hynes, senior director of business development for StorageDNA, and Michelle Maddox, marketing director of Imagine Products, joined me to discuss what they had ready for the show.

The introduction of LTFS during NAB 2010 allowed LTO tape to be accessed as if it was a hard drive. Since LTO tape is linear, executing multiple operations at once and treating it like a hard drive results in performance falling off of a cliff. It also could cause the drive to engage in shoeshining, or shuttling of the tape back-and-forth over the same section.

Imagine Products’ main screen.

Eight years later, these performance and operation issues have been addressed by StorageDNA’s creation of HyperTape, which is their enhanced Linear File Transfer System that is part of Imagine Products’ myLTOdna application. My first question was “Is HyperTape yet another tape format?” Fortunately for myself and other users, the answer is “No.”

What is HyperTape? It is a workflow powered by dnaLTFS. The word “enhanced” in the description of HyperTape as an enhanced Linear File Transfer System refers to a middleware in their myLTOdna application for Mac OS. There are three commands that can be executed to put an LTO drive into either read-only, write-only or training mode. Putting the LTO drive into an “only mode” allows it to achieve up to 300MB/s of throughput. This is where the Hyper in HyperTape comes from. These modes can also be engaged from the command line.

Training mode allows for analyzing the files stored on an LTO tape and then storing that information in a Random Access Database (RAD). The creation of the RAD can be automated using Imagine Products’ PrimeTranscoder. Otherwise, each file on the tape must be opened in order to train myLTOdna and create a RAD.

As for shoeshining, or shuttling of the tape back-and-forth over the same section, this is avoided by intelligently writing files to LTO tape. This intelligence is proprietary and is built into the back-end of the software. The result is that you can load a clip in Avid’s Media Composer, Blackmagic’s DaVinci Resolve or Adobe’s Premiere Pro and then load a subclip from that content into your project. You still should not load a clip from tape and just press play. Remember, this is LTO tape you are reading from.

The target customer for myLTOdna is a DIT with camera masters who wants to reduce how much time it takes to backup their footage. Previously, DITs would transfer the camera card’s contents to a hard drive using an application such as Imagine Products’ ShotPut Pro. Once the footage had been transferred to a hard drive, it could then be transferred to LTO tape. Using myLTOdna in read-only mode allows a DIT to bypass the hard drive and go straight from the camera card to an LTO tape. Because the target customer is already using ShotPut Pro, the UI for myLTOdna was designed to be comfortable and not difficult to use or understand.

The licensing for dnaLTFS is tied to the serial number of an LTO drive. StorageDNA’s Krueger explained that, “dnaLTFS is the drive license that works with stand alone mac LTO drives today.” Purchasing a license for dnaLTFS allows the user to later upgrade to StorageDNA’s DNAevolution M Series product if they need automation and scheduling features without having to purchase another drive license if the same LTO drive is used.

Krueger went on to say, “We will have (dnaLTFS) integrated into our DNAevolution product in the future.” DNAevolution’s cost of entry is $5,000. A single LTO drive license starts at $1,250. Licensing is perpetual, and updates are available without a support contract. myLTOdna, like ShotPut Pro and PrimeTranscoder, is a one-time purchase (perpetual license). It will phone home on first launch. Remote support is available for $250 per year.

I also envision myLTOdna being useful outside of the DIT market. Indeed, this was the thinking when the collaboration between Imagine Products and StorageDNA began. If you do not mind doing manual work and want to keep your costs low, myLTOdna is for you. If you later need automation and can budget for the efficiencies that you get with it, then DNAevolution is what you can upgrade to.


Jonathan S. Abrams is the Chief Technical Engineer at Nutmeg, a creative marketing, production and post resource, located in New York City.