Tag Archives: Rising Sun PIctures

Storage in the Studio: VFX Studios

By Karen Maierhofer

It takes talent and the right tools to generate visual effects of all kinds, whether it’s building breathtaking environments, creating amazing creatures or crafting lifelike characters cast in a major role for film, television, games or short-form projects.

Indeed, we are familiar with industry-leading content creation tools such as Autodesk’s Maya, Foundry’s Mari and more, which, when placed into the hands of creatives, the result in pure digital magic. In fact, there is quite a bit of technological magic that occurs at visual effects facilities, including one kind in particular that may not have the inherent sparkle of modeling and animation tools but is just as integral to the visual effects process: storage. Storage solutions are the unsung heroes behind most projects, working behind the scenes to accommodate artists and keep their productive juices flowing.

Here we examine three VFX facilities and their use of various storage solutions and setups as they tackle projects large and small.

Framestore
Since it was founded in 1986, Framestore has placed its visual stamp on a plethora of Oscar-, Emmy- and British Academy Film Award-winning visual effects projects, including Harry Potter, Gravity and Guardians of the Galaxy. With increasingly more projects, Framestore expanded from its original UK location in London to North American locales such as Montreal, New York, Los Angeles and Chicago, handling films as well as immersive digital experiences and integrated advertisements for iconic brands, including Guinness, Geico, Coke and BMW.

Beren Lewis

As the company and its workload grew and expanded into other areas, including integrated advertising, so, too, did its storage needs. “Innovative changes, such as virtual-reality projects, brought on high demand for storage and top-tier performance,” says NYC-based Beren Lewis, CTO of advertising and applied technologies at Framestore. “The team is often required to swiftly accommodate multiple workflows, including stereoscopic 4K and VR.”

Without hesitation, Lewis believes storage is typically the most challenging aspect of technology within the VFX workflow. “If the storage isn’t working, then neither are the artists,” he points out. Furthermore, any issues with storage can potentially lead to massive financial implications for the company due to lost time and revenue.

According to Lewis, Framestore uses its storage solution — a Pixit PixStor General Parallel File System (GPFS) storage cluster using the NetApp E-Series hardware – for all its project data. This includes backups to remote co-location sites, video preprocessing, decompression, disaster recovery preparation, scalability and high performance for VFX, finishing and rendering workloads.

The studio moved all the integrated advertising teams over to the PixStor GPFS clusters this past spring. Currently, Framestore has five primary PixStor clusters using NetApp E-Series in use at each office in London, LA, Chicago and Montreal.

According to Lewis, Framestore partnered with Pixit Media and NetApp to take on increasingly complicated and resource-hungry VR projects. “This partnership has provided the global integrated advertising team with higher performance and nonstop access to data,” he says. “The Pixit Media PixStor software-defined scale-out storage solution running on NetApp E-Series systems brings fast, reliable data access for the integrated advertising division so the team can embrace performance and consistency across all five sites, take a cost-effective, simplified approach to disaster recovery and have a modular infrastructure to support multiple workflows and future expansion.”

BMW

Framestore selected its current solution after reviewing several major storage technologies. It was looking for a single namespace that was very stable, while providing great performance, but it also had to be scalable, Lewis notes. “The PixStor ticked all those boxes and provided the right balance between enterprise-grade hardware and support, and open-source standards,” he explains. “That balance allowed us to seamlessly integrate the PixStor into our network, while still maintaining many of the bespoke tools and services that we had developed in-house over the years, with minimum development time.”

In particular, the storage solution provides the required high performance so that the studio’s VFX, finishing and rendering workloads can all run “full-out with no negative effect on the finishing editors’ or graphic artists’ user experience,” Lewis says. “This is a game-changing capability for an industry that typically partitions off these three workloads to keep artists from having to halt operations. PixStor running on E-Series consolidates all three workloads onto a single IT infrastructure with streamlined end-to-end production of projects, which reduces both time to completion and operational costs, while both IT acquisition and maintenance costs are reduced.”

At Framestore, integrating storage into the workflow is simple. The first step after a project is green-lit is the establishment of a new file set on the PixStor GPFS cluster, where ingested footage and all the CG artist-generated project data will live. “The PixStor is at the heart of the integrated advertising storage workflow from start to finish,” Lewis says. Because the PixStor GPFS cluster serves as the primary storage for all integrated advertising project data, the division’s workstations, renderfarm, editing and finishing stations connect to the cluster for review, generation and storage of project content.

Prior to the move to PixStor/NetApp, Framestore had been using a number of different storage offerings. According to Lewis, they all suffered from the same issues in terms of scalability and degradation of performance under render load — and that load was getting heavier and more unpredictable with every project. “We needed a technology that scaled and allowed us to maintain a single namespace but not suffer from continuous slowdowns for artists due to renderfarm load during crunch times or project delivery.”

Geico

As Lewis explains, with the PixStor/NetApp solution, processing was running up to 270,000 IOPS (I/O operations per second), which was at least several times what Framestore’s previous infrastructure would have been able to handle in a single namespace. “Notably, the development workflow for a major theme-park ride was unhindered by all the VR preprocessing, while backups to remote co-location sites synched every two hours without compromising the artist, rendering or finishing workloads,” he says. “This provided a cost-effective, simplified approach to disaster recovery, and Framestore now has a fast, tightly integrated platform to support its expansion plans.”

To stay at the top of its game, Framestore is always reviewing new technologies, and storage is often part of that conversation. To this end, the studio plans to build on the success it has had with PixStor by expanding the storage to handle some additional editorial playback and render workloads using an all-Non-Volatile Memory Express (NVMe) flash tier. Other projects include a review of object storage technology for use as a long-term, off-premises storage target for archival data.

Without question, the industry’s visual demands are rapidly changing. Not long ago, Framestore could easily predict storage and render requirements for a typical project. But that is no longer the case, and the studio finds itself working in ever-increasing resolutions and frame rates. Whereas projects may have been as small as 3TB in the recent past, nowadays the studio regularly handles multiple projects of 300TB or larger. And the storage must be shared with other projects of varying sizes and scope.

“This new ‘unknowns’ element of our workflow puts many strains on all aspects of our pipeline, but especially the storage,” Lewis points out. “Knowing that our storage can cope with the load and can scale allows us to turn our attention to the other issues that these new types of projects bring to Framestore.”

As Lewis notes, working with high-resolution images and large renderfarms create a unique set of challenges for any storage technology that’s not seen in many other fields. The VFX will often test any storage technology well beyond what other industries are capable of. “If there’s an issue or a break point, we will typically find it in spectacular fashion,” he adds.

Rising Sun Pictures
As a contributor to the design and execution of computer-generated effects on more than 100 feature films since its inception 22 years ago, Rising Sun Pictures (RSP) has pushed the technical bar many times over in film as well as television projects. Based in Adelaide, South Australia, RSP has built a top team of VFX artists who have tackled such box-office hits as Thor: Ragnarok, X-Men and Game of Thrones, as well as the Harry Potter and Hunger Games franchises.

Mark Day

Such demanding, high-level projects require demanding, high-level effects, which, in turn, demand a high-performance, reliable storage solution capable of handling varying data I/O profiles. “With more than 200 employees accessing and writing files in various formats, the need for a fast, reliable and scalable solution is paramount to business continuity,” says Mark Day, director of engineering at RSP.

Recently, RSP installed an Oracle ZS5 storage appliance to handle this important function. This high-performance, unified storage system provides NAS and SAN cloud-converged storage capabilities that enable on-premises storage to seamlessly access Oracle Public Cloud. Its advanced hardware and software architecture includes a multi-threading SMP storage operating system for running multiple workloads and advanced data services without performance degradation. The offering also caches data on DRAM or flash cache for optimal performance and efficiency, while keeping data safely stored on high-capacity SSD (solid state disk) or HDD (hard disk drive) storage.

Previously, the studio had been using an Dell EMC Isilon storage cluster with Avere caching appliances, and the company is still employing the solution for parts of its workflow.

When it came time to upgrade to handle RSP’s increased workload, the facility ran a proof of concept with multiple vendors in September 2016 and benchmarked their systems. Impressed with Oracle, RSP began installation in early 2017. According to Day, RSP liked the solution’s ability to support larger packet sizes — now up to 1MB. In addition, he says its “exceptional” analytics engine gives introspection into a render job.

“It has a very appealing [total cost of ownership], and it has caching right out of the box, removing the need for additional caching appliances,” says Day. Storage is at the center of RSP’s workflow, storing all the relevant information for every department — from live-action plates that are turned over from clients, scene setup files and multi-terabyte cache files to iterations of the final product. “All employees work off this storage, and it needs to accommodate the needs of multiple projects and deadlines with zero downtime,” Day adds.

Machine Room

“Visual effects scenes are getting more complex, and in turn, data sizes are increasing. Working in 4K quadruples file sizes and, therefore, impacts storage performance,” explains Day. “We needed a solution that could cope with these requirements and future trends in the industry.”

According to Day, the data RSP deals with is broad, from small setup files to terabyte geocache files. A one-minute 2K DPX sequence is 17GB for the final pass, while 4K is 68GB. “Keep in mind this is only the final pass; a single shot could include hundreds of passes for a heavy computer-generated sequence,” he points out.

Thus, high-performance storage is important to the effective operation of a visual effects company like RSP. In fact, storage helps the artists stay on the creative edge by enabling them to iterate through the creative process of crafting a shot and a look. “Artists are required to iterate their creative process many times to perfect the look of a shot, and if they experience slowdowns when loading scenes, this can have a dramatic effect on how many iterations they can produce. And in turn, this affects employees’ efficiency and, ultimately, the profitability of the company,” says Day.

Thor: Ragnarok

Most recently, RSP used its new storage solution for work on the blockbuster Thor: Ragnarok, in particular, for the Val’s Flashback sequence — which was extremely complex and involved extensive lighting and texture data, as well as high-frame-rate plates (sometimes more than 1,000fps for multiple live-action footage plates). “Before, our storage refresh early versions of this shot could take up to 24 hours to render on our server farm. But since installing our new storage, we saw this drastically reduced to six hours — that’s a 3x improvement, which is a fantastic outcome,” says Day.

Outpost VFX
A full-service VFX studio for film, broadcast and commercials, Outpost VFX, based in Bournemouth, England, has been operational since late 2012. Since that time, the facility has been growing by leaps and bounds, taking on major projects, including Life, Nocturnal Animals, Jason Bourne and 47 Meters Down.

Paul Francis

Due to this fairly rapid expansion, Outpost VFX has seen the need for increased capacity in its storage needs. “As the company grows and as resolution increases and HDR comes in, file sizes increase, and we need much more capacity to deal with that effectively,” says CTO Paul Francis.

When setting up the facility five years ago, the decision was made to go with PixStor from Pixit Media and Synology’s NAS for its storage solution. “It’s an industry-recognized solution that is extremely resilient to errors. It’s fast, robust and the team at Pixit provides excellent support, which is important to us,” says Francis.

Foremost, the solution had to provide high capacity and high speeds. “We need lots of simultaneous connections to avoid bottlenecks and ensure speedy delivery of data,” Francis adds. “This is the only one we’ve used, really. It has proved to be stable enough to support us through our growth over the last couple of years — growth that has included a physical office move and an increase in artist capacity to 80 seats.”

Outpost VFX mainly works with image data and project files for use with Autodesk’s Maya, Foundry’s Nuke, Side Effects’ Houdini and other VFX and animation tools. The challenge this presents is twofold, both large and small: concern for large file sizes, and problems the group can face with small files, such as metadata. Francis explains: “Sequentially loading small files can be time-consuming due to the current technology, so moving to something that can handle both of these areas will be of great benefit to us.”

Locally, artists use a mix of HDDs from a number of different manufacturers to store reference imagery and so forth — older-generation PCs have mostly Western Digital HDDs while newer PCs have generic SSDs. When replacing or upgrading equipment, Outpost VFX uses Samsung 900 Series SSDs, depending on the required performance and current market prices.

Life

Like many facilities, Outpost VFX is always weighing its options when it comes to finding the best solution for its current and future needs. Presently, it is looking at splitting up some of its storage solutions into smaller segments for greater resilience. “When you only have one storage solution and it fails, everything goes down. We’re looking to break our setup into smaller, faster solutions,” says Francis.

Additionally, security is a concern for Outpost VFX when it comes to its clients. According to Francis, certain shows need to be annexed, meaning the studio will need a separate storage solution outside of its main network to handle that data.

When Outpost VFX begins a job, the group ingests all the plates it needs to work on, and they reside in a new job folder created by production and assigned to a specific drive for active jobs. This folder then becomes the go-to for all assets, elements and shot iterations created throughout the production. For security purposes, these areas of the server are only visible to and accessible by artists, who in turn cannot access the Internet; this ensures that the files are “watertight and immune to leaks,” says Francis, adding that with PixStor, the studio is able to set up different partitions for different areas that artists can jump between easily.

How important is storage to Outpost VFX? “Frankly, there’d be no operation without storage!” Francis says emphatically. “We deal with hundreds of terrabytes of data in visual effects, so having high-capacity, reliable storage available to us at all times is absolutely essential to ensure a smooth and successful operation.”

47 Meters Down

Because the studio delivers visual effects across film, TV and commercials simultaneously, storage is an important factor no matter what the crew is working on. A recent film project like 47 Meters Down required the full gamut of visual effects work, as Outpost VFX was the sole vendor for the project. So, the studio needed the space and responsiveness of a storage system that enabled them to deliver more than 420 shots, a number of which featured heavy 3D builds and multiple layers of render elements.

“We had only about 30 artists at that point, so having a stable solution that was easy for our team to navigate and use was crucial,” Francis points out.

Main Image: From Outpost VFX’s Domestos commercial out of agency MullenLowe London.

The VFX Industry: Where are the women?

By Jennie Zeiher

As anyone in the visual effects industry would know, Marvel’s Victoria Alonso was honored earlier this year with the Visual Effects Society Visionary Award. Victoria is an almighty trailblazer, one of whom us ladies can admire, aspire to and want to be.

Her acceptance speech was an important reminder to us of the imbalance of the sexes in our industry. During her speech, Victoria stated: “Tonight there were 476 of you nominated. Forty-three of which are women. We can do better.”

Over the years, I’ve had countless conversations with industry people — executives, supervisors and producers — about why there are fewer women in artist and supervisory roles. A recent article in the NY Times suggested that female VFX supervisors made up only five percent of the 250 top-grossing films of 2014. Pretty dismal.

I’ve always worked in male-dominated industries, so I’m possibly a bit blasé about it. I studied IT and worked as a network engineer in the late ‘90s, before moving to the United States where I worked on 4K digital media projects with technologists and scientists. One of a handful of women, I was always just one of the boys. To me it was the norm.

Moving into VFX about 10 years ago, I realized this industry was no different. From my viewpoint, I see about 1/8 ratio of female to male artists. The same is true from what I’ve seen through our affiliated training courses. Sadly, I’ve heard of some facilities that have no women in artist roles at all!

Most of the females in our industry work in other disciplines. At my workplace, Australia’s Rising Sun Pictures, half of our executive members are women (myself included), and women generally outweigh men in indirect overhead roles (HR, finance, administration and management), as well as production management.

Women bring unique qualities to the workplace: they’re team players, hard working, generous and empathetic. Copious reports have found that companies that have women on their board of directors and in leadership positions perform better than those that don’t. So in our industry, why do we see such a male-dominated artist, technical and supervisory workforce?

By no means am I undervaluing the women in those other disciplines (we could not have functioning businesses without them), I’m just merely trying to understand why there aren’t more women inclined to pursue artistic jobs and, ultimately, supervision roles.

I can’t yet say that one of the talented female artists I’ve had the pleasure of working with over the years has risen to the ranks of being a VFX supervisor… and that’s not to say that they couldn’t have, just that they didn’t, or haven’t yet. This is something that disappoints me deeply. I consider myself a (liberal) feminist. Someone who, in a leadership position, wants to enable other women to become the best they can be and to be equal among their male counterparts.
So, why? Where are the women?

Men and Women Are Wired Differently
A study by LiveScience suggests men and women really are wired differently. It says,  “Male brains have more connections within hemispheres to optimize motor skills, whereas female brains are more connected between hemispheres to combine analytical and intuitive thinking.”

Apparently this difference is at its greatest during the adolescent years (13-17 years), however with age these differences get smaller. So, during the peak of an adolescent girl’s education, she’s more inclined to be analytical and intuitive. Is that a direct correlation to them not choosing a technical vocation? But then again I would have thought that STEM/STEAM careers would be something of interest to girls if they’re brains are wired to be analytical?

This would also explain women having better organizational and management skills and therefore seeking out more “indirectly” associated roles.

Lean Out
For those women already in our industry, are they too afraid to seek out higher positions? Women are often more self-critical and self-doubting. Men will promote themselves and dive right in, even if they’re less capable. I have experienced this first hand and didn’t actual recognize it in myself until I read Sheryl Sandberg’s Lean In.

Or, is it just simply that we’re in a “boys club” — that these career opportunities are not being presented to our female artists, and that we’d prefer to promote men over women?

The Star Wars Factor
Possibly one of the real reasons that there is a lack of women in our industry is what I call “The Star Wars factor.” For the most part, my male counterparts grew up watching (and being inspired by) Star Wars and Star Trek, whereas, personally, I was more inclined to watch Girls Just Want to Have Fun and Footloose. Did these adolescent boys want to be Luke or Han, or George for that matter? Were they so inspired by John Dykstra’s lightsabers that they wanted to do THAT when they grew up? And if this is true, maybe Jyn, Rae and Captain Marvel —and our own Captain Marvel, Victoria Alonso — will spur on a new generation of women in the industry. Maybe it’s a combination of all of these factors. Maybe it’s none.

I’m very interested in exploring this further. To address the problem, we need to ask ourselves why, so please share your thoughts and experiences — you can find me at jz@vfxjz.com. At least now the conversation has started.

One More Thing!
I am very proud that one of my female colleagues, Alana Newell (pictured with her fellow nominees), was nominated for a VES Award this year for Outstanding Compositing in a Photoreal Feature for X-Men: Apocalypse. She was one of the few, but hopefully as time goes by that will change.

Main Image: The woman of Rising Sun Pictures.
——–

Jennie Zeiher is head of sales & business development at Adelaide, Australia’s Rising Sun Pictures.

Reviews & Approvals: Cospective’s Rory McGregor talks cineSync, Frankie

By Randi Altman

Many of you are likely familiar with the remote review and approval tool cineSync from Australia-based software company Cospective. cineSync has been used on some big-name film and television projects (Christopher Nolan is one outspoken fan) since its introduction in 2005. Back then the company was called Rising Sun Research, which grew out of VFX studio Rising Sun Pictures. So, users solving a problem by developing their own solution.

Well the minds behind the Academy Award-winning cineSync technology have Continue reading

RSP’s Tim Crosbie gets VFX Oscar nom for ‘X-Men: Days of Future Past’

VFX supervisor Tim Crosbie, from Australia’s Rising Sun Pictures (RSP) is part of the team nominated for an Academy Award for Best Achievement in Visual Effects for the film X-Men: Days of Future Past.

Crosbie supervised the team of RSP artists involved in creating the film’s “Kitchen” scene where time appears to stand still as the super-fast Quicksilver moves around the Pentagon cafeteria distracting guards who are attacking a group of mutants. Also named in the nomination are the film’s VFX supervisor Richard Stammers, Digital Domain’s Lou Pecora and Object Inc.’s Cameron Waldbauer.

The X-Men “Kitchen” scene involved a blend of live-action, CG objects and visual effects. RSP collaborated with Stammers and director Bryan Singer to bring the scene to life through the production of scores of CG props, including frying pans, knives, pots of boiling soup, carrots and bullets, as well as the cascades of water droplets. Each of these elements needed to be rendered in near microscopic detail, placed precisely within the geometry of the kitchen and choreographed to move and react realistically to lighting, other objects and characters.

RSP also integrated Quicksilver into the near frozen environment. That illusion was accomplished through a combination of live action, a stunt double, greenscreen photography, a partial CG body replacement and a shimmering “rain tunnel” that forms around Quicksilver (caused by his swift passage through the near motionless falling water). All of this had to work properly in 2D and stereo 3D. The studio’s main tool is Houdini.

Crosbie reports that pulling off a sequence like Quicksilver’s visit to the Pentagon is more than a matter of managing data. “This work is grounded in realism,” he says. “Even though it’s a fantastical event you still want to feel as though you are there. The biggest challenge is to find that balance between an exciting, magical event and one that looks real.”

X-Men_TwoShot_Gun

“All of us at RSP are extremely proud of our work on X-Men: Days of Future Past and to have it recognized by our peers is very gratifying,” says RSP executive producer Tony Clark. “We are very grateful to director Bryan Singer, visual effects supervisor Richard Stammers and 20th Century Fox for giving us the opportunity to work on such a fun, exciting and challenging project.”

RSP’s X-Men work has been recognized by other organizations as well. Crosbie is also named in a BAFTA Award nomination for Best Achievement in Special Visual Effects.  RSP artists are named in two VES Award nominations, Outstanding Virtual Cinematography in a Photoreal/Live Action Motion Media Project (Dennis Jones) and Outstanding Effects Simulations in a Photoreal/Live Action Feature Motion Picture (Adam Paschke, Premamurti Paetsch, Sam Hancock and Timmy Lundin). X-Men: Days of Future Past is also nominated for one of the VES Awards’ Outstanding Visual Effects in a Visual Effects-Driven Photoreal/Live Action Feature Motion Picture.