Tag Archives: Chris Healer

VFX Storage: The Molecule

Evolving to a virtual private local cloud?

By Beth Marchant

VFX artists, supervisors and technologists have long been on the cutting-edge of evolving post workflows. The networks built to move, manage, iterate, render and put every pixel into one breathtaking final place are the real super heroes here, and as New York’s The Molecule expands to meet the rising demand for prime-time visual effects, it pulls even more power from its evolving storage pipeline in and out of the cloud.

The Molecule CEO/CTO Chris Healer has a fondness for unusual workarounds. While studying film in college, he built a 16mm projector out of Legos and wrote a 3D graphics library for DOS. In his professional life, he swiftly transitioned from Web design to motion capture and 3D animation. He still wears many hats at his now bicoastal VFX and VR facility, The Molecule —which he founded in New York in 2005 — including CEO, CTO, VFX supervisor, designer, software developer and scientist. In those intersecting capacities, Healer has created the company’s renderfarm, developed and automated its workflow, linking and preview tools and designed and built out its cloud-based compositing pipeline.

When the original New York office went into growth mode, Healer (pictured at his new, under-construction facility) turned to GPL Technologies, a VFX and post-focused digital media pipeline and data infrastructure developer, to help him build an entirely new network foundation for the new location the company will move to later this summer. “Up to this point, we’ve had the same system and we’ve asked GPL to come in and help us create a new one from scratch,” he says. “But any time you hire anyone to help with this kind of thing, you’ve really got to do your own research and figure out what makes sense for your artists, your workflows and, ultimately, your bottom line.”

The new facility will start with 65 seats and expand to more than 100 within the next year to 18 months. Current clients include the major networks, Showtime, HBO, AMC, Netflix and director/producer Doug Limon.

UKS-beforesmall      UKS-aftersmall
Netflix’s Unbreakable Kimmy Schmidt is just one of the shows The Molecule works on.

Healer’s experience as an artist, developer, supervisor and business owner has given him a seasoned perspective on how to develop VFX pipeline work. “There’s a huge disparity between what the conventional user wants to do, i.e. share data, and the much longer dialog you need to have to build a network. Connecting and sharing data is really just the beginning of a very long story that involves so many other factors: how many things are you connecting to? What type of connection do you have? How far away are you from what you’re connecting to? How much data are you moving, and it is all at once or a continuous stream? Users are so different, too.”

Complicating these questions, he says, are a facility’s willingness to embrace new technology before it’s been vetted in the market. “I generally resist the newest technologies,” he says. “My instinct is that I would prefer an older system that’s been tested for years upon years. You go to NAB and see all kinds of cool stuff that appears to be working the way it should. But it hasn’t been tried in different kinds of circumstances or its being pitched to the broadcast industry and may not work well for VFX.”

Making a Choice
He was convinced by EMC’s Isilon system, based on customer feedback and the hardware has already been delivered to the new office. “We won’t install it until construction is complete, but all the documentation is pointing in the right direction,” he says. “Still, it’s a bit of a risk until we get it up and running.”

Last October, Dell announced it would acquire EMC in a deal that is set to close in mid-July. That should suit The Molecule just fine —most of its artists computers are either Dell or HP running Nvidia graphics.

A traditional mass configuration on a single GigE line can only do up to 100MB per second. “A 10GigE connection running in NFS can, theoretically, do 10 times that,” says Healer. “But 10GigE works slightly differently, like an LA freeway, where you don’t change the speed limit but you change the number of lanes and the on and off ramp lights to keep the traffic flowing. It’s not just a bigger gun for a bigger job, but more complexity in the whole system. Isilon seems to do that very well and it’s why we chose them.”

His company’s fast growth, Healer says, has “presented a lot of philosophical questions about disk and RAID redundancy, for example. If you lose a disk in RAID-5 you’re OK, but if two fail, you’re screwed. Clustered file systems like GlusterFS and OneFS, which Isilon uses, have a lot more redundancy built in so you could lose quite a lot of disks and still be fine. If your number is up and on that unlucky day you lost six disks, then you would have backup. But that still doesn’t answer what happens if you have a fire in your office or, more likely, there’s a fire elsewhere in the building and it causes the sprinklers to go off. Suddenly, the need for off-site storage is very important for us, so that’s where we are pushing into next.”

Healer honed in on several metrics to help him determine the right path. “The solutions we looked at had to have the following: DR, or disaster recovery, replication, scalability, off-site storage, undelete and versioning snapshots. And they don’t exactly overlap. I talked to a guy just the other day at Rsync.net, which does cloud storage of off-site backups (not to be confused with the Unix command, though they are related). That’s the direction we’re headed. But VFX is just such a hard fit for any of these new data centers because they don’t want to accept and sync 10TB of data per day.”

A rendering of The Molecule NYC's new location.His current goal is simply to sync material between the two offices. “The holy grail of that scenario is that neither office has the definitive master copy of the material and there is a floating cloud copy somewhere out there that both offices are drawing from,” he says. “There’s a process out there called ‘sharding,’ as in a shard of glass, that MongoDB and Scality and other systems use that says that the data is out there everywhere but is physically diverse. It’s local but local against synchronization of its partners. This makes sense, but not if you’re moving terabytes.”

The model Healer is hoping to implement is to “basically offshore the whole company,” he says. “We’ve been working for the past few months with a New York metro startup called Packet which has a really unique concept of a virtual private local cloud. It’s a mouthful but it’s where we need to be.” If The Molecule is doing work in New York City, Healer points out, Packet is close enough that network transmissions are fast enough and “it’s as if the machines were on our local network, which is amazing. It’s huge. It the Amazon cloud data center is 500 miles away from your office, that drastically changes how well you can treat those machines as if they are local. I really like this movement of virtual private local that says, ‘We’re close by, we’re very secure and we have more capacity than individual facilities could ever want.’ But they are off-site and the multiple other companies that use them are in their own discrete containers that never crosses. Plus, you pay per use — basically per hour and per resource. In my ideal future world, we would have some rendering capacity in our office, some other rendering capacity at Packet and off-site storage at Rsync.net. If that works out, we could potentially virtualize the whole workflow and join our New York and LA office and any other satellite office we want to set up in the future.”

The VFX market, especially in New York, has certainly come into its own in recent years. “It’s great to be in an era when nearly every single frame of every single shot of both television and film is touched in some way by visual effects, and budgets are climbing back and the tax credits have brought a lot more VFX artists, companies and projects to town,” Healer says. “But we’re also heading toward a time when the actual brick-and-mortar space of an office may not be as critical as it is now, and that would be a huge boon for the visual effects industry and the resources we provide.”

The Molecule: VFX for ‘The Affair’ and so much more

By Randi Altman

Luke DiTommaso, co-founder of New York City’s The Molecule, recalls “humble”
beginnings when he thinks about the visual effects, motion graphics and VR studio’s launch as a small compositing shop. When The Molecule opened in 2005, New York’s production landscape was quite a bit different than the tax-incentive-driven hotbed that exists today.

Rescue Me was our big break,” explains DiTommaso. “That show was the very beginning of this wave of production that started happening in New York. Then we got Damages and Royal Pains, but were still just starting to get our feet wet with real productions.”

The Molecule partners (L-R) Andrew Bly, Chris Healer and Luke DiTommaso.

Then, thanks to a healthy boost from New York’s production and post tax incentives, things exploded, and The Molecule was at the right place at the right time. They had an established infrastructure, talent and experience providing VFX for television series.

Since then DiTommaso and his partners Chris Healer and Andrew Bly have seen the company grow considerably, doing everything from shooting and editing to creating VFX and animation, all under one roof. With 35 full-time employees spread between their New York and LA offices — oh, yeah, they opened an office in LA! — they also average 30 freelance artists a day, but can seat 65 if needed.

While some of these artists work on commercials, many are called on to create visual effects for an impressive list of shows, including Netflix’s Unbreakable Kimmy Schmidt, House of Cards and Bloodline, Showtime’s The Affair, HBO’s Ballers (pictured below), FX’s The Americans, CBS’ Elementary and Limitless, VH1’s The Breaks, Hulu’s The Path (for NBC and starring Aaron Paul) and the final season of USA’s Royal Pains. Also completed are the miniseries Madoff and Behind the Magic, a special on Snow White, for ABC.

Ballers-before      Ballers-after

The Molecule’s reach goes beyond the small screen. In addition to having completed a few shots for Zoolander 2 and a big one involving a digital crowd for Barbershop 3, at the time of this interview the studio was gearing up for Jodie Foster’s Money Monster; they will be supplying titles, the trailer and a ton of visual effects.

There is so much for us to cover, but just not enough time, so for this article we are going to dig into The Molecule’s bread and butter: visual effects for TV series. In particular, the work they provided for Showtime’s The Affair, which had its season finale just a few weeks ago.

The Affair
Viewers of The Affair, a story of love, divorce and despair, might be surprised to know that each episode averages between 50 to 70 visual effects shots. The Molecule has provided shots that range from simple clean-ups to greenscreen driving and window shots — “We’ll shoot the plates and then composite a view of midtown Manhattan or Montauk Highway outside the car window scene,” says DiTommaso — to set extensions, location changes and digital fire and rain.

One big shot for this past season was burning down a cabin during a hurricane. “They had a burn stage so they could captFire-stageure an amount of practical fire on a stage, but we enhanced that, adding more fire to increase the feeling of peril. The scene then cuts to a wide shot showing the location, which is meant to be on the beach in Montauk during a raging hurricane. We went out to the beach and shot the house day for night — we had flicker lighting on the location so the dunes and surrounding grass got a sort of flickering light effect. Later on, we shot the stage from a similar angle and inserted the burning stage footage into the exterior wide location footage, and then added a hurricane on top of all of that. That was a fun challenge.”

During that same hurricane, the lead character Noah gets his car stuck in the mud but they weren’t able to get the tires to spin practically, so The Molecule got the call. “The tires are spinning in liquid so it’s supposed to kick up a bunch of mud and water and stuff while rain is coming down on top of it, so we had our CG department create that in the computer.”

Another scene that features a good amount of VFX was one that involved a scene that took place on the patio outside of the fictitious Lobster Roll restaurant. “It was shot in Montauk in October and it wasn’t supposed to be cold in the scene, but it was about 30 degrees at 2:00am and Alison is in a dress. They just couldn’t shoot it there because it was just too cold. We shot plates, basically, of the location, without actors. Later we recreated that patio area and lined up the lighting and the angle and basically took the stage footage and inserted it into the location footage. We were able to provide a solution so they could tell the story without having the actors’ breath and their noses all red and shivering.”

Lobster_Roll-before      Lobster_Roll-after

Being on Set
While on-set VFX supervision is incredibly important, DiTommaso would argue “by the time you’re on set you’re managing decisions that have already been set into motion earlier in the process. The most important decisions are made on the tech scouts and in the production/VFX meetings.”

He offers up an example: “I was on a tech scout yesterday. They have a scene where a woman is supposed to walk onto a frozen lake and the ice starts to crack. They were going to build an elaborate catwalk into the water. I was like, ‘Whoa, aren’t we basically replacing the whole ground with ice? Then why does she need to be over water? Why don’t we find a lake that has a flat grassy area leading up to it?’ Now they’re building a much simpler catwalk — imagine an eight-foot-wide little platform. She’ll walk out on that with some blue screens and then we’ll extend the ice and dress the rest of the location with snow.

According to DiTommaso being there at the start saved a huge amount of time, money and effort. “By the time you’re on set they would have already built it into the water and all that stuff.”

But, he says, being on set for the shoot is also very important because you never know what might happen. “A problem will arise and the whole crew kind of turns and looks at you like, ‘You can fix this, right?’ Then we have to say, ‘Yeah. We’re going to shoot this plate. We’re going to get a clean plate, get the actors out, then put them back in.’ Whatever it is; you have to improvise sometimes. Hopefully that’s a rare instance and that varies from crew to crew. Some crews are very meticulous and others are more freewheeling.”

Tools
The Molecule is shooting more and more of their own plates these days, so they recently invested in a Ricoh S camera for shooting 360-degree HDR. “It has some limitations, but it’s perfect for CG HDRs,” explains DiTommaso. “It gives you a full 360-degree dome, instantly, and it’s tiny like a cell phone or a remote. We also have a Blackmagic 4K Cinema camera that we’ll shoot plates with. There are pros and cons to it, but I like the latitude and the simplicity of it. We use it for a quick run and gun to grab an element. If we need a blood spurt, we’ll set that up in the conference room and we’ll shoot a plate.”

The Molecule added John Hamm’s head to this scene for Unbreakable Kimmy Schmidt.

They call on a Canon 74 for stills. “We have a little VFX kit with little LED tracking points and charts that we bring with us on set. Then back at the shop we’re using Nuke to composite. Our CG department has been doing more and more stuff. We just submitted an airplane — a lot of vehicles, trains, planes and automobiles are created in Maya.”

They use Side Effects Houdini for simulations, like fire and rain; for rendering they called on Arnold, and crowds are created in Massive.

What’s Next?
Not ones to be sitting on the sidelines, The Molecule recently provided post on a few VR projects, but their interest doesn’t end there. Chris Healer is currently developing a single lens VR camera rig that DiTommaso describes as essentially “VR in a box.”