Thanks to today’s available technology, more and more artists are embracing experimental storytelling. One of those filmmakers is Bernie Su, creator, executive producer and director on the Twitch series, Artificial.
Artificial, which won Twitch its first Primetime Emmy for “Outstanding Innovation in Interactive Media, features a doctor and his “daughter — a human-looking artificial intelligence creation named Sophie. Episodes air live with actors reacting to audience input in realtime. This is later edited into clips that live on Twitch.
The unique live broadcast and “choose your own adventure” aspects of the show created a need for a very specific workflow. We reached out to Su to talk about the show, his workflow and his collaboration with show editor Melanie Escano.
Where did the idea for Artificial come from, and what was its path to its production?
The original story came from my co-creator Evan Mandery. When we partnered, we looked at how we could present it in an innovative way. We identified Twitch pretty early in the process as a place to really push some groundbreaking storytelling methods. What would an original series on Twitch look like? What makes it Twitch and not Amazon video (Amazon owns Twitch by the way)? Once we pitched it to Twitch it was all systems go.
Can you talk about what it’s shot on and how you work with your DP to get the look you were after?
We shot on Panasonic Lumix GH4 cameras. We kept it pretty simple. DP Allen Ho and I have worked together a lot, and we tried to make Artificial feel real-yet-polished. Because we were on Twitch and that’s a platform where everything is livestreamed, we wanted it to feel like a real livestream yet have touches of a cinematic look. Every scene we shot in our show… we would always discuss why the camera is even there in the first place and how the characters react to them. The show called Artificial had to feel immersive.
Were you at all intimated by the interactive aspect of the show?
Nervous but not intimidated. I’ve done several interactive shows, but the live element is a different animal. A live interactive series is built around chaos, and if you aren’t embracing that chaos that the audience is going to throw at you, then you shouldn’t be making a live interactive series.
What went into making this interactive? Can you talk about the challenges and how you overcame those?
Well the first step is figuring how you’re building the audience into the story while still maintaining an arc. The second is how you make that audience consequential. You can always let the audience choose something non-consequential, like should someone drink tea or coffee.
Yes the audience made a choice, but it’s not consequential to the story. Now when we have the audience choose what a character’s relationship to another will be, or even if a relationship will end or not, now you’re letting the audience play with fire, and once they do it’s our responsibility to honor the consequences of that. The simplest way I can describe our solution is that we as the storytellers accepted every result we presented. We dared the audience to play with fire and if they burned a character, then that’s the consequences.
Your team used Adobe Creative Cloud to make this a reality. Can you talk about that and how you worked with your editor and post team? How involved were you?
Oh yeah, Premiere, Photoshop, After Effects and Audition were all in play for us. We don’t have a big team, but we have an incredibly versatile team. Any of us could comfortably jump into several of those tools and be able to knock something out quickly. We were all about speed and efficiency.
Once we got the systems in place, I wanted to stay at a very high level and let my team play. I trust them, if I didn’t, they wouldn’t be on my team. Co-producer Jen Enfield-Kane worked closely with our editor Melanie Escano and our writer/sound editor Micah McFarland. They would go back and forth with cuts and mixes. Then upon approval, it would go to creative producer Rachel Williams, who could implement final effects for broadcast. If everything is going smoothly, then we’re good to go.
But because of the speed of weekly broadcasts consisting of 30 to 45 minutes of edited content a week, and the fact that the post-team is literally four people, there were many times that someone would have to assist, and that’s fine. That’s what a great team does.
How did you work with the editor, specifically? What was your workflow?
Pretty straight forward. When you’re innovating in the live format, you purposely make the post system as simple and easy as possible. If the show is chaotic, you don’t want your post to be that. Melanie would do a rough pass using the script supervisor notes, Jen would give her notes on that and she would come back with another cut. After that, they might discuss color correction on a particular scene and get that done but that was rare. We always kept it simple.
This interactive film’s editor talks challenges as well as how Netflix’s Branch Manager tech made it all possible.
By Karen Moltenbrey
In any film, or web/television series for that matter, the final presentation is the culmination of many choices. The director’s, the scriptwriter’s, the editor’s… just about everyone’s but the viewers. However, Netflix changed that with Black Mirror: Bandersnatch, a special interactive TV movie during which viewers are prompted to make selections that affect the decision-making and, ultimately, determine the fate of the main character: a young video game programmer.
Alas, while the viewer is tasked with making certain decisions at various intervals in the movie, that certainly did not mean the workload was any less for those on the project. In fact, they had to devise a plethora of paths that could be selected — so many, in fact, that a new tool, called Branch Manager, was devised and integrated into the workflow to maintain order and elegance to what could easily have become a tangled web on so many fronts.
Black Mirror, a British science-fiction series of stand-alone stories, mostly focuses on the consequences of new technologies. It was created by Charlie Brooker, who serves as showrunner along with Annabel Jones. The very first episode debuted in late 2011, and after four “series,” the pair introduced the interactive movie Black Mirror: Bandersnatch on December 28, 2018, in which reality and fantasy merge together for programmer Stefan Butler (Fionn Whitehead) as he adapts a choose-your-own-adventure type novel into a video game. Soon Butler’s life begins to resemble that of the tragic author, and he begins to break down mentally, slipping further down the rabbit hole as he tries to make a seemingly impossible release date for the game.
Within this storyline, viewers are tasked with making certain decisions, each leading them along various paths in the narrative. In addition, there are a number of possible story endings. Viewers have 10 seconds to make decisions or one is made for them. But, once a play-through ends, viewers can go back and make a different choice. According to Netflix, the average viewing time for the movie is 90 minutes.
With 150 minutes of unique footage divided into 250 segments, just about every aspect of production was impacted in some way, perhaps none more so than editing. And that task was given to Tony Kearns, a veteran editor who calls Bandersnatch “the biggest challenge of my editing career” thus far. In terms of production, Kearns estimates the shoot to have been two to three weeks longer than a regular Black Mirror episode, and the edit took five to seven weeks longer than a show of similar duration, lasting 17 weeks.
“When we started working on Bandersnatch, we realized we were doing something that none of us had done before by making an interactive movie — especially one of such complexity,” says Kearns. “I think everyone who had a major role in the production grasped quite early on the need to be very organized and to get our heads around the structure of the script and the segments, as well as the implications of a nonlinear storytelling based on viewers’ choices at every choice point.”
Then, as the group worked through the movie and began getting more footage in the can, “it was obviously clear we had to work out a way of keeping track of things and getting the right results from the edit and how we were constructing it,” he continues. “Having the Branch Manager software [developed in-house by Netflix] enabled us to watch the movie with the various choices, and while making them, seeing the implications for editing — particularly at the end of a segment and when starting another. That’s because you weren’t moving to just one thing; you were going to two things, and both had to work. Some of the segments had six variations, so you had to make sure they all worked. It was a novel experience and very intense. We had to be on our toes all the time.”
Bandersnatch is not Netflix’s first interactive show. In fact, the company has experimented with more simplified interactive, or “branching narrative,” children’s shows since 2017. However, Bandersnatch marked the first time it has done so for live action and for an adult audience — and to resounding success based on audience reaction. On the heels of that success, Netflix has followed up with the live-action interactive show You vs. Wild, putting viewers into the tracks of adventurer Bear Grylls as they make decisions for him while he tries to survive an adventure in the wilderness.
For Kearns, though, Bandersnatch was his first interactive “adventure.” (He is currently editing the Netflix drama The End of the F***ing World, Season 2.) He found the process “very, very different from a linear experience.” Making things even more daunting was the level of interactive complexity that was introduced in Bandersnatch. “We had no idea how it was going to be received. Would people become too frustrated, or would the emotional aspects of the story come through within all the choices?”
One of the biggest considerations was in terms of structure — making sure there weren’t too many recaps and that they balanced out with the story’s complexity, lest viewers give up on the movie. Another big focus was ensuring that the performances within this structure maintained the empathy, or humanity, that would keep viewers engaged and invested in the characters and story.
As a result, the nonlinear process fostered closer communication among the group, with script supervisor Marilyn Kirby and assistant director Jay Author invaluable on set, and a particularly crowded editing room. “That prevented us from going mad while trying to get our heads around things,” says Kearns.
While the editor and director always work closely on projects, at times director David Slade, executive producers Brooker and Jones, producer Russell McLean, assistant editor John Weeks and VFX editor Will Howden were all working together in the cutting room. “Everyone was contributing. It wasn’t that it made things difficult; it was essential and made things more interesting and exciting,” says Kearns.
As Kearns points out, a typical TV show, drama or film has a main cut and that’s it. Not so for Bandersnatch, which had segments that at times had upward of 14 cuts, all of which had to be tracked and organized.
The script was divided into eight sections, and each segment in those sections was assigned a four-character alphanumeric number, along with the corresponding variant. “The workflow was based on keeping track of the segments. We knew by the number which section of the script it belonged to,” Kearns explains. “The workflow was dependent on us keeping a record, spreadsheets. While editing, we had to know which was the latest version, or cut, because they were constantly being reworked. And the latest one went into Branch Manager to be viewed on our laptops. That was an important part of the workflow.”
Using Branch Manager, however, required some technical savvy, and helping the editing team navigate the software was assistant editor John Weeks, who just happened to be an experienced coder and worked his magic with the QuickTime files for each segment and ingested them into Branch Manager. “He was able to be so proactive and communicate with the engineers and developers at Netflix. He really took to this like a duck to water,” Kearns says. “I know it took up a lot of his time, but it ended up being essential for us in terms of making decisions for the edit and structure.”
Kearns would receive footage to a particular segment after each day of filming, do an assembly and then integrate it into the system. Then he would work on further edits as the segment progressed and as the structure was reworked and aspects re-aligned. “The numbering system for each segment was kind of the spine of the process and helped us keep track of what we were doing,” he adds. “You have to be prepared to pull things apart and reassemble them because the experience is different.”
For the movie, Kearns edited on Adobe Premiere, since it allowed him to open more than one sequence at a time. “It was essential to have more than one segment edit up at a time and switch between them just to see how [the segments] flowed,” he says. He also used Adobe Premiere for the VFX work.
The big star in terms of software on the project, though, was Branch Manager, developed by Netflix, which enabled the editing team to play with various options, choice points and timing, to ensure that the viewer was presented with the correct next segment based on the selection he or she had made. “You have a viewer’s experience, rather than looking at it on the edit system,” Kearns says of using the software. “We could view the movie in an interactive way on our laptops. We could see how the segments were working with each other, which was very useful.”
He explains: “We’d basically do a pass and watch it, make notes and adjust the edit accordingly, because sometimes you see things in isolation and think, ‘Oh, they’re working,’ and then using Branch Manager, we were able to see that, well, maybe they aren’t working so well. It was an essential platform. It made the process more fluid and creative, and easy to understand the structural and editorial aspects. We wouldn’t have been able to do this movie without it.”
According to Carla Engelbrecht, Netflix director of product innovation, her team met with Brooker and Jones in May 2017 and introduced them to the interactive storytelling technology, which, at the time, enabled Netflix to tell various interactive stories. A few months later, the pair returned with what would be the beginnings of Bandersnatch, “and we could see this was going to be a much more complex story than what we had previously done for our interactive titles for kids, which contained simple maps with just 15 or so choices, and each often led to maybe three different endings,” she says.
Not so with the plans for Bandersnatch. “The complexity of the stories Charlie [Brooker] wanted to tell, as well as the complexity of the stories adults can tackle in general, was partly what really drove us to create Branch Manager.”
Initially, the Bandersnatch scriptwriters began their process using Twine, an open-source interactive fiction engine, but it was easy to see that would not be sufficient for the planned complexity of Bandersnatch. This prompted Engelbrecht’s group to begin developing its own software. Moreover, production teams were developing their own mapping systems, often using spreadsheets for the interactive content. “We knew we could smooth this out and make the process easier for everyone by creating a common language so we could all be on the same page,” she explains.
As Engelbrecht notes, Branch Manager is a visualization tool that is used throughout the production process, from viewing an outline to creating a flowchart of the story structure, within which pieces of the outline are embedded as a script is formed. During the shoot, rough (or even fine) cuts are added to the software. “Then you can start watching it and experiencing all of the pieces, whether for continuity or choices.”
After a few months, the software was up and running and ready to be migrated over for use on the project. “That become the ongoing tool, as we used it during the rest of scriptwriting through the actual production and even into post production,” she says. “We were sort of beta-testing it on the fly [with Bandersnatch]. As we got script deliveries, we would also get notes on Branch Manager and on other features they wanted us to add.”
Engelbrecht points out that for some in production and post, the new workflow was seamless, involving “just more” — in essence, one big linear file. (She estimates that the final file ingested into the system is approximately five hours long due to the various options.) But for others, like the DP and actors, scenes had to be reshot with slightly different takes, and editing had to track and assemble those different options. “Throughout development and beyond, we had conversations and tried to be mindful of where problems could occur at the various stages. We wanted the software to be as minimally disruptive [to the production workflow] as possible, given what we were doing.”
While Netflix hasn’t specifically quantified the time-savings that Branch Manager brought to Bandersnatch, Engelbrecht notes that it was significant and allows for the telling of much more complex interactive stories.
Listen to Netflix and Black Mirror execs discuss how Branch Manager helped drive Bandersnatch’s production and innovation.
An Interactive Future
Kearns attributes the project’s success to the group carefully considering how it would approach the movie and managing to avoid major “teething” problems by making the right decisions along the way. He notes it was important, as well, to stay on top of what was going on at any given moment in terms of how a particular segments of story. “There were so many dimensions that, mentally, it was really taxing, but exciting as well,” he says. “I had to be able to recalibrate my editing brain to not think of the story overall, but rather from the point of view of individual segments, and keep them coherent.”
Looking to the future, Kearns expects an uptick in interactive projects but believes the key to their success — as evidenced with Bandersnatch — is to develop good scripts that suit the format, rather than trying to do it as a gimmick. He warns: “You really need people in important roles to be at the top of their game. It’s not for the faint of heart. And, you have to be prepared to make those tough decisions, which are made even tougher due to the nature of the interactive structure.”
He adds, “No matter how difficult your next job is, it is going to be so easy after Bandersnatch.”
Meanwhile, Engelbrecht’s team is working on improvements to Branch Manager. “On Bandersnatch, we were building the airplane as we were flying it,” she says. “We’re now moving into Version 2, better integrating the software with external tools to make the work even more seamless. We’re also looking to improve the onboarding experience to make the learning curve shorter, so it’s not like learning a new programming language. We want it to feel more drag-and-drop.”
For instance, the group has made Branch Manager compatible with Final Draft screenwriting software, enabling a script to be imported directly into Netflix’s tool. The team is still working on the interface. “We have a long wish list just pertaining to the visualization experience with the tools. And, we’re working on how to better integrate it on the other end, so when we ingest files into the system, the metadata flows from Branch Manager directly into our [production] system, whereas right now we still have to create a spreadsheet to negotiate part of the process.”
Thanks to Branch Manager, the team on Bandersnatch was able to negotiate a complex web of shifting directions. So, too, for the executives at Netflix, who are able to explore and more easily navigate new directions for content.
Karen Moltenbrey is a veteran VFX and post writer.