Tag Archives: Adobe Character Animator

Konee Rok creates animated DJ Aktive The City music video on his own

By Randi Altman

Filmmaker Konee Rok likes helping to promote the work of artists. Rok, who specializes in making music videos, animation and documentaries featuring athletes, musical artists and dancers, recently created an animated music video for DJ Aktive called The City, featuring Common, Freeway, Bri Steves and DJ Jazzy Jeff.

Konee Rok

The video begins with spinning heads of the featured artists transitioning to the spinning of an LP by DJ Aktive. We then see rapper Freeway moving from one upside-down city building to another, like a kid working the handlebars at a playground. The viewer is then taken on a tour of Philadelphia, with locations including the Liberty Bell and animated recreations of the Eagles winning the Super Bowl. Another Philly artist, Bri Steves, raps some more about the City of Brotherly Love, and that transitions to Common in his hometown of Chicago. Finally, there is Jazzy Jeff, spinning the globe like a turntable. It’s all very bright, colorful and cinematic.

Rok completed The City, from start to finish, by himself during a two-month period. He relied heavily on Adobe’s Character Animator and the rest of the Adobe suite. We reached out to him to find out more.

How early on did you get involved in the video? Concept stage?
I was contacted to make the video by Michael McArthur, who manages DJ Aktive, who is an incredible DJ and producer from Philly. He spins for Janet Jackson, Alicia Keys, Nas, Kanye, Common, Puffy and more. Mike sent me Aktive’s album, which is composed of songs he produced with Ivan “Orthodox” Barias and features many artists Aktive has worked with. I chose the song “The City” featuring Common, Freeway, Bri Steves and DJ Jazzy Jeff as a standout. They also wanted me to work on that song. They left it up to me from there. With that, I went ahead and made the video. So, I was involved as early as possible.

Assuming you did previz/storyboards? Can you talk about this part of the process?
I normally listen to a song and storyboard a video as quickly as I can think, almost as if I’m “writing” down pictures instead of words. It’s an organic process, and I think it brings the most truth to a presentation because it’s unfiltered. I generally try to stay as close as I can to the initial ideas unless something is undeniably better or it’s technically necessary to make a change.

Can you walk us through how you created the lipsync, face-tracking and movements? How did you decide this would be the best way to tackle all of that, and how challenging was this?
I designed and drew multiple head positions in Adobe Photoshop for each vocal artist: Common, Freeway and Bri Steves. These were in forward-facing views, three-quarter views and profile views. For each of those views, I drew 14 different mouth positions. That’s a lot of mouths. Then, I brought those into Character Animator and rigged them so they moved how I wanted. After that, I laid down the acapella to the song and recorded the heads moving their mouths to the track using Character Animator’s audible lipsync method.

Because it was rap audio and not normal speech audio, I had to spend some time doing specific tweaks, so the mouth movements matched with the vocals. Once the lipsync was right, I used my Mac’s camera to motion track their head turns and expressions using my own head and face. For different scenes, I performed specific movements and exported them out of Character Animator as Mov video files with an alpha channel. Next, I brought them into Premiere Pro to stitch it together.

I love the section of the video that gives a nod to the game Operation. Can you talk about how that idea came about?
Thanks! Yeah, that was a fun one. My intention was to tell Freeway’s actual story quickly and lightly. In real life, he had kidney failure and received a transplant — a serious thing, no doubt. I thought, what’s the most efficient and appealing way to get this across to the audience? Since the board game Operation is ubiquitous across generations, when you watch the video, you get it right away, and because it’s unexpected, it demands a smile. It’s a great example of a best-case-scenario marriage between information and entertainment.

A funny thing about that scene is that a few days before the video was released, Freeway shared an Instagram video of himself at a kidney health awareness event playing with a life size Operation-style game. Crazy. On top of that, there’s another animated scene wherein Freeway is hammering a bell at the 76ers arena to open an NBA game, a new tradition for Philly natives before each game. That was imagined months before, and then literally the last 76ers game before coronavirus shut down the NBA, Freeway was asked to ring the bell, and he shared it on social media right before the music video release. All coincidence! Life imitates art imitates life.

How closely did you work with DJ Aktive? How often were you showing him scenes? Was that done remotely?
It was entirely remote. I was at home in Chicago, and Aktive was DJing in Las Vegas for Janet Jackson. I sent a few behind the scenes of unfinished animated line art but kept the video mostly a mystery. I made that choice because the way the video is structured; it lends itself to be viewed as a whole for maximum impact instead of spoiling it in bits. Thankfully, he was patient and trusting that I was doing something he would like. It worked out because I was not asked to change anything, and Common, Freeway, Bri Steves and DJ Jazzy Jeff all loved the video.

Had you used Character Animator before, and how did this tool allow you to accomplish this all yourself and in two months’ time? And what about the other Adobe tools?
I used Character Animator sparingly on a documentary we premiered at Cannes called Beyond Driven, directed by Vincent Tran and Riyaana Hartley. This is where I started to become familiar with it. That experience encouraged me to dive in fully and alter my traditional workflow to incorporate it.

Committing to it for this project forced me to learn more, which I always try to do with every project. Motion capture and lipsync automated a lot of the parts that I normally do manually, which made it more efficient in many ways. There’s also a unique quality of performance you get in the characters which I really like.

Premiere is where I structure the presentation. Photoshop is where I drew and developed the ideas for the final visuals. There’s heavy usage of the “animated timeline” in Photoshop and great attention to framing and timing in Premiere Pro. The back and forth between them is a sweet relationship. I would say they are more than friends.

Had you ever created anything like this before?
I wouldn’t say I created anything like The City before, necessarily, because I always want to do something new. It’s definitely a cousin of other projects. Right now, a video I did for Avery Sunshine called I Got Sunshine comes to mind when I think about a project I’ve done that also has continuous movement from start to finish. You can check that out on my site.

You’ve done both live-action and animated projects. How do you decide which way to go and can you talk about what you like about working in both areas?
Since I love both, I do a lot of live-action/animation hybrid videos. What I love about live action is the adventures you have working with other people in a physical environment. It’s very challenging and there is always a story about overcoming struggles.

Whenever I get together with my cinematographer, Jason “Intel” Deuchler, we’re always laughing about crazy situations we’ve been in. I love that. Deciding on live or animation is sometimes about what someone is asking for in a project, or other times it’s simply about logistics. I originally wanted The City to be live action, but it was too difficult to get everyone in one place. Oftentimes, it’s also about sensibility; what does this song feel like it should be visually?

Finally, any words of wisdom for young creators just starting out?
Strive to do what you haven’t seen. Always try something that you don’t know how to do so you force yourself into a corner where you have no choice but to figure it out. Keep working even when it seems like it’s not going well. You can gain a lot of satisfaction if you go by those affirmations. I know I have.

Randi Altman is the founder and editor-in-chief of postPerspective. She has been covering production and post production for more than 20 years. 

Making an animated series with Adobe Character Animator

By Mike McCarthy

In a departure from my normal film production technology focus, I have also been working on an animated web series called Grounds of Freedom. Over the past year I have been directing the effort and working with a team of people across the country who are helping in various ways. After a year of meetings, experimentation and work we finally started releasing finished episodes on YouTube.

The show takes place in Grounds of Freedom, a coffee shop where a variety of animated mini-figures gather to discuss freedom and its application to present-day cultural issues and events. The show is created with a workflow that weaves through a variety of Adobe Creative Cloud apps. Back in October I presented our workflow during Adobe Max in LA, and I wanted to share it with postPerspective’s readers as well.

When we first started planning for the series, we considered using live action. Ultimately, after being inspired by the preview releases of Adobe Character Animator, I decided to pursue a new digital approach to brick filming (a film made using Legos), which is traditionally accomplished through stop-motion animation. Once everyone else realized the simpler workflow possibilities and increased level of creative control offered by that new animation process, they were excited to pioneer this new approach. Animation gives us more control and flexibility over the message and dialog, lowers production costs and eases collaboration over long distances, as there is no “source footage” to share.

Creating the Characters
The biggest challenge to using Character Animator is creating digital puppets, which are deeply layered Photoshop PSDs with very precise layer naming and stacking. There are ways to generate the underlying source imagery in 3D animation programs, but I wanted the realism and authenticity of sourcing from actual photographs of the models and figures. So we took lots of 5K macro shots of our sets and characters in various positions with our Canon 60D and 70D DSLRs and cut out hundreds of layers of content in Photoshop to create our characters and all of their various possible body positions. The only thing that was synthetically generated was the various facial expressions digitally painted onto their clean yellow heads, usually to match an existing physical reference character face.

Mike McCarthy shooting stills.

Once we had our source imagery organized into huge PSDs, we rigged those puppets in Character Animator with various triggers, behaviors and controls. The walking was accomplished by cycling through various layers, instead of the default bending of the leg elements. We created arm movement by mapping each arm position to a MIDI key. We controlled facial expressions and head movement via webcam, and the mouth positions were calculated by the program based on the accompanying audio dialog.

Animating Digital Puppets
The puppets had to be finished and fully functional before we could start animating on the digital stages we had created. We had been writing the scripts during that time, parallel to generating the puppet art, so we were ready to record the dialog by the time the puppets were finished. We initially attempted to record live in Character Animator while capturing the animation motions as well, but we didn’t have the level of audio editing functionality we needed available to us in Character Animator. So during that first session, we switched over to Adobe Audition, and planned to animate as a separate process, once the audio was edited.

That whole idea of live capturing audio and facial animation data is laughable now, looking back, since we usually spend a week editing the dialog before we do any animating. We edited each character audio on a separate track and exported those separate tracks to Character Animator. We computed lipsync for each puppet based on their dedicated dialog track and usually exported immediately. This provided a draft visual that allowed us to continue editing the dialog within Premiere Pro. Having a visual reference makes a big difference when trying to determine how a conversation will feel, so that was an important step — even though we had to throw away our previous work in Character Animator once we made significant edit changes that altered sync.

We repeated the process once we had a more final edit. We carried on from there in Character Animator, recording arm and leg motions with the MIDI keyboard in realtime for each character. Once those trigger layers had been cleaned up and refined, we recorded the facial expressions, head positions and eye gaze with a single pass on the webcam. Every re-record to alter a particular section adds a layer to the already complicated timeline, so we limited that as much as possible, usually re-recording instead of making quick fixes unless we were nearly finished.

Compositing the Characters Together
Once we had fully animated scenes in Character Animator, we would turn off the background elements, and isolate each character layer to be exported in Media Encoder via dynamic link. I did a lot of testing before settling on JPEG2000 MXF as the format of choice. I wanted a highly compressed file, but need alpha channel support, and that was the best option available. Each of those renders became a character layer, which was composited into our stage layers in After Effects. We could have dynamically linked the characters directly into AE, but with that many layers that would decrease performance for the interactive part of the compositing work. We added shadows and reflections in AE, as well as various other effects.

Walking was one of the most challenging effects to properly recreate digitally. Our layer cycling in Character Animator resulted in a static figure swinging its legs, but people (and mini figures) have a bounce to their step, and move forward at an uneven rate as they take steps. With some pixel measurement and analysis, I was able to use anchor point keyframes in After Effects to get a repeating movement cycle that made the character appear to be walking on a treadmill.

I then used carefully calculated position keyframes to add the appropriate amount of travel per frame for the feet to stick to the ground, which varies based on the scale as the character moves toward the camera. (In my case the velocity was half the scale value in pixels per seconds.) We then duplicated that layer to create the reflection and shadow of the character as well. That result can then be composited onto various digital stages. In our case, the first two shots of the intro were designed to use the same walk animation with different background images.

All of the character layers were pre-comped, so we only needed to update a single location when a new version of a character was rendered out of Media Encoder, or when we brought in a dynamically linked layer. It would propagate all the necessary comp layers to generate updated reflections and shadows. Once the main compositing work was finished, we usually only needed to make slight changes in each scene between episodes. These scenes were composited at 5K, based on the resolution off the DSLR photos of the sets we had built. These 5K plates could be dynamically linked directly into Premiere Pro, and occasionally used later in the process to ripple slight changes through the workflow. For the interactive work, we got far better editing performance by rendering out flattened files. We started with DNxHR 5K assets, but eventually switched to HEVC files since they were 30x smaller and imperceptibly different in quality with our relatively static animated content.

Editing the Animated Scenes
In Premiere Pro, we had the original audio edit, and usually a draft render of the characters with just their mouths moving. Once we had the plate renders, we placed them each in their own 5K scene sub-sequence and used those sequences as source on our master timeline. This allowed us to easily update the content when new renders were available, or source from dynamically linked layers instead if needed. Our master timeline was 1080p, so with 5K source content we could push in two and a half times the frame size without losing resolution. This allowed us to digitally frame every shot, usually based on one of two rendered angles, and gave us lots of flexibility all the way to the end of the editing process.

Collaborative Benefits of Dynamic Link
While Dynamic Link doesn’t offer the best playback performance without making temp renders, it does have two major benefits in this workflow. It ripples change to the source PSD all the way to the final edit in Premiere just by bringing each app into focus once. (I added a name tag to one character’s PSD during my presentation, and 10 seconds later, it was visible throughout my final edit.) Even more importantly, it allows us to collaborate online without having to share any exported video assets. As long as each member of the team has the source PSD artwork and audio files, all we have to exchange online are the Character Animator project (which is small once the temp files are removed), the .AEP file and the .PrProj file.

This gives any of us the option to render full-quality visual assets anytime we need them, but the work we do on those assets is all contained within the project files that we sync to each other. The coffee shop was built and shot in Idaho, our voice artist was in Florida, our puppets faces were created in LA. I animate and edit in Northern California, the AE compositing was done in LA, and the audio is mixed in New Jersey. We did all of that with nothing but a Dropbox account, using the workflow I have just outlined.

Past that point, it was a fairly traditional finish, in that we edited in music and sound effects, and sent an OMF to Steve, our sound guy at DAWPro Studios http://dawpro.com/photo_gallery.html for the final mix. During that time we added other b-roll visuals or other effects, and once we had the final audio back we rendered the final result to H.264 at 1080p and uploaded to YouTube.

Mike McCarthy is an online editor/workflow consultant with over 10 years of experience on feature films and commercials. He has been involved in pioneering new solutions for tapeless workflows, DSLR filmmaking and multi-screen and surround video experiences. Check out his site.