Your Shot: Ryan Staake’s Outlandish Ode to the VFX Breakdown for Ed Sheeran

Ryan> I’ve always been interested in the look of a motion capture stage as a location, there’s something so visually pleasing about the austere minimalism, pragmatic colours of tape on black, and reflective balls on the suits. I’m a huge fan of VFX breakdowns and have always geeked out on the look of the various passes, the bones, the pre-renders, the lighting passes, etc. It’s such a clear way of showing the step-by-step process of something being created.

Ryan> I honestly didn’t think he’d be interested in this idea and was incredibly happy when Riff Raff & Joceline Gabriel told me he was. It’s a bit of a fringe idea for anyone, and the fact that a pop powerhouse like him was into it was incredibly exciting. Lyrically, I was playing off of the “cross me cross her” line, almost re-interpreting its meaning to play with the idea of crossing from one reality to another, or one person controlling another. It’s a bit of a stretch, but I think it plays well. Furthermore there’s the fact that the lyrics are about important women in Ed & Chance’s lives, and it was interesting to give a female dancer the power to quite literally control male performers.

Ryan> MPC used every tool in their toolkit on this, which was fascinating to see. I'll ramble for a bit… we tracked both Courtney and the Steadicam rig within the motion capture volume, which allowed editor Sam Bould and I to cut from live action to CG at any time and have a pretty close match cut. As far as scene modeling, the scenes were created in a mix of Maya, Houdini, Cinema4D, Unreal Engine, and probably software I was never even told about. Full body models of Ed, Chance and PNB Rock were created from photographic reference, a bit of stock and a trip to the Ed wax statue at Madame Tussauds. We did a bit of secondary lip sync using an iPhone X as a facial tracking camera to give a bit of life to the models. We made a trip to visit Ed on the set of another video and took more photo references of his arm tattoos because we couldn’t find good enough ones online. Tons of different wireframe/process passes were rendered out to allow us to glitch between different looks in Flame in both the performance sequence and the fourth-wall breaking ending sequence. Rumour has it that the bloated project files were beginning to crash Flame by the time we were delivering.

Dominic> Cross Me is one of those examples where timing played a big part. It was the first time I had worked with Ryan and flame op Zdravko Stoitchkov so we set ourselves up in a suite and bashed over Ryan's brief and thumbnails and the creative juices were just allowed to flow - suffice to say we got on well. There was literally no interference at this stage and I think this was partly because Ryan's idea was so whacky and experimental, making it harder for someone to judge the work at an early stage. It was very much a case of trusting the process. 

Usually when working on an advert or film, it’s quite clear how you go about creating a visual effect. Most of the time it’s something realistic and clearly plotted in storyboards or moodboards, so it can become second nature. In this case we really did have a chance to play in an open sandbox and throw in ideas and solutions given we were on a very tight deadline. It was like being back at art school and having the chance to experiment, only this time with top talent at your disposal. Ryan clearly knows what can be done in this growing, technically driven art form and that a little trust in the artists will allow an idea to grow and find its feet. This is quite rare and to some risky but the results speak for themselves.

Dominic> This year has seen MPC undertake quite a bit of character work with Elton John's Christmas ad, Cross Me and another very big character job currently in production. To that end we are developing new techniques to achieve detailed lip sync and character animation. In this case, to achieve the level of lip sync we thought we'd need, we used an in-house app developed for the iPhone that takes the technology for the 3D emoji tool and re-purposes it to capture the data of a moving face. That data is then fed into our face rig tool which has over 52 expressions. The face rig tool then uses the captured data to figure out which expressions need to be triggered and how much of each should be dialed in. 

The icing on the cake is that we are able to transfer these expressions onto different heads, which in this case was very necessary. From an animation perspective the motion capture was very clean so we spent only a week in clean-up mainly fixing arms, fingers, legs and the odd neck movement. The camera was however not tracked to a level that would work in production so we had to rely on our talented camera tracking team to track a new camera, which meant nine minutes of tracking as the edit is made up of mainly three different takes. The 3D was developed in Houdini, Maya and Cinema4D which meant we had to make sure we had our three hero characters working in both Houdini and Maya and that, when rendered, they looked the same. Having Arnold at our disposal made this process less painful. Caching out animation, adding lip sync and getting the timing right was certainly a massive undertaking.

Ryan> I’d created a second-by-second spreadsheet of the sequence with sketches and little descriptions of beats in the choreography as well as what we were planning to animate. I think everyone on set thought it was hilariously nerdy to block a dance with a spreadsheet, but it worked! In the end, a good amount of the animation evolved and changed as we started to see first passes in post. For example the ‘gold queen’ scene was initially going to be a massive queen bee, but it didn’t seem to match Courtney’s motion well enough so we went with the gold scene.

Dominic> Like most jobs we start with a live action edit and add in VFX requirements from there, and in this case, that was no different. I must say though, the edit of Courtney's dance was so good that we actually didn't want to put anything over it initially! Ryan's idea was that the video was meant to play like a single take, but there are a number of clever cuts in there to make the action more dramatic. We then started going through the edit, breaking down into internal shots, colour coding over the video wherever we thought we'd cut into CG. 

The edit then just slowly took on its own life. We'd update shots with Ryan's sketches or a reference image and then repopulate time and time again with new material, firstly with just the motion capture on a female CG model and then slowly with the CG work we were developing for each shot. This way, at the end of the day we could quickly start to see the development and progress we were making and whether it was working with the audio track. It was a very creative process, a lot of fun and it kept the momentum going - which was important given we had three-and-a-half weeks of production.

Ryan> I liked the idea that the characters and environments would kind of switch roles at times. For example the deflating gold figures at the beginning allude to the gold queen we’ll see later, the car that the muscle person lifts has a ‘little’ cameo later on, the coloured particles are both an environment and a character and then simply a character. Looking back, a lot of these scenes are visual ideas I’ve had rattling around in my head for some time and have finally spilled out into this visual trip. I did find that I was describing a lot of these scenes to the animators with movie references: “the heart pulses like in The Mask… the arrows all hit like in 300… it’s a mirrored dance duet like the end of Annihilation… the phones all fly onto him like the goo that goes down Neo’s throat in The Matrix… lasers like in Entrapment… motion lines like in Akira… etc.” 

Ryan> Erin Murray, our choreographer, did an incredible job of interpreting my ideas (and spreadsheet) into a tightly choreographed routine for Courtney. There was an incredible freedom at points where we could say, “okay she walks up stairs, and they’ll turn into… maybe a hill, maybe arms, maybe boulders, we’ll figure out the details later, we just need her to walk up something”. This allowed us to choreograph quickly, which was critical because we only had two days of rehearsal before shooting.

Ryan> I gave the reference of “a modern PC game running on a top of the line graphics card”. I think MPC’s team went beyond this to hit a sort of hyperreal look in which we made it clear that this wasn’t intended to be photoreal, yet had a ton of detail and humanity.

Ryan> I felt the performance idea for a full 3:30 would just be too long and drawn out and wanted to switch it up with something for the final third. This idea of what’s real/what’s fake seems to be one of the most important themes of our times (both in politics, media, science, climate, etc.) and I wanted to explore that idea loosely. It was also just a really fun way to take the story out of the mocap studio into a slightly more gritty, realistic world. We alluded to this idea of little CG remnants entering reality throughout the edit as well, a little playful foreshadowing.

Ryan> When Courtney is waiting for the subway, she sees someone on the train… guess who?! There are Ed related posters on the train. I wanted to add in a 3D scan of myself into the performance finale, but we ran out of time.

Ryan>  The label was very concerned over which moves we would show Ed doing versus Courtney. Luckily this format allowed us to elegantly ebb and flow from one to the other.

Dominic> By far the trickiest job from my perspective was dealing with all the data from the motion capture, the lip sync and the character rigs and managing this so that the artists could quickly and efficiently pick up caches and work without being slowed down or hindered by technical issues. This meant a lot of spreadsheets recording edit information and frame ranges and a lot of painstaking back and forth doing quick playblasts to check animation and lip sync alignment in the edit. Getting our CG characters to line up with the dancer was also a tricky process requiring a lot of manual edits to the CG scenes. 

We also needed a fast way to switch between characters so our character rigs had to be efficient in that we could reference in any of the characters whether the dancer, Ed or Chance the Rapper and they would all work seamlessly with the animation / mocap data. The trickiest character was definitely the rainbow particle lady and she was developed in Houdini which allowed us to control the movement and size of the particles but also how they should interact with the environment, be it colliding with Ed or falling with gravity.

Ryan> Our team at Pomp&Clout made a Snapchat AR filter to go along with the video and it was #5 on the Snap Lens community, you can download it here.

And here’s a video of Ed trying it:

Dominic> Cross Me is one of those rare projects in that it really allows artists to open up their tool box and explore the possibilities. It's these kinds of projects that push us into new directions so we are thankful for the opportunity and hope that more directors take note. VFX artists are not just technical operators, they are a very unique breed of creative artist and technician and they know so much about the entire filmmaking process, from camera lenses to composition and of course making the unimaginable. As the industry continues to develop and grow, it's my belief that this kind of artist will only become more instrumental. Thanks again to Ryan and RiffRaff and, of course, Ed Sheeran for trusting in us.