IDLES Reimagines Iconic Coldplay Video Using AI

British rock band IDLES has unveiled a surprise collaboration, bringing together the video from Coldplay’s classic song ‘Yellow’ with IDLES’ new single ‘Grace’.

Commissioner John Moule approached production studio Joyrider with the project - posing the challenge of taking the renowned ‘Yellow’ music promo (which was released in 2000) and making it look like Chris Martin was singing IDLES’ new track.

Research and development was performed by Joyrider’s director Jonathan Irwin, who utilised various AI tech solutions to land on the best way of making Chris Martin look like he was actually singing ‘Grace’, whilst maintaining the integrity of Chris’s original performance.

Initial tests on the mouth, face and full head replacement tests were trialled using AI tools such as wac2lip, which generated mouth shapes based on audio inputs in Stable Diffusion. However, this didn’t deliver the consistency and quality Joyrider required: Chris Martin sings front and centre for the entire duration of the track without cutting away, so there could be no compromise on the quality or convincingness of the video.

The team then moved to an approach that used a filmed mouth singing the song, which the AI Deepfake training then used, before this pre-trained face went through around one million iterations to get to a very decent level.

The mouth had to be synced together seamlessly with the original video, augmenting Chris’ head shape and manipulating neutral faces in order to apply new mouth positions. To achieve this, Joyrider approached VFX specialists, Stone Dogs, to harness their flame compositing expertise. In a cyclic turn of events, Brian Carbin, senior VFX artist and co-founder of Stone Dogs who had worked on the original ‘Yellow’ video, was part of the team crafting this re-imagined iteration.

Wanting to make the video look more like Chris did over 20 years ago, the team were able to take the production to the next level and film the “real” Chris Martin in the band’s studio, singing the song at double speed (50fps) with three 4K cameras. The video was then played back and slowed down, adding an additional layer of subtlety to match the original plate. The filmed material was then Deepfaked further, helping to deliver a convincing AI performance of Chris Martin, filmed as if it actually happened in 2000.

The VFX team of Brian Carbin, Dave Kiddie, Rufus Blackwell and Danny Coster were so deeply immersed in the process that they learned the differences between the nasolabial crease, philtrum, cupid’s bow, upper lip, lateral commissure, vermillion border and the lower lip - even finding themselves on occasion singing the lyrics whilst looking at a mirror in order to mimic the phenoms needed for convincing animated lip sync.

Jonathan Irwin, director and AI technical lead, at Joyrider, led weeks of further research and development and refining machine-learning approaches saying, “To create the training set, we fed every frame of the original video into DeepFaceLab and ran for five million iterations, approximately one and a half months of processing time, running 24/7. At each stage of improvement, there would be backwards and forwards of new AI training runs and flame VFX work, tracking, extensions, clean up, more tracking, test composites, more AI, more clean up etc.”

In edit, there was a further challenge of how to cut the original one shot ‘Yellow’ promo into a shorter IDLES track, without losing the opening sea at night setting or the bright walking away end scene.

Careful timing adjustments were needed to find the best fit for the IDLES lyrics to match Chris’ delivery as the footage is all in one take. A time/speed ramp effect was applied to a section of the film to bring the ‘Yellow’ and ‘Grace’ promos in sync. This speed effect was in keeping with the original creative narrative of time advancing. Additional time warp effects were positioned over certain points of the video, to take the viewer away from the original Coldplay ‘Yellow’ video and into an IDLES-esque world.

The grade was undertaken by Stone Dogs’ colourist Mark Meadows, who seamlessly blended in the new deep fake faces with the original footage, matching flesh tones and grain without affecting the original promo. The video begins with the beach and sky being somewhat dark until sunlight rises at around the video’s midpoint, therefore the new AI derived face had to match the changing colour grade.

Original video for ‘Yellow’ by Coldplay directed by James Frost and Alex Smith. Special thanks to directors James Frost and Alex Smith and the rest of the crew behind the original Yellow video, Coldplay and their management team.

Contains a sample from the music video for ‘Yellow’ as performed by Coldplay, licensed courtesy of Parlophone Records Limited.