Compositing is the keystone of visual effects (to find out more, read our Complete Noob's Guide to Creative Compositing
) - and VFX breakdown videos make it look like magic. CG creatures, live action actors, a fantastical backdrop, delicately floating clouds, layers of moss and rust seem to whizz together of their own accord, gradually building an immersive world that feels real and cohesive.
The reality, though, is that while compositing can create magic, it’s a magic that requires skill, care, craft, a careful attention to detail and hard work. Following from last week’s introduction to the compositor’s art, it’s time to dig into the crunchy creative challenges and debates that compositors find themselves tackling on every job they get.
What We Do in the Shadows
As you become involved in more VFX-heavy projects as a creative, producer or marketer, you may find yourself transition from ‘bewildered noob’ to ‘interested outsider’. The craft is subtle and it’s layered - pun not entirely intended - and the challenges involved can be fascinating.
Perhaps the most mind-bending challenge that compositors have to tussle with is the question of shadows. At its purest, compositing is all about tricking the eye into believing that several objects that were not created together belong together - and the surest way to shatter that illusion is to overlook the lighting and shadows of each plate or object.
So much depends on what you’re working with. In some cases, the compositors have to confect shadows from nothing, which requires a fastidious understanding of the natural ways that shadows interact with the environment. And so, in some cases, it’s more convincing to extract and manipulate the pre-existing shadows - if all of the separate objects have been correctly supervised and lit during filming.
This is a method preferred by Alter Ego co-founder David Whiteson - though it also has its own challenges. “I try to maintain and keep the real shadows from the shot footage. Even if it means spending extra time keying, extracting and rotoscoping it. There’s a natural way that shadows react to an environment in real life that is really challenging to replicate. If I can salvage even a small portion of the real shadow I’ll always go that route first.”
David’s Alter Ego co-founder Darren Achim says that, as with most compositing decisions, the route taken often comes down to the kind of look the director desires and, of course, time and money.
“If you have real shadows that can be extracted, either through a key or roto would be the ideal scenario. Rebuilding non-existent shadows would require rotoscoping skewing tricks, or now with the advent of AI motion capture from video clips, you could ‘match move’ a CG double to then re-project accurate shadows in 3D. The trick used would depend on how much time and budget is at your disposal and the impact that the shadow has on the overall storytelling visual.”
Often, of course, compositors aren’t just dealing with one object’s shadow cast from one light source. Sometimes new shadows added by the compositor can interact with existing shadows in the footage in undesirable ways. According to Becky Clay at Untold Studios, it’s all a careful balancing act
“Shadows can be a pain! They are especially tricky if you have existing shadows in your plate and your shadow passes over them, you don't want to double up on the shadows there. You have to make a shadow clean plate, by turning the whole area into the same darkness as your ‘in plate’ shadows. You can do this with luma keys and roto and usually a bit of edge work. Then if the ground is uneven you might have to warp your shadow as it travels on the surface.”
Federico Quiros at La Posta Argentina agrees, “Pay special attention to interactions with real shadows, and take into account the possibility of more than one light source.”
But thankfully, new technologies - particularly those driven by AI - are making it easier for compositors to make shadows that are more realistic - or, indeed if the job calls for it, more creative and stylised. Duncan Malcolm, head of 2D and creative director at Glassworks London reflects on these old methods and how artists can combine a whole palette of techniques nowadays.
“It used to be a case of making them up from shapes and animating those shapes over objects in the scene. In fact, rotoscoping people and flipping the matte to make a shadow is still viable in some circumstances. Luckily, the sophistication of compositing software has moved on a great deal, with the inclusion of 3D scenes, geometry and tracked cameras,” he says. “Where shadows are heavily featured and falling across objects in a scene, the compositor can rebuild an approximate model of the scene and use that to deform the shadows. This can get very time consuming with complex shapes and so we use this to get the general layout, then adopt hand-painting techniques to get around building very detailed CG models. As always, the technique adopted depends on so many variables and often we use many techniques to get the final result.”
And as with many of these sorts of challenges, prevention is so much better than a cure. That means that compositors believe in open and early communication, particularly with the lighting team.
Michael Baker, compositing supervisor at Framestore explains, “Shadows are tricky and actually have a bit of complexity to them. Depending on the type of light they can have all sorts of different levels of density and softness and they always have a colour to them (which is often overlooked). Good communication with your lighting team and constantly studying references, either from something real in the shot or an image with the look you are after, is how we get them looking right.”
When a project involves a live action shoot, the shadow work starts on set. In fact, marketers and clients may have seen VFX supervisors prepping shadows during a shoot without even realising it.
“If we are compositing CG elements then the CG team is able to create accurate shadows from the information we collect on set,” explains Jim Allen, creative director, VFX at No.8. “That camera, iPad and shiny ball we carry around on set is performing a function!”
Moreover compositors constantly consult reference shots taken on set as well studying the original plates closely. Liciani Vargas, composting supervisor at Rodeo FX agrees. “Understanding how the lighting was done on set is fundamental to recreating the right look. We need to analyse the type of light that was used, whether it creates a hard or soft light, the amplitude, the direction, the tone it creates. This way, we will understand all the aspects that the shadow needs to have.”
Ultimately, building shadows into a scene - and being able to judge exactly what’s working and what isn’t - is a muscle that’s built up through time and experience.
“If you have a reference shot in the same lighting, then that is a great place to start,” says Joe Tang, senior Flame artist at Absolute Post in London. “If you can match that as close as possible, then you won’t be far off. Otherwise, look at the direction of the lighting and find other references of how the shadows may look in a similar scene. Finally, trust your eyes and the eyes of others, as they will usually tell you if it doesn’t quite look right.”
As the challenge of shadows demonstrates, compositors are meticulous observers and they love to use references as much as possible to ensure that the image they’re creating is believable and convincing. However, even when working on a project that’s based on a true story, exact references aren’t always possible. In that case, compositors have to rely on their own imagination and logic to figure out how various elements might interact within a scene.
That’s what Becky Clay at Untold found when working on the hit Netflix series ‘The Crown’. “I think the shots that you don't see for real anywhere or have very little reference to find online are the trickiest. Take an underwater explosion, like the one we did for The Crown’s 'Gold Stick' episode. You would never see one in real life and live long enough to comp it. You have to piece together bits of reference that you can find from other sources, and often it's taking little bits of inspiration from similar situations and mashing them altogether. We referenced above ground explosions and miniature underwater charges going off, then imagined them scaled.”
As VFX is a collaborative process that brings together multiple disciplines, communication is key to make sure that everyone’s imagination is running along roughly the same lines, Becky continues. “If your shot involves a lot of disciplines, FX, DMP [digital matte painting], lighting and comp, for example, and you're doing something quite abstract, getting concept work done up front can get everyone on the same page very quickly and give everyone something visual to work towards, because my idea of what an underwater explosion looks like could be very different to the next person’s. I think these are the trickiest but also the most fun shots to work on.”
But this creative freedom also allows artists to get cheeky and playful in unexpected ways, as Jessica Bagby, head of the roto prep department at Rodeo FX explains. “Recreating something from scratch [is an interesting challenge] because this is where you can be creative and have fun. Like having to place text or graffiti in the background but you don’t know what the writing was in the first place… you make it up! I put my cat’s name once. This is how you end up with Easter eggs.”
Where things get even trickier is when trying to build fantastical, magical worlds. How do shadows work on planets with three suns? How would our green-screened hero’s enchanted sword and shield reflect and respond to the evil sorcerer’s magic spell? Perhaps you’re working on a shot in an alternative dimension where the laws of physics operate completely differently. It sounds like fun - and it is - but the more detached from reality a shot is, the more the compositor has to make their way through countless obscure questions, all in service of weaving an illusion for the viewer.
Genevieve Camilleri,VFX supervisor with Alt VFX, says that these works of fantasy have an element of subjectivity and underline the artistic side of compositing.
“One of the greatest challenges of composting is when it comes to working with make-believe characters and environments. As there is no real life reference for these types of projects, it does heavily rely upon your own imagination and interpretation of what you think these characters and environments may look like. There is never a right or wrong approach so the outcomes are endless as each individual has their own unique interpretation.”
From the fantastical to the mundane, perhaps the most challenging and yet most overlooked facet of compositing is that which has been created not to be noticed. It’s inevitable that when things have been created to be invisible that we won’t see them, but it does mean that compositors (and indeed the wider community of VFX artists) often don’t get the praise and recognition for the their most nuanced and delicate work.
“Everyone thinks about blockbuster comic book visual effects extravaganzas when they think about VFX, but often overlooked is the art of invisible VFX,” says Nick Whiteley, VFX and compositing supervisor at The Embassy. “Adding, or removing, something from a plate or adding a subtle effect, yet one which is important to the story, takes a lot of techniques and technical skill. There is a real craft in being able to really look at the plate, to peer into all of the imperfections and nuances, and apply that to your work, especially on long lingering or close-up shots. Subtle lens effects, tiny camera vibrations and disturbances. What does the image do at the side or the edge of the frame? How does that react as the camera pans? The aim of all of this is for the audience not to notice. In many ways, big VFX shots with lots of CG elements are easier.”
The Filth and the Fury of Compositing
Perhaps the most delicate challenge is figuring out just how much to dirty up a scene. In many cases, compositors will add layers of texture to make objects feel more embedded in a scene, as well as more lived in. This could be smears on a window, clouds of dust around a horse’s hooves or layers of grime to turn a recently-built set into a worn down, post-apocalyptic medical facility.
“Reality isn't clean, it's full of imperfections. If you want to give realism, the shot must have some ‘dirty’ spots, scratches, bumps and asymmetries,” says Liciani Vargas. “Some aspects of the lenses can also help to break the perfect look from the CG elements, like having chromatic aberration and distortions.”
David Whiteson at Alter Ego shares an example of a particuarly dynamic shot that a bit of noise and grime really added some believability to. “I recently composited a shot where the camera would fly from the exterior of a car, through the front windshield to the back seat. We shot the car without any windshield and created that portion in CG. Although it looked great, it was too clean and perfect. We needed to add some glares, dirt, and a milkiness to the window so that we could sense the glass. Our world is full of imperfections that our eye sees and if those imperfections are not in the composite, the viewer subconsciously notices that in a negative way.”
Moreover, compositors may, if appropriate, build in effects that change how the footage itself looks, adding film grains or distortions or JJ Abrams’ favourite, the lens flare. Sometimes it’s about ensuring that a digital object matches the particular quirks of the original footage as it was shot in-camera by mimicking flaws - though in other occasions it’s about trying to evoke a particular style or aesthetic.
Nick Whitely at Embassy explains, “It should start by looking for the imperfections, usually inherent in the lens, that are in the original plate. Chromatic aberration, lens halation, colour bloom, light-wrap, light scattering, lens flare and light leaks.”
“These days, modern lenses attempt to reduce a lot of these imperfections and some effects are more pronounced when shooting on film, however there is a definite trend towards using ‘vintage’ anamorphic lenses just to get back some of these desired optical imperfections,” Nick continues, while warning that it can be tempting to splash around those effects with wanton abandon. Nick cautions that usually less is more. “There is a tendency to overdo all of these effects in the comp, however. So, my rule of thumb is to add what you think is the right amount… then reduce it by 50%. Then, when a supervisor looks at it, they will probably ask you to reduce it by 50% again.”
Indeed, these pseudo lens effects and layers of dirt and clouds of dust can become a crutch for inexperienced VFX artists.The compositing experts interviewed all say that there’s a careful balance to strike between conjuring up the details that make a scene immersive and being heavy handed.
“A bit of added texture is mostly a good thing, but it’s still in the details,” says Joe at Absolute. “If you don’t have the fundamentals correct then no amount of scuzziness will save you!”
Federico Quiros at La Posta agrees. “I agree, you should give elements ‘used’ or ‘lived’ properties whenever possible or applicable,” he says. “If we are talking about a full CG shot, I would try to emulate a real camera - distortions, aberrations, flares, noise etc. - just don't over do it.”
Indeed, if the to-scuzz-or-not-to-scuzz debate shows anything, it’s that compositing is a highly personal and creative art, rather than a by-the-numbers science. There’s so much variety between jobs that there’s no hard and fast rule, as Duncan Malcolm discusses.
“A shot where dirt and debris can be used to help embed elements is a bit of a gift for compositors. Unfortunately, arbitrarily adding lots of muck doesn’t always work!” he says. “Also it's never only ‘dirt’. What’s usually needed is many layers of different techniques, each one breaking, covering, softening and sharpening to differing degrees. The combination of all of these layers leads the viewer to believe what is in the comp really happened. Most compositors have their own secret recipe for this, and although it's seldom the same, they’ll start in a similar place, then will use their knowledge of the real world and specific shot plates as their guide to tweak the look.”
Genevieve Camilleri suggests that the best way to figure out if you’re on the right track is to really hone your powers of observation and to seek out real world references. “This [added grime] certainly does help, especially when working with full cg shots to help bring a sense of realism to it. However, sometimes this can easily be over done to an extent that actually breaks the reality of your work instead. I am always a huge fan of finding real life references to use as your source of inspiration and as a guide to the finer details that help make a shot look realistic. For example, if you’re trying to composite a cg plane then referencing and trying to match the dirtiness or scuzziness of a real plane is always your best point of call.”
Making and Breaking the Rules
This subjectivity extends across the craft of compositing - while it’s highly technical and involves an understanding of how forces like light and gravity interact with objects, compositors don’t have to be slaves to reality. Sometimes unrealism is much more fun and relevant to the project. But, again, knowing when to let loose is something that comes with experience.
Michael Baker at Framestore says, “I think knowing when to get creative and break the rules is a bit is of a challenge in itself. In compositing we can get very hung up on 'does it look real?' or 'would that really happen?' that we often miss opportunities to just make it look cool! I don't mind deviating a little bit from the real world if it gives the shot a great moment and the struggle is finding where you can make that impact in a shot you've been staring at for a month.”
The Joy of the Journey
Speaking to compositors and VFX supervisors, it’s clear that it’s the challenges that keeps the work fresh and creatively satisfying. Beyond those challenges outlined above, there are a whole host of other fascinating puzzles that can crop up for compositors - and it’s that variety and novelty that excites them.
“It ranges from project to project, which is what keeps things interesting!” says Joe Tang at Absolute. “Sometimes you can be using your creativity to elevate or enhance a scene aesthetically by comping 3D or 2D elements. Other times, it’s about using creative thinking to problem solve in order to bring assets together! It’s never the same, but it’s always fun.”
“Each shot is a different challenge,” agrees Liciani Vargas. “Sometimes our goal will be to create something completely surreal and magical, while in other cases it will be to recreate a real event, person or animal. Therefore, for each type of scene, a different technical and artistic approach will be required.”
Duncan Malcolm reflects that the most satisfying challenges to work on are those where he gets to work closely with the director to bring wild and difficult scenes to life - rather than being brought in last minute to rescue a job and ‘fix it in post’.
“For years I spent many a long day working through projects that arrived in my Flame suite and it was a great sense of achievement to get people out of creative trouble, to crunch together interesting images from what was there - sometimes building a forgotten shot from nothing. This still can happen, but nowadays I find the most interesting part of my job is working with directors to plan the shots ahead of time, then bringing the shots through the whole process to the end,” he says. “Without the pre-planning and early chats, it's easy to spend all the time we have fixing stuff instead of improving or adding extra value to the screen. The pre-planning part lets us know what we need to build ahead of time, what we need to shoot and helping producers know what we don't need them to spend their shoot time doing.
“The short answer is that experiencing the whole journey is the most interesting and creative part of compositing.”