Moving Mountains: How 2020 Accelerated Virtual Production’s Challenge to the Green Screen

It’s easy to tie yourself in knots trying to explain in layman’s terms what virtual production means. But this year producers have had plenty of practice at it. JP Colombo, head of production at RESET, has a snappy description: “Any time you can mix both live-action elements and CGI elements in real time, and in-camera.” Blackfish Studios EP Maciej Żemojcin has an evocative line: “Placing actors' film scenes inside a computer game environment in real time.”

They’ve had time to rehearse these lines because in 2020, placing actors “inside a computer game environment” has often been a lot more appealing than placing them inside a real-life pandemic environment. To say that shooting on set or location has proved challenging this year is an understatement that insults practically every producer who’s managed to shoot something. The uptick in animated films we’ve been enjoying this year is delightful, but it’s clearly the result of live-action shooting being disrupted by a deadly virus. 

Not all ideas work as animations though. And there’s only so far that an actor can project themselves into another situation from a socially distanced empty film studio. Thankfully, due to advances in various technologies that have been converging for years, it’s now achievable to surround real actors as they perform, and directors as they shoot, with seemingly real environments that are in fact totally simulated. And more and more producers have been paying attention to this prospect this year.

“We are at a watershed moment where technology has opened the door to possibility,” says The Mill creative director Adam Grint. Game engines such as Unreal and Unity are now able to render virtual worlds in real time to a remarkably realistic level. Adam describes the technology as at “a tipping point” with the quality achievable. Combine that with more robust camera tracking, advances in LED screen technology and user-friendly options to capture real-time body movement and a new mode of almost magical scope for creativity is possible. “The door suddenly opens to a vast new playground,” he says.

At the extreme end of this playground, pushing all these technologies to the max, is a set-up like the one used to make much of Disney’s Star Wars spin-off phenomenon The Mandalorian (“Everyone calls out The Mandalorian because it's obviously great,” says Dave Zeevalk, Head of VFX Creative & Technology at Break + Enter). Over 50% of its first season was shot with the real actors performing in a studio, surrounded by giant LED screens onto which dynamic, photo-real digital landscapes were projected, physically immersing the performers in a galaxy far, far away as they acted.

Back in the ‘70s, George Lucas had to physically build the world of Tatooine in Tunisia. Today, Jon Favreau can have any world he can imagine built in a game engine and brought to him, animated in real time, on these giant screen walls. That’s got to be quite a power trip for a filmmaker.

LED wall capture is popping up in studios all over the world, meaning it’s not just Jon that gets to play this game. In a year when travel to access filmmaking capabilities has been throttled, LED wall production has become conveniently available for more and more producers. Production Service Network now lists locations of these screens as part of its Global Production Capability Directory. So whether you’re shooting in Chile or China you can bring environments, real or imagined, to your doorstep.

For service-oriented businesses like PSN, the idea of locations delivered to a studio of your convenience is exhilarating, particularly in a year when flying to the location of your choice has become fraught with difficulty. “The idea you can jump from a desert, to snow, to a rainforest in one shoot day is pretty amazing,” says managing director Michael Moffett. “Or conjure up a locale or backdrop that doesn’t exist in the real world, and is completely CG. It’s quite exciting to see how this tech will evolve and flourish over the coming months.” Add in the variables of weather and light and Michael’s eyes have been opened to a future where virtual production makes PSN busier than ever, capturing high-demand locations and beaming them into producers as and when they need them.

In fact, the production service community has been providing this kind of service, albeit in a less high-tech form, for years. PSN Norway collaborated with the VFX team at Marvel Studios to shoot plates for 2013 superhero epic ‘Thor, The Dark World’. And last year, Blackfish Studios asserted that ten locations in a day is “not a problem anymore” thanks to similar techniques. 

 

 

Space-age as it the tech may seem, there’s actually something quite nostalgic about the process. Remember those janky back projections that ‘30s Hollywood used to use? Mountains and cacti whizzing past in the background as a pair of studio stars exchanged witty repartee while pretending to drive a car? You could use this tech to do exactly that. It was a method used for decades until CGI and the rise of the green screen put it out of business. Now that idea’s been reborn - but with more processing power behind it. “It’s interesting to find a forward looking solution by looking backwards,” reflects Dave at Break + Enter.

Giant LED walls are expensive, of course. But if you don’t have the budget of The Mandalorian, an iPad can also work wonders. Visual effects supervisor Daniel Fraass at Mackevision paints a picture of how almost any space can become any environment you want it to be, with a little tech and imagination: “You’re walking around in an empty room or studio with an iPad mounted to a camera rig. The iPad works as a virtual camera through which you can see your virtual world. The motion of the camera is captured in real time. This enables us to simulate our virtual shoot and generate footage very fast.” This technique doesn’t provide the immersion for talent that LED walls do, but it does democratise the dream of virtual production for filmmakers.

This all makes the process of CGI filmmaking less monopolised by experts, in a way that visual effects experts like him welcome. “We can play with the footage in editing and create much more dynamic and surprising results,” he says. It’s an intuitive set-up, which means whoever has the vision for the film can explore the CG possibilities without the interference of a CG artist.

Virtual production’s sudden seriousness could be the catalyst that everyone in the VFX world has been waiting for to firm up the workflows they’ve been calling for decades. Adam at The Mill likes to think of it as bringing elements of post-production back into pre-production or production. When the virtual assets are so present and malleable on set, there’s no more need to ‘fix it in post’. You can fix it there and then. 

Whether the virtual world appears on a tablet screen or a vast wall, the exciting thing is that the virtual elements of a film are there to play with. “The director can ask to move a mountain, change a camera angle or fly a car through a virtual world,” says Carsten Keller, head of CG at MPC. Or, as Adam suggests, the DOP can say "move those clouds to the right, and change the lighting and sky to golden hour". The powers it gives filmmakers in real time are almost godlike - they need never worry about the weather interrupting the perfect take.

Break + Enter senior VFX supervisor Gabriel Regentin is thankful for the chance to put these powers in the hands of on-set crew. “I feel that for the first time, I can actually bring VFX to the set,” he says. “No longer will I have to have the two, three or four conversations with various people on set saying, ‘Trust me, there’s going to be a big dinosaur right about... there. And he’ll be about that tall and looking really mean.’” The actors can see the dinosaur right there, projected all-too-realistically, onto the wall next to them. And the advertising client paying for that dinosaur can see just how mean he looks.

Framestore’s Alex Webster loves how in-camera melding of the real and virtual can lead to “the ability for an actor to play their role with a far greater comprehension of their environment, whilst also allowing for more natural and interactive lighting.”

Adam at The Mill adds: “What we are doing is that we are empowering directors, DOPs and actors to make the right choices with more context by allowing them to visualise what their final shot and edit could be in a much better way than before.”

All of this was already happening this time last year, when The Mandalorian Season 1 was just about to be released. Framestore even pioneered a lot of the techniques virtual production relies on making Gravity, for which they won the 2014 VFX Oscar. But the pandemic that defined 2020 the world over has propelled virtual production into a new era. Not only was it suddenly extremely difficult to fly a whole crew halfway across the world to the perfect location for your smart watch commercial - the CMO of that smart watch company was suddenly more open to the idea of her shoot happening in a partially digital environment. “[Covid-19] has been a catalyst to how accepting we all are of working with and even inhabiting virtual spaces,” says Adam. “Even the entire population communicating over Zoom helps our evolution towards embracing virtual worlds.”  

Of course, as anyone who’s sick of Zoom meetings knows, there are downsides to working in these ‘new normal’ ways. Virtual locations, whether captured from real life or crafted in a VFX house, don’t come fast and they don’t come cheap. “You are essentially front loading the VFX process before filming,” says JP at RESET. In many ways that process remains largely the same, no matter when it happens.

There will also be times when production needs the flexibility to decide in post production what is needed for the story. Fixing it in post might not be dead… “The one thing virtual production will definitely need more of than any green screen is pre-production time,” says Gabriel.

And for commercials, “current costs to produce virtually are beyond the budget of most projects,” says Michael. “If it is a quick turnaround commercial that doesn’t have any post time, this approach really doesn’t work,” says JP.

That’s why virtual production has mostly been seen used in series that reuse locations, getting more bang for their buck. Building a fantastical world and beaming it onto a gigantic LED wall for your 30-second sportswear commercial might be considered extravagant. 

Thanks to the progress in technology and understanding made this year, JP can imagine sensible practical applications of virtual production for advertising purposes. Imagine an automaker that wants to shoot a full campaign for each of its vehicles, with a variety of breathtaking locations to suit each model. “This could be a great approach and can be more cost effective than traveling multiple crew members and talents across the world,” he says.

Even a single car ad could benefit, though. Particularly if it’s big and glossy enough. Imagine filming on location with talent on a process trailer with a car. A take goes wrong (say an actor misses a line). As JP points out, the crew would have to reset the whole process trailer - turn around all of the vehicles, return to the starting point - and begin filming another take. Depending on the location, such as a picturesque windy road in the mountains, this could take a considerable amount of production time to reset. If this was done on stage in front of an LED screen or panel, it would be as simple as rewinding the displayed elements and starting another take almost immediately. 

Epic Games, the company behind Unreal Engine powering so much virtual production, is convinced that the ad production industry is ready to dive into this more and more. It’s hoping to use its London Innovation Lab as a chance to demonstrate the power of its new curved LED Volume to commercial clients. 

Framestore have recently found themselves using LED panels on a number of commercial shoots as an alternative to green screen, says Alex, who extolls the virtues of its ability to  “provide in camera compositing and as a means of capturing realistic lighting in camera, for which LED panels were perfect.”

And then there’s the more emotional argument. “I think it depends on how fantastical you want the commercial to be,” says Carsten at MPC. If your beer ad is set on an alien jungle planet with giant purple flora and fauna that you want your protagonist to interact with, you could have a lot of fun - and probably get a more convincing performance - producing that spot virtually.

When the world emerges from the pain of Covid-19, the way we think about all sorts of things will have changed irrevocably. The way the filmmaking world is engaging with virtual production is one of those things. As Maciej states soberly, “people were pushed out of their comfort zone and this is the new and safer way forward. If not for the Covid-19 situation we would be talking about it in the next five to six years.”

“We’re in the early days of what this is going to look like,” says Dave at Break + Enter. The tech is there, but the post-Covid world is bright and shiny and will no doubt bring leaps and bounds in improvement as the filmmaking world starts to take virtual production seriously. “We need to solidify best practices and workflows; seed to get clients with the idea of this new methodology that isn’t fully fleshed out and proven,” he adds. “No one has all the solutions yet. There are no real experts yet. Everyone is building expertise and confidence as they experience and encounter new problems on set.” 

Progress has been made and virtual production’s proponents are excited about what’s to come. “The tipping point on these has been reached,” says Gabriel. “Soon it’ll just be faster, better and cheaper.”