Crowd Pleasers: How Zoic Studios Mastered the Art of Lifelike Digital Audiences
Have you noticed how 2023's most popular TV shows like Ted Lasso and Daisy Jones & The Six are using digital replication to simulate hyper-realistic crowd scenes? No? Well that’s because the technology in the hands of this industry’s most skilled VFX artists has reached new heights, resulting in unthinkable on-screen wizardry.
According to Insider, the largest on-screen crowd scene ever shot was for the 1982 film, Gandhi. Filmed in India, the iconic funeral procession scene reportedly featured 300,000 extras. While this has gone down in movie history as a record breaking effort, crowd scenes far smaller than this filmmaking feat can prove problematic to direct. On top of the coordination of hundreds of extras, the additional expense makes real life crowd scenes an unappealing prospect for studios.
Luckily, the visual effects artists toiling away behind the scenes, have found an ideal sweet spot - both in terms of budget and practicality - with the most realistic payoff. The best solution for filmmakers in 2023 is using a combination of real people and computer-generated avatars. By filming a small group of real people - also known as plate extras - acting out a wide range of movements in front of a green screen, VFX artists can digitally replicate them en masse to fit the requirements of a specific crowd scene.
Eager to unearth the craft behind this increasingly in-demand area of VFX, LBB’s April Summers speaks to Rocco Passionino about conjuring computer-generated crowds for Emmy-nominated musical drama, Daisy Jones & The Six.
LBB> How did the digital craft of this project differ from other crowd scenes you have worked on?
Rocco> Every crowd I’ve ever created has been a unique experience. Frustratingly, it’s very rare for us to be able to lift a toolset from one project and drop it into another – there is always a twist. The size of the crowds and the time period definitely made Daisy Jones & The Six different from other shows we have worked on.
As the series progresses, the band’s popularity leads to them performing at larger and larger venues, culminating with their final performance at Chicago’s Soldier Field Stadium. For this, we had to fill up the entirety of a 1970’s replica (Soldier Field was remodelled in 2002) of the 50,000 seat stadium with a digital crowd - in period clothing - matching to the performance of the foreground live action extras in the scene. Almost every shot in the series had a live action extras component to it which challenged us to make our crowd match in look, feel, and action to the real people in the scene.
LBB> Can you tell us about the creative process of crafting large crowd shots? What technology is used and how does it lend itself to the creative process?
Rocco> At the beginning of this show we did a deep dive into what exactly concert footage from the 70s looked like. Surprisingly, outside of clothing, average age, and the use of lighters, what the crowd itself was doing was very similar to today: lots of people standing around listening to music. We then looked into the locations the band were playing and made stand-in proxy’s for the venues. In the case of Soldier Stadium, it had been remodelled in 2002 so that meant we had to reference what the location looked like back then, rebuilding it from those specifications.
In terms of technology, the backbone of our crowd creation for this project was Houdini. All the population density, animation, performance swapping, simulation, variety and randomization, and wardrobe selection happened in Houdini. We developed a library of crowd animations for all the characters that allowed them to walk, cheer, wave, dance, and other typical behaviours needed to fill out the crowd.
Next came implementing the crowd dynamics to create the illusion of organic movement. Parameters such as density of the crowd, speed of the dancing, waving, and distribution of where everyone was standing needed to be coded to create a natural-looking crowd flow to avoid a repetitive and artificial look.
Lastly, iterating based on feedback from directors, producers, and other team members can prove challenging. Each individual note of feedback requires us to revert back a few stages and start the process all over again. Overcoming these challenges involves ensuring the right artistic skill, technical expertise, and effective collaboration among different departments required to make it all come together.
LBB> What are some of the common challenges that VFX artists come across when working on computer-generated crowd scenes?
Rocco> Crafting lifelike digital characters is always a challenge. Creating these diverse characters with varied appearances, clothing, and behaviours requires meticulous attention to detail for modelling, texturing, shading, and lighting. Ensuring that the movements and interactions of each crowd member appears natural while avoiding repetitive patterns - making it not look like a wheat field vs a random collection of people enjoying a concert - is a significant challenge. Seamlessly integrating CG crowds with live-action footage is a crucial aspect of pulling off a convincing crowd. Achieving proper lighting, shadows, and reflections that match the live elements, especially in a stage environment where the lighting is constantly changing and flowing. Rendering a large number of crowd members can be computationally intensive. We went through several rounds optimising the rendering process to get times down to meet production deadlines and hardware limitations without sacrificing quality.
LBB> Zoic has a rich history of 20 years – how have technological advancements in VFX altered the way that artists approach digital replication during this time?
Rocco> Epic Game’s Metahumans has definitely changed the creation of CG characters. It has allowed us to have a starting point for generating unique faces and bodies in a crowd without having to use a scanning process. The use of motion capture and performance capture technologies has revolutionised the way characters and movements are replicated digitally. Instead of having to go to a stage and do a full motion capture day, artists can capture the nuances of real performances from our office or their homes, translating them into lifelike animations for digital characters.
Houdini’s sophisticated simulation and dynamics systems has allowed us to create more realistic and dynamic digital replicas. This is particularly evident in the simulation of fluid dynamics, cloth, hair, and crowd behaviour, making digital elements behave more like their real-world counterparts.
3D scanning and photogrammetry technologies, such as LIDAR, have gotten more affordable and are an excellent starting point for artists to capture real-world objects, environments, and even actors in 3D with high precision. The use of cloud computing has increased the scalability and flexibility of our rendering pipeline. Instead of maintaining our render farm in the facility, we can now access machines remotely, enabling them to handle larger and more complex scenes. Real-time rendering engines, like Unreal Engine, helped us rapidly previz out some of the scenes allowing us to see immediate results and make adjustments on the fly. These are just a few examples, the list goes on…
LBB> Digitally generated crowds are becoming increasingly popular in film and TV. What do you attribute this increase in popularity to?
Rocco> There are several factors at play - cost efficiency, advancements in technology, changes in production techniques, and the evolving demands of storytelling. Digital crowds can be more cost-effective than organising and coordinating large groups of extras or hiring crowds for practical effects. It allows shows to achieve the scale and spectacle of crowd scenes without the logistical challenges and expenses associated with managing real crowds (hiring extras, wardrobing them, catering, wrangling and coordinating them, separate film crew to shoot tile plates).
COVID also influenced the use of digital crowds, as it prompted changes in on-set practices and safety protocols. There have been several projects where digital crowds provide a viable alternative to traditional crowd scenes when physical distancing measures are necessary. Filmmakers often seek to create epic, visually stunning scenes that involve large crowds, whether it's a battle, a concert, or a cityscape. Digital crowd generation allows for the portrayal of massive and awe-inspiring scenarios that may be impractical or impossible to achieve with practical effects alone. Audiences today often expect high levels of visual spectacle in their films and TV shows. Digital crowds allow filmmakers to deliver impressive and visually striking scenes that captivate audiences and contribute to the overall cinematic experience.
LBB> Do you have any thoughts on the ethical use of digitally replicated background actors?
Rocco> If we are talking about the process of scanning extras on set and then using their likeness for crowd scenes - that process has been used for many, many years and is not something new. It is also not as simple and easy as everyone makes it out to be! The scan’s themselves are a great starting point to build a CG character from, but a considerable amount of time and effort (and therefore money) goes into generating a photorealistic human being whose performance does not look uncanny as just a background actor! And at a certain point, that level of detail falls apart and you need to go through an exponentially higher level of detail to make them look better.
Plus, these computations aren’t cheap - cloth sims, skin textures and render passes, behavioural algorithms, etc – and they take time. As far as re-using said asset for another project, like I mentioned earlier, every crowd system has a unique twist that makes plug and play challenging: different era’s, different clothing, different skin tones, different percentages male-to-female, different accessories. There are many different factors that make it hard to use an avatar again. Will advancements in technology give us better tools to replace them… possibly? But it’s still not going to be a cheap and easy solution.
