Everything You Wanted to Know About Virtual Reality But Were Too Afraid to Ask

Common Misconceptions

Virtual reality is becoming increasingly common in the advertising industry – for example at Little Black Book we see about five new branded VR projects a month, and there are certainly others we don’t see. But the boom has happened in a relatively short space of time – alongside the hype and excitement there’s also been an explosion in misconceptions.

Mike Woods is the founder of Framestore’s Digital Department and dedicated VR studio and for him, the myth that he keeps coming up against is that virtual reality experiences are put together in the same way as traditional film. “It’s a whole new way of doing things,” he says. “You can’t just put someone in that environment and tell them a story in the same way you would with film. There’s no frame; you could be looking in a completely different direction when someone’s trying to tell you something important. There are certain creative choice moments when it can be a bit more like a film.”

At Happy Finish, Matt Morgan’s biggest frustration is that VR is often seen by marketers or agencies as a fly-by-night novelty. “[One big misconception is] that the format may be a fad or restricted to early adopters who invest in the tech. With the advent of Google cardboard we will be able to deliver the VR content to the masses,” he says.

Indeed, over at Nexus, co-founder Chris O’Reilly likens the VR revolution to the transition animators underwent in the ‘90s when they had to shift their skills from 2D animation to computer-supported animation and 3D animation. He even compares the change to the leap the Lumiere Brothers made when they first brought entertainment to the silver screen.

“It’s like being in 1895 at the birth of cinema; people came to that with a lot of experience from theatre and Vaudeville – but there’s a lot we still don’t know. It took filmmakers another 20 years for cinema to develop editing! So this is a really exciting opportunity; we are the first generation – not just at Nexus but collectively as a creative community – we’re the first bunch of people who can explore what this can be and potentially invent the things that might end up becoming commonplace.”

Think Strategically

Understanding why you’re doing a virtual reality project will go quite some way towards figuring out how that experience will be produced and what platforms it will appear on. Are you looking to create buzz among a handful of opinion formers? Or are you keen to pump it out to as many people as possible? Do you want your audience to take an active or more passive role in the experience? 

Happy Finish’s Ainsley Henn has a few suggestions to get you thinking. “It is always important to use the right technology to solve the right problem. We have no doubt all seen installations where an inappropriate technology has been shoehorned in when something else would have suited better. Potential clients should ask themselves why they are using VR. Common responses include that they want to create a high adrenalin experience (adrenalin); to place someone in a place they couldn’t otherwise be (telepresence); to place someone in someone else's shoes (empathy); to create a piece of digital art (conceptual).”

One great example for 'telepresence' is the platform Immersiv.ly, which has been created as a means of spreading 360 news and editorial content. Immersiv.ly has teamed up with Amplified Robot, part of the Three Wise Monkeys hub in London's Soho, and they're hoping to use VR and 360 technology to create a new form of journalism that escapes the confines and bias of the cameraman's framing.

Try opening this on your smartphone for the 360 experience...

Of course it’s tricky to know what you want to achieve if you don’t know what’s possible – and that’s all about getting stuck in and having a go. If you’ve only experienced the Tron-like polygons of the ‘80s, then you might understandably have your expectations anchored around that point. Or perhaps you’ve been entranced by the polished photoreal CG of the high-end Oculus Rift experiences and are wedded to that ideal. One of the most surprising recent VR projects is Vincent Morisset’s ‘Way to Go’, an experience that feels closer to an experimental art film than a flashy tech experiment. It combines simple animation, grainy footage and ingenious generative technology and was created by a team of just four people. 

“Way To Go demonstrates the potential of VR for artistic expression and the fact that it’s a democratic medium open to indie creatives who learn the required skills. For me, the most interesting evolution of VR over the coming months will be when we start to see user-generated VR content going viral in the same way videos currently do,” says UNIT9’s Henry Cowling. “Way To Go was a labour of love for Vincent Morisset and the team. Their warmth and creativity really lights up the piece. I think the lesson here for other creators is that VR can deliver a whole spectrum of emotions, it’s not just about showing off the tech; VR experiences like Way To Go can have a real heart.”

Timescales

There’s no such thing as an average VR project, so it stands to reason that there are no average timescales and budgets. As with a TV commercial, the budget and length of time a project will require will depend entirely on what it is you’re hoping to achieve. A passive live action film shot on GoPro or a fully interactive, immersive world? Speaking to the experts at various VR studios, the shortest turnarounds from initial brief to delivery varied from four to eight weeks, with more complex projects taking anything from three to eight months. Elements that tend to drive up the production schedules include complex installations with sensory feedback (temperature, smell, rumbling floors) as well as the sheer duration and complexity of the experience. Timelines have come down considerably over the past year-and-a-half as studios refine their pipelines and processes and have become better at anticipating the kinds of hurdles and issues that might arise.

Choose Your Weapon: Headsets

Possibly the most needlessly off-putting part of virtual reality production are the headsets themselves. The key thing to understand is that ultimately each different headset is just a slightly different delivery platform – and while the choice of headset will depend on who you want to reach and how, it makes surprisingly little difference to the overall production process. 

“The first thing to decide is what is the intended user. Morpheus by Sony will be targeted for the PlayStation, therefore limited to running PS games and applications. Using the Oculus Rift allows users to run an experience through a PC, and others such as Samsung Gear VR and the HTC Vive turn mobile phones into virtual reality headsets. So determining the target audience and usage of the experience will determine the best product for the project,” explains Happy Finish’s Arnaud Baernhoft.

Google Cardboard is probably the easiest headset to deliver at scale – it’s a device made of folded cardboard that holds a smart phone and can be sent out to consumers through the post or handed out like sweeties at events. At the other end of the scale is the Oculus Rift, which is powered by the PC and tends to deliver more high-end, beautifully rendered experiences (at the moment an Oculus Rift Dev Kit costs about $350, so it’s not widely available to consumers but can be an affordable-yet-powerful option for installations, conference demos or glossy launch events). The Samsung Gear and HTC Vive are powered by smartphones and are seen as a sort of middle ground. Excitingly, the Altergaze, created by Nexus’ Alexandra and Liviu is an ‘open-source’ headset that can be personalised and 3D printed (by brands, universities and even consumers).

The little sister to the VR headset is the 360 video platform, which allows users to experience a 360 story but without affixing anything to their face. Google Spotlight and YouTube 360 are well worth exploring – an experience created for Oculus Rift can find a longer shelf life and a wider audience if it’s transferred to 360 video. In the long term it’s predicted that truly great experiences will become ‘hardware agnostic’.

It’s Sort of Like Filmmaking… Only It’s Really Not

While many of the companies creating content in the VR space come from some sort of filmic background (whether animation, post production or live action) the skills and vocabulary of filmmaking don’t translate directly to virtual reality. That means that there has been just as much ‘un-learning’ as learning over the past couple of years.

The fast cuts between locations that we’re used to seeing in film can be disorientating in a virtual reality environment – one of the biggest mistakes is to think of the viewer as ‘a camera’ rather than a person.

For Nexus co-founder Chris O’Reilly, it’s been an exercise in problem solving. “The thing that we’re bringing to it is the background in storytelling, but what we’ve been thinking about quite a lot is how do we tell stories in an environment where a lot of the tools that a director would traditionally use aren’t available? You don’t know where people are looking, you don’t control the framing, you can’t use filmic techniques like cutting and zooming.” At Nexus, the team has been drawing a lot of insight and expertise from the world of immersive theatre. What’s more their dedicated Virtual Reality and 360° Storytelling department is headed up by Liviu Berechet Antoni and Alexandra Stancu, co-creators of the Altergaze headset. Liviu’s education includes degrees in both architecture and game design, while Alexandra studied Geophysical engineering before doing a degree in 3D digital animation. An understanding of three-dimensional space is crucial for creating virtual worlds.

At Framestore, the world of gaming has provided a valuable resource for both talent and clever tricks. “We always look to the games industry even though we’re keen to point out to clients that we’re not making games,” says Mike Woods. “The fact is that the games industry has been doing this for years and years, and have genius narrative devices to help tell a story while giving you the impression that you have a say in what’s going on around you. There aren’t too many things that you can bring over from the film world. So many things you’ve learned at film school, like framing and shot composition and lighting, all goes out the window because you’re not trying to make this perfect, evocative frame. In that respect it becomes more like a console game where characters can be in a world that they can explore at their own leisure but still be part of a story arc.”

Even the most simple and well-established conventions of cinema and theatre need to be re-thought in the VR space. Amplified Robot exemplified this dramatically when they paired cutting edge VR technology with the master of storytelling, Shakespeare. They’ve been working on a project to bring Much Ado About Nothing to virtual reality platforms and found that the Bard’s words needed a completely new approach. On stage, Shakespearian actors often find a rhythmic groove and bounce off each other in a clear-cut back and forth, while putting the audience right amongst the action means that the actors are able to include more cross-talking. The same project was filmed not against a green screen but within a ‘green box’, so that the actors could be transplanted into different 3D environments – which presented its own unique set of challenges, not least that the director had to direct from outside of the ‘box’.

Pre-production

The differences between a traditional film project and a VR project emerge right from the off. For one thing it’s much more iterative. Clients are still getting their heads around the possibilities and limitations of VR, which means their initial concept or idea might not be fully formed – or it may turn out that there is a more effective way to achieve their desired results. What’s more, a successful VR experience relies on your viewers responding to cues and behaving in a desired way – if your audience isn’t doing what you want them to, you’ll have to go back and tweak things. What this means is that the pre-production stage is often a much more collaborative affair between agency/brand and production company. 

Even that old stalwart of preprod, the storyboard, has to be drastically re-thought. Traditionally it’s a way of conveying shots, framing, timing and possible edits before a project has been committed to film (or, err, digital files). To add to that, storyboards are usually… flat. Flattening out or ‘unwrapping’ a 360 environment for that purpose can be more confusing than clarifying as the lines of the image become wavy and distorted. In VR it’s often more effective to go straight to an example animatic or even a rough example VR experience.

The Script

The script itself is less linear and much more complex. It has to take into account the fact that actions are triggered by certain behaviours and that the viewer might need to be nudged in a certain direction by cues, that there may be multiple possible trees of action and that the pace of the story is largely in the hands of the viewer.

At Framestore, the team tends to use a ‘cue script’ which sits above the main script and is a tightly planned map of the various cues which gently push the viewer to direct their attention to a particular spot or to perform a certain action. These cues can involve sound, smell or even rumbling floors.

Nexus’ Chris O’Reilly has a background in script development and it seems he’s been particularly enjoying stretching and challenging himself in this area. “I’ve been working a lot on scripts recently. VR has a lot to do with triggers and you need to prevent actions from happening when you’re not looking in the right direction. That has implications for the timings. I’m working on one script now that has one main narrative but lots of stories going on around it. It’s a bit like game design, the audience has choices but we keep bringing back to the main through-line of the narrative. In games they do this all the time, often using cul-de-sacs which can be physical or ‘interest cul-de-sacs’ that lead you back once you’ve explored everything there is to explore in a particular area.”

Creating the Image

There are several ways to capture and create visual assets for virtual reality – and more variations emerging all the time. We can’t cover all the technical specs and intricacies here, so we’ll give you a rough outline of some of the possibilities and the sort of jargon you might come across.

At its simplest, 360 video can be captured using a rig of spherically-arranged cameras (often GoPros) – and the images captured by these cameras can be stitched together, removing overlapping, to form a 3D virtual sphere of imagery. That might sound basic but it requires a great deal of ingenuity from both the film crew and the post production team. “We were once asked to re-use a traditional film set for a VR shoot but a quick recce revealed that half the set didn’t exist. With VR you have to think about a full 360 point of view,” recalls Unit9’s Henry Cowling. “When we were filming for the Canadian Tourist Board, we hid all the crew and equipment under military camouflage nets and the 360 camera flew overhead mounted on a drone.”

Once captured, the footage has to then be smoothed out and calibrated. “Because the shot is made up of multiple cameras with different light exposure, the film needs a grade to level out the final output. A good understanding of the camera set up is required; we have now built up the internal knowledge and experience to make the process quicker,” explains Happy Finish’s Matt Morgan.

Matt’s colleague Arnaud Baernhoft agrees and suggests that the tricky bit is not so much ‘the stitch’ but ensuring that the content is filmed correctly which then allows the editor to process the image more efficiently.

Another possible route, which allows for more interactive experiences, is to build your VR project using a games engine (the tech that underpins and drives video games). The most commonly used of these are Unreal and Unity. These engines also allow for greater dynamism and responsiveness in light and audio. And while that may sound terrifying, it’s the sort of decision that a client or agency producer doesn’t have to worry themselves about particularly. “We use both – Unreal for the bigger, installation based projects and Unity for the mobile projects tends to be how we break it down currently but that could all change. It could change based on style; it’s not so different to sitting down at the start of a commercial project and deciding what camera you’re going to use because of the creative requirement. I think our choice of games engine is similar to that,” explains Framestore’s Mike Woods.

Depending on the desired experience there are lots of alternative approaches that studios should be able to navigate with you. These include inserting live action footage into a games engine-environment or mapping animated elements onto live action film, or even feeding real-time data into a CG environment.  With animation, 'traditional' methods (or as traditional as things can really be in computer-aided animation) have had to be adapted and re-thought - rotoscoping or motion tracking can be tricky when the 'flattened out' image is all wobbly lines and curves.

Lights, Camera, Action!

In the world of traditional filmmaking, the Director of Photography and the Colourist are two highly respected roles, both absolutely pivotal to ensuring the final image fits the desired ‘look’. Both play, in their own way, with lighting and colour – the bedrock of photography and cinema. In the world of VR, light is just as important… but it works very, very differently.

In film, lighting is about achieving a particular look from a particular perspective. In VR, however, it’s an entirely dynamic thing that changes depending on where the viewer is looking from. What’s more, if aesthetics win out over some sort of grounded physics, the result can be confusing rather than beautiful.

“In a VR experience, for example, there should be real time lighting – you need to know if you want to show afternoon light or evening light, for example, otherwise it will be a surreal experience,” explains Nexus’ Alexandra Stancu. “And that can be very difficult for an art director. Light is a graphic element but the light has to respond naturally. If you’re outdoors, there has to be sun. Things will be lit differently if you look at it from different angles or directions. These things that you don’t experience in films start becoming an issue.”

To strike the balance between beauty and dynamism, Framestore have invented a new role, the Virtual DP. As with many of these things in the ‘wild west’ of VR, there are not many industry conventions but the Virtual DP could prove crucial for maintaining consistent aesthetics in an ever-changing environment. “Instead of a lighting TD – technical directors in a CG world or DoP on a shoot – we tend to have this ‘Virtual DP’ role. We like this idea of the virtual DP because obviously if you’ve got this ability to look anywhere and move anywhere then light becomes dynamic, looks become dynamic, grades become dynamic. They’re not baked into a frame like they are for a DP or a telecine grader. How evocative your piece is comes down to algorithms ultimately, especially in a games engine. That’s a clever new role – it’s part TD in a traditional CG sense but also strongly DP in terms of cinematography.”

Audio

Sound design, the Cinderella of film production, can make the difference between a spot that leaves you flat or one that grips you and doesn’t let go. In virtual reality, where the desired effect is immersion, it’s even more important. As with traditional film, though, it’s often an oversight or after thought.

“Sound is too often the first thing to get squeezed when budgets are tight. Without spatial sound design, VR is just a screen too close to your face. Immersive sound is the foundation of a transcendent VR experience – this applies as much to live action as to CGI experiences,” says UNIT9’s Henry Cowling.

There are also some misconceptions about sound, reckons Framestore’s Mike Woods. There’s much excitement of binaural sound design, for example, but this process essentially ‘bakes in’ the recording to the left and right ear channels meaning that the sound can’t respond dynamically. Instead the source of the sound is anchored to a certain point in the 3D environment and how the user hears that sound depends on the position of each ear in relation to that.

“It has to be real time audio post production, you can’t just use stereo audio. You can’t just bake in the audio channels like you normally would. It has to be dynamic. The better 360 video players are all real time, they have to be because you want the sounds to ‘stay’ in place and respond to the user’s movement,” explains Woods. “With the recent Avengers job, it’s a linear stereoscopic 360 video that you’re watching, but as you move your head around the video being played back by the unity game engine, it allows you to do a dynamic binaural mix. You can have elements of sound that stay in place. That sells it. There’s a big race towards this idea of presence, feeling like you’re actually somewhere. Your body is responding to more than just eyesight and it’s a key thing to get right.”

Try opening this on your smartphone for the 360 experience...

Hack, Hack and Hack Again

Studios that have been working in VR over the past couple of years have started to see patterns emerge and are now quite adept at predicting issues that could affect a production. For more straightforward jobs, some studios are starting to develop more consistently structured pipelines – with the caveat that a degree of flexibility is still key and that some projects really do require a completely fresh approach.

“The VR pipeline is broadly similar to traditional production, with a few more steps added. For example the process of producing a 100 per cent CGI experience as a traditional film requires you to build, texture and animate only the elements within your defined camera frame. For VR this work has to be done a full 360 degrees. It’s more similar to game world creation than regular CGI,” explains Henry Cowling.

That doesn’t mean that production studios are easing off on the innovation. New projects throw up new challenges and those in the field are constantly hacking the available technology to create new solutions.

“I won’t go into the tech specifics but we’ve worked out how to render 360 live action in a way that would absolutely look as real as it possibly can,” says Framestore’s Mike Woods. “Ultimately 360 3D is a hack. Cameras can’t simulate what your eyes are doing as your head pivots on your neck, but we can do hacks to make it look as good it can. We’ve got other methods now that are less labour intensive.”

Woods also notes that though Framestore's breakthrough VR project for HBO's Game of Thrones was revealed a mere 15 months ago, he feels like the team have progressed at supersonic speeds thanks to a constant process of refinement and invention. "It’s only been 14 or 15 months but it feels like it’s been five years. We’ve learned so much. We’re still very proud of Game of Thrones but we can see things in it that we’d do different;y now. It was so good that it was received the way it was received – it really kick started us".

Some recent Unit9 projects have been particularly ingenious. One experience for Wrigley involved hacking an e-cigarette to deliver (non-nicotine) scents into the user’s head set. Another, for bookmaker William Hill, involved pulling live data from GPS trackers embedded in the saddles of racehorses, to recreate an inch accurate, real time virtual reality version of the horserace as it’s taking place. The dream of creating live-streamed, live-action 360 experiences is still many years off, but the combination of live data and a CG environment is a smart hack that allows sports fans to immerse themselves in live events.

Get stuck in

All of the studios we’ve spoken to are taking a highly pro-active role in educating brands and agency clients and are open to speculative chats and questions – and it’s well worth doing. It’s a great way to de-tangle some of the misconceptions and it’s a useful way to figure out just how much you will need to understand for your particular job role, whether you’re an agency producer, writer, art director, marketer or strategist. What's also apparent is that there are multiple 'ways in' to a VR project and the different studios we spoke to all take slightly different approaches and prioritise slightly different elements, so it's also worth asking around (armed with a little bit of jargon!) to suss out which companies might be most appropriate for your needs.

"That’s the first thing to do, to just start a conversation," says Chris at Nexus. "There are surprising things that can be done; one of the interesting things is that this is a moving image experience being told on a device that can do more than just render images. This is a connected device. There’s potential to bring data into it, geolocation too. The dynamism of it is quite exciting. It’s not merely about immersing yourself in a 360 world, these experiences can be unique every time."

But, ultimately, broadening your personal experience with VR is an unbeatable way to familiarise yourself with the limitations and surprising possibilities that it entails.

“I just say DO lots of it,” says Framestore’s Mike Woods. “I’m always amazed when we get briefs come in from people who have maybe played about once with their mate’s Google Cardboard. That’s the thing that needs to change. Go and watch stuff. Buy an Oculus Rift and plug it into a PC. Spend the weekend downloading 278 demos that people have made and throw yourself into it. Figure out what you think works and what doesn’t. People who put together a film script have probably spent the past 30 years of their lives watching films every single day. They’ve been familiar with linear narrative since they were five-years-old and are completely informed about how that works. That’s the thing. You want to get yourself into a position where you’re familiar with what’s possible. Do it. And it’s brilliant fun.”