God Mode: Virtual Reality’s Power as a Creative Tool
Most people know virtual reality as a platform now, although it still lives on the nerdier side of society. Since the Oculus Rift announced its crowdfunding project in 2012, practically every major player in tech has released a headset. Many industries have been abuzz with talk of VR and the experiences in it that thrill audiences experiencing the immersion in alternate universes that it can offer. That buzz seems to have died down more recently as the tech world found new subjects to distract it in augmented reality, artificial intelligence and whatever the hell blockchain is.
“You can tell when someone’s experiencing VR for the first time” says Liam Walsh, Creative Technology Director at Nexus Studios, “because they sort of have their head up in the air looking around and marvelling at the tech.” That’s the power of gimmickry - a force that has diminishing returns as more and more people experience that first ‘wow’ moment. With that mostly behind us in 2018, VR content is having to work ever harder to impress audiences. It’s no longer the star of future entertainment - just another platform that experiences will exist on.
But VR’s history predates its use as a medium. It began life as a tool. Gavin Fox, who devises rides and immersive experiences as Creative Director at Framestore, uses VR as a creative tool regularly. “We’ve been using it longer for visualisation than we’ve been seeing it in entertainment,” he says. In its earliest iteration, VR was developed largely for military applications. And Gavin remembers the first video game applications in the ‘90s. “I’m sure there were practical industrial applications for it before then,” he says.
“When you use it as a tool it doesn’t need to be quite as slick,” Gavin observes. “Back in the day they would use the old CRT-based headsets that were really heavy, but it still would be useful. You wouldn’t give it to a consumer because they’d say ‘this looks awful!’”
A sanctuary for creativity
While VR’s popularity in the consumer space has moved past its hype cycle and levelled off, as the tech has progressed its usefulness has only increased. When creating a universe for a film or experience in the distraction-filled world of 2018, Liam at Nexus sees the value of putting on a VR headset as a unique creative experience. “It’s an odd thing in that it requires 100% of your attention because it blocks out the rest of the world,” he says. “That’s a great place to get into, as an adult with a wife and two-year-old child. I very rarely have the luxury of giving 100% of my attention to any media I’m consuming. As someone who works in an open-plan office, it’s fantastic to be able to block out the world. The main shift for me is that one. People don’t stand over my shoulder and interrupt me when I have the headset on because they know I can’t see them and I’m clearly working.”
Aside from that intriguing sociological benefit, VR’s capacity as a creative tool for pre-visualisation is massive. Once creative people have tools, they will find a use for them. “The same way people are used to making models to test something, now they’ll use VR to do the same kind of thing,” says Gavin.
Essentially, it’s another way to supplement imagination. Once there were sketchbooks, then CAD technologies allowed people to build worlds virtually. Now creators can immerse themselves in those worlds.
“There’s only so much you can have in your head. If you can lay it all down in a rough 3D model that you can look around then you can make it feel a lot more real,” says Gavin. Of course you could look at something with a virtual camera on a screen, but VR takes it much further. “A VR headset puts you more in the mindset of what the audience is going to feel like. It’s a much more emotional connection to your design.”
“It does bring a huge leap,” says Liam. “There’s just one less layer of abstraction. It feels more direct, so the part of your brain that thinks creatively about manipulating space doesn’t have to translate that back into the tools that you’re working in. Just being able to physically step back and look at something from a different angle is far more intuitive than the indirect way of doing it with tools. Don’t get me wrong, everyone becomes a dab hand with keyboard shortcuts and the mouse and so on, but it feels quicker because you can be more direct.”
Inhabiting a world as you’re creating it, with almost limitless capability - it’s a concept we’ve encountered in the fantastical realms of cinema, like ‘The Construct’ in The Matrix or the malleable dreamscapes of Inception. And of course there’s the Steven Spielberg adaptation of Ernest Cline’s VR fantasy Ready Player One.
The laws of physics don’t need to apply in VR spaces, meaning a director or creative can teleport or fly around the space, manipulating his or her surroundings instantaneously. “It’s really, really powerful,” says Liam, “‘God mode’ is a good way of looking at it. You really feel like you’re manipulating time, space and matter in a way that you don’t feel like you have control of just clicking buttons and moving a mouse around.”
“I have complete control over that world,” says Gavin. “If I want to get rid of all the lights I can do that. If I want to flood it I can do that. If I want a forest there suddenly I can do that.”
Nexus utilised this godlike power to create Smith & Foulkes’ stunning 2D animated trailer for the BBC’s Winter Olympics coverage.
While the directors’ imaginations could have dreamed up and storyboarded the shots of these various action sequences with nothing more than a pencil and sketch pad, they instead created rough 3D scenes and immersed themselves in them so they could look around for the best shots and transitions to capture the action. The speed advantage this offers is amazing. “You look at three or four different camera angles right there by just moving your head around or making a little frame with your fingers,” says Liam. In the past, this amount of iteration at the storyboarding stage would have involved a very full waste paper basket.
Gavin is working on a VR ride at the moment. His team at Framestore has built tools into the version they’re building so he can move around it and review the work, take screen grabs, pause it and annotate over them and give that to the layout artists and animators.
Like so many technological advances, VR is making the world smaller for creators. Matthew O'Sullivan, Head of Worldwide FX UK works with Bulgaria’s Nu Boyana studios on visual effects for major film titles. In a filmmaking landscape dominated by green screens, allowing anyone in the world being able to inhabit a virtual space is transformative. “You can plug into multiple formats from multiple locations,” he says, “preventing people from having to travel and improving general accessibility. It provides accessibility to people who may be immobile or who live in remote locations, and allows immersion and exploration in a virtual world.”
Nexus are more than aware of this. They’ve been developing a tool for animation in VR called VLO (Virtual Flow), that aims to make animation for any output easier and faster. None of the tools already out there had multi-user capabilities, so they built this into their own tool. “It means the director can be in there with the animator, actually in the space, rather than looking over their shoulder or waiting for rushes, they can dive in. They can both be moving things and you can imagine integrating a motion capture suit to that so people can kind of act out stuff for each other.”
Liam was part of a team that developed a “slightly ridiculous” (in his words) augmented reality app called HotStepper. Nexus created it as an internal R&D project, animating the app’s bizarre hero in VLO. Partially, they wanted a proof of concept for VLO, but they also needed to use VLO because their character animator was in Mexico and the directors were in the UK. This allowed them to be in a shared space where they could perfect the HotStepper’s walk in a much shorter period of time than if they had to wait for files to cross the Atlantic.
This means it’s not only the most creative or techy person on a team that can work in VR. It lowers the barrier to entry to someone being in there as a collaborator in a creative space. “I’m not suggesting I’m going to put all our clients into VR, looking over your shoulder while you work,” jokes Liam. “But having a creative in that space who doesn’t need to be fantastic at drawing or understand how to use a complicated piece of software is great. They can just be in there and say ‘up a bit’ or ‘what about more like this?’” As Matthew puts it, “once set up, the technology is easy to use. You don't necessarily need to be technical.”
This makes VR perfect for demonstrating work in progress to clients or people not directly involved in a project. Gavin is excited about this application: “We can tell you what the basic backstory is, go ‘here you go’ and we can see what you think. That’s what VR’s useful for.”
Designing experiences beyond the rectangle
Entertainment can take so many forms now and in a world where content isn’t confined to rectangular screens, VR is a much more suitable pre-vis platform.
This applies particularly to the kind of projects Gavin’s team works on - rides and immersive live experiences that often integrate various media with reality. “The only way to really appreciate a piece of media or animation that’s going to be designed to work in harmony with a building is to try to visualise it in VR,” he says. For these projects a screen is a limiting window on the experience you’re creating. VR allows him and his team to stress test experiences from every possible perspective.
The Framestore team have a ride opening soon in China that they’ve been using VR on extensively. Audiences will travel through a massive physical set of “fantastical caves” on a ride vehicle. And at the end of the cave network will be a screen - “an extension into that world,” says Gavin. “It’s sort of 3D, stereo-rendered media and we warp and distort perspective so it looks correct no matter where you are. It’s similar to the old Hitchcock films where you have a painting at the end of a hotel corridor - that sort of feeling. The only way to test to see how that feels is VR. So that’s why we’ve made a VR pass-through of the ride and all the animators are putting their work into that version. It’s stress testing everything that we do. You need to be able to see where any errors might be.”
Likewise, AR experiences extend way beyond the screen. As Liam puts it, the ‘camera angles’ are dictated by the user in AR, so content has to look good from all perspectives. “VR makes that a lot clearer and easier, compared to a storyboard. Because you’re already in the medium that’s a bit like what it’s going to be like later. We can do things like make a little square frame and hold that in our hands in VR to mimic what it’s going to be like in AR,” he says.
An antidote to the green screen
It’s not just the swooshy futuristic media that are benefiting from VR’s creative capabilities. Matthew at Worldwide FX works mainly in the traditional entertainment world of feature films. He points out that VR can help actors now too. With contemporary cinema so dominated by VFX, actors’ imaginations are stretched further than ever. “Most of the time an actor is acting in a blue screen or green screen space, where they can’t visualise the scene,” says Matthew. “What VR could do is the actor would put the headset on and it would put them in the scene so they can interact with their surroundings.” They can rehearse a scene in a virtual space to get more of an understanding of how their character might feel. And using markers, the actor’s movements can be tracked to help put together the final scene in post production.
Matthew also feels the tech is there for more intuitive filmmaking processes. “I think it could replace the need for heavy expensive camera equipment and labour costs when filming on set,” he says. Using a VR headset to drive a drone could provide huge benefits to filming large sequences in tricky locations. “I believe this is already possible due to drone capability rapidly improving also lightweight and powerful cameras that now exist.”
The next steps
With all this going on, what technological breakthroughs are the creative community waiting for? Gavin is relatively satisfied with the level VR is at now. “You don’t need that much more than we have. We’re not gagging for too many things. You can accept being tethered to a computer or the resolution not being too high [when using VR] as a tool. For the consumers that’s where you want wireless headsets and all that sort of stuff.”
Liam agrees that for pre-vis purposes, it’s already phenomenal - a creative holodeck where anything is possible. “If you take the photorealism away, we’re already there,” he says. “If you just accept that you have to use your imagination in that space then we’ve already achieved that. It’s just we’ve produced it with stick figures and sketches. At the ideas phase, when you want it to be immediate, we’re kind of there.”
But for the sheen of a finished product the work still has to be done elsewhere. “None of the VR tools are ready to be the whole thing end to end yet,” he says. “The whole ecosystem around the software that isn’t there yet with VR just because it’s not been around as long.”
That will soon change though. Matthew is excited about developments from a London company called Foundry who are working on a tool called Cara VR that could be a step towards taking VR’s applications beyond pre-vis. “It’s a compositing tool,” he explains. “So an artist can put a headset on and comp in a scene while also being in that scene. For an artist to get into a scene themselves would be brilliant because the creative control once you’re in that world would be fantastic. There are plugins for [compositing software] Nuke for that already.”
Framestore, Nexus, Worldwide FX and companies like them are now using VR as both a tool and a destination for content. The tech has potential in both of these roles, but it’s conceivable that VR actually has more of a future as a mass market creative medium than as a mass market consumer medium. “As someone who works in it both as a tool to create and also making content for it, I think it’s better as a creative tool than it is a creative medium for people to consume,” says Liam. “In my mind, that’s the thing that will live on and be more commonly used. “
VR will continue to entertain and inform consumers for years. No doubt there will be many more ‘wow’ moments to come now it’s largely over its awkward gimmicky phase. But it’s not always the most comfortable of platforms, as Liam and other VR creators are well aware. “We don’t want to trap people in a helmet of capitalism that they can’t escape from. If you’re not careful, that’s what you do with commercial VR.”
Even Gavin, who works on some of the most exciting VR projects in the world, feels the tech’s potential is more solid as a tool. “I think it’s always going to be there,” he says. “It’s been there for a lot longer than it has been for consumers and I think it will last longer as a tool. The ability to be able to go into a space that you can completely control is invaluable.”
