by Evan Yionoulis

For decades, the name Raitt has been synonymous with music makers and creators of illusion on stage. These days, Bay Raitt—digital animator, 3-D graphic novelist, and CEO of the Spiraloid Workshop Company—continues the family tradition through digital magic. Among Raitt’s first professional jobs were coloring the comics Spawn and Akira. He went on to create the digital facial system for Gollum in the Lord of the Rings film trilogy. Raitt spoke recently with SDC Executive Board Secretary and SDC Journal Editorial Committee Member Evan Yionoulis about breaking down the fourth wall through virtual reality and augmented reality, and how to incorporate new technology into theatremaking and life.

EVAN | Let’s start at the beginning. Your family has been active in the performing arts for a long time. Can you talk about that and how it led to your work today?

BAY | My background is a bit of a strange one. My grandfather, John Raitt, was a Broadway singer, so I grew up around theatre people. I was that kid who was always in the wings, watching the adults rehearse. I ended up having some hearing problems, so I didn’t pursue the musical side of things because I couldn’t quite hear as well as I needed to.

I dove into the visual and the science side. I was something like Employee Number 30 at Weta Digital in New Zealand. Weta is the preeminent visual effects facility for film; even without seeing their films, you can see something from them every week on a website, TV ad, or billboard. When American high school seniors were interviewed, more wanted to work at Weta Digital than be doctors or lawyers—combined. I helped build the company as it grew from 30 to thousands of people.

The home of Weta Digital is one of the largest computer installations in the world. Think of machine rooms with 85,000 computers and armies of computer graphics geeks. I did everything on the tech side in terms of designing and building Gollum’s face for The Lord of the Rings. I studied human faces and got deep into how faces can work on a screen if you’re building them from scratch.
I also worked at Weta Workshop as a concept artist and sculptor. The Workshop is a design facility that makes physical props and wardrobe, and where miniatures are sculpted.

Then I worked for about 10 years at Valve Software, a famous video game company here in the US. They are one of the largest PC gaming companies in the world. So, I do left-brain and right-brain stuff. I like to design the tools that make new things possible.

EVAN | It’s not surprising to me that you spent so much time watching from the wings as a child: you’ve been applying all of the arts of the theatre, plus coding, to all of your technology works.

BAY | All of the stuff that gets run backstage now is very similar to the computer graphics happening in video games and film. We live in a time where the worlds of film and television and gaming are colliding with the worlds of Broadway and theatre, and then expanding out into the real world. The borders of the box are coming down. That’s very interesting for people like us who are thinking about how to build moments.

EVAN | People say directing a VR [virtual reality] experience is much closer to what a theatre director does than to what a film director does. Do you agree?

BAY | Absolutely. I’ve seen firsthand how a film director grapples with the fact that in VR, they can’t control where the audience is looking; there is no frame. In AR [augmented reality] and VR, you lose two very important colors from the palette. You lose the ability to remove time, because it all has to happen continuously, or else you really disorient your visitor; and you don’t have a frame. You don’t have a square that you can put around everything, because it’s happening all around them. You have a theatre-in-the-round problem.

 

Bay Raitt giving a keyote at the Unity Vision VR/AR Summit PHOTO c/o Bay Raitt

WHAT ARE VIRTUAL REALITY AND AUGMENTED REALITY?

VIRTUAL REALITY (VR) is a computer-generated simulation of a threedimensional image or environment that can be explored and interacted with by a person in a seemingly real or physical way. Created with a mixture of interactive hardware and software, VR utilizes electronic devices such as goggles, helmets, gloves, and treadmills to create a sensory experience of a virtual, 360-degree environment. Users become immersed within the environment, can manipulate objects or perform a series of actions, and experience it as if they were really there.

AUGMENTED REALITY (AR) is a technology that superimposes computergenerated elements on a user’s view of the real world. While VR completely replaces the user’s real-world environment, AR enhances it with computergenerated perceptual information. AR can add graphics, video, sound, touch feedback, smell, and more to the natural world in real time. The overlaid sensory information is seamlessly interwoven with the physical world and can be presented on a variety of displays, from screens, monitors, smartphones, and tablets to special glasses or contact lenses. In addition to gaming (such as the viral sensation Pokémon GO), AR has many other technological and educational applications, such as marking landing routes for flight navigation, displaying anatomical information during surgical procedures, or providing detailed location information for travelers or emergency rescue teams.

EVAN | Can you tell in the 360-degree view where a person’s (or the player’s) head is looking? Do you have eye tracking?

BAY | Some devices have eye tracking and, depending on the device, you can know an awful lot about what’s happening with the person. You know where the head is looking; you know where the hands are. You know if the person is making noise or not. It can track facial expressions and even what’s happening under the skin in terms of brain scanning.

EVAN | Can you detect breathing rhythms and someone’s emotional state?

BAY | The current generation of VR headsets has very limited things they can do. But people have built some prototypes where they’ve taken high-end brain scanners and hooked them up. Because the VR device is actually touching the face, you can get a good amount of brain activity. What you do with it and how you interpret it is another thing. But even with just some rudimentary things, such as where they’re looking, how their hands are moving, and how they’re interacting, you can get a good idea of how engaged or activated the audience is.

EVAN | What are some of the challenges of working in VR?

BAY | Disorientation is a big deal. If you’re putting people into a world where the rules of reality are relaxed, you need to take them back to being a baby again and teach them how to walk. You have to guide them. There’s a sense of presence there that’s jarring.

Back when cinema was first being invented, theatre owners showed that famous train reel where it looked as though the train was coming straight out at you. People would run screaming from the theatre because they thought the train was about to hit. That works in VR. If I put a giant sphere over your head in VR, you’re going to feel a sense of dread that it might fall and crush you. You’re going to want to try to get out from underneath it. Right now, that piece is a gimmick that I think people are overusing. What’s going to be required is a new kind of talent where people are taking a much more theatrical bent of “How do I actually bring some truth and some engagement and not just go for the cheap trick?”

On the positive side, you have paraplegics wearing VR headsets to escape the bed that they’re stuck in. Doctors are noticing that patients are having unconscious nerve impulses and twitches when they experience vertigo in VR.

They were able to use that reinforcement cycle to help a paraplegic child walk by rebuilding his neural pathways using VR; they could tell that his skin was still connecting, even though his spine wasn’t. I’m probably horribly mangling the medicine behind this.

But similarly, they can take a whole classroom full of kids and make a VR experience that shows the world through the eyes of someone with autism so the kids can see what it’s like to be in another person’s shoes.

EVAN | That’s fascinating.

In terms of theatre, some people are experimenting with VR with live actors in the room. But, obviously, those things are difficult to scale.

BAY | I think there are some interesting options for intersection points between the live stage and the VR and performance with a video camera, and it just doesn’t hold up.

EVAN | No, it often doesn’t.

BAY | To put it kindly: I love live theatre and the live event and live performances. They’re always best experienced in person and always will be. But there are people who are stuck in a hospital or who are just too far away and can’t make it to a live performance. Or people who just want to have a different kind of experience. The idea of turning the lights off in your house and watching a play on your coffee table hasn’t existed before, but we’re just on the cusp of that being technically possible for the general public.

It’s an exciting idea when you think about the popularity of the stage, and how it could break free from being just a Broadway Street experience to being a Main Street experience.

EVAN | I want to see that volumetric capture stage!

BAY | It’s interesting. It captures every angle from e very point of view and every kind of light—
all at the same time. Once you have the 3-D object, you can place it—or those characters or those actors—anywhere you like, relative to the audience. When the person puts on the headset or wears their AR device, the “virtual actors” can be performing all around them, directly to them. You could be looking at the performers, holding them in your hand. They could be the size of giants moving through the clouds around you. Once you have that kind of data captured, it represents a new kind of camera.

This is a kind of photography that has no frame. Because it doesn’t have a frame, you can place it in space however you like. We know where the audience is looking, so we can have our characters make eye contact. Not only can we have the actor—we call them NPCs (non-player characters)—walk over and be in the player’s line of sight, but we can have the NPC’s eyes look directly into the player’s eyes. Or even have them look at their mouth and lips the way a lover might. Those kinds of nuances are a strange color palette to be handed as a creator, right?

Have you done VR?

EVAN | A little bit. I’ve gone to the Future of StoryTelling symposium in New York for a couple of years. I’ve experienced “Tree,” in which, essentially, you are a tree growing from a seed in a rain forest. You put on a SubPac vest, which transmits sound vibrations to your body, and you rise higher and higher and higher. At the end, the forest starts to burn. They do some olfactory magic so that you smell the fire. It’s sort of terrifying. And I’ve gotten to try out others there as well.
I understand that you’ve created the first 3-D graphic novel: Nanite Fulcrum. Tell me about that.

BAY | Having worked on comics, movies, and games, I’m at that weird spot in my career where I thought, “Hey, what is the thing that I want to put my heartbeats to while I’m still on this planet?”

I grew up being inspired by graphic novels. I love diving into my favorite art moments and favorite panels. So, I thought, “What if I could actually dive into those moments?” In a comic book, the audience controls time. It makes a good on-ramp into a much deeper experience. I try hard to be very aware of that audience minute, that entertainment minute. And how I can engage someone in that minute—in magical, creative, and inspiring ways—to embellish the moment.

Because of my career path, I’ve been that weird guy who has learned how to do everything across all these different disciplines. I don’t want to sculpt as well as Michelangelo, act as well as Andy Serkis, or be a photographer the way Annie Leibovitz is. I just want to be just good enough so that I can make those moments.

I feel that, because of VR devices, creators have a chance for the fi rst time to make any kind of experience they want, spanning any type of media. On a VR device, you can watch a play, go to a movie, play a game, or make things. The differences between a stage show, a movie, a play, a circus, or a ride are evaporating. It’s a surreal evolution of “Video Killed the Radio Star,” but with all the barriers between legacy entertainment experiences—radio and video—all melting together.

And if it’s really a reality, anything you do can be done in that device. You can make experiences that are going to be experienced in the device. It’s a closed loop. But that also means that every consumer, every member of the audience, has been handed a Broadway stage, an acting troupe, and all the tools and all the means of production.

Human beings essentially like to make, watch, and play. We like to watch a show or watch a story or watch a play. We like to draw with our crayons and paint and make art. Make stuff to share with each other and make stuff together when we have free time. If we can come up with a way of being useful so we can turn our work into play, then suddenly a lot gets done.

If you’re an entertainer alive today, you owe it to yourself to step back and think about what are these lines that we’ve drawn between the different formats—between stage and screen, between the couch with a game pad and the movie theatre? Is it part of the entertainment experience? Or is that just an accident of production?

EVAN | If somebody is just starting out, what kind of background do you think is useful for them? Is there training? You’ve managed to stay so agile in your work.

BAY | I didn’t go to school. I am a self-taught guy. I would say find the thing that you love, that you really like doing, that you think is really cool. Then share with the world why you think it’s cool.

And then listen when the world reacts. If the world goes, “Oh, that one thing you did? That is really cool,” then go, “Oooo, it is cool. How else can I make it cool, and how can I draw you in even more?”

In the world that we live in, it’s easy to think that you’re just separated by the tool. It’s easy to fall into the mental trap of thinking if you only had that magic computer, or if you only had that really cool piece of software, or if you only went to that one school, your dreams can come true.

A lot of times, people miss what’s right next to them, which is that the fastest computers that have ever been built are sitting in our pockets and on our couches and on our desks right now. The Pope himself could not ask for a faster computer than the one that every 12-year-old has. That’s just the reality of the world that we live in.

We live in this time—for the time being—where if you want to know a thing, you can access pretty much any information that you want. If you’re good at finding it and if you have a good grasp of the forest and language, you can Google-search your way, Wikipedia-read your way, and comment-link your way into finding out just about anything. You need to know what not to ask; we’re in a world where there are deep rabbit holes in every direction, which is where having mentors and going to school helps.

But this stuff doesn’t have to be done at school. A lot of the time, people get into creative arts because it’s a personal expression of their own growth. I used to get really angry as a little kid, and I would draw monsters and it calmed me down. And I ended up finding a career making Gollum and the Balrog and the Cave Troll.

EVAN | For older people like us—like me—who have taken a deep dive in certain areas, how can we incorporate these new technologies into our work if we know we’re not going to be coders?

BAY | That’s a great question. First off, it’s like photography. At first glance, it’s super complex. But as you get used to it, you find there’s a “point and shoot” angle, and you realize it’s not as complicated as it first appears.

We’re standing on the shoulders of giants. Take these 3-D graphic novels I’m making. I’m an adult and pretty far along in my career. I can do every aspect of a given production: the visuals, the story, the performance, the animation, and the edit. So why not try and teach myself how to write the code and put it on a few platforms and put on my own show? That’s what I spent the last few years doing, and now it’s a bit like I’ve eaten a magic mushroom in a Mario game. Suddenly, I’m able—without anybody else, just with me and the computer—to make a thing that can reach tens of thousands, if not millions, of people. It’s surreal. Like building a theatre to sell your own tickets. For me, it was like, “This isn’t as hard as I thought it was!”

There’s an event called the Global Game Jam where people from all skill levels—from total beginners to hardcore game developers—get together on a weekend and make a game with strangers in three days. I did that a few years back at the behest of some friends. I had such a good time; it was incredible. I didn’t know much about it at the time and, obviously, I’ve got some background. I was struck by how many young people and old people were entering into this world of computer literacy. There were people there who had never programmed before a day in their lives, trying to help make a video game and talking through the process and teaching each other for fun.

A lot of people have a stigma around programming. But, at its heart, programming is the phrase “if-then-else.” If this, then this, else that. It’s not magic; it’s actually a fairly simple, mechanical tool. One of my former coworkers liked to refer to learning to program as a literacy issue—like learning to read. It doesn’t necessarily mean you’re a brilliant programmer just because you can write or read code; it just means that you know a language.

You don’t have to become a programmer now. You don’t have to go deep into those waters if you don’t want to. But I would urge anybody who’s interested in the creative—or problem—space of programming to not let it intimidate you. It’s actually not that big of a climb from where you already are. Our brains already work this way.

EVAN | I suppose also that there is the possibility that programmers might want to collaborate with old-fashioned theatre artists…

BAY | Yes, there is. I did a lot of theatre as a kid, and I have a real love of rehearsals. That world and that spirit are very welcomed in tech spaces. As the frame starts to disintegrate, there’s an open world of assumptions that are being rewritten. There’s a lot of creative space to explore here.
I would urge anybody of any age, at any skill level, to get your hands on software programming and look at what it is, what it isn’t, and where it’s going. It’s an interesting area that’s unfolding in front of us, both creatively from an audience point of view and from a creator’s point of view.