Lighting Design and Technology Student Explores XR Extended Reality Production

The Lighting Design and Technology program at UC's College-Conservatory of Music is teaching students how to create extended reality productions that use digital media as virtual sets for actors on stage.

By now, you may have seen The Mandalorian on Disney+. You know, the Star Wars spin-off featuring "Baby Yoda" (no spoilers on his real name) and out-of-this-world visual effects. The series was one of the first to use a groundbreaking extended reality (xR) production method that is now being explored at CCM.

Extended reality (xR) is a production technique for the stage and screen where virtual scenery is rendered around an actor or subject in real-time from the perspective of the video camera capturing it. The technique is similar to working with a green screen that uses digital environments, but unlike a green screen, xR environments are actually placed in front of the camera lens.

"When first picturing this concept, it is easy to think about this workflow in terms of a backdrop," says MFA Lighting Design and Technology student Jeremy Mayo, who is exploring xR production for his master's thesis project. "Instead of painting a backdrop for actors to stand in front of, we can replace that backdrop with a video screen of sorts, which is able to change to show any content that is needed."

The digital scene is programmed onto screens that make up the set behind the actor. Directors and performers in an xR production can see the digital environment in real time instead of imagining it. Additionally, the xR digital sets create their own light that matches the environment, whereas green screen productions require significant lighting and post-production editing in order to create realistic lighting effects.

"Since the computer systems needed for xR are extremely powerful, it is possible to rapidly recreate digital worlds," Mayo adds. "Artists working with extended reality setups are able to rebuild scenes in a few minutes or even a matter of seconds depending on the change needed. This is an incredible tool as directors can request changes to the shots and environments they are filming without wasting any additional production time."

Mayo first explored xR production techniques over the summer while researching The Mandalorian. He was immediately captivated by the technology and wanted to explore how it could be used in the performing arts. He set out to create three virtual scenes for his "Project Holodeck": a graveyard, a firework display in the woods and a studio production set.

"When the pandemic started, it completely decimated the entertainment industry (and it still is)," Mayo says. "Typically, in a normal timeline, I would design a large-scale production as a thesis project. However, the pandemic prevented CCM from staging [large-scale productions] this year. After discovering xR via Disney's The Mandalorian, I immediately knew xR could and would have a massive effect on how we create art, particularly during a pandemic, so I chose to see if I could learn and produce a production using an xR stage that I built myself."

Last spring, CCM approved a major media software and hardware acquisition for the lighting department to make sure students have access to professional-quality tools of the industry. This made it possible for Mayo to explore xR at CCM this fall.

To create a proof of concept functional xR stage, Mayo used UnReal Engine, a 3D video game creation tool and management software. He also used the Disguise Media Server, which is a software/hardware solution for managing complex video and projection systems. Additionally, Mayo used an HTC Vive Tracker so the virtual environment on stage and the video camera shared positional data with each other - this made the set responsive so when the camera moved the whole 3D world moved on stage.

"This was my first time working with many of the tools and working specifically with xR," Mayo says. "Though there is some crossover into traditional lighting and projection design. I'd say my experience as an entertainment designer allowed me to approach this project more easily."

CCM Professor of Lighting Design and Technology Sharon Huizinga says it's important that students are familiar with this technology because it's being used in a myriad of ways throughout the industry. Recent examples include Katy Perry's performance on American Idol, the 2020 MTV Video Music Awards and Billie Eilish's "Where do we go? The Livesteam" concert. "Although it's existed before, now it's sort of blown up because it's the way forward if you want to do a major festival or live concert or awards show," Huizinga says.

"I look at my program in terms of maximum employability for my students and to me, it would be unconscionable to ignore the media aspect of it," she adds. "There's a million jobs in lighting, but there is also a million jobs in projections. If someone isn't into extended reality, you can still make content for rock shows, or maybe you want to be a Broadway projection designer, or maybe you don't like any of that and just want to get into the equipment."

She believes in partnerships with manufacturers so that students have unique access to the latest technology available in the industry. She has a partnership with Disguise to give CCM student access to the necessary hardware and software needed to explore xR production techniques. Huizinga has also hosted master classes with guest artists from Disguise to connect students with a professional network.

"I would like to create a framework where students have maximum access, and they have pathways to develop relationships with designers, experts, specialists, manufacturers and rental houses," she adds. "I'm just saying 'hey you need to know that, I'm going to bake it into one of my classes so that you have exposure to it.' It feels like a great way forward."

Mayo plans to pursue xR jobs as one option after he graduates, thanks to the opportunity to work with the production technique at CCM under Huizinga's guidance. "A good professor makes you a better student, a great professor makes you a better person, the best does both," he adds. "That's Sharon."

"This is one major success story of 'wow, here's a student who is using this equipment that no more than a tiny handful of people on an academic level in the U.S. have access to," Huizinga says of Mayo's work. "Students are walking out with a big degree of confidence that is uncommon. The field of choice just got massive - do you want to go into Broadway or dance performances, live concerts, architecture lighting or award shows, or light art festivals? I feel like we just doubled the job opportunities for every student in the lighting department."

To post a comment, you must register and login.