NOVEMBER 2022 // EBOOK

svconline.com

SOUND&VIDEO
CONTRACTOR
R E V I E W S | T E C H N O LO GY | A P P L I C AT I O N S

www
ww

THE NEW WORLD OF

HYBRID AND
VIRTUAL
PRODUCTION
 VIRTUAL AND HYBRID PRODUCTION

REAL NOT REAL

F or Karate Combat’s KC35 at Universal Studios, disguise
partner Meptik brought hybrid realities to a live audience/
live-streamed sports environment. The event merged the
ancient art of karate with xR, Unreal Engine VFX workflows, and set
extension for next-level engagement with live audiences and at-home
viewers.
For one night, Universal Studios turned into a dystopian Paris in the
distant future. The virtual set showcased a rough cobblestone road lead-
ing to the Eiffel Tower, the physical fighting ring placed right in the
center of the road.
The virtual set was surrounded by physical props that provided the
illusion of being on a dingy side street in Paris. The entire set seam-
lessly blended together the virtual and the physical to create one cohe-
sive environment.
Not only did the in-person audience feel immersed in the environ-
ment, set extension allowed the broadcasts to extend the physical set into
the virtual world using in-camera workflows.
At home, viewers did not see the studio’s lighting fixtures and trusses,
but experienced the live event in an detailed setting, bridging the best disguise, enabling creatives to make changes on the spot and enlivening
of both worlds. a virtual set extension just as Karate Combat principles had envisioned
The Meptik team integrated the xR and Unreal Engine workflow using See the full video here.

02
SVCONLINE.COM
Temporary or Installed
Planar Venue Pro VX Series LED video walls have
mechanical features to suit temporary (set up)
applications or fixed installation. Cabinets and
modules are removable from the front or rear for
easy service, and quick-install features like locking
pins, magnetic points an embedded quick-lock
system support fast setup and teardown for
mobile studios.

www.planar.com/venue-pro-vx �
�
 VIRTUAL AND HYBRID PRODUCTION

EXPAND THE
MESSAGE
W
hen Ford wanted to communicate in a fresh way with inves-
tors, disguise partner Meptik created a presentation that not
only impressed the intended audience with the new technol-
ogy inside Ford’s new line-up, it immersed investors in custom-designed,
fully branded virtual sets that took the audience on a visual journey.
The event kicked off with a high-level keynote in which the speaker
welcomed the audience and presented the newest developments on a vir-
tual stage. The virtually created reflections coming off the LED wall
immersed the speaker in a fully virtual environment and placed specta-
tors of a realistic setting that could shift realities to support the content.

With the press of a button, the location changes from a virtual stage
setup to the inside of a virtual innovation lab. The 90’ wide LED Vol-
ume was extended to infinity by virtually extending the digital content
beyond the volume.
Inside the virtual innovation design lab, Ford’s real-life keynote
speaker was able to show the full line-up of cars, inside and out without
ever having to physically take apart or re-assemble an actual vehicle.
A detailed, virtual replica of the technology behind the vehicles vir-
tualized the product and illustrated key pieces in detail to support the
speaker’s explanation of the technology.
Set extension around the 90ft. wide LED stage, realistic reflections
generated in software, and skilful natural lighting, allowed this extended
reality production to be truly immersive.

04
SVCONLINE.COM
 VIRTUAL AND HYBRID PRODUCTION

THE NEW WORLD

OF HYBRID
LIVE EVENTS

Moving from Meetings to Productions

By Eric Wenocur, Lab Tech Systems

06
SVCONLINE.COM
 VIRTUAL AND HYBRID PRODUCTION
n the wake of the coronavirus pandemic there is new enthusiasm for hybrid live events, in which
I presenters and a live audience are joined by remote participants, and usually a streaming audience
as well. While meetings in conference rooms might have some of the same aspects, in order to run
smoothly a hybrid event may require some of the complexity of live TV production.
The reality of hybrid live events is so new, with so much constant
change, that every setup is likely to be different. So this is not a simple
“how-to” article. The aim is to provide food for thought about what func-
tionality complex events may require, and what kinds of technologies
might be used. These concepts can apply to fixed or portable systems,
and to other situations, such as classrooms.
BUILDING UP THE SYSTEM
Imagine, if you will, a large conference space or small auditorium.
There’s a stage area in front with a presenter, and audience seating in
rows or tables. An A/V system handles sound reinforcement for the
presenter in the room, and projection or video screens to show the pre-
senter’s computer content. There might also be mics placed for audience
members to ask questions via the room PA. Pretty standard stuff.
Even before the pandemic, live streaming of events was becoming
common, so there might be one or more cameras, and other production
equipment, that feed a streaming encoder for viewers to watch online via
a CDN, Youtube, etc. That’s a basic one-way stream--viewing only--not
a video conference.
This scenario also applies to concerts, house of worship, etc. Before
internet streaming this would have been considered the “broadcast” out-
put of the event and, depending on the client, might still feed other deliv-
ery paths, such as fiber or satellite transport.
Now we’ll add a remote guest, connecting from home using a UCC
video conferencing app such as Zoom or Teams. We’ll call this video
teleconference (VTC) generically. At the venue, the app runs on a com-
puter and connects to the rest of the system via USB devices using UVC
A
and UAC protocols (generally built into the operating system). The
remote guest must be seen and heard by the live audience and presenter
in the venue and must be able to see and hear what’s going on in the
venue. The presenter and guest might engage in conversation. If the live
audience is going to ask questions the remote guest may need to hear
them. And if there is more than one remote guest all of this multiplies.
In addition, depending on the “production values” desired by the cli-
ent doing the event, there might be a need to play video clips or graphic
elements, such as show opens, intro and break slides, music, or back-
ground videos. These may be seen by the streaming audience and/or the
in-room audience.
Let’s also add teleprompter for the live presenter, or perhaps an event
host. The prompter output is intended only for the person reading (using
a stage-facing display of some sort) but it may also be useful to see,
or possibly control, the prompter in the central control room for the
production.
Finally, if the event is to run smoothly it’s likely that a crew will be
needed for stage management, wrangling mics, keeping an eye on the
07
NOVEMBER 2022
audience, perhaps running cameras, etc. And all of these people need
to communicate. This is typical for any live event or show, but adding
remote guests is a new twist.
PRODUCTION CONTROL
At this point let’s assume that there’s a central location where all the
incoming and outgoing signals meet, where there are operators running
equipment, and someone is directing the event. This could be the back
of the hall, a control room, or a remote truck. If the venue is built for live
events there might already be a control area and typical operator posi-
tions such as house audio and lighting. For the hybrid scenario described
here, you might also need:
• Producer who oversees the action and process of the live event.
• Possibly an Assistant Producer to handle communications with
inside and outside personnel.
• Director for the “broadcast” stream, to call camera shots and video
sources, call up graphics, etc.
• Technical Director to run the stream production equipment (or this
might be the Director).
• Camera Control operator for shading or robotic control (or this
might be the Director or TD).
• Playback operator to roll in videos and graphics (could also run
lower-third titles if used).
• One or more VTC operators to manage the computers and remote
guests.
• Someone to control what’s on the screens in the venue (or could be
shared with another job).
• A Production Audio operator to handle levels, mix-minuses, play-
back sound, etc.
• Prompter operator (could be shared if the demand is minimal)
• Someone to monitor the live stream.
Some of the job titles are kind of traditional from broadcast TV, but
the bottom line is that all of these functions need to happen, whether
assigned to individuals or shared. Some choices may depend on what
equipment is being used for certain functions, and how particular
devices interact. For example, if video clip or graphics playback is built
into the production switcher the TD might end up running playback--
even for what goes to screens in the venue.
One role that should not be shared is managing remote guests (VTC
Op). This is because the remote guests need to be kept “in the loop” as if
they were at the venue. Nobody wants to be hanging on a call, wondering
when they’ll be live. Plus VTC platforms tend to require some attention,
such as muting audio, watching chat boxes, and simply making sure the
connection is solid.
 VIRTUAL AND HYBRID PRODUCTION

Outgoing ("Broadcast") Stream Production

Separate devices or all-in-one platform
Conceptual Block Diagram of Described System

Playback Device Character

or Computer Generator

Local
Cameras Recording

Live Presenter on Stage

Video Production Switcher
Streaming Outgoing
Encoder Stream
(or other feeds)
Live Presenter
Computer

IFB Prompter Prompter

Belt Display Computer
pack

VTC
Computer

In-Room
Display Proc

Room A/V System VTC

(or direct connections Computer
Audience to production system)
Mics

In-Room
Displays
Producer
Talkback

Production Mixer

In-Room
Speakers
Lab Tech Systems
17413 Monitor Drive Olney, MD 20832
301-438-8270

Title:
Sound & Video
Contractor
Hybrid Event Article Illustrations

Notes Design EHW 8/22/22

Revise EHW

Refresh EHW

Dwg No.
Sheet 1 of 1

How much equipment is required, and what types, can vary greatly.
If the outgoing stream is supposed to have the polish of a professional
TV show there should be a production switcher, character generator (for
titles) and playback device, or a production platform (such as the New-
tek Tricaster) that does a combination of these functions. If the event is
being recorded on site that might require another device or be done in
the platform. Robotic cameras should have a physical control panel, and
of course there’s an audio mixer.
There will be computers for bringing in the remote guests--generally
one for each (although new options are appearing that allow separate
video of each guest from a single UCC computer). There might be a com-
puter or wall processor feeding in-room displays, depending on what
content they will show. Prompting will have a computer, whether it’s run
in the control room or in the venue.
Something to consider with all-in-one production products (such as
Tricaster, VMix, etc.) is that just because the platform CAN do every-
thing doesn’t mean it’s a good idea! For one, the operator will have to
divide attention between running the outgoing stream and other func-
tions. Secondly, there are cases where a particular signal needs to go to
several places. This can be a challenge if the production platform has
limited input and output capacity, and also creates complicated delay
scenarios (coming up). For example, some platforms include the ability
to connect with Zoom, Skype or other VTC options internally. But using
this feature may not be practical in a hybrid production when the remote

08
SVCONLINE.COM
 VIRTUAL AND HYBRID PRODUCTION
guest feeds are also sent to in-room displays (and makes it difficult to
offer good guest management).
SPEAKING OF TIMING
By now everyone understands that the live stream of an event is always
delayed. The processing required to encode at the venue, decode at the
CDN, re-encode for different viewing streams, etc. takes time (far more
than sending through the internet). In my experience Youtube Live is
generally 20-30 seconds behind. But that doesn’t usually matter because
the viewers have no knowledge of the real time event--they only know
what they get at their time.
When two-way communication is added, however, there are different
timing issues. As we’ve all found doing virtual meetings, the latency
of VTC platforms such as Zoom is surprisingly low, so not usually a
big problem--even with real-time conversations between presenters and
remote guests. This can vary unpredictably, and with changes in video
quality, but still is usually acceptable. (Ain’t nothin’ you can do about
it anyway. For best control over remote guest participation, and highest
quality, you must move away from popular UCC apps and into more
proprietary signal transport. That’s another story.)
Contact info@usabsen.com or reach out to Absen representatives for more information
09
NOVEMBER 2022
The issue now is keeping people’s picture in sync with their sound
(known as “lip-sync” in TV), because of processing delays in the pro-
duction equipment. We are talking here about delay in video frames,
which is 33mS/frame for 30fps video. I would suggest that a 2-frame
lip-sync error is on the edge of acceptable; beyond that things start to
look “off” to the average viewer. A self-contained production platform
might have 6-12 frames of latency--input to final output--which must be
accounted for in some cases. The most obvious is if audio is being mixed
outside the platform. If the audio is sent into the platform, and combined
with video, the delay will be handled internally. But if audio and video
are combined downstream the audio will need to be delayed to match.
This is also true for video through a conventional production switcher,
but the latency is often a frame or less.
Lip-sync issues also arise because remote guests are being seen in
several places at the same time. When guest audio and video leave the
UCC computer, via a USB converter of some sort, they should be (nomi-
nally) in sync. That a/v goes into the production system and becomes
part of the outgoing stream.
But the remote guest also has to appear on in-room screens, with
matching audio coming from the house PA. The guest audio coming
SCAN TO GET MORE DETAILS
 VIRTUAL AND HYBRID PRODUCTION
through the production mixer will be real time. But if the video gets to
the screens by way of an output from the production system, it will be
late and the audience will notice a lip-sync problem. With a digital mixer
it may be possible to add delay to particular inputs or outputs to line the
a/v back up in the room.
Another scenario is if the in-room screens are fed from their own
computer, let’s say to give the audience a finished-looking image, or
show several guests together, TV-news style. Again, the remote guest
audio must be delayed to match the video that finally gets to the screens.
Fortunately, when it comes to lip-sync, there’s only one variable:
Latency in video paths. Nothing in any audio path--not preamps, mixers,
converters, cable or any processing--will delay audio significantly. If
your audio is behind your video there’s a delay dialed into some piece of
equipment, or something is broken. (Yes, wireless mics can sometimes
take 10-20 mS, but that is still much less than a video frame.)
Also keep in mind that evaluating lip-sync can be tricky with all the
types of displays and signal paths. Assume any flat-screen display might
be a frame behind all the time (an issue CRT monitors did not have).
Also assume a basic video production switcher (not an all-in-one plat-
form) will take a frame for processing non-genlocked sources. And the
“multiview” output of any switching device is at least a frame behind,
probably more. So even under the best conditions monitoring lip-sync
in the control room may be sloppy by a few frames--generally not a big
deal. As with any TV production, the ideal is to have a “reference” dis-
play that shows the final output signal--unadulterated by any additional
processing--to use for judging image quality and lip sync.
COMMS
Communication within the control room, and to venue crew, is critically
important and can be handled fairly easily with any number of wired or
wireless intercom systems. A single “party-line” (where everyone is on
one channel) may be fine. Sometimes it’s handy to have two channels,
so that tech and producer conversations can be separate. Communicating
with presenters on stage and remote guests is where it gets complicated.
A live presenter may wear an earpiece so that IFB (interrupted fold-
back) can be used. In this case they will get a feed from the production
mixer (often known as “IFB Program”) that includes whatever they need
to hear--typically the remote guests. The producer or director inter-
rupts (or talks over) that audio when necessary. IFB with true interrupt
requires a more sophisticated intercom, while simply talking over the
Program audio can be done via the mixer. (This does not address how the
producer or director audio gets into the mixer or intercom.)
The ability to talk to a remote guest from the control room (not
through a mic out in the venue) is tricky, but quite important as part of
guest management. They need to know what’s going on, and high-profile
guests, who are used to doing TV news interviews, will certainly expect
communication from the producer or director. In general, as of this writ-
10
SVCONLINE.COM
ing, UCC platforms using USB converters assume one audio input and
one audio output (“stereo” notwithstanding). The remote guest computer
gets a mix that contains the on-stage presenter, possibly audience mics,
and playback audio. This should be a mix-minus feed, meaning the
remote guests get a full mix minus themselves, to avoid echoes. Here
again, if the producer needs to speak to the guest, that audio must be
added to their mix-minus, or must interrupt the mix-minus somewhere
between the mixer and the remote guest computer.
Audio and comms is one of the hardest areas to implement with UCC
conferencing because these platforms were not designed for individual
communication. Everyone who joins the conference hears all the audio,
and what comes out of the UCC computer is a mix of all participants
(although that is slowly changing). So even when done well, the back-
channel communication between a producer and guest is actually heard
by anyone else on the same call. In a case where remote guests should
not interact with each other it’s necessary to connect each guest to a
separate call and combine them in the control room to keep their audio
isolated. Another comms option comes in the form of internet-based ser-
vices, such as Unity Intercom, that allow cell phones to be part of an
intercom network. The process is administered from a central applica-
tion running on a computer, perhaps in the control room, or can be cloud-
based. It can also be interfaced to conventional wired intercoms. An app
like Unity can add functionality when personnel need to participate in
the production from outside the venue but can’t take the place of a dedi-
cated intercom system for handling specific individual communication.
A fully functional intercom that covers all the needs may require more
equipment and expertise than one might expect.
IMPORTANT POINTS TO NOTE
• A full-bore live-streamed event, with live audience and remote
guests, is more like a television broadcast production than a
teleconference.
• There are essentially three audiences being served simultaneously:
The live audience in the venue, the live stream or broadcast view-
ers, and the remote guests. All three require somewhat different
content and communication.
• Don’t underestimate the need for managing remote guest(s), includ-
ing the ability for people in the control room to talk to them.
• The production mixer will need to create several different mixes for
different purposes: Final show output to stream, IFB for live pre-
senter, mix-minus(es) for remote guests, possibly room PA. Some
inputs or outputs may require delays added to fix lip-sync issues.
• As convenient and ubiquitous as they are, UCC platforms create
specific limitations for including remote guests in a production.
Other approaches (such as using a platform like VMix as a remote
guest gateway) may offer more flexibility and control, but will
require guests to use different applications or equipment.
 VIRTUAL AND HYBRID PRODUCTION

VIRTUAL
PRODUCTION
FOR PRO AV
Not just for Hollywood
any more
By Cynthia Wisehart

irtual production (VP) is riding on two waves—the technical

V advances made with cinema blockbusters like The Mandalo-

rian, and the rising demand for professional video to deliver
engaging content to hybrid and remote audiences.
Throughout pandemic, corporations, live entertainment venues,
churches, schools, and eSports arenas became familiar settings for vir-
tual and hybrid events and productions. Users and technical teams saw
the benefits as they became more comfortable with digital interactions
and virtual or hybrid workflows. It became clear that virtual and hybrid
experiences could reach remote audiences while deepening engagement
and interactivity, supporting new business models, and widening access.
Now people want and expect better production values to continue that
trend forward, providing a more professional approach to live streamed
and recorded hybrid content, and offering options beyond the customary world demonstrations of VP stages. These demonstrations showed that
talking head and Zoom square configurations. even a small footprint can support a powerful virtual stage, and how
This is an opportunity, and it can also be a challenge without the familiar gear such as LED displays, cameras, and media servers can
proper technology and support. Specifically, the Pro AV channel needs come together in this new type of platform. Pro AV always had their
VP partners who understand Pro AV–the unique hardware and software hands on these pieces; this puts a different spin on it A virtual studio
components, and the importance of sustainability and service after the can be a value proposition for Pro AV end users and an opportunity for
sale. This VP trend also introduces a new kind of collaborator into the integrators to provide a differentiated and forward-looking service.
mix—content experts who understand dynamic content, extended real- Whether in its simplest or most ambitious form, VP has a few impor-
ity (XR), and hybrid live experiences. tant underpinnings for success.
A VP environment can be everything from a simple green screen
replacement to an extended reality studio to a virtual world. It can be KEY COMPONENTS
used to expand worship outreach across the country and the world, it can In September, Planar expanded the VP and XR product portfolio with
provide business leaders with a platform to reach a global workforce and the Venue Pro VX Series, a cost-effective solution with enhanced fea-
shareholder community; it can bring a virtual concert experience to far- tures for maximized in-camera visual performance.
flung audiences. Pro AV can draw from techniques that broadcasters and
filmmakers have learned on the XR stage and apply them to applications LED Screens: Early applications of virtual production adapted pretty
closer to home. Imagine for example, a medical instrument company successfully to the existing capabilities of LED screens, many of them
demonstrating their tech from within a virtual operating theater, or a from the rental and staging markets. However since then, manufactur-
CEO addressing employees, customers, and shareholders from inside a ers have added and refined new capabilities that support the in-camera
digital model of the company’s factory. required by VP, installation of virtual studios, and more permanent
The potential for VP was made visible at InfoComm22 with real- integration with components including cameras, servers, and motion

12
SVCONLINE.COM
 VIRTUAL AND HYBRID PRODUCTION
“Virtual production is
where the physical and
digital worlds meet.”
—Weta Digital
capture. Many of these features reflect feedback from users. Today’s
screens are more flexible; cameras and displays work better together and
excel for the specific demands of on-camera performance.
LED screens have been used for this purpose for a long time in broad-
cast; moire has been vanquished. But there are other artifacts that can
affect a virtual studio set up: communication between display and cam-
era, and attention to the optimal synchronization matters. Modern LEDs
can deliver curved XR and VR environments as well as floor and ceiling
screens for truly immersive applications. These days you can choose
displays that have been designed for VP and tested for it.
By creating an entire scene live on an LED screen, visible both
13
NOVEMBER 2022
through the camera and to the presenter on set, it’s possible to avoid
“green-screen fatigue,” a common problem with actors or broadcasters
who perform against little more than a mark on a green scrim. This can
occur even more so for users in corporate, worship, and even live events
settings–a flat, unchanging screen can be too disconnected. The right
live LED video wall gives people clear visual context, keeping them as
engaged as they would be in a real place.
Best Practice: Layer the visuals on stage to add to depth of field.
The screen carries the load as the dynamic background, but adding
physical elements like a table or potted plant bring additional reality
to the visual.
Cameras, Lighting & Camera Tracking: Recent years have seen con-
siderable advancement in tracking cameras; depending on the desired
complexity of a virtual studio there are automation levels and manu-
ally driven options for any budget or application. Studio lighting doesn’t
have to be expensive but it does have to be optimally specified. The
incorporation of motion capture camera tracking in virtual sets signifi-
cantly improves the perspective of the presenter, making the set move
appropriately to convey accurate environments.
Best Practice: Have enough lights and flexible options. Light the fore-
ground and the people. Storyboard the shoot to understand how cam-
era movements (if any) and different lighting scenarios might be needed.
Allow studio time to tweak the stage and rehearse, and to deploy more or
different lighting. This is one of the benefits of a permanent virtual stage
where you can refine lighting needs and lock them in through repeat use.
Media Engines: The media engine is the heart of any virtual studio. The
Unreal Engine has grabbed the consumer spotlight and raised awareness
of the incredible capabilities of these processors and their accompanying
software. However, closer to home for Pro AV, familiar media servers
provide a range of capabilities. Those manufacturers are natural partners
in specifying a virtual studio.
Best Practice: Include your display and media engine partners early.
They can help you optimize a small, highly functional studio or go big
with a brand-defining virtual stage. Either way they can help deliver
accessible, professional, repeatable infrastructure for an ROI-friendly
studio.
KEY EXPERTISE
Content: Just as the right technology partners can help you design a
studio that is right-sized for your client, content experts can help set the
all-important dynamic content strategy. As virtual production expands
into Pro AV, so does networking. Relationships–among manufacturers,
integrators, and content experts–set the (virtual) stage for success. As
just one example, disguise recently purchased Meptik studios, a leading
provider of virtual content, XR, and VR. Look for more of that kind of
cross-pollination. Virtual production does not exist in a vacuum. Under-
stand if your client is going to do in-house or third-party content and
media management. Engage with your client’s content needs and be a
resource for other experts.
 VIRTUAL AND HYBRID PRODUCTION
In September, Planar expanded the VP and XR product portfolio with
the Venue Pro VX Series, a cost-effective solution with enhanced
features for maximized in-camera visual performance.
Integration and Calibration: Understanding content strategy also
means understanding the visual language the client wants to use. Will
the camera be moving—now or in the future? Do you want it to be
tracked? Will the speakers be moving? Will your client use fixed cam-
era plate shots similar to greenscreen, or will they need a “world”, an
XR environment? In between those options is a range of choices that
affect everything from technology choices to media partners. Wherever
your client lies on that spectrum, your manufacturers can help network
14
SVCONLINE.COM
around their needs. Today’s VP customers may want to do camera tests
against LED backdrops to put together the right combination. Some
LED manufacturers can make facilities and mockups available for that
exploration, and extend into calibrating the finished stage. The camera,
motion capture, virtual world, display, and lighting can and should flow
and look natural.
Distribution: Where will your audience see your content? YouTube?
Intranet? Broadcast television? Part of a content strategy is to determine
the distribution platform. For some corporate clients that may mean
multiple platforms, for internal and external communications, so needs
assessment is key. There are different levels of customers–some who
want to simply elevate from green screens to a modest virtual stage and
streaming, which most clients can transition to pretty readily. For those
who want to make content from a virtual world for distribution across
multiple live and post-produced platforms, expertise is more specialized
and knowledgable partners can help.
CASE STUDY: INSIDE PLANAR STUDIOS
Last June the company opened Planar Studios, an initiative designed
to revolutionize the virtual production (VP) and extended reality (XR)
markets. Planar saw the change to combine advanced technologies,
industry partnerships, and personnel to create realistic on-screen con-
tent for VP and XR industries in entertainment, corporate, and education
 VIRTUAL AND HYBRID PRODUCTION
market. The cornerstone is the company’s in-house fine pitch LED dis-
play and motion capture expertise through sister company OptiTrak, as
well as relationships with key industry partners on the processing side.
Planar solutions for VP and XR are being deployed worldwide, includ-
ing at Orbital Virtual Studios, a Los Angeles-based VP studio dedicated
to producing movies, music videos and commercials from pre-produc-
tion design and development, to production, photography and all the way
through post-production.”
Through the Planar Studios initiative, the company leverages Planar’s
established leadership in on-camera broadcast studio installations. The
company has worked directly with creative and technical professionals
to design LED solutions that deliver unmatched in-camera visual perfor-
mance and deployment versatility needed to develop life-like recorded,
streamed or broadcast video content.
Planar Studios is supported by a dedicated VP and XR team and
strengthened by collaboration with industry-leading partners who spe-
cialize in digital production and developing next-generation visual
storytelling tools. While LED display technology is a significant compo-
nent of VP and XR environments, successful studio implementation also
requires a broad range of hardware, software and services. Planar works
closely with leaders delivering in these areas, including Epic Games
15
NOVEMBER 2022
Unreal Engine, disguise, Planar’s sister company OptiTrack, Brompton
Technology, Colorlight and more.
“We are proud and honored to work with the world’s leading creative
and technological professionals as we set out to perfect the art of VP,”
said Executive Vice President Adam Schmidt. “Recent developments
in VP and XR are already unlocking wonderful capabilities for story-
tellers, but we’re just scratching the surface. We recognize advanced
technological development, productive industry partnerships and a dedi-
cated team of experts to be key in helping customers overcome creative
limitations. Planar Studios is our response and will be a driving force in
unlocking the future of VP and XR.”
WHERE DO I START?
Virtual production is on the rise and rapid advancements that make it
better, easier, and more efficient for Hollywood will continue to adapt
into Pro AV. It’s shaping up to be a collaborative future–look for pro-
viders who are attracting expertise, building connections, working with
studios, universities, and agencies. Like many advancements in Pro AV,
this is not about selling boxes or screens, it’s about building a new AV
platform together and adapting one of the most exciting trends in tech-
nology to our industry.
 VIRTUAL AND HYBRID PRODUCTION

VIRTUAL PRODUCTION
IN THE REAL WORLD
Tying it all together with PTZ cameras and LED Volumes
By Erik “Wolfie” Wolford

’ve never had a harder time explaining to my friends

I what I do for a living: “I’m a cinematographer. I

specialize in shooting LED Volumes. It’s like a
greenscreen, but instead of the greenscreen we
have these huge video walls that are 80 feet wide by 20 feet
tall—we play the background material, and we film the actor
standing in front of it…” Blank stares are the most common
response.
The virtual world seems to be at the top of conversations
about the future of entertainment, community, and social
experiences. But, for those of us in production, we’re
no strangers to new visual effects technology. I’ve shot
greenscreen. I’ve filmed using rear screen projection. I’ve even
filmed using front screen projection back in the late ‘90s.
But, there’s no denying that virtual productions have
exploded in just the last two years. One of the main drivers
of this was Covid, as suddenly many companies specializing
in music tours and the like suddenly needed to repurpose their
video walls. Combined with the trend of actors being willing
to film locally on a soundstage, as it meant they didn’t need
to climb on an airplane and risk exposure, virtual productions
took off. In terms of big budget options for virtual production, we’ve all seen
And beyond these necessary pivots, the technological barriers to using the magnificent achievements of The Mandalorian, which introduced the
this method have recently reduced significantly—we’ve seen improved virtual production concept to a broad audience who found themselves
camera technologies and innovations powering growth, such as PTZ “fooled” by the sweeping desert vistas, all composed of tiny pixels
cameras, LED walls, game engines, increased computational power, instead of tiny grains of sand.
and an explosion in the quality and quantity of new higher quality LED But LED volumes are being used at all budget levels of production,
panels. even in unexpected places like live comedy shows. I recently shot a
We are now in a defining moment of how far virtual production has comedy series with Laugh Lounge in which we employed a wide range
come and where it’s going next. of Canon cameras to capture the live performances. We also usually
have one or two Canon CR-N500 PTZ cameras with the Laugh Lounge
VOLUME WORK IS HERE TO STAY Comedy Club, which produces 150 live stream sets a year.
In my work as a cinematographer, I’m witnessing volumetric stages and Typically, these performances need to feature differentiated sets and
virtual production stages popping up all over America—Los Angeles backgrounds that fit the style and tone of the comedian, but they don’t
alone has a large and varied selection of stages to choose from. Elements typically need to be hyper photo-realistic or complex—it’s more of a
of virtual productions are being integrated into productions across graphic aesthetic. We might have an LED wall as a passive background,
the country. In fact, I just finished shooting a season of a TV show at but also might include segments utilizing the power of a game engine.
Brandstar stages in Florida, which featured practical sets with virtual set One of the things we found in our live shows is when you get two world
extensions. Volume work is here to stay. class comedians improvising with each other, you can find spontaneous

16
SVCONLINE.COM
 VIRTUAL AND HYBRID PRODUCTION

I recently shot a comedy series with Laugh Lounge in

which we employed a wide range of Canon PTZ cameras to
capture the live performances.

comedy gold. To take advantage, we’ve crafted graphic 3D elements that

can be used in an improv segment to be visually stimulating and a leap
off point for the comedians’ banter.
PTZ cameras, which now mostly have genlock and allow us to control
and match color in real time and produce a cinema quality image, shine
in settings where there is a passive, non-tracked LED wall that is purely
decorative. And when shooting these shows, the need for both high and
low angles, as well as pans of the audience for reactions, all lends itself
nicely to the use of PTZ cameras. They sit in an arsenal of cameras
available for virtual production, and can be a unique tool when used
correctly.

VIRTUAL FLOOR AND CEILING

Another valuable use case for virtual production is for large-scale live
concerts, in which we typically set up a tracked camera and utilize an
LED wall with graphic imagery and animations. I was cinematographer
for a similar production for a Riot Games event and its musical guest
Porter Robinson—his background animations were highly graphic and
esoteric and harken back to the animated quality of the IP that we were
tying in. We created a virtual floor and virtual ceiling so that we could
grab high angle shots and dramatic low angle views and still make sure
everything was cohesive. We had a short window of time to get all our
shots. From those constraints, I realized that sometimes I’ll need more
coverage.
I subsequently began testing out a PTZ camera to see how it would
look in such an aggressively LED environment. The PTZs are great—
easy to rig, easy to remote control, and nowadays much better sensors
and glass than one would expect to find in a PTZ. They now have camera
tracking built in—as long as the camera base itself stays put and doesn’t
move. All of this works together to provide fantastic coverage and a final
piece of content that reflects the imagination and visual wonder of an
artistic performance.
With the explosion of the aforementioned technologies including LED
video wall technology and remote controllable cameras, we’re seeing the
possibility of remote directing or remote cinematography operating outside
of the venue and even out-of-state—again, a great option for producing
comedy shows, live music, or other monetizable live performances.
For example, the Laugh Lounge is looking at remote content recording.
Being able to install a handful of PTZ cameras in a comedy club in
Austin, log in from a control room in L.A., operate and live cut a comedy
show and have it up on the Laugh Lounge channel streaming live is an
amazing use case for the PTZs. The color science in these cameras has
improved greatly and can mix in well with cinema cameras like a Canon
Cinema EOS C300MKIII or C500MKII so with certain live streaming
situations, it totally fits the bill.
WHEN YOU CAN’T BE THERE IN PERSON
“Content is king” is still the mantra and for us in production, it means
that it needs to get made, no matter what. Remote options become more
attractive for producers and clients who might not be able to make it to
set for health or financial reasons.
Virtual production with telecommunication through a PTZ camera is
a way to help ensure that clients can still have an in, and it’s a huge boon
for working with above and below the line talent from around the globe.
Recently, I was the cinematographer on a commercial/ music video shoot
and the director couldn’t make it to the set—instead of shutting down
for the day, we opted for him to watch and direct the set via remote
control PTZ camera. It all felt a bit futuristic, but ultimately valuable in
the moment. PTZs can be a “Swiss Army Knife” in this way.
The last couple of years have introduced the power of virtual
productions to many in our industry. While virtual productions won’t be
right for every project, it can greatly improve the process and open doors
for new endeavors. I’ll be riding the wave of technological innovation
and excitement in the industry to make virtual production even more of
a reality.

In my work as a cinematographer, I’m witnessing

volumetric stages and virtual production stages
popping up all over America
17
NOVEMBER 2022
 VIRTUAL AND HYBRID PRODUCTION

SMPTE STRATEGY
n response to requests from members to speed up its work on an area that could benefit from SMPTE’s expertise and because the tim-

I new technologies, SMPTE has launched a new initiative, the

Rapid Industry Solution (RIS) and laid out plans to provide
guidance and educational materials that will provide ways to
ing was right. 
“The time to do this is now, because the ecosystem is very much in
technical and creative flux,” she explained. “It’s early enough that we
improve and simplify virtual productions.  can watch for new developments,” and at the same time, “take areas that
SMPTE also introduced an advisory panel that will be working on its are approaching maturity” to develop a standard or best practices in cer-
new On Set Virtual Production initiative.  tain areas that can be more widely used in virtual production. 
The panel will tackle a host of issues that the industry is grappling with SMPTE announced that the first members of the advisory board are: 
as virtual production is more widely used. These included such issues • Raed Al Tikriti, chief product officer at Disguise
as interoperability for virtual production technologies, workflows, best • Wyatt Bartel, executive vice president of production at Lux Machina
practices, developing a unifying common language, artificial intelligence, • Des Carey, head of cinematic innovation at Samsung Research
augmented reality, using higher quality video and audio in virtual produc- America
tions, the need for better training and the involvement of universities, both • Chris Swiatek, co-founder and chief of product at ICVR
in terms of training students and in conducting research.  • Pat Griffis, vice president of technology, Dolby Laboratories
SMPTE president Hans Hoffman and head of media fundamentals • Chaitanya Chinchlikar, vice president, business head, CTO and
and production at the European Broadcasting Union noted that SMPTE head of emerging markets at Whistling Woods International
members had been telling the organization that it needed to find ways • David Long, director of MAGIC Center/MAGIC Spell Studios and
to speed up the pace of its work on new technologies. The new Rapid a professor at Rochester Institute of Technology
Industry Solution initiative addresses those concerns by enabling “us to • Richard Welsh, governor at large of SMPTE Board
be more agile and faster, delivering value to the industry at a much faster Looking forward, Raed Al Tikriti, chief product officer at disguise,
scale than with the work we’ve done in the past,” Hoffman said.  noted that “we see significant growth in this space given all the benefits
The organization chose virtual production as its first RIS effort that virtual production has to offer.” But as it becomes more widely used,
because the volume of virtual production has exploded and because vir- the content being produced “is getting bigger and more demanding” with
tual production was at a point in its development where work on such higher resolution video formats and improved audio systems, he said. 
issues as interoperability, workflows, and best practices would provide That is causing people to push the envelope with new technologies and
an immediate benefit to its members and the industry. makes “the ability to collaborate to create content increasingly impor-
“It’s kind of the wild west out there and if you look at virtual produc- tant,” he said. 
tion, everybody’s driving, or using a different type of transportation, This also highlights “the need for interoperability between content
from as simple as a bicycle to as advanced as a space shuttle,” explained creation tools,” he added.  
Kari Grubin, a consultant and a long-time media and entertainment Des Carey, head of cinematic innovation at Samsung Research Amer-
executive who presented a detailed outline of the RIS and virtual pro- ica added that Samsung is “taking everything we know from postpro-
duction initiatives.   duction, from theatrical exhibition, from home displays and bringing
Grubin said that SMPTE’s “role is not to design these vehicles but to [that knowledge] into virtual production” to achieve more consistency. 
really provide the recommendations and the rules of the road to accom- Chris Swiatek, co-founder and chief of product at ICVR added, “I
modate whatever vehicles the creative community” chooses to use.  think we’re really going to see the formation of a more standardized con-
One major challenge is interoperability.  “The file formats don’t trans- tent pipeline between real time engines and stages. “Doing that...without
late across all platforms, there’s over 300 file formats and game engines loss of visual fidelity or metadata...is going to be really important.”
combined,” Grubin said. “More than half of those are proprietary, so Swiatek also stressed that “blending real and virtual environments
they don’t necessarily move very easily between system to system.” seamlessly” will become more important as LED wall virtual produc-
Grubin stressed that SMPTE chose virtual production because it was tion becomes more widely adopted.

18
SVCONLINE.COM
 VIRTUAL AND HYBRID PRODUCTION
VIRTUAL ESPORTS
M
o-Sys Engineering has successfully supplied its LED content
server solution, VP Pro XR, via regional partner Cgangs for
a major esports championship production in Asia.
Cgangs was selected by American games developer Valve to handle
the production of the opening sequence and top-four team player intro-
duction videos for the International Dota 2 Championships, a major
esports competition held in Singapore during October 2022, with a
reported prize pool of $18.9 million.
The integrator was introduced to the Dota 2 project in July 2022 and
shortly after welcomed the Valve software team to its studios in Singa-
pore. After shooting a sample video based on an animatic demo provided
using green screen virtual production, Valve was able to decide on the
same day to work with Cgangs.
“The project made extensive use of Live AR, and it all needed to be
shot in a single day without post-production,” Alvin Lim, Cgangs stra-
tegic partnership and business development director explained. We only
had about 1.5 hours per team to shoot and capture the opening and intro
videos, which meant we were shooting real-time ICVFX and needed
rock-solid camera tracking and a powerful LED content server solution
that wouldn’t let us down.”
Cgangs utilized VP Pro XR, its LED content server solution designed
specifically for film and broadcast production, and combined with Mo-
Sys StarTracker camera tracking, delivered high-quality Live AR. The
entire system was integrated with real-time DMX controlled lighting and
19
NOVEMBER 2022
an LED ceiling to increase the production value and to assist the cinema-
tographer by providing real-time ‘scene spill’ on the talent. Mo-Sys also
introduced Garden Studios who provided expert on-site technical support.
“We had limited time on-set so needed to work quickly,” Lim added.
“We were able to do this after ample pre-production planning and select-
ing products that would enable rapid ICVFX production. StarTracker and
VP Pro XR were perfect for this project, both in terms of set-up, work-
flow and reliable performance. It was also a great pleasure to work with
the DOP from Valve, who was using LED virtual production for the first
time. He had a very clear vision and worked collaboratively with our
technical team to achieve it, and this was essential to making the project
a successful one, with very happy clients.”
“It has been a great pleasure to support Cgangs, and it’s fantastic to see
virtual production together with StarTracker and VP Pro XR utilized for
such a high-profile eSports project,” said Stephen Gallagher, Mo-Sys’
marketing director. “VP Pro XR is enabling more and more projects to
achieve exceptionally high production value through its cost-effective
and dedicated feature set for film and broadcast.”
Following the DOTA 2 project, Cgangs has announced plans to posi-
tion itself as a technology enabler to support and train media students
and industry professionals. The company plans to set up a permanent
LED VP Studio to enable local production companies to have the oppor-
tunity to learn and apply VP for their own projects, as well as to become
the leading VP technology production company in the Asia region.