news

Society | DOI:10.1145/3565978 Esther Shein

Preserving the Past with

Immersive Technologies
Digital technologies such as virtual reality, augmented reality,
and holograms allow museums and other institutions to preserve
historical events and tell stories in an engaging way.

I
Holocaust Me-
N S I D E T H E U.S.
morial Museum in Washing-
ton, D.C., is an exhibit called
the Tower of Faces (https://bit.
ly/3cxI3Ik), which uses aug-
mented reality to tell stories behind
some of the 1,041 photos of people
from the small town of Eishishok,
in what is now Lithuania. The tower
soars 50 feet high across 30 rows,
displaying the faces of the town’s in-
habitants, nearly 4,000 of whom were
massacred when the Germans invad-
ed during World War II.
When visitors walk into the tower,
they can pick up one of several iPads
and hold it up to an image on the
wall, which will then play a video that
transports them into the town. The
video first appears in color and then
fades to black and white, while a nar-
rator reads a brief script about the
person. One tells the story of Szeina,
an actress who was fluent in five lan-
guages and owned a hotel on the mar- The interactive Heroes and Legends attraction at NASA’s Kennedy Space Center Visitor
ket square the Nazis took over to use Complex in Florida, featuring the U.S. Astronaut Hall of Fame.
as their local headquarters.
“It’s a beautiful experience … there bring them to life, just for a moment, historical events and places that are
are many, many photos looking out and have it not just be a memorial but long gone in an immersive way—kind
at you—people riding bikes, outside have victims of the Holocaust come to of like IMAX taken to the next level,’’
in the snow, at a wedding banquet, life and treat them with respect and explains Tuong Nguyen, a senior prin-
and on the stairs of their houses,’’ engage visitors to really see them as cipal analyst at Gartner. “So instead of
says Sarah Lumbard, director of mu- people,” Lumbard says. just seeing it on a flat screen, like TV
seum experience and digital media at Digital technologies such as virtual or movies, or seeing it all around you
the Holocaust Museum. The photos reality (VR), augmented reality (AR), like an IMAX movie, VR can potential-
depict people of all ages prior to the and three-dimensional (3D) graphics ly enable people to explore that space
massacre in September 1941. are making it possible for museums with 360 degrees of freedom.”
Many of the photos were supplied and other institutions to preserve his- AR can change how someone expe-
IMAGE COURTESY OF KSC VISITOR COM PLEX/NASA

by Yaffa Sonenson Eliach, whose
grandmother was a photographer in
Eishishok and is herself a survivor.
The immersive experience opened in
April 2022 and the idea was to “cre-
ate a spark of life” so the residents of
Eishishok will be remembered, Lum-
bard says.
“Our question was: How do we
torical events and tell the stories of
those events in an engaging way. In
the case of VR, the technology actual-
ly takes them to another time or place
away from where they physically are.
As research firm Gartner says, “the
future of digital experience is immer-
sive” (see https://gtnr.it/3BqmqU2).
“In this way, we can experience
riences the world in front of them or
around them, usually in a visual way,
integrating information such as text,
graphics, and audio with real-world
objects. “The idea is to show users
how things looked in the past with ac-
tual video and photos from these time
periods,” Nguyen says.
For the Smithsonian’s National

DEC E MB E R 2 0 2 2 | VO L. 6 5 | N O. 1 2 | C OM M U N IC AT ION S OF T HE ACM 15

news
Museum of Natural History, also
in Washington D.C., the impetus
behind developing a mobile app
called “Skin and Bones” (https://s.
si.edu/3wGy0aT) using 3D augmented
reality and 3D tracking was to attract
more visitors to its Bone Hall by tell-
ing stories about some of the speci-
mens on display.
Visits and dwell time in the hall
and the experience people were hav-
ing in the Bone Hall “fell far short of
any measure of what a visitor experi-
ence should be in a modern-day exhi-
bition,’’ says Robert Costello, nation-
al outreach program manager, who
developed the mobile experience.
In fact, most visitors were using the
Bone Hall as a passageway from one
section of the museum to another,
rather than a destination, Costello
says. The average time spent in a
modern exhibition at the museum is
between 10 and 20 minutes, and most
of those visitors were spending less
than two minutes in the hall, which
has a storied history, he says.
The Bone Hall opened in 1881
as one of the first exhibitions at the
Smithsonian, displaying skeletons of
fish, reptiles, amphibians, birds, and
mammals. Costello’s challenge was
to create a new experience for visi-
tors without changing anything about
the exhibit. “That seems like a pretty
impossible task, but through technol-
ogy, it’s quite doable,’’ he says. “With
augmented reality in particular, you
can superimpose the graphics onto
the skeletons to extract the stories
Communications of the ACM you think are of relevance to the pub-
is looking for writers in our
lic,’’ he says. “And that stays true to
the exhibit.”
community to contribute One story that comes to life in the
sci-fi short stories, between app is that of a blue catfish skeleton
1,000 and 1,200 words, and cast, which were part of the mu-
seum’s original exhibit. Costello
for our quarterly “Future
Tense” section.
tracked down the skeleton’s history,
went looking for the cast of the fish
from the 1881 exhibit, and found it in
Do you have
a great story to tell?
storage in the museum’s attic. He had
the cast optically scanned, and that
Make contact at became the AR model officials super-
LastByte@cacm.acm.org imposed on the skeleton.
“I thought it was an important his-
torical story and as far as I could tell,
Association for
the skeleton and cast had not been
Computing Machinery together since the 1880s,’’ Costello
says. The fish is an invasive species
and is still around. Once the free Skin
16 COM MUNICATIO NS O F TH E AC M | D EC EM BER 2022 | VO L . 65 | NO. 1 2
and Bones AR app is downloaded, a
user in the hall can direct their de-
vice’s camera at the skeleton, trigger-
ing 3D graphics to see how the live
fish moved, and learn its history.
The idea was to “have an experi-
ence in a hall that was underperform-
ing that would wow you and surprise
you and entertain you, so you’d walk
away with memories you wouldn’t
otherwise have,’’ he says.
It took approximately one year
to develop the content for the app,
which also includes an interactive
activity that challenges a user to iden-
tify bats from their calls, and objects
from a bat’s location. The app was
released in 2015 and has been down-
loaded “tens of thousands” of times,
Costello says.
The Kennedy Space Center devel-
oped an interactive experience called
“Heroes and Legends” (https://bit.
ly/3wItpF0) to educate visitors about
the early years of the U.S. space pro-
gram, “while exploring the concept of
heroism and the qualities that define
the pioneers who inspired a genera-
tion,’’ says Rebecca Burgman, senior
manager of public relations/commu-
nications for the center.
The experience requires the use
of a number of technologies, such
as holograms and AR. There is also a
“Through the Eyes of a Hero” custom-
built theater that features a multi-
sensory 4D experience during which
“guests vicariously join NASA’s he-
roes and legends on the most peril-
The idea was to
“have an experience …
that would wow
you and surprise you
and entertain you,
so you’d walk away
with memories
you wouldn’t
otherwise have.”

ous stages of their missions,” says
Burgman. Unlike 3D presentations
that engage the visual and auditory
senses, 4D incorporates additional
stimuli to address other senses such
as smell, touch, and motion.
A display of the Gemini 9 space
capsule offers a realistic hologram of
astronaut Gene Cernan climbing out
of the tiny spacecraft, Burgman says.
Visitors also can download an app
called “Edge of Home,” where they
can see the International Space Sta-
tion in the way an astronaut would
see it. “They will also be able to expe-
rience an extravehicular activity walk
around the outside of the ISS while
learning facts and figures about each
module,’’ she says.
The app KSC 360 expedition
supplies facts about each rocket
in the Rocket Garden (https://bit.
ly/3cSzlow), where visitors can view
authentic rockets that have never
flown in space, gaze at Space Shuttle
Atlantis as it floats in orbit, take a
ride on the Moon in a lunar rover at
the Apollo/Saturn V Center, and sit in
the commander’s seat of the Mercury-
Atlas 8, Gemini 9, and Apollo 14 space
capsules, according to Burgman.
“Users will see every switch, gauge,
and lever as only astronauts Wally Sch-
irra, Gene Cernan, and Alan Shephard
have seen before,” she says.
Schools and enrichment programs
also can take advantage of immer-
sive technologies with VR headsets,
so students can experience history
as if they were there. Claudia White,
founder and executive director of the
nonprofit Youth Global Perspectives
(YGP), brought in a platform from
TechRow (www.techrow.org) that
provides 3D content on VR headsets,
to help disadvantaged youths expe-
rience historical events—including
from their own history—about which
they may not know much, or anything
at all. The students use the headsets
at YGP’s headquarters and other com-
munity centers in Indianapolis and
South Bend, IN.
“Our big focus is to teach students
history and culture mainly through
the lens of people of color, because
our students are mainly African-
American,’’ White says. A former so-
cial studies teacher, White says she
saw the same issue with her students
Feedback about the
Tower of Faces has
been focused on how
the exhibit honors
the community of
Eishishok and
brings its inhabitants
back to life.
that she herself experienced: “not be-
ing able to identify with their culture
and making derogatory comments
about themselves. They’re not learn-
ing positive material … and they are
starting to believe the narrative that
Black people are dumb, violent, and
poor,’’ she says. “We want them to see
themselves in a positive way.”
One lesser-known period in Black
American history is the Tulsa Race
Massacre of May 31, 1921. Students
wearing headsets can watch the the
actual event.
“TechRow uses the technology to
tell the stories that are not being por-
trayed in schoolbooks the way they
should,’’ White says. “They go out of
their way to include information that
almost feels like it’s being withheld
from students. It gets people includ-
ed in something bigger than them-
selves.”
The question of how effective im-
mersive technologies are at preserv-
ing historical events and telling those
stories comes down to the developer
of the content, says Costello.
“There’s a lot of technologies that
preserve historical artifacts or infor-
mation, but it’s really the intent of
the developer that matters because
it’s the developer, the person doing
the preservation work, who creates
the content for communicating about
that object or event,” he says.
Technology is merely the vehicle
for doing so, albeit while serving an
important purpose. Costello says “3D
is a great insurance policy” that can
be used to preserve museum speci-
mens and artifacts. He cites, as an ex-
DEC E MB E R 2 0 2 2 | VO L. 6 5 | N O. 1 2 | C OM M U N IC AT ION S OF T HE ACM
news
ample, the National Museum of Brazil
in Rio de Janeiro, which was heav-
ily damaged by fire in 2018. “That’s
a huge cultural loss,’’ Costello says.
“Had the specimens and artifacts
been preserved digitally through 3D
technology, the legacy of those collec-
tions could have lived on.”
The Holocaust Museum’s Tower
of Faces exhibit opened to the pub-
lic in April 2022. The content took 18
months of research to prepare, to en-
sure “we knew who was in the photo
and their fate,” Lumbard says.
The use of AR creates “an experi-
ence that’s easy to use that will pull
you into the history and lives of these
people living in a lovely town and, over
the course of less than a week, almost
all of them were massacred … not in
[concentration] camps and in some
cases, by people they know,’’ she says.
Feedback about the Tower of Faces
has been focused on how the exhibit
honors the community of Eishishok
and brings its inhabitants back to life,
Lumbard says. “What better way to
serve the memory of the victims than
to have people see them as people
murdered by Nazis; not just numbers,
but people with dreams.”
Further Reading
Kennedy Space Center
Virtual Reality Applications
https://www.kennedyspacecenter.com/info/
vr-applications
Leslie, S.
The Key Role Of VR In Preserving
Cultural Heritage, Arts Management &
Technology Laboratory, 2022.
Carlsson, R.
How virtual reality is bringing historical
sites to life. Museum Next, 2020.
Crutcher, M.
Artificial reality is the likely future of
preservation. National Council on Public
History, 2020.
Melnick, K.
Preserving Historical Landmarks in VR
Should Be A Priority. VRScout, 2019.
Voinea, G.D., Girbacia, F.,
Postelnicu, C.C., and Marto, A.
Exploring Cultural Heritage Using
Augmented Reality Through Google’s
Project Tango and ArCore. Springer. 2018.
Esther Shein is a freelance technology and business
writer based in the Boston area.
© 2022 ACM 0001-0782/22/12 $15.00
17
Copyright of Communications of the ACM is the property of Association for Computing
Machinery and its content may not be copied or emailed to multiple sites or posted to a
listserv without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.