EARLY ONLINE RELEASE

Note: This article was posted on the Archives Web site as

an Early Online Release. Early Online Release articles
have been peer reviewed, copyedited, and reviewed by the
authors. Additional changes or corrections may appear in
these articles when they appear in a future print issue of
the Archives. Early Online Release articles are citable by
using the Digital Object Identifier (DOI), a unique number
given to every article. The DOI will typically appear at the
end of the abstract.

The DOI for this manuscript is doi: 10.5858/arpa.2017-0189-OA

The final published version of this manuscript will replace

the Early Online Release version at the above DOI once it is available.

© 2018 College of American Pathologists

 Original Article
Augmented Reality Technology Using Microsoft HoloLens
in Anatomic Pathology
Matthew G. Hanna, MD; Ishtiaque Ahmed, BS; Jeffrey Nine, MD; Shyam Prajapati, DO; Liron Pantanowitz, MD
 Context.—Augmented reality (AR) devices such as the
Microsoft HoloLens have not been well used in the medical
field.
Objective.—To test the HoloLens for clinical and
nonclinical applications in pathology.
Design.—A Microsoft HoloLens was tested for virtual
annotation during autopsy, viewing 3D gross and micro-
scopic pathology specimens, navigating whole slide imag-
es, telepathology, as well as real-time pathology-radiology
correlation.
Results.—Pathology residents performing an autopsy
wearing the HoloLens were remotely instructed with
real-time diagrams, annotations, and voice instruction.
3D-scanned gross pathology specimens could be viewed as
holograms and easily manipulated. Telepathology was
supported during gross examination and at the time of
intraoperative consultation, allowing users to remotely
access a pathologist for guidance and to virtually annotate
areas of interest on specimens in real-time. The HoloLens
T echnology that offers an augmented reality (AR)
experience (eg, Google Glass [Mountain View, Cal-
ifornia], Microsoft HoloLens [Redmond, Washington])
overlays a virtual world on top of a user’s existing (‘‘real’’)
surroundings. AR technology differs from devices for virtual
reality (VR) that completely immerse a user within a virtual
experience (eg, Oculus Rift [Menlo Park, California], HTC
Vive [Taoyuan, Taiwan]).1 Most AR and VR devices include
a head-mounted display, where an ergonomically assem-
bled headset provides a viewable display for human-
computer interaction. The Microsoft HoloLens is a wearable
headset that projects a visible image in the user’s point of
view. It superimposes an image (eg, hologram) on the user’s
surrounding real-life environment to create a mixed-reality
experience.
Accepted for publication August 15, 2018.
From the Department of Pathology, University of Pittsburgh
Medical Center (UPMC), Pittsburgh, Pennsylvania (Drs Hanna, Nine,
and Pantanowitz and Mr Ahmed); Icahn School of Medicine at
Mount Sinai, The Mount Sinai Hospital, New York, New York (Dr
Prajapati).
The authors have no relevant financial interest in the products or
companies described in this article.
Corresponding author: Liron Pantanowitz, MD, Department of
Pathology, University of Pittsburgh Medical Center, Shadyside
Hospital, Suite 201, 5150 Centre Ave, Pittsburgh, PA 15232 (email:
pantanowitzl@upmc.edu).
Arch Pathol Lab Med
permitted radiographs to be coregistered on gross speci-
mens and thereby enhanced locating important pathologic
findings. The HoloLens also allowed easy viewing and
navigation of whole slide images, using an AR workstation,
including multiple coregistered tissue sections facilitating
volumetric pathology evaluation.
Conclusions.—The HoloLens is a novel AR tool with
multiple clinical and nonclinical applications in pathology.
The device was comfortable to wear, easy to use, provided
sufficient computing power, and supported high-resolution
imaging. It was useful for autopsy, gross and microscopic
examination, and ideally suited for digital pathology.
Unique applications include remote supervision and
annotation, 3D image viewing and manipulation, tele-
pathology in a mixed-reality environment, and real-time
pathology-radiology correlation.
(Arch Pathol Lab Med. doi: 10.5858/arpa.2017-0189-
OA)
AR technology has the potential to radically redefine how
humans interact with their environment. AR solutions have
been investigated for various applications including their
feasibility in education, engineering, chemistry, environ-
mental sciences, and tourism.2,3 AR devices such as the
HoloLens have been sparsely used in the medical field, and
to the best of our knowledge hitherto not in pathology.
Potential uses for AR in the medical field include education,
simulation, and clinical care such as telemedicine. To date,
AR devices have been tested with image-guided and
minimally invasive surgical procedures.4–6 Case Western
Reserve University School of Medicine (Cleveland, Ohio)
with Cleveland Clinic (Cleveland, Ohio) started a pilot
program with the Microsoft HoloLens to improve courses in
medical school, such as 3D holographic content to teach
human anatomy.7 The aim of this study was to test the
HoloLens in a variety of clinical and nonclinical applications
in pathology.
METHODS
Internet Connectivity
A Microsoft HoloLens (Development Edition) was connected to
the Internet via our institution’s (Pittsburgh, Pennsylvania) secure
wireless network. The HoloLens contains an 802.11ac, 232 wireless
fidelity (WiFi) radio, which means it can only connect to a network
supporting the 5-GHz band. Connecting the device to a WiFi
network was similar to connecting a laptop or mobile device to a
WiFi network. The institution’s secure wireless network required
Augmented Reality in Anatomic Pathology—Hanna et al 1
specifications such as Encrypted and authenticated Transport Layer
Security (ETLS) or Protected Extensible Authentication Protocol
(PEAP). The secure network provided a 5-Mbps minimum
bandwidth connection. To gain access to this network, we required
a service account in the institution’s Identity Management System;
once approved, the mobile device could access the secure network
through a designated username and password.
Hardware
The HoloLens device is a lightweight (579 g), wearable
holographic computer, enabling human-computer interaction in a
mixed-reality environment. Holographic computer images can be
viewed within a user’s point of view of native surroundings. It
consists of a widescreen stereoscopic head-mounted display
(resolution 1268 3 720 pixels per eye, 16:9 aspect ratio, and 60-
Hz refresh rate) containing tinted holographic lenses, a depth
camera (1208 3 1208), and 4 additional cameras with built-in
sensors for environment sensing and detecting ambient light. The
sensors receive user input through inertial measurement units that
include an accelerometer, gyroscope, and magnetometer. It also
has integrated speakers and 4 microphones for 2-way communi-
cation. The device’s Holographic Processing Unit can handle 1
trillion calculations per second, and has 2 gigabyte (GB) of random
access memory (RAM). Further specifications include 64 GB of
internal flash storage and support for wireless connectivity with
WiFi (IEEE 802.11ac) and Bluetooth (4.1 Low Energy). Battery life
carries up to 3 hours of active use or a standby time of 2 weeks. The
HoloLens has a clicker accessory, connected via Bluetooth,
allowing users to make physical commands instead of using gaze,
voice commands, and/or hand gestures.
AR sessions during testing were captured with a Microsoft
Surface Pro 3 computer (weighing 1.76 lb). This is a 12’’ high-
definition (2160 3 1440) touchscreen tablet computer (64-bit) with
64 GB of storage and 4-GB RAM. The tablet was powered by a
fourth-generation Intel Core i5 processor (Santa Clara, California)
and used the Windows 10 operating system.
Software
The HoloLens used a Windows Holographic platform under the
native Windows 10 operating system. The platform included a
Microsoft application repository. Applications specific to the
HoloLens were directly downloaded and installed on the device
via the Windows store. The device’s interface used gaze input (head
tracking), gesture (bloom, air tap, press and hold), and voice
commands.8 Three gestures were used to interact with the AR
environment:
1. Bloom: upward facing palm, starting with fingertips together,
then spreading fingers outward; for application start-up and
closure.
2. Air tap (tap and release): with the dorsal aspect of the user’s
hand facing them, raising and flexing the index finger (ie, up,
down, and up again) in a pinchlike fashion (press and release);
for selecting an operation.
3. Tap and hold: raising and flexing the index finger to the thumb
and motioning the pressed fingers together (press and hold); in
the user’s 3D space (ie, up then down)—for manipulation of
selected objects.
A Bluetooth Clicker was also provided with the device that can
mimic hand gestures. Voice commands were interchangeably used
(eg, ‘‘select’’ instead of the air-tap gesture, or ‘‘take a picture’’ to
capture a live point-of-view mixed-reality screen-capture). Internet
connectivity was established via the secure hospital network, which
caused no problems. No patient health information data were
recorded. Photos and videos were captured with the HoloLens and
backed up on our institution’s Microsoft OneDrive cloud storage.
Voice command instructions using Microsoft’s software intelligent
personal assistant, Cortana, were used to capture photos and video.
With a remote desktop agent the HoloLens was used to create a
dynamic virtual workstation. By means of this virtual ‘‘cockpit’’
2 Arch Pathol Lab Med
users were able to access and work with various applications (eg,
Microsoft Outlook, laboratory information system, electronic
medical record, whole slide image [WSI] viewer).
Pathology Use Cases
Autopsy.—After establishing secure Internet connectivity in the
autopsy suite, Skype (version 7.26.0.101, Microsoft) was installed
on the Surface Pro with a Skype for HoloLens plug-in, and Skype
Beta was installed on the HoloLens device. These applications
allowed for simultaneous live streaming and bidirectional virtual
annotation between both devices. By annotating in real-time, users
could virtually point an arrow or freehand draw on their
surrounding environment. By using these applications, remote
communication was initiated for users to contact and interact with
one another. Our director of autopsy was selected as an expert
pathologist user to virtually guide pathology trainees through their
autopsy procedure.
3D Holographic Gross Pathology Specimens.—A Shining 3D
Einscan-S 3D scanner (Hangzhou, China) with Einscan 3D
software (version 1.7.3) was used to capture 3D images of gross
specimens. Detailed methods of specimen scanning and software
enhancement to produce a 3608 3D virtual image have been
previously reported.9 Digital files of scanned 3D specimens were
saved in object file format (.OBJ) and imported into open-source
Blender software (Amsterdam, the Netherlands) to be scaled,
edited, and oriented for viewing. The final 3D specimen image was
then exported from Blender in Filmbox (.FBX) file format. The
Filmbox format was the only readable 3D holographic file format
that could be natively imported into the Microsoft HoloLens 3D
Viewer Beta application. 3D viewer Beta application was used to
view 3D model holograms of our scanned specimens. The 3D
viewer allowed users to interact with holographic images (eg,
resize, rotate, flip plane axes, mirror).
Telepathology.—Similar to the autopsy setup, the HoloLens
and Surface Pro devices with Skype Beta for HoloLens were used
for live streaming and bidirectional virtual annotation to support
gross specimen telepathology. Pathologists’ assistants and trainees
wearing the HoloLens received instructions (eg, selection of
representative areas from a specimen to obtain tissue sections)
from a remote pathologist using audio, visual communication, and
virtual annotations.
Specimen Radiograph Coregistration.—Conventional and AR
workflows were compared. Pathologists’ assistants wearing the
HoloLens device at the grossing workbench were asked to
manipulate (eg, move, scale) virtual radiographs (eg, mammogram)
and overlay them atop corresponding gross specimens. Findings in
the radiograph (eg, calcification, biopsy marker clip) were used to
locate corresponding tissue areas within the gross specimen (eg,
mastectomy, breast lumpectomy excision). Patient specimens
included 12 total mastectomies and 12 excisional breast resections
(ie, lumpectomies, excisional breast biopsies); 9 patients underwent
neoadjuvant chemotherapy with or without radiation divided
among both workflow methods. Pathology diagnoses included
invasive ductal carcinoma (n ¼ 20), invasive lobular carcinoma (n ¼
1), atypical ductal hyperplasia (n ¼ 1), and papilloma (n ¼ 2).
Specimen radiographs from the picture archiving and communi-
cation system (PACS) were de-identified and imported to the
Microsoft HoloLens as JPEG image files through the device’s
OneDrive cloud storage account. Conventional workflow included
manual detection during specimen dissection and pathologists’
assistants having the specimen radiograph display on the monitor if
retrieved from the PACS.
Whole Slide Image Viewing.—Microsoft Edge Web browser
was used on the HoloLens to access the Internet. The remote
desktop application Remote Desktop Preview App was download-
ed and investigated for remote navigation of digital slides stored on
an institutional server. Several Web-based WSI viewers were tested
with the HoloLens (Table). All viewers were freely accessible.
Supported gestures for image navigation included scroll, pan, and
zoom. Compatibility and usability of each WSI viewer was
documented. Features of interest include loading of WSI Viewer
Augmented Reality in Anatomic Pathology—Hanna et al
 List of Whole Slide Image Viewers Tested With the
HoloLensa
Compatible View Only Incompatible
DigitalScope AJAX DBViewer
ImageZoomer Google Maps API Webscope
OpenSeadragon OpenLayers Zoomify
View-only whole slide image viewers were not compatible with native
navigation gestures.
a
DigitalScope (Aptia Systems, Houston, Texas), ImageZoomer (Thiet-
kewebDrupal, Hai Ba Trung, Hanoi, Vietnam), OpenSeadragon (open
source), AJAX (MSTechnology, Charlotte, North Carolina), Google
Maps API (Google, Mountainview, California), OpenLayers (open
source), DBViewer (Hanover, Germany), Webscope (Aperio, Buffalo
Grove, Illinois), Zoomify (Zoomify, Aptos, California).
Web page, rendering of WSI, and gesture functionality (ie, scroll,
pan, and zoom navigation of WSI).
Volumetric Pathology.—Formalin-fixed paraffin-embedded
blocks of human and animal tissue samples were processed by
3Scan (San Francisco, California) with their knife-edge scanning
microscopy technology. This process generates images from
thousands of regular, serial tissue sections with fiber optic
microscopy. The images are subsequently reconstructed in 3D
(OBJ image files) to study cellular and subcellular structures.10 The
HoloLens was used to view these images with the 3D Viewer Beta
application.
RESULTS
Overall, the Microsoft HoloLens device was light and
comfortable to wear, easy to use, and provided sufficient
computing power. Gesture input, although limited to only 3
gestures, did provide adequate interaction for all pathology
use cases to be performed successfully. For all use cases in
Figure 1. Augmented reality using Microsoft HoloLens (Redmond, Washington) allows remote access of multiple applications, as demonstrated at
this gross pathology workbench where the user has opened (clockwise from top left) a Web-based procedure manual, microscopic image from an
online reference Web site, radiology image from the Picture Archive and Communication System (PACS), and laboratory information system (LIS).
Arch Pathol Lab Med
this study, the bloom, air tap, and tap and hold were used as
well as associated interchangeable voice commands or
calling on Cortana for expedited command operation. The
device facilitated remote, real-time, bidirectional audio, and
mixed-reality annotation. Image resolution was sufficient to
support digital pathology including telecommunication
during autopsy, telepathology during grossing, viewing
and manipulating 3D pathology specimens, and WSI.
Captured photos, using the HoloLens, averaged 300 KB in
size, with a resolution of 1408 3 792, and 72 dots per inch.
The creation of a virtual workstation successfully allowed
multiple virtual windows to be displayed by the HoloLens in
the user’s visual field. Interacting with these windows
allowed users to resize and reposition them. Setting up a
virtual desktop permitted a variety of secure clinical
applications (eg, laboratory information system, electronic
health record, PACS) to be remotely accessed from any
location (eg, at the gross workbench) (Figure 1).
Autopsy Findings
A pathologist was able to successfully establish remote
communication by using Skype with pathology housestaff
wearing the HoloLens device through Skype for HoloLens
application during an autopsy in the autopsy suite located in
the basement of the hospital. The Microsoft Surface Pro
tablet was used by the attending pathologist, whereas the
HoloLens was donned by housestaff. Face shields were
worn in such a fashion as to not obstruct the HoloLens
sensors. Audio quality and visual resolution were sufficient
for both users. For example, the attending pathologist was
able to easily determine that there was no pulmonary
embolism in one case while remotely directing the trainee
during dissection of the pulmonary vasculature. The
Augmented Reality in Anatomic Pathology—Hanna et al 3
Figure 2. A hologram of a scanned human
brain that was removed during autopsy is
being manipulated while wearing the Holo-
Lens device (Microsoft HoloLens, Redmond,
Washington).
pathologist and trainees both used the air tap and the tap
and hold to create virtual arrows and freehand drawings
during AR sessions. Using the HoloLens by means of
interactive gestures, trainees performing an autopsy suc-
cessfully recorded organ weights, pathology notes, and
voice-commanded image acquisition of gross findings
including screen-captures with annotations. For attending
pathologists, advantageous uses of Skype included remote
portability, remote supervision, quick access to other
pathologists if needed, a more timely response to trainee
inquiries that avoided going to the morgue, and the ability
to record or document the interaction. Educational autopsy
live video streaming or recordings could be used for future
teaching and education.11
3D Holographic Gross Pathology Specimen Experience
Multiple gross pathology specimens (eg, whole brain
including subsequent coronal slices, prostate, amputated
digit, kidney with tumor) were scanned with the Einscan 3D
scanner. A 3608 3D rendition of these scanned gross
specimens were generated. Specimens with a coarse surface
texture, as opposed to those with a shiny reflective surface,
were most successfully visualized. Gray and white matter
tracks of coronal brain slices were also less clear. Imported
3D gross specimens were successfully viewed with the
HoloLens (Figure 2). Using hand gestures, these virtual
specimens could be scaled and manipulated (eg, rolled
around the x-, y-, and z-axes). Live streaming allowed 3D
gross specimens in the AR field to be shared for educational
purposes.
Telepathology Results
Virtual images remotely transmitted for review by a
pathologist demonstrated adequate resolution for a variety
of gross specimens (eg, nephrectomy, prostatectomy).
Telepathology was performed, without difficulty, by pathol-
ogy residents and pathologists’ assistants wearing the
HoloLens. Bidirectional annotation via hand gestures was
used to successfully guide gross specimen examination. For
example, a pathologist was able to remotely specify tissue
areas to be sectioned during intraoperative consultations
and remote gross dissections.
4 Arch Pathol Lab Med
Specimen Radiograph Coregistration Feasibility
Both methodologies for coregistration of breast resec-
tion specimen radiographs and gross specimens for
biopsy clip detection and correlation were performed
by using conventional workflow and the head-mounted
AR device. The HoloLens, worn by pathologists’ assis-
tants during gross examination of breast excision
specimens, was effectively used to retrieve corresponding
specimen radiographs from the PACS. There was
sufficient resolution in these virtually displayed radiology
images to clearly identify breast lesions and metallic
biopsy clips. The imported radiographs were easily
manipulated with gestures in order for them to be
accurately coregistered over gross specimens in real-time
(Figure 3, A through D). This process was used to
rapidly identify the precise location of a biopsy clip or
lesion such as calcifications in the breast resection
specimen. Pathologists’ assistants were successfully able
to manually coregister the radiograph and gross speci-
men by using hand gestures and voice commands to
scale and position the radiograph atop the gross
specimen. Identification of the metallic biopsy clip
averaged 10.6 minutes when using conventional work-
flow applications and 1.5 minutes when using the
Microsoft HoloLens. One hundred percent of users rated
the Microsoft HoloLens 4 or greater on a 5-point scale
in being a useful and usable technology for radiograph-
specimen coregistration.
Whole Slide Image Viewing Outcome
Whole slide image viewing in 2 dimensions within the
HoloLens AR environment was possible. A range of
gestures permitted users to navigate digital slides and zoom
in on selected fields of view. Drag mode allowed users to
pan around the WSI by directing their fingers toward the
intended direction of the digital slide. Zooming in was
performed by directing the end-users’ fingers upward and
zooming out, by directing their fingers downward. Of the 9
WSI viewers tested (Table), 3 were not compatible owing to
Flash plug-in encoding built into the viewer (DBViewer
[Hanover, Germany], Webscope [Aperio, Buffalo Grove,
Illinois], Zoomify [Zoomify, Aptos, California]). These
Augmented Reality in Anatomic Pathology—Hanna et al
Figure 3. Real-time radiology-pathology correlation using the Microsoft HoloLens (Redmond, Washington). A, A breast specimen radiograph is
shown with a clip (inset) in the lower quadrant. B, The same radiograph is now shown imported into the HoloLens and coregistered with the
corresponding gross pathology specimen at the gross workstation. C, A pathology assistant wearing the Microsoft HoloLens is shown working at the
gross bench. D, The clip in this specimen was identified in the precise location indicated by the overlying specimen radiograph.

viewer Web pages did not load in the HoloLens device Edge
browser, or render the WSI for navigation. Three WSI
viewers (AJAX [MSTechnology, Charlotte, North Carolina],
Google Maps API [Google, Mountainview, California], and
OpenLayers) permitted viewing, but the gestures in all
navigation modes did not function as intended. For
instance, while attempting to use the tap and hold gesture
in the zoom navigation mode, the WSI would continuously
pan in one direction and not zoom in. The remaining
compatible 3 WSI viewers (DigitalScope [Aptia Systems,
Houston, Texas], ImageZoomer [ThietkewebDrupal, Hai Ba
Trung, Hanoi, Vietnam], and OpenSeadragon) permitted
viewing, and all gestures worked as intended in all
navigation modes. A Bluetooth-connected keyboard and
mouse were also successfully used for navigation input.
Whole slide images can successfully be viewed on the
HoloLens device through Web-enabled viewers.
Volumetric Pathology
Tissue from mouse gastrointestinal tract, lungs, and blood
vessels was successfully serially sectioned, imaged, and
reconstructed by using the knife-edge scanning microscopy
Arch Pathol Lab Med
technique.10 3D image file data sets were converted from
.OBJ files to .FBX files to be imported into the Microsoft
HoloLens (Figure 4, A and B). Image file sizes ranged from
17 MB to 1 GB. Large files (.50 MB) were not compatible
with HoloLens 3D viewer applications. Performance load
capabilities were also limited to 150 000 vertices or more
than 400 meshes. While optimization of image files allowed
them to be imported into the HoloLens, they could not be
manipulated.
DISCUSSION
While the capability to deliver AR experiences began
decades ago, only recently has this technology become
easily accessible, affordable, and transportable. Compared
to AR, VR solutions are more widely developed and
used, even for use in pathology.1,12–17 To the best of our
knowledge, we are not aware of any other studies
demonstrating AR-specific applications using the Hol-
oLens in pathology.
AR is ‘‘active’’ technology, whereby users can interact
with virtual objects relative to real life. One of the
drawbacks of using Google Glass for telepathology was
Augmented Reality in Anatomic Pathology—Hanna et al 5

Figure 4. Demonstration of volumetric pathology image sets using 3D-
reconstructed histologic sections. A, Murine gastrointestinal segment. B,
Vascular network of blood vessels.
poor image resolution.18 Similarly, low image resolution
of the Oculus Rift device was a major limitation for
viewing WSIs.1 This study showed that image quality
with the HoloLens was sufficient for viewing gross and
microscopic (eg, WSI) pathology images. The resolution
of the HoloLens of 2536 3 1440 (1268 3 720 per eye)
surpasses that of earlier VR/AR devices. For example, the
Oculus Rift (Development Kit 1) by comparison only has
an image resolution of 1280 3 800 (640 3 800 per eye).
AR also has other advantages over VR devices. For
instance, it was reported that users were able to
complete tasks related to 3D object manipulation up to
22.5% faster in an AR environment than with VR.19
The Microsoft HoloLens is a novel AR tool with potential
for clinical and educational applications in health care.
Medical education is transforming by using safer simulation
technologies.20 A recent study21 reported higher achieve-
ments and lower cognitive burden from medical students
learning neuroanatomy with AR than with traditional
textbook methods. Educational content specific to the
Microsoft HoloLens today is mostly devoted to anatomy
(eg, HoloAnatomy, HoloHeart [Made in Point GmbH,
Erlangen, Germany]). There is still a gap regarding
educational content for pathology. AR also has novel clinical
applications such as 3D volumetric reconstruction to
coregister radiology, gross anatomy, and histopathology to
support radiation treatment planning in patients with
cancer.22 AR using 3D-integrated intraoperative imaging
has proved to be reliable in performing minimally invasive
procedures in a variety of neurosurgical diseases, showing
superior accuracy of hardware placement as compared to
free-hand techniques.23,24
Security for health care–related AR or VR head-mounted
devices is critical. Indeed, the main vulnerability when
6 Arch Pathol Lab Med
connecting these wearable devices to the Internet is security.
Usually, for a mobile device to have access on our
institution’s network, a passcode for the device is necessary.
However, the HoloLens does not have login entry
password-protected capability. This makes it hard to lock
down. Also, for mobile devices (eg, smartphones) our
institution uses MobileIron (Mountain View, California)
software to secure content on these devices. The MobileIron
app, unfortunately, was not yet available or compatible with
the HoloLens. Hence, with repeated use of this ‘‘open
device’’ on our network, hospital information technology
security officers may not recognize the HoloLens and hence
remove it from the network. Fortunately, we did have any of
these problems. The other problem with relying on WiFi
connectivity was the geographic limitation of using the
device only within our network.
Our experience shows that wearing the HoloLens is
feasible and permits pathologists, pathologists’ assistants,
and trainees to easily manipulate their surrounding mixed-
reality environment for a variety of clinical and nonclinical
use cases. This includes supervising prosectors and pathol-
ogy residents during an autopsy or while grossing surgical
pathology specimens, evaluating 3D gross specimens as
holograms, remotely interacting with WSIs, reconstructed
microscopic images (volumetric pathology), facilitating
telepathology, and aiding in real-time pathology-radiology
correlation. Not all WSI viewers are yet configured for use in
such a mixed-reality environment. Future directions should
include developing WSI viewers for easy navigation with AR
and/or VR devices. Remote telepathology applications for
the HoloLens has tremendous potential. Volumetric pa-
thology represents the study of pathology in 3 dimensions,
where 3D visualization of hundreds or thousands of
coregistered histopathology WSIs is possible. Coregistration
of radiology images (eg, computed tomography, magnetic
resonance imaging) and digitized slides from postmortem
human brain specimens has been successfully mapped with
an accuracy of 0.56 6 0.39 to 0.87 6 0.42 mm.25 It remains
to be seen if this can be improved with the HoloLens. The
HoloLens also supports a wide variety of hands-free tasks
(eg, voice commands, documentation, annotation, video
recording) and digital imaging applications (eg, whole slide
imaging). A limitation of using both AR and VR head-
mounted devices is a type of visually induced motion
sickness, better known as simulator sickness. Simulator
sickness has been described in VR more than AR devices,
possibly due to native surrounding visualization in AR. This
discomfort is characterized by nausea, disorientation, eye
strain, or other oculomotor symptoms, and can negatively
impact the user’s experience, acceptance, performance, and
safety. We did not find this in our testing, perhaps because
the HoloLens was not used for prolonged periods of time.
Individual consecutive user times ranged from a few
minutes (eg, during WSI navigation, rad-path correlation,
3D gross specimen manipulation) up to 1 hour (eg, during
an autopsy). Users did not report episodes or symptoms of
simulator sickness. Indeed, some evidence shows that
shorter footage duration viewed on a head-mounted display
may be better tolerated in VR.26
In summary, we found that using AR in pathology for
clinical and nonclinical reasons is promising. The
HoloLens introduces many novel opportunities that
could enrich the practice of pathology, such as creating
virtual workstations and exploiting dynamic ways to
manipulate spatial digital data. However, further testing
Augmented Reality in Anatomic Pathology—Hanna et al
is required to validate the HoloLens for use in routine
clinical practice.
We thank 3Scan for generating and sharing their 3D images.
References
1. Farahani N, Post R, Duboy J, et al. Exploring virtual reality technology and
the Oculus Rift for examining digital pathology slides. J Pathol Inform. 2016;7:
22.
2. Johnson L, Smith R, Levine A, Haywood K. The 2010 Horizon Report:
Australia–New Zealand Edition 2010. Austin, TX: The New Media Consortium;
2010.
3. Zhu E, Hadadgar A, Masiello I, Zary N. Augmented reality in healthcare
education: an integrative review. PeerJ. 2014;2:e469.
4. De Paolis LT, Pulimeno M, Aloisio G. An augmented reality application for
minimally invasive surgery. In: Katsahev A, Dekhttyar Y, Spigulis J, eds. 14th
Nordic-Baltic Conference on Biomedical Engineering and Medical Physics.
International Federation for Medical and Biological Engineering (IFMBE)
Proceedings. Berlin Heidelberg: Springer-Verlag; 2008:489–492.
5. De Paolis LT, Ricciardi F, Dragoni AF, Aloisio G. An augmented reality
application for the radio frequency ablation of the liver tumors. In: Murgante B,
Gervasi O, Iglesias A, Taniar D, Apduhan B, eds. International Conference on
Computational Science and its Applications (ICCSA) 2011. Berlin Heidelberg:
Springer. 2011;6785:572–581.
6. Wacker FK, Vogt S, Khamene A, et al. An augmented reality system for MR
image-guided needle biopsy: initial results in a swine model. Radiology. 2006;
238(2):497–504.
7. Case Western Reserve University. CWRU takes the stage at Microsoft’s
Build conference to show how HoloLens can transform learning. In: Research
News. http://case.edu/hololens/. Accessed September 29, 2016.
8. Microsoft. Windows Development Center. Mixed reality. Gestures. https://
developer.microsoft.com/en-us/windows/mixed-reality/gestures. Accessed March
24, 2017.
9. Farahani N, Duboy J, Ahmed I, Yousem S, Hartman D, Pantanowitz L.
Automated 3D scanning of gross surgical pathology specimens. J Pathol Inform.
2016;7:S30.
10. Mayerich D, Abbott L, McCormick B. Knife-edge scanning microscopy for
imaging and reconstruction of three-dimensional anatomical structures of the
mouse brain. J Microsc. 2008;231(pt 1):134–143.
11. Brebner EM1, Brebner JA, Norman JN, Brown PA, Ruddick-Bracken H,
Lanphear JH. Intercontinental postmortem studies using interactive television. J
Telemed Telecare. 1997;3(1):48–52.
Arch Pathol Lab Med
12. Trelease RB, Nieder GL, Drup J, Hansen MS. Going virtual with quicktime
VR: new methods and standardized tools for interactive dynamic visualization of
anatomical structures. Anat Rec. 2000;261(2):64–67.
13. Prajapati S, Madrigal E. (Virtual) reality check. The Pathologist. 2016;(916):
501.
14. Zito FA, Marzullo F, D’Errico D, et al. Quicktime virtual reality technology
in light microscopy to support medical education in pathology. Mod Pathol.
2004;17(6):728–731.
15. Sakellariou S, Charissis V, Grant S, Turner J, Kelly D, Christomanos C.
Virtual reality environment as knowledge enhancement tool for musculoskeletal
pathology. In: Virtual and Mixed Reality: Systems and Applications. Lecture Notes
in Computer Science. Berlin Heidelberg: Springer; 2011;6774:54–63.
16. Melin-Aldana H, Sciortino D. Virtual reality demonstration of surgical
specimens, including links to histologic features. Mod Pathol. 2003;16(9):958–961.
17. Erel E, Aiyenibe B, Butler PE. Microsurgery simulators in virtual reality:
review. Microsurgery. 2003;23(2):147–152.
18. Syed M, Nine J, Ahmed I, et al. Applications of Google Glass [GLKSSTM]
in autopsy pathology. J Pathol Inform. 2014;5:S43–S44.
19. Krichenbauer M, Yamamoto G, Taketomi T, Sandor C, Kato H. Augmented
reality vs virtual reality for 3D object manipulation [published online ahead of
print January 25, 2017]. IEEE Trans Vis Comput Graph. doi:10.1109/TVCG.2017.
2658570.
20. Gorman PJ, Meier AH, Rawn C, Krummel TM. The future of medical
education is no longer blood and guts, it is bits and bytes. Am J Surg. 2000;
180(5):353–356.
21. Küçük S, Kapakin S, Göktas Y. Learning anatomy via mobile augmented
reality: effects on achievement and cognitive load. Anat Sci Educ. 2016;9(5):411–
421.
22. Dahele M, Hwang D, Peressotti C, et al. Developing a methodology for
three-dimensional correlation of PET-CT images and whole-mount histopathology
in non-small-cell lung cancer. Curr Oncol. 2008;15(5):62–69.
23. Meola A, Cutolo F, Carbone M, Cagnazzo F, Ferrari M, Ferrari V.
Augmented reality in neurosurgery: a systematic review [published online ahead
of print May 7, 2016]. Neurosurg Rev. doi:10.1007/s10143-016-0732-9.
24. Elmi-Terander A, Skulason H, Söderman M, et al. Surgical navigation
technology based on augmented reality and integrated 3D intraoperative
imaging: a Spine Cadaveric Feasibility and Accuracy Study. Spine (Phila Pa
1976). 2016;41(21):E1303–E1311.
25. Ohnishi T, Nakamura Y, Tanaka T, et al. Deformable image registration
between pathological images and MR image via an optical macro image. Pathol
Res Pract. 2016;212(10):927–936.
26. Madrigal E, Prajapati S, Hernandez-Prera JC. Introducing a virtual reality
experience in anatomic pathology education. Am J Clin Pathol. 2016;146(4):462–
468.
Augmented Reality in Anatomic Pathology—Hanna et al 7