th

Proceedings of the 5 Virtual Reality International Conference (VRIC2003)

pp.69/1-69/9, Laval Virtual, France, May 13-18, 2003

Telexistence and Retro-reflective Projection

Technology (RPT)
Susumu TACHI, Ph.D.
Professor
The University of Tokyo
Department of Information Physics & Computing
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656 Japan
http://www.star.t.u-tokyo.ac.jp/

Abstract. This paper reviews how the original telexistence (tele-existence) technology has been developed, and describes a
newly developed method of mutual telexistence using projection technology onto retro-reflective objects dubbed RPT
(Retro-reflective Projection Technology). Telexistence is fundamentally a concept named for the general technology that enables
a human being to have a real-time sensation of being at a place other than where he or she actually exists, and being able to
interact with the remote environment, which may be real, virtual, or a combination of both. It also refers to an advanced type of
teleoperation system that enables an operator at the control to perform remote tasks dexterously with the feeling of existing in a
surrogate robot working in a remote environment. Telexistence in the real environment through a virtual environment is also
possible. Although conventional telexistence systems provide an operator the real-time sensation of being in a remote
environment, persons in the remote environment have only a feeling that a surrogate robot is presented, not the operator. Mutual
telexistence aims to solve this problem so that the existence of the operator is apparent to persons in the remote environment by
providing mutual sensations of p resence. This has become possible by the use of RPT.

1. Introduction robot in the remote environment. Although

Telexistence (tele-existence) is a technology that
enables us to control remote objects and
communicate with others in a remote environment
with a real-time sensation of being there by using
surrogate robots, remote/local computers and
cybernetic human interfaces. This concept has
been expanded to include projection of ourselves
into computer-generated virtual environments, and
also the use of a virtual environment for the
augmentation of the real environment.
The author proposed the concept of the telexistence
in 1980, and the concept of telexistence became the
fundamental principle of the eight year Japanese
National Large Scale Project of "Advanced Robot
Technology in Hazardous Environments," which
started in 1983 together with the concept of Third
Generation Robotics. Through this project, we
made theoretical considerations, established
systematic design procedures, developed
experimental hardware telexistence systems, and
demonstrated the feasibility of the concept.
Thanks to the efforts of twenty years of research and
development, it has nearly become possible for
humans to use a humanoid robot in the remote
environment as if it was an other self, i.e. they are
able to have the sensation of being just inside the
existing telexistence systems succeeded in providing
an operator a real-time sensation of being in a remote
environment, persons in the remote environment did
not have the feelings that the human operator is
presented, but only a surrogate robot. Mutual
telexistence aims at solving this problem so that the
existence of the operator is apparent by the persons
in the remote environment by providing mutual
sensations of presence.
This paper reviews the original telexistence
technology, and introduces a method of mutual
telexistence based on the projection of real-time
images of the operator on the surrogate robot, based
on RPT (Retro-reflective Projection Technology).
2. Short History of Telexistence
Figure 1 shows the concept of telexistence in real
environments, virtual environments, and the real
environment through a virtual environment
(augmented telexistence). The following describes
the research and development conducted in order to
realize the concept.
Our first report [1] proposed the principle of the
telexistence sensory display, and explicitly defined
its design procedure. The feasibility of a visual
display with a sensation of presence was

VRIC 2003 Proceedings –69/1

demonstrated through psychophysical measurements
using experimental visual telexistence apparatus.
A method was also proposed to develop a mobile
telexistence system that can be driven remotely with
both an auditory and visual sensation of presence.
A prototype mobile televehicle system was
constructed and the feasibility of the method was
evaluated [2].
Fig.1 Concept of Telexistence.
In 1989, a preliminary evaluation experiment of
telexistence was conducted with the first prototype
telexistence master slave system for remote
manipulation. An experimental telexistence system
for real and/or virtual environments was designed
and developed, and the efficacy and superiority of
the telexistence master -slave system over
conventional master-slave systems was
demonstrated experimentally [3, 4, 5].
Fig. 2 Telexistence Surrogate Anthropomorphic Robot
(TELESAR) at Work (1988).
Augmented telexistence can be effectively used in
numerous situations. For instance, to control a
slave robot in a poor visibility environment, an
experimental augmented telexistence system was
developed that uses a virtual environment model
constructed from design data of the real environment.
VRIC 2003 Proceedings –69/2
To use augmented reality in the control of a slave
robot, a calibration system using image
measurements was proposed for matching the real
environment and the environment model [6,7].
The slave robot has an impedance control
mechanism for contact tasks and to compensate for
errors that remain even after calibration. An
experimental operation in a poor visibility
environment was successfully conducted by using a
humanoid robot called TELESAR (TELExistence
Surrogate Anthropomorphic Robot) (Fig.2) and its
virtual dual. Figure 3 shows the virtual TELESAR
used in the experiment and Figure 4 shows the
master system for the control of both real TELESAR
and virtual TELESAR.
Experimental studies of tracking tasks demonstrated
quantitatively that a human being can telexist in a
remote and/or a computer -generated environment by
using the dedicated telexistence master slave system
[5].
A networked telexistence paradigm called R-cubed
(Real-time Remote Robotics) was proposed in 1986,
and several research efforts towards the goal are
being conducted, including a real-time remote robot
manipulation language dubbed RCML [6,7].
Fig. 3 Virtual TELESAR at Work (1993).
Fig. 4 Telexistence Master (1989).
Telexistence technology was adapted in METI
(Ministry of Economy, Trade and Industry)'s national
five year HRP (Humanoid Robotics Project:
1998-2003) for a new type of cockpit system to
control a humanoid bipedal robot, as shown in
Figure 5. The telexistence cockpit was completed
for this project in March 2000 (Fig.6). It consists of
three main subsystems: an audio/visual display
subsystem, a teleoperation master subsystem, and a
communication subsystem between the cockpit and
the humanoid robot [10,11,12].
Fig.5 HRP Humanoid Robot at Work (2000).
Various teleoperation tests using the developed
telexistence master system confirmed that
kinesthetic presentation through the master system
with visual imagery greatly improves both the
operator's sensation of walking, and dexterity at
manipulating objects.
If the operator issued a command to move the robot,
the robot actually walked to the goal. As the robot
walked around, real images captured by a wide field
of view multi-camera system were displayed on four
screens of the surrounded visual display. This
made the operator feel as if he or she was inside the
robot, walking around the robot site.
VRIC 2003 Proceedings –69/3
Fig.6 Telexistence Cockpit for Humanoid Control (2000).
A CG model of the robot in the virtual environment
was represented and updated according to the current
location and orientation received from sensors on the
real robot. The model was displayed on the
bottom-right screen of the surround visual display,
and when augmented to the real images captured by
the camera system, it supported the operator's
navigation of the robot. Since the series of real
images presented on the visual display are integrated
with the movement of the motion base, the operator
feels the real-time sensation of walking, or stepping
up and down.
Through these efforts of more than twenty years, it
has become nearly possible for a human to use a
humanoid robot in the remote environment as if it
was his or her other self. Persons can control the
robot by just moving their body naturally, without
using verbal commands. The robot conforms to the
person’s motion, and through the sensors on board
the robot the human can see, hear and feel as if they
sensed the remote environment directly. Persons
can virtually exist in the remote environment without
actually being there in a sense.
For observers in the remote environment, however,
the situation is different. They see only the robot
moving and speaking. Although they can hear the
voice and witness the behaviour of the human
operator through the robot, it does not actually look
like him or her. This means that the telexistence is
not yet mutual. In order to realize mutual
telexistence, we have been pursuing the use of
projection technology with retro-reflective material
as a surface, which we call RPT (Retro-reflective
Projection Technology).
3. Retro-reflective Projection Technology
(RPT)
A Head Mounted Display (HMD) and a CAVE
(CAVE Automatic Virtual Environment) are two
classic virtual reality visual displays. Although
they are quite useful displays, it is also true that
they have some shortcomings. The former has a
tradeoff of high resolution and wide field of view,
and the latter has problems concerning the user’s
body casting shadows on a virtual environment,
and the interaction between the user's real body
and the virtual interface. In addition, both
displays have problems concerning occlusion when
in use under the augmented reality condition, i.e.
virtual objects and real objects are mixed. The
problems of conventional displays HMD and
CAVE are shown in Figure 7 (C) and (D),
respectively.
Figure 7 (A) shows a virtual vase and a virtual
ashtray on a virtual desk. When a real hand is
placed between two virtual objects, an ideal
occlusion should be depicted as in Figure 7 (B), i.e.,
the real hand occludes the virtual vase and is
occluded by the virtual ashtray. However, a real
hand cannot occlude the virtual vase nor be occluded
by the virtual ashtray when an optical see-through
HMD is used to display virtual objects, and the hand
and the ashtray look as if they are transparent. This
is simply due to the fact that the physical display
position is always just in front of the eyes of the user
when HMD is used.
Fig. 7 (A) A virtual vase and a virtual ashtray on a virtual desk;
(B) An ideal occlusion when a real hand is placed between two
virtual objects; (C) Unfavorable results when optical
see-through HMD is used; (D) Unfavorable results when IPT
(Immersive Projection Technology) like CAVE is used.
VRIC 2003 Proceedings –69/4
On the other hand, the virtual ashtray cannot occlude
a real hand when IPT (Immersive Projection
Technology) like CAVE is used, as shown in Figure
7 (D). This is due to the fact that the display
position of virtual objects is always at the screen
surface, which is one to two meters away from the
human user when CAVE and other IPT (Immersive
Projection Technology) displays are used.
In our laboratory at the University of Tokyo, a new
type of visual display is being developed called X'tal
(pronounced crystal) vision [16,17], which uses
retro-reflective material as its projection surface.
We call this type of display technology RPT
(Retro-reflective Projection Technology).
Under the RPT configuration, a projector is arranged
at the axial symmetric position of a user's eye with
reference to a half-mirror, with a pinhole placed in
front of the projector to ensure adequate depth of
focus, as shown in Figure 8. Images are projected
onto a screen that is either made, painted, or covered
with retro-reflective material [17].
Projector
Pin Hole
Retroreflector
Half Mirror
Operator
Fig.8 The Principle of RPT system.
A retro-reflective surface projects back the projected
light just to the direction of projection, while
conventional screens normally used for IPT scatter
projected lights in all directions ideally as a
Lambertian surface (Fig.9). Figure 10 shows how a
retro-reflective surface behaves . It is covered with
microscopic beads of about 50 micrometers in
diameter, which reflect back the incident light to the
incident direction. It can also be realized with a
microstructure of prism-shaped retro-reflectors
densely placed on a surface.
 Fig. 9 Three typical reflections.

Fig. 12 Head Mounted Projector (HMP) with RPT.

Figure 13 shows an example of an image projected

on a sphere painted with retro-reflective material.
Fig. 10 Retro-reflective surface densely covered with Figure 14 shows an example of projecting a virtual
microscopic beads with about 50 micrometer diameter. Ideally, cylinder onto a Shape Approximation Device
refractive index should be 2. (SAD) [18], which is a haptic device that enables
the user to touch geometrical shapes as if they
The retro-reflector screen, together with the pinhole, were real. The use of the SAD as the
ensures that the user always sees images with retro-reflective screen enables us to feel just as we
accurate occlusion relations. In the construction of see by using a HMP.
an RPT system, screen shapes are arbitrary, i.e., any
shape is possible. This is due to the characteristics of
the retro-reflector, and the pinhole in the conjugate
optical system. By using the same characteristics of
an RPT system, binocular stereovision becomes
possible using only one screen with an arbitrary
shape. The projector can be mounted on the head
of a user, which we call an HMP (Head Mounted
Projector) system as shown in Figure 11. Figure 12
shows a general view of a prototype HMP.

Fig.13 Projected image on a spherical retro-reflective screen.

In the figure, (A) illustrates the principle of SAD,

(B) shows an image to be displayed, (C) is an
actual SAD, and (D) indicates the image projected
onto SAD, which can be touched as it is seen.

Fig.11 Principle of stereo display using RPT with Head Mounted

Projector (HMP).

VRIC 2003 Proceedings –69/5


Fig.14 Projected image on a Surface Approximation Device (SAD).
A: Principle of SAD; B: Image; C: Actual SAD; D: Image
projected on SAD, which can be touched as it is seen.
Figure 15 shows an example of the use of RPT for
augmented reality. Pre-captured x-ray and/or MRI
data can be superimposed onto a human patient so
that a surgeon can have open surgery related
information even under the minimally invasive
surgery environment. By superimposing
ultra-sonic data, real-time presentation of inner
body information is also possible through RPT.
Fig.15 An Augmented Reality Application using PRT.
Figure 16 shows another example of applying RPT
to medicine.
VRIC 2003 Proceedings –69/6
Fig. 16 Another Example of the application of RPT to medicine.
Figures 17 and 18 show application examples of
using RPT for optical camouflage. In Fig. 17,
pre-captured a background image is projected so that
a retro-reflective object grasped by a human appears
to be transparent.
Fig. 17 An optical CamouflageApplication using RPT.
Figure 18 shows how optical camouflage can be
achieved using real-time video information. Figure
19 shows how RPT is applied to realize the situation
of Figure 18. The coat is made of retro-reflective
material so that the coming light is reflected back to
the same direction that it comes from. Microscopic
beads on the surface of the coat have the function of
retro-reflection.
A half mirror makes it possible for a spectator to see
virtually from the position of the projector. An HMP
projects an image of the background scenery
captured by the video camera behind the
camouflaged subject. A computer calculates the
appropriate perspective and transforms the captured
image to the image to be projected on the subject
using image-based rendering techniques. Since the
cloak the subject is wearing is made of a special
retro-reflective material, which reflects back the
incident light just the same direction it comes from,
an observer looking through a half mirror sees a very
bright image of the scenery so that he is virtually
transparent.
robots A' and B', and are sent to the HMPs of human
users A and B respectively, both with a sensation of
presence. Both telexistence robots A' and B' are
seen as if they were their respective human users by
projecting the real image of the users onto their
respective robots.

Motion Control / Tactile Control

Visual & Auditory Information

HMP 3D image observed by

Telexistence Robot B’
3D image observed by is projected and observed
Telexistence Robot A ’ by Human User B with
is projected and observed by
a sensation of presence.
Human User A with a
sensation of presence.

Human Telexistence
User A Robot A’

Telexistence Robot B ’ Human

Telexistence Robot A’is HMP User B
Telexistence Robot B’is
Fig. 18 Another example of application of RPT to optical observed as if it is Human
User B by the projection
observed as if it is Human
User A by the projection
of real image of B on of real image of A on
camouflage. the robot B ’
. the robot A’.

Visual & Auditory Information

Motion Control / Tactile Control

Fig. 20 Concept of Robotic Mutual Telexistence (adopted from

[13]).

Fig. 19 A schematic diagram of the RPT system used for the

optical camouflage in Fig.18.

4. Mutual Telexistence using RPT

More than twenty years have passed since we
initially proposed the concept of telexistence, and it
is now possible to telexist in the remote and/or
virtual environment with a sensation of presence.
We can work and act with the feeling that we are
present in several real places at once. However, in
the location where the user telexists, people see only
the robot but cannot feel that the person is actually
present. Simply placing a TV display on board the
robot to show the face of the user is not very
satisfying, since it appears mostly comical and far
from reality.
By using RPT, the problem can be solved as shown
in Figure 19 [13]: suppose a human user A uses his
telexistence robot A' at the remote site where another
human user B is present. The user B in turn uses
another telexistence robot B', which exists in the site
where the user A works. 3-D images of the remote
scenery are captured by cameras on board both
Fig. 21 (A) Miniature of the HONDA Humanoid Robot,
(B) Painted with Retro-reflective Material,
(C) Example of Projecting a Human onto it,
(D) Another Example (adopted from [13]).
Figure 21 presents an example of how mutual
telexistence can be achieved through the use of RPT.
Figure 20(A) shows a miniature of the HONDA
Humanoid Robot, while Figure 21(B) shows the
robot painted with retro-reflective material. Figures
21 (C) and (D) show how they appear to a human
wearing an HMP. The telexisted robot looks just
like the human operator of the robot, and
telexistence can be naturally performed [13]. We
are currently in the process of a feasibility study for

VRIC 2003 Proceedings –69/7

the proposed method using TELESAR.
5. Conclusions
Projection technology on Retro-reflective surfaces is
called RPT (Retro-reflective Projection technology),
which is a new approach to augmented reality (AR).
The first demonstration of RPT was made at
SIGGRAPH98, followed by demonstrations at
SIGGRAPH99 and SIGGRAPH2000. In this
keynote paper, the principle of RPT is explained, and
applications to AR are discussed.
Mutual telexistence is one of the most important
technologies for the realization of networked
telexistence, because users "telexisting" in a robot
must know whom they are working with over the
network. A method using RPT, especially an HMP
(Head Mounted Projector) and a robot with
retro-reflective covering was proposed and proved to
be a promising approach toward the realization of
mutual telexistence.
Acknowledgments: The research presented here
has been conducted in part under the CREST
Telexistence Communication Systems Project,
supported by JST (Japan Science and Technology
Corporation). This work has been supported by
many members and ex-members of Tachi
Laboratory of the University of Tokyo. Special
thanks to Dr. Naoki Kawakami, Dr. Masahiko
Inami and Dr. Dairoku Sekiguchi for their
contribution to the project.
References
[1] S.Tachi, K.Tanie, K.Komoriya and M.Kaneko:
Tele-existence (I): Design and evaluation of a visual
display with sensation of presence, Proceedings of the 5th
Symposium on Theory and Practice of Robots and
Manipulators (RoManSy '84), pp.245-254, Udine, Italy,
(Published by Kogan Page London), June 1984.
[2] S.Tachi, H.Arai, I.Morimoto and G.Seet: Feasibility
experiments on a mobile tele-existence system, The
International Symposium and Exposition on Robots (19th
ISIR), Sydney, Australia, November 1988.
[3] S.Tachi, H.Arai and T.Maeda: Development of an
Anthropomorphic Tele-existence Slave Robot,
Proceedings of the International Conference on Advanced
Mechatronics (ICAM), pp.385-390, Tokyo, Japan, May
1989.
[4] S.Tachi, H.Arai and T.Maeda: Tele-existence master
VRIC 2003 Proceedings –69/8
slave system for remote manipulation, IEEE International
Workshop on Intelligent Robots and Systems (IROS'90),
pp.343-348, 1990.
[5] S.Tachi and K.Yasuda: Evaluation Experiments of a
Tele-existence Manipulation Sysytem, Presence, vol.3,
no.1, pp.35-44, 1994.
[6] Y.Yanagida and S.Tachi: Virtual Reality System with
Coherent Kinesthetic and Visual Sensation of Presence,
Proceedings of the 1993 JSME International Conference
on Advanced Mechatronics (ICAM), pp.98-103, Tokyo,
Japan, August 1993.
[7] K.Oyama, N.Tsunemoto, S.Tachi and T.Inoue:
Experimental study on remo te manipulation using virtual
reality, Presence, vol.2, no.2, pp.112-124, 1993.
[8] S.Tachi: Real-time Remote Robotics - Toward
Networked Telexistence : IEEE Computer Graphics and
Applications, vo.18, Nov-Dec98, pp.6-9, 1998.
[9] Y.Yanagida, N.Kawakami and S.Tachi: Development
of R-Cubed Manipulation Language - Accessing real
worlds over the network, Proceedings of the 7th
International Conference on Artificial Reality and
Tele-Existence (ICAT'97), pp.159-164, Tokyo, Japan,
December 1997.
[10] T.Nishiyama, H.Hoshino, K.Suzuki, R.Nakajima,
K.Sawada and S.Tachi, "Development of Surrounded
Audio-Visual Display System for Humanoid Robot
Control", Proc.of 9th International Conference of
Artificial Reality and Tele-existence (ICAT'99), pp.
60-67, Tokyo, Japan, December 1999.
[11] T.Nishiyama, H.Hoshino, K.Suzuki, K.Sawada and
S.Tachi, "Development of Visual User Interface
Embedded in Tele-existence Cockpit for Humanoid Robot
Control", Proc. of IMEKO 2000 World Congress, Vol.XI
(TC-17 & ISMCR2000), pp.171-176, Vienna, Austria,
September 2000.
[12] S.Tachi, K.Komoriya, K.Sawada, T.Nishiyama,
T.Itoko, M.Kobayashi and K.Inoue: Development of
Telexistence Cockpit for Humanoid Robot Control,
Proceedings of 32nd International Symposium on
Robotics (ISR2001), pp.1483-1488, Seoul, Korea, April
2001.
[13] S.Tachi: Augmented Telexistence, Mixed Reality,
pp.251-260, Published by Springer-Verlag, 1999.
[14] S.Tachi: Toward Next Generation Telexistence,
Proceedings of IMEKO-XV World Congress, vol.X
(TC-17 & ISMCR'99), pp.173-178 , Tokyo/Osaka, Japan,
June 1999.
[15] S.Tachi: Toward the Telexistence Next Generation,
Proceedings of the 11th International Conference on
Artificial Reality and Tele -Existence, (ICAT2001), pp.1-8,
Tokyo, Japan, December 2001.
[16] N.Kawakami, M.Inami, T.Maeda and S.Tachi: Media
X'tal -Projecting virtual environments on ubiquitous
object-oriented retro -reflective screens in the real
environment-, SIGGRAPH'98, Orlando, FL, July 1998.
[17] M.Inami, N.Kawakami, D.Sekiguchi, Y.Yanagida,
T.Maeda and S.Tachi: Visuo-Haptic Display using
Head-Mounted Projector, Proceedings of IEEE Virtual
Reality 2000, pp. 233-240, New Brunswick, NJ, March
2000.
[18] S.Tachi, S.Tachi, T.Maeda, R.Hirata and H.Hoshino:
A construction method of virtual haptic space, Proceeding
of the 4th International Conference on Artificail Reality
and Tele-Existence (ICAT'94), pp.131-138, Tokyo, Japan,
July 1994.
[19] M. Inami, N. Kawakami, Y. Yanagida, T. Maeda and
S. Tachi, Method of and Apparatus for Displaying an
Image, US PAT. 6,283,598 2001.
[20] M. Inami, N. Kawakami, Y. Yanagida, T. Maeda and
S. Tachi, Method and Device for Providing Information,
US PAT. 6,341,869 2002.
[21] S.Tachi: Two Ways of Mutual Communication:
TELESAR and TWISTER, in S.Tachi ed.
Telecommunication, Teleimmersion and Telexistence,
IOS Press, ISBN 1-58603-338-7, pp. 3-24, 2003.

VRIC 2003 Proceedings –69/9