2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)
Reproducing Material Appearance of Real Objects
using Mobile Augmented Reality
Seiji Tsunezaki*
Saitama University
Shoji Yamamoto§
Tokyo Metropolitan College of Industrial Technology
Figure 1: Mobile augmented reality system for reproducing material appearance of real objects
A BSTRACT
In this paper, we propose a system that can reproduce the material
appearance of real objects using mobile augmented reality (AR).
Our proposed system allows a user to manipulate a virtual object,
whose model is generated from the shape and reflectance of a real
object, using the user’s own hand. The shape of the real object is
reconstructed by integrating depth images of the object, which are
captured using an RGB-D camera from different directions. The
reflectance of the object is obtained by estimating the parameters of
a reflectance model from the reconstructed shape and color images,
assuming that a single light source is attached to the camera. We
measured the shape and reflectance of some real objects and pre-
sented the material appearance of the objects using mobile AR. It
was confirmed that users were able to obtain the perception of ma-
terials from changes in gloss and burnish of the objects by rotating
the objects using their own hand.
Keywords: reflectance property, reflectance measurement, object
manipulation, mobile display, RGB-D camera
Index Terms: Human-centered computing—Human conputer in-
teraction (HCI)—Interaction paradigms—Mixed / augmented real-
ity
1 I NTRODUCTION
With the spread of online shopping, it has become possible to see
and purchase products through the Internet. However, since users
cannot see a product with taking it in their hand, they often fail to
* e-mail: tsunezaki@is.ics.saitama-u.ac.jp
† e-mail:
nomura@is.ics.saitama-u.ac.jp
‡ e-mail: komuro@mail.saitama-u.ac.jp
§ e-mail: yamasho@metro-cit.ac.jp
¶ e-mail: tsumura@faculty.chiba-u.jp
978-1-5386-7592-2/18/$31.00 ©2018 IEEE
DOI 10.1109/ISMAR-Adjunct.2018.00126
Ryota Nomura† Takashi Komuro‡
Saitama University Saitama University
Norimichi Tsumura¶
Chiba University
obtain the perception of material. Therefore, the system that repro-
duces the realistic material appearance of real objects is required.
To reproduce the realistic material appearance, there have been
mobile augmented reality (AR) systems which allow a user to ma-
nipulate a virtual object using his or her hand, which is superim-
posed in the real space and that is presented on a mobile display or
a head-mounted display [3, 4]. However, these systems use man-
ually designed models for virtual objects and do not use models
generated from real objects.
There have been studies for measuring reflectance properties of
real objects using light sources and cameras that are put in various
directions around a target object [5, 6]. It is possible to measure
exact reflectance properties by providing incident light from vari-
ous directions and observing reflected light to various directions.
However, this requires large-scaled apparatus and long measure-
ment time.
On the other hand, studies that estimate the shape, light sources
and reflectance property only from images captured by a monocu-
lar camera have been conducted [1, 9]. These studies do not need
a large-scaled apparatus or long measurement time. However, the
amount of information obtained from the images of a monocular
camera is limited, which may cause unstable estimation and incor-
rect result.
Based on these studies, we propose a mobile AR system that
reproduces the material appearance of a real object, whose shape
and reflectance are measured using a simple apparatus. A user can
obtain the perception of material by manipulating a virtual object
that is displayed on a mobile display using his or her own hand.
2 M OBILE AR SYSTEM FOR REPRODUCING MATERIAL AP -
PEARANCE OF REAL OBJECTS
Figure 1 shows the concept of our proposed system. A user can ma-
nipulate a virtual object that is displayed on a mobile display using
his or her own hand. The model of the virtual object is generated
from the shape and reflectance of a real object, which are measured
using a simple apparatus consisting of an RGB-D camera, a light
source that is attached to the camera, and a turntable.
421
Figure 2: A user is manipulating an object whose model is generated from the shape and reflectance of a real object, using his or her own hand.
2.1 Providing perception of materials using Mobile AR
We use the mobile AR system developed by Nomura et al. [8] for
presenting material appearance of objects. To realize natural inter-
action with virtual objects, the system provides visual information
that is consistent with the user’s somatic sensation by displaying
user-perspective images on a mobile display. The system allows
the user to manipulate a virtual object using his or her own hand
with six degrees of freedom (three degrees of freedom for trans-
lation and three degrees of freedom for rotation) by mapping the
hand pose to the object’s translation and rotation. The user can ob-
tain the perception of materials from changes in gloss and burnish
of the objects by rotating the objects using his or her own hand.
2.2 Measuring shape and reflectance of real objects
To measure reflectance properties of real objects, the system cap-
tures depth and color images of a rotating object on the turntable us-
ing the RGB-D camera. The shape of the object is reconstructed by
integrating the depth images of the object captured from different
viewpoints. The reflectance of the object is obtained by estimating
the parameters of a reflectance model from the reconstructed shape
and color images.
Figure 3 shows the measurement scene. We connected Microsoft
Kinect to a PC and used it for capturing the images of a target ob-
ject. We used a small LED light as a single light source and fixed
the light to the camera.
.LQHFW
2EMHFW
/('/LJKW
7XUQWDEOH
Figure 3: Measurement scene
The 3D shape of the target object is reconstructed by integrat-
ing the depth images captured from different viewpoints using the
RGB-D camera. We used KinectFusion [7] for 3D shape recon-
struction. Based on ICP (Iterative Closest Point) algorithm, Kinect-
Fusion aligns point clouds with high accuracy and estimates the
camera motion.
The system calculates the correspondence between each vertex
of the reconstructed 3D model and color pixels in all the captured
frames. By reprojecting the vertex to all the camera viewpoints, it is
determined which pixel in each color image the vertex corresponds
422
to. By picking the corresponding pixel values, intensity change at
the vertex over all frames is obtained.
The reflectance property of the target object is estimated from the
reconstructed shape, acquired change intensity, and the light source
information. We used the Blinn-Phong model [2] as the reflectance
model. To obtain parameters of this model, the least squares method
is applied to the reflectance model and observation values.
3 D EMO D ESCRIPTION
In our demonstration, participants can experience a new mobile AR
system that reproduces the material appearance of a real object. The
participants can manipulate a virtual object with six degrees of free-
dom, whose model is generated from the shape and reflectance of
the real object, using their own hand. The participants feel like they
are taking the object in their hand by grasping and manipulating the
virtual object using their hand. In addition, the participants can see
the object from various directions such as from the side and from
the back by rotating the object.
Figure 2 shows examples of presenting the material appearance
of real objects which were measured using our measurement sys-
tem. A user is manipulating the virtual object whose model is gen-
erated from the shape and reflectance of a real object, using his or
her own hand.
The unique point of our demonstration is that our system real-
izes reproduction of the material appearance of real objects using
mobile augmented reality. Our system can be used in online shop-
ping such as users can see the material appearance of products using
the system. Our demonstration will draw a crowd since the system
is easy to use and presents the realistic material appearance to the
participants.
R EFERENCES
[1] J. T. Barron et al. Shape, illumination, and reflectance from shading.
IEEE Transactions on Pattern Analysis Intelligence, 37(8):1670–1687,
2015.
[2] J. F. Blinn. Models of light reflection for computer synthesized pictures.
In Proc. of SIGGRAPH 1977, pages 192–198, 1977.
[3] M. L. Caballero et al. Behand: Augmented virtuality gestural interac-
tion for mobile phones. In Proc. of MobileHCI 2010, pages 451–454,
2010.
[4] A. Hettiarachchi et al. Annexing reality: Enabling opportunistic use of
everyday objects as tangible proxies in augmented reality. In Proc. of
CHI 2016, pages 1957–1967, 2016.
[5] H. Li et al. Automated three-axis gonioreflectometer for computer
graphics applications. Optical Engineering, 45(4), 2005.
[6] G. Muller et al. Rapid synchronous acquisition of geometry and ap-
pearance of cultural heritage artefacts. In Proc. of VAST 2005, pages
13–20, 2005.
[7] R. A. Newcombe et al. Kinectfusion: Real-time dense surface mapping
and tracking. In Proc. of ISMAR 2011, pages 127–136, 2011.
[8] R. Nomura et al. Mobile augmented reality for providing perception of
materials. In Proc. of MUM 2017, pages 501–506, 2017.
[9] R. Xia et al. Recovering shape and spatially-varying surface reflectance
under unknown illumination. TOG, 35(6):1–12, 2016.