An Interactive Virtual Reality System

with a Wireless Head-Mounted Display

Shujia Hao, Wei Song, Kaisi Huang, Yulong Xi, Kyungeun Cho
and Kyhyun Um

Abstract Virtual reality (VR) with head-mounted display (HMD) device provides
an immersive experience for novel multimedia applications. This paper develops an
interactive virtual reality system with a wireless HMD to enable a natural VR
operation interface. In a server, the system utilizes the Kinect as a motion detection
device to estimate VR user’s location and gesture information in real time. Through
a WiFi network, the user’s information is transferred to the HMD as a client, where
the user controls an avatar following his motion. The controlled avatar interacts
with the virtual environment in real time. The proposed system is implemented
using a Samsung Gear VR, Kinect 2.0 and Unity3D environment. This system is
compatible with serious games, virtual and physical collaboration, natural user
interfaces, and other multimedia applications.

Keywords Virtual reality
Natural user interface
Head-mounted display

Wireless sensor network

1 Introduction

Virtual Reality (VR) technology has been actively researched in various multimedia
fields. HMD is widening rapidly in recent years. HMDs allow VR user to observe
the virtual environment at both left and right viewports, which generate a stereo
image in human’s brain [1]. HMD devices are also available for Natural User
Interface (NUI) applications, which aim to implement multimedia operation with
human gesture controlling [2]. The interdisciplinary projects combining NUI, VR
and HMD provide users with natural operation and intuitive observation experience.

S. Hao
W. Song (&)
K. Huang

Department of Digital Media Technology, North China University of Technology,
Beijing, China
e-mail: sw@ncut.edu.cn
Y. Xi
K. Cho
K. Um
Department of Multimedia Engineering, Dongguk University, Seoul, Korea

© Springer Science+Business Media Singapore 2016 203

J.J. (Jong Hyuk) Park et al. (eds.), Advanced Multimedia and Ubiquitous
Engineering, Lecture Notes in Electrical Engineering 393,
DOI 10.1007/978-981-10-1536-6_27
204 S. Hao et al.

Recently, several human motion sensors are developed for NUI input devices,
such as Microsoft Kinect and Leap Motion [3]. The Kinect is a popular body
gesture detection device for touch-less operation interface. By combining HMD
with the Kinect, this paper proposes a wireless interactive virtual reality system. As
computing in a server, the Kinect detects the user’s motion information, as an NUI.
Different from video stream transmission method, this system only transmit the
motion signal and the VR visualization process is generated by the HMD client.
This method with small data volume enables the real-time wireless network
transmission so as to promote the HMD VR commercialized. The proposed system
provides several function interfaces for NUI application, such as serious games,
medical experiments, and multimedia lectures.
The remainder of this paper is organized as follows. Section 2 overviews related
work. Section 3 discusses the architecture of the interactive virtual reality system.
Section 4 develops a serious game using the proposed system. Section 5 concludes
this paper.

2 Related Work

Currently, Oculus Rift is a popular display device in VR researching areas.

Draganov [4] proposed a disable patient assistant system with the Oculus Rift. In
the system, he utilized a sensor glove equipped on the hands for hand gestures
recognition, so as to interact with virtual objects. Kenneth [5] utilized a leap motion
to develop a molecular graphics controlling toolkit, called Aquaria. By hand motion
recognition, the interaction between real and virtual world was visualization in
scene of the Oculus Rift display. Avila [6] utilized a Virtuix Omni VR treadmill to
extend the user’s movement scope by walking, jumping, and strafing on it. The
interaction methodologies between real and virtual environments are always
detected through wearable sensors, such as data gloves and inertial measurement
units. These NUI controllers need data transmission cables to communicate with a
server computer. The cables limit the scope of user’s movement and reduce the VR
immersion.
Some researchers implemented the sensors computation by a computer carried
on the user’s back. Although this method enable the VR user’s free movement, the
user felt tired for a long time walking with the heavy computer. These methods with
body mounted sensors caused the operation uncomfortable, thus wireless human
gesture sensing device is necessary for VR enhancement. To solve this problem,
Sony PlayStation VR, a wireless HMD, was developed so that wearable devices
were not required for VR development [7]. These technologies transmitted the VR
visualization result to the HMD via a wireless network. Due to the limited band-
width, the display signal was unstable for real-time video transmission, while the
user moves. To enhance natural user experience, this paper develops a wireless
HMD technology for real-time interactive VR approaches.
An Interactive Virtual Reality System with a Wireless … 205

3 Interactive Virtual Reality System

We proposed an interactive virtual reality system with wireless HMD, as shown in

Fig. 1, which provides VR developers with wireless motion recognition interface.
The system is mainly combined with a motion detection server and a wireless
HMD VR client.
In the motion detection server, we utilize a Kinect to detect user’s skeleton
motion and analyze gesture information from infrared video sequences. We display
the user’s location and gesture information in a monitor. The system delivers the
computed motion and gesture datasets to the wireless HMD via WiFi network.
Different from transmitting large dataset of the VR visualization buffers, the
delivery dataset size by this method is within 1 KB per frame.
In the wireless HMD VR client, an avatar is controlled by the received gesture
data. The avatar interacts with virtual objects computed by the smart phone, which is
installed in a HMD. We utilize the Unity Legacy Integration, provided by Oculus, to
realize the immersive experience for VR display. The virtual environment is cap-
tured by left and right virtual cameras, which provide a stereo scene to the VR user.

4 Experiments

Using the proposed system, we developed a serious game, shown in Fig. 2. We

utilized Microsoft Kinect 2 to detect user’s gesture on a server, which is a 2.60 GHz
Intel® Core™ i5-4210 M CPU laptop with a GeForce GTX 850 M graphics card
and 4 GB RAM. The client is implemented on a Samsung Gear VR with a Samsung

Fig. 1 The architecture of the proposed interactive virtual reality system

206 S. Hao et al.

Fig. 2 A serious game

development using the
proposed interactive VR
system. a Real world capture.
b Avatar controlling. c Avatar
in HMD VR. d Game sense

galaxy Note 4 in it. The Note 4 has a 2.7 GHz Qualcomm Snapdragon Quad CPU,
3 GB RAM, 2560 × 1440 resolution, and android 4.4 operation system.
As shown in Fig. 2a, the user wore the HMD and played the game in front of a
Kinect. The server monitor displayed the human gesture recognizing result joint
with an avatar, as shown in Fig. 2b, in order to monitor and correct the Kinect
performance. Using the TCP/IP, the gesture information is transmitted to the client.
We developed a serious game using Unity3D, which was exported to android
operation system in the HMD VR. In the game, the player was allowed to hit virtual
monsters and walked in a virtual world with his hand. Besides monsters and virtual
An Interactive Virtual Reality System with a Wireless … 207

environment, the player was able to see his avatar body at the first person viewport,
as shown in Fig. 2c, d.
The proposed system allows users to interact the virtual objects by their gestures,
without wearing any motion detection device. It has a bright prospect in digital
media technologies, such as entertainment, gaming, film and video, and education.

5 Conclusions

To provide a natural and convenient interface for the HMD VR, this paper
described an interactive VR System using the Kinect, as a gesture detection sensor.
By transmitting the gesture information from the server to the HMD VR client, the
system realized that the user interacted with the virtual environment by his gesture
operation. The user was able to control and browse his avatar in the HMD VR in
real time due to the small transmission datasets. The developed serious game
verified that such wireless NUI operation approach enhanced the immersive
experience.
In our system, the obstacles in the real world are not visualized in the VR scene,
thus the user is easy to hit and knock onto the obstacles. In future, we will
reconstruct the real world to a virtual 3D background model using a LiDAR sensor.
The VR scene will provide obstacle warning graphics interface to show the point
clouds of the real world.

Acknowledgments This research was supported by the National Natural Science Foundation of
China (61503005), and by SRF for ROCS, SEM.

References

1. Gordon NS, Merchant J, Zanbaka C et al (2011) Interactive gaming reduces experimental pain
with or without a head mounted display. Comput Hum Behav 27:2123–2128
2. Song W, Cai X, Xi Y et al (2015) Real-time single camera natural user interface engine
development. Multimedia Tools Appl, pp 1–17
3. Beattie N, Horan B, McKenziea S (2015) Taking the LEAP with the oculus HMD and CAD—
plucking at thin air? Procedia Technol 20:149–154
4. Draganov IR, Boumbarov OL (2015) Investigating oculus rift virtual reality display
applicability to medical assistive system for motor disabled patients. In: The 8th IEEE
international conference on intelligent data acquisition and advanced computing systems:
technology and applications, pp 24–26
5. Kenneth S, Christian S, Bruce T et al (2013) The molecular control toolkit: controlling 3D
molecular graphics via gesture and voice. In: 2013 IEEE symposium on biological data
visualization, pp 49–56
6. Avila L, Bailey M (2014) Virtual reality for the masses. Comput Graph Appl 34:103–104
7. Ito M, Kihara K, Yoshida S et al (2015) Patient’s self-monitoring of transurethral surgical
images using a head-mounted display. Urol Case Rep 3:27–29