Research Report

On

AGENTS AND UBIQUITOUS COMPUTING

IN

MICROSOFT KINECT: CONTROLLER FREE INTERACTION

Submitted by
SUNIL KUMAR KAR

Intake No: Ptl00782225

Asia Pacific Institute of Information Technology

Faridpur Road, Paniapt, Haryana-132103

 Agents and Ubiquitous Computing
In
Microsoft Kinect: Controller Free Interface
Sunil Kumar Kar
B.E. Final year (Computing)
Int ak e -Pt l 00782225 (Le v e l 3)
Asia Pacific Institute of Information Technology
Staffordshire University, Panipat-132103, Haryana
Email Id – sunilkarcse@gmail.com

Abstract-The idea of ubiquitous computing is an environment dominated by computing and communication

devices of different scales which are seamlessly integrated to the user activities. This research paper describes
a middleware to facilitate developers to manage some of the complexities associated with the development of
ubiquitous computing systems by means of use of autonomous agents, which enable ubiquitous computing
technology to respond to users’ particular conditions and demands. Autonomous agents are used to
implement the desirable features of ubiquitous computing systems and for enhancing th interactions of the
users with environment. In particular, the paper focuses on agents in ubiquitous and controller free gaming
interface environments which need to be context-aware so that they can adapt themselves to different
situations. Another key issue in these environments is allowing autonomous, heterogeneous agents to have a
common semantic understanding of contextual information. [1]

Keywords- Ubiquitous Computing, Controller Free Interface, Agents, Gaming, Microsoft Kinect,
Xbox 360

1. INTRODUCTION
Microsoft Kinect is the code name for a "controller-free gaming and entertainment experience" for the Xbox 360
video game platform. It enables users to control and interact with the Xbox 360 without the need to touch a game
controller through a natural user interface using gestures, spoken commands, or presented objects and images. [2]
 Fig.1 The user interaction with the Microsoft Kinect through body movement

 What makes this accessory distinct from the others is of the fact that player is the one who controls the games
through making his/her own body movements throughout the game. Moreover, it lets the players get access to
an easier interface that's capable of reading one's body movements, motion and voice commands. This can be
done through a motion sensor which is capable of recording the player's movements accessorized with a multi-
array mic for the voice commands, a camera with CCD sensors, and a depth sensor also.[3]
 But for one thing, Kinect doesn’t just record your movements. Its system of cameras, microphones, sensors and
software algorithms also records (and recognizes) your voice and can recognize faces and objects.[3]

2. SYSTEM OVERVIEW
 The device features an "RGB camera, depth sensor, multi-array microphone, and custom processor running
proprietary software", which provides full-body 3D motion capture, facial recognition( by RGB Camera), and
voice recognition capabilities running proprietary software.[4]
 The depth sensor is an infrared projector combined with a monochrome CMOS sensor that lets “Microsoft
Kinect” sees the room in 3-D (as opposed to inferring the room from a 2-D image) under any lighting
conditions.[4]
 The multi-array microphone in “Microsoft Kinect” locates voices by sound and can extract ambient noise. This
allows headset-free Xbox LIVE party chat and more.[4]
 Proprietary software at the heart of “Microsoft Kinect” is what makes the magic possible. This is what distances
“Microsoft Kinect” from any other technology on the market, enabling human body recognition and extract
other visual noise.[5]
 In addition to “Microsoft Kinect” tracking your full body movement in 3-D, it also recognizes your face and
facial expressions. It can even detect the change of emotion in your voice.
 Microsoft Kinect” remembers your face and voice so that you can connect to Xbox LIVE without the need to
reach for a controller. This is the future!
2.1 Technology [6]
  The Kinect sensor is a horizontal bar connected to a small base with a motorized pivot, and is designed
to be positioned lengthwise below the video display. The device features an "RGB camera, depth
sensor and multi-array microphone running proprietary software", which provides full-body 3D motion
capture, facial recognition, and voice recognition capabilities. The Kinect sensor's microphone array
enables the Xbox 360 to conduct acoustic source localization and ambient noise suppression, allowing
for things such as headset-free party chat over Xbox Live.
 The depth sensor consists of an infrared projector combined with a monochrome CMOS sensor, and
allows the Kinect sensor to see in 3D under any ambient light conditions. The sensing range of the
depth sensor is adjustable, with the Kinect software capable of automatically calibrating the sensor
based on game play and the player's physical environment, such as the presence of furniture.
 Kinect is based on software technology developed internally by Microsoft and range camera
technology by Israeli developer PrimeSense, which interprets 3D scene information from a
continuously-projected infrared pattern. The software technology enables advanced gesture
recognition, facial recognition, and voice recognition. Kinect is capable of simultaneously tracking up
to six people, including two active players for motion analysis with a feature extraction of 20 joints per
player.
2.2 Working [7]
 Kinect is a sensor that recognizes motions and voice pattern changes around it with the help of cameras
and microphones in it. Let’s start with a simple example of playing Netflix movies on XBox. If you
have XBox Kinect attached to your XBox 360 console, you can come into the room and say XBox Play
and your movie starts playing or you can say XBox Pause and your movie pauses or you can say XBox
Stop and your movie stops. So you can see how XBox Kinect reacts to voice commands. It works on
some pre-defined commands that your XBox 360 console can act on. No, you can't say, XBox Clean
my room. Well, may be in few years XBox Kinect will help you do that, till then you are stuck with
picking up after you.

Fig.2 Microsoft Kinect Xbox 360

 Now its move on to how XBox Kinect helps in playing games. For example if you are playing Wii Sports,
then moving of your arms with Wii controller in your hand, helps you perform actions like bowling, play
tennis, play baseball and things like that. XBox Kinect is very much the same principal but the difference is
that there is no controller. When you want to play Kinect game like Kinect Sports, you will simply jump in
front of Kinect device or sensor; it will recognize you as a player. And then you will perform the game
actions like swinging your arms, kicking your feet etc. and Kinect will send the signals to your XBox
Console and you are on roll and being next Wimbledon champion.
2.3 Advantages

 The system is also designed to be compatible and used for music listening and video chats. Users can share
and view movies together and use voice commands to activate the device to start the movie, no buttons or
any kind of physical touching is necessary. [8]
 Your Shape" is a workout program which lets users exercise with a digital trainer and electronically track
workout progress.
 "Kinect Adventure" engages the player in a rafting adventure traveling down a digital river.
 "Dance Central" gives players the opportunity to move to a choreographed beat of popular music videos,
integrating top songs with games, probably an attractive feature for many.

2.3 Disadvantages [9]

 There are advantages to this whole controller free experience. But there are some downsides to it as well.
As I mentioned about voice recognition to control playing of movies. Well we all know power of remote
control in our hands. Now imagine, two siblings in same room watching movies with Kinect attached to
XBox console. One wants to pause and other one wants to continue. Well, currently there is no locking
mechanism to prevent Kinect from obeying voice commands based on certain person's voice. So the
actions based on voice could be a little problem with whole bunch of kids and people in same room.
Hopefully by the time XBox Kinect hits the stores, Microsoft will make some improvements and make the
experience more interesting.
 Kinect is truly impressive in our early hands-ons, no doubt. It can track your full body as you spike
volleyball, or it can just watch your hands as you mime a steering wheel. But there's a perpetual, slight
lag. The frame rates in even these somewhat simple games can suffer, most probably because Kinect
requires the Xbox 360 to process all of its data, there's no internal processor in their final build of the
device.

3. AGENTS

“An agent is a computer system that is capable of flexible, autonomous action on behalf of its user or
owner in order to meet its design objectives in a designated environment.”[10]
3.1 Types of Agents in use [11]
 Interface Agents: An interface agent to be a program that can also affect the objects in a direct interaction
with interface, but without explicit instruction from the user. The interface agents reads input that the user
presents to the interface, and it can make changes to the objects the user sees on the screen, though not
necessarily one-to-one with user actions
 Autonomous: An autonomous agent is an agent program that operates in parallel with the user.
Autonomous agents run at every time. Here in our system
 Reactive: An agent can be viewed as an entity that is able to perceive its environment and to act according
to its own decisions.
 Smart: Agents are a new form of software agent that interfaces with other agents forming an artificial
intelligence system. The acronym "SMART" stands for "System for Managing Agents in Real Time".
 Intelligent: A computer program that is given a specific task that can be done without user assistance.
 Mobile: A mobile agent is a composition of computer software and data which is able to transfer
information from one computer to another autonomously and continue its execution on the destination
computer.

3.2 How agent will work in smart home?[12]

Agents in Microsoft Kinect works collaboratively to have a successful controller free gaming interface.
With the power of such agents, the Xbox 360 will be able to perform tasks very easily like by just waving your
hand from the Xbox 360 dashboard, you'll be able to sign into Xbox Live. Wave again and you'll call up a
Kinect-specific interface, and then just say (for example) "Xbox, Zune" to browse your music, or "Xbox,
Facebook" to see what your Facebook pals are up to. You'll also be able to pick Netflix movies to stream or
scan to a specific scene by waving your hand across the interface. There are various agents working in the
Microsoft Kinect Xbox 360 which are described very clearly below:

Fig.3 Microsoft Kinect gets its hardware dissected

 Gesture interface (Interface and Reactive Agent) are graphical user interfaces which accept input in a
form of hand gestures, or mouse gestures sketched with a computer mouse or a stylus.
 Motion tracking interfaces (Interface and Reactive Agent) monitor the user's body motions and translate
them into commands.
 Voice user interfaces (Interface and Reactive Agents) which accept input and provide output by
generating voice prompts. The user input is made by pressing keys or buttons, or responding verbally to the
interface. When Kinect is equipped with this interface, user will be provided a separate menu system on the
Xbox 360. It's simpler, ditching tabs for simpler iconography. To load Netflix, you can wave to hit the
Netflix button, or you just say "Netflix", pause by telling it to pause or reaching in the air to grab a virtual
pause button.
 Motion Sensor (Reactive Agent ): According to T3, the motion sensor in the Kinect is made up of two
elements: a projector that fires out a laser to scan your living room and an infrared VGA camera that tracks
that laser and records what it bounces off of. The Kinect then interprets the pixels from that VGA camera
image and run it through a software filter, which figures out what objects the camera is seeing are
humanoid and which aren’t.
 Smart Camera (Reactive and Intelligent Agent): Kinect is an Xbox 360 peripheral. Think of it as a
webcam on steroids that plugs into the console's USB port that uses IR to track you in 3D space.
 So Kinect Camera is smarter than your average webcam. First, it has excellent sight thanks to its secret
trick: spotlighting the room with invisible infrared light. The camera sees you wonderfully thanks to
this infrared. And, coupled with some advanced software that will run on the 360, it can track 48 point
of your body in real-time for up to two players simultaneously.
 You can fit as many people in that camera as possible, and the computer will see all of them and can
even recognize them as human shapes. But it can only run calculations on two people at a time, just
because the processing power required to track all of the body's locations and movements is so great
(Update: See below). During our testing with the device, a person moving in front of the camera was
able to "steal focus," but the computer can also be told through gestures to keep focus on a certain
person.
 The setup is able to determine depth because the camera is paired with a projector that emits invisible,
infrared light in a complicated speckle pattern. The camera picks up how the pattern is distorted when
it hits objects. At a distance of 7 feet, it can distinguish depth differences of less than an inch. That's
enough to decipher gestures and hand movements. At a distance of 3 feet, the resolution is finer, and
facial expressions can be distinguished
 Video chat via the Kinect camera will be another key Xbox Live feature, including the ability to watch
videos with your video chat partners or conduct over-the-Xbox video calls with Windows Live users.

 Microphone and Microsoft Software (Reactive and Interface Agent): Kinect is equipped with a
microphone so you can talk to the Xbox 360. Also, it doesn't just see you in IR; it can also film you in full
RGB color, recognizing your face to automatically sign you in. And its tilt? Fully motorized to track you.
Kinect also has a microphone that picks up the location of sounds, helping the system understand its
 surroundings. Microsoft's software ties the components together. These two agent collaboratively work
together to perform this task.

Software Agents

Software agents in use are:

 "Videokinect" Chat Software: Whether your friends are on Xbox Live or Windows Live Messenger, you
can video chat with them. But you can also do some other neat stuff, like watch a video together. And as
mentioned above, it can track you in real-time thanks to motion sensing and the motorized tilt.

4. CHALLENGES IN DESIGNING MICROSOFT KINECT?

 It should be user friendly so they can easily use the system.

 It should be able to provide basic capabilities for negotiation, control, reactivity and adaptation.
 Design proactive and responsive interfaces. Applying basic principles of software and knowledge
engineering to develop and deploy agent based systems.
 Already highly competitive market situations in the controller free gaming interface domain

5. CONCLUSION

By the contextual and current study researcher conclude that ubiquitous computing can be implemented by
using the autonomous agents as its main components. Autonomous agents have characteristics that can be
used to implement some of the desirable features of ubiquitous computing system. In these systems
autonomous agents are mainly used to enhance the user’s interaction with the ubiquitous computing
environment in which autonomous agents are the software components that represents users, devices, and
services. In ubiquitous computing, intelligent agents and agent operations must be contextualised within a
broader conceptual discursive space.

6. ACKNOWLEDGEMENT

First of all I would like to thank almighty God whose blessings have inspired me throughout my research
paper. Next I would like to thank my module lecturer Mr. Ankur Singla who has guided me throughout my
research. It is his class room teaching which provided me with the knowledge of ubiquitous computing and
its agents. I would also like to thank the director of our college Prof. Dr. R. K. Choudhary for providing an
educational environment and infrastructure in the college. In the last but not the least I would also like to
thank my dear friends who have continuously inspired me throughout the research work.
 REFERENCES

[1] http://www.worldscibooks.com/compsci/l004.html (Accessed on 10th August 2010, Time- 10:30 pm)

[2] http://gizmodo.com/5563047/what-is-xbox-360-kinect (Accessed on 15th August 2010, Time- 9:30 pm)

[3] http://kotaku.com/5279531/microsoft-project-natal-can-support-multiple-players-see-fingers(Accessed on 20th August 2010, Time-

8:30 pm)

[4] http://venturebeat.com/2009/06/01/microsoft-shows-off-motion-sensing-game-control-project-natal/(Accessed on 25th August 2010,

Time- 7:30 pm)

[5] http://www.tgdaily.com/games-and-entertainment-features/50273-will-microsofts-kinect-appeal-to-core-gamers(Accessed on 30th

August 2010, Time- 10:30 am)

[6] http://sandbox.xerox.com/ubicomp/ (Accessed on 10th sept 2010, Time- 2 pm)

[7] http://www.byteblocks.com/post/2010/06/22/How-XBox-Kinect-Works.aspx(Accessed on 11th sept 2010, Time- 11:30 pm)

[8] http://gizmodo.com/5563047/what-is-xbox-360-kinect(Accessed on 12th sept 2010, Time- 12:30 pm)

[9] http://www.usatoday.com/tech/gaming/2010-06-14-vidgame14_ST_N.htm(Accessed on 15th August 2010, Time- 10:30 pm)

[10] http://www.geek.com/articles/games/thoughts-halo-reach-for-xbox-360-20100913/(Accessed on 10th August 2010, Time- 10:30 pm)

[11] http://gizmodo.com/5633272/where-is-object-scanning-and-controllers-in-xbox-360-kinect(Accessed on 10th August 2010, Time-

10:30 pm)

[12] http://www.xbox.com/en-GB/news-features/news/Project-Natal-in-detail-050609.htm(Accessed on 14th sept 2010, Time- 10:30 pm)