This action might not be possible to undo. Are you sure you want to continue?
SUNIL KUMAR KAR Intake No: Ptl00782225
Asia Pacific Institute of Information Technology Faridpur Road, Paniapt, Haryana-132103
Agents and Ubiquitous Computing In
Microsoft Kinect: Controller Free Interface
Sunil Kumar Kar
B.E. Final year (Computing) Int ak e -Pt l 00782225 (Le v e l 3) Asia Pacific Institute of Information Technology Staffordshire University, Panipat-132103, Haryana Email Id – firstname.lastname@example.org Abstract-The idea of ubiquitous computing is an environment dominated by computing and communication devices of different scales which are seamlessly integrated to the user activities. This research paper describes a middleware to facilitate developers to manage some of the complexities associated with the development of ubiquitous computing systems by means of use of autonomous agents, which enable ubiquitous computing technology to respond to users’ particular conditions and demands. Autonomous agents are used to implement the desirable features of ubiquitous computing systems and for enhancing th interactions of the users with environment. In particular, the paper focuses on agents in ubiquitous and controller free gaming interface environments which need to be context-aware so that they can adapt themselves to different situations. Another key issue in these environments is allowing autonomous, heterogeneous agents to have a common semantic understanding of contextual information. 
Keywords- Ubiquitous Computing, Controller Free Interface, Agents, Gaming, Microsoft Kinect, Xbox 360
Microsoft Kinect is the code name for a "controller-free gaming and entertainment experience" for the Xbox 360 video game platform. It enables users to control and interact with the Xbox 360 without the need to touch a game controller through a natural user interface using gestures, spoken commands, or presented objects and images. 
Fig.1 The user interaction with the Microsoft Kinect through body movement
What makes this accessory distinct from the others is of the fact that player is the one who controls the games through making his/her own body movements throughout the game. Moreover, it lets the players get access to an easier interface that's capable of reading one's body movements, motion and voice commands. This can be done through a motion sensor which is capable of recording the player's movements accessorized with a multiarray mic for the voice commands, a camera with CCD sensors, and a depth sensor also.
But for one thing, Kinect doesn’t just record your movements. Its system of cameras, microphones, sensors and software algorithms also records (and recognizes) your voice and can recognize faces and objects. 2. SYSTEM OVERVIEW
The device features an "RGB camera, depth sensor, multi-array microphone, and custom processor running proprietary software", which provides full-body 3D motion capture, facial recognition( by RGB Camera), and voice recognition capabilities running proprietary software.
The depth sensor is an infrared projector combined with a monochrome CMOS sensor that lets “Microsoft Kinect” sees the room in 3-D (as opposed to inferring the room from a 2-D image) under any lighting conditions.
The multi-array microphone in “Microsoft Kinect” locates voices by sound and can extract ambient noise. This allows headset-free Xbox LIVE party chat and more. Proprietary software at the heart of “Microsoft Kinect” is what makes the magic possible. This is what distances “Microsoft Kinect” from any other technology on the market, enabling human body recognition and extract other visual noise.
In addition to “Microsoft Kinect” tracking your full body movement in 3-D, it also recognizes your face and facial expressions. It can even detect the change of emotion in your voice. Microsoft Kinect” remembers your face and voice so that you can connect to Xbox LIVE without the need to reach for a controller. This is the future! 2.1 Technology 
The Kinect sensor is a horizontal bar connected to a small base with a motorized pivot, and is designed to be positioned lengthwise below the video display. The device features an "RGB camera, depth sensor and multi-array microphone running proprietary software", which provides full-body 3D motion capture, facial recognition, and voice recognition capabilities. The Kinect sensor's microphone array enables the Xbox 360 to conduct acoustic source localization and ambient noise suppression, allowing for things such as headset-free party chat over Xbox Live.
The depth sensor consists of an infrared projector combined with a monochrome CMOS sensor, and allows the Kinect sensor to see in 3D under any ambient light conditions. The sensing range of the depth sensor is adjustable, with the Kinect software capable of automatically calibrating the sensor based on game play and the player's physical environment, such as the presence of furniture.
Kinect is based on software technology developed internally by Microsoft and range camera technology by Israeli developer PrimeSense, which interprets 3D scene information from a continuously-projected infrared pattern. The software technology enables advanced gesture recognition, facial recognition, and voice recognition. Kinect is capable of simultaneously tracking up to six people, including two active players for motion analysis with a feature extraction of 20 joints per player.
2.2 Working  Kinect is a sensor that recognizes motions and voice pattern changes around it with the help of cameras and microphones in it. Let’s start with a simple example of playing Netflix movies on XBox. If you have XBox Kinect attached to your XBox 360 console, you can come into the room and say XBox Play and your movie starts playing or you can say XBox Pause and your movie pauses or you can say XBox Stop and your movie stops. So you can see how XBox Kinect reacts to voice commands. It works on some pre-defined commands that your XBox 360 console can act on. No, you can't say, XBox Clean my room. Well, may be in few years XBox Kinect will help you do that, till then you are stuck with picking up after you.
Fig.2 Microsoft Kinect Xbox 360
Now its move on to how XBox Kinect helps in playing games. For example if you are playing Wii Sports, then moving of your arms with Wii controller in your hand, helps you perform actions like bowling, play tennis, play baseball and things like that. XBox Kinect is very much the same principal but the difference is that there is no controller. When you want to play Kinect game like Kinect Sports, you will simply jump in front of Kinect device or sensor; it will recognize you as a player. And then you will perform the game actions like swinging your arms, kicking your feet etc. and Kinect will send the signals to your XBox Console and you are on roll and being next Wimbledon champion. 2.3 Advantages The system is also designed to be compatible and used for music listening and video chats. Users can share and view movies together and use voice commands to activate the device to start the movie, no buttons or any kind of physical touching is necessary.  Your Shape" is a workout program which lets users exercise with a digital trainer and electronically track workout progress. "Kinect Adventure" engages the player in a rafting adventure traveling down a digital river. "Dance Central" gives players the opportunity to move to a choreographed beat of popular music videos, integrating top songs with games, probably an attractive feature for many. 2.3 Disadvantages  There are advantages to this whole controller free experience. But there are some downsides to it as well. As I mentioned about voice recognition to control playing of movies. Well we all know power of remote control in our hands. Now imagine, two siblings in same room watching movies with Kinect attached to XBox console. One wants to pause and other one wants to continue. Well, currently there is no locking mechanism to prevent Kinect from obeying voice commands based on certain person's voice. So the actions based on voice could be a little problem with whole bunch of kids and people in same room. Hopefully by the time XBox Kinect hits the stores, Microsoft will make some improvements and make the experience more interesting. Kinect is truly impressive in our early hands-ons, no doubt. It can track your full body as you spike volleyball, or it can just watch your hands as you mime a steering wheel. But there's a perpetual, slight lag. The frame rates in even these somewhat simple games can suffer, most probably because Kinect requires the Xbox 360 to process all of its data, there's no internal processor in their final build of the device.
“An agent is a computer system that is capable of flexible, autonomous action on behalf of its user or owner in order to meet its design objectives in a designated environment.”
3.1 Types of Agents in use  Interface Agents: An interface agent to be a program that can also affect the objects in a direct interaction with interface, but without explicit instruction from the user. The interface agents reads input that the user presents to the interface, and it can make changes to the objects the user sees on the screen, though not necessarily one-to-one with user actions Autonomous: An autonomous agent is an agent program that operates in parallel with the user. Autonomous agents run at every time. Here in our system Reactive: An agent can be viewed as an entity that is able to perceive its environment and to act according to its own decisions. Smart: Agents are a new form of software agent that interfaces with other agents forming an artificial intelligence system. The acronym "SMART" stands for "System for Managing Agents in Real Time". Intelligent: A computer program that is given a specific task that can be done without user assistance. Mobile: A mobile agent is a composition of computer software and data which is able to transfer information from one computer to another autonomously and continue its execution on the destination computer. 3.2 How agent will work in smart home? Agents in Microsoft Kinect works collaboratively to have a successful controller free gaming interface. With the power of such agents, the Xbox 360 will be able to perform tasks very easily like by just waving your hand from the Xbox 360 dashboard, you'll be able to sign into Xbox Live. Wave again and you'll call up a Kinect-specific interface, and then just say (for example) "Xbox, Zune" to browse your music, or "Xbox, Facebook" to see what your Facebook pals are up to. You'll also be able to pick Netflix movies to stream or scan to a specific scene by waving your hand across the interface. There are various agents working in the Microsoft Kinect Xbox 360 which are described very clearly below:
Fig.3 Microsoft Kinect gets its hardware dissected Gesture interface (Interface and Reactive Agent) are graphical user interfaces which accept input in a form of hand gestures, or mouse gestures sketched with a computer mouse or a stylus.
Motion tracking interfaces (Interface and Reactive Agent) monitor the user's body motions and translate them into commands. Voice user interfaces (Interface and Reactive Agents) which accept input and provide output by generating voice prompts. The user input is made by pressing keys or buttons, or responding verbally to the interface. When Kinect is equipped with this interface, user will be provided a separate menu system on the Xbox 360. It's simpler, ditching tabs for simpler iconography. To load Netflix, you can wave to hit the Netflix button, or you just say "Netflix", pause by telling it to pause or reaching in the air to grab a virtual pause button. Motion Sensor (Reactive Agent ): According to T3, the motion sensor in the Kinect is made up of two elements: a projector that fires out a laser to scan your living room and an infrared VGA camera that tracks that laser and records what it bounces off of. The Kinect then interprets the pixels from that VGA camera image and run it through a software filter, which figures out what objects the camera is seeing are humanoid and which aren’t. Smart Camera (Reactive and Intelligent Agent): Kinect is an Xbox 360 peripheral. Think of it as a webcam on steroids that plugs into the console's USB port that uses IR to track you in 3D space. So Kinect Camera is smarter than your average webcam. First, it has excellent sight thanks to its secret trick: spotlighting the room with invisible infrared light. The camera sees you wonderfully thanks to this infrared. And, coupled with some advanced software that will run on the 360, it can track 48 point of your body in real-time for up to two players simultaneously. You can fit as many people in that camera as possible, and the computer will see all of them and can even recognize them as human shapes. But it can only run calculations on two people at a time, just because the processing power required to track all of the body's locations and movements is so great (Update: See below). During our testing with the device, a person moving in front of the camera was able to "steal focus," but the computer can also be told through gestures to keep focus on a certain person. The setup is able to determine depth because the camera is paired with a projector that emits invisible, infrared light in a complicated speckle pattern. The camera picks up how the pattern is distorted when it hits objects. At a distance of 7 feet, it can distinguish depth differences of less than an inch. That's enough to decipher gestures and hand movements. At a distance of 3 feet, the resolution is finer, and facial expressions can be distinguished Video chat via the Kinect camera will be another key Xbox Live feature, including the ability to watch videos with your video chat partners or conduct over-the-Xbox video calls with Windows Live users. Microphone and Microsoft Software (Reactive and Interface Agent): Kinect is equipped with a microphone so you can talk to the Xbox 360. Also, it doesn't just see you in IR; it can also film you in full RGB color, recognizing your face to automatically sign you in. And its tilt? Fully motorized to track you. Kinect also has a microphone that picks up the location of sounds, helping the system understand its
surroundings. Microsoft's software ties the components together. These two agent collaboratively work together to perform this task. Software Agents Software agents in use are: "Videokinect" Chat Software: Whether your friends are on Xbox Live or Windows Live Messenger, you can video chat with them. But you can also do some other neat stuff, like watch a video together. And as mentioned above, it can track you in real-time thanks to motion sensing and the motorized tilt. 4. CHALLENGES IN DESIGNING MICROSOFT KINECT?
It should be user friendly so they can easily use the system. It should be able to provide basic capabilities for negotiation, control, reactivity and adaptation. Design proactive and responsive interfaces. Applying basic principles of software and knowledge engineering to develop and deploy agent based systems. Already highly competitive market situations in the controller free gaming interface domain
By the contextual and current study researcher conclude that ubiquitous computing can be implemented by using the autonomous agents as its main components. Autonomous agents have characteristics that can be used to implement some of the desirable features of ubiquitous computing system. In these systems autonomous agents are mainly used to enhance the user’s interaction with the ubiquitous computing environment in which autonomous agents are the software components that represents users, devices, and services. In ubiquitous computing, intelligent agents and agent operations must be contextualised within a broader conceptual discursive space.
First of all I would like to thank almighty God whose blessings have inspired me throughout my research paper. Next I would like to thank my module lecturer Mr. Ankur Singla who has guided me throughout my research. It is his class room teaching which provided me with the knowledge of ubiquitous computing and its agents. I would also like to thank the director of our college Prof. Dr. R. K. Choudhary for providing an educational environment and infrastructure in the college. In the last but not the least I would also like to thank my dear friends who have continuously inspired me throughout the research work.
 http://www.worldscibooks.com/compsci/l004.html (Accessed on 10th August 2010, Time- 10:30 pm)
http://gizmodo.com/5563047/what-is-xbox-360-kinect (Accessed on 15th August 2010, Time- 9:30 pm) http://kotaku.com/5279531/microsoft-project-natal-can-support-multiple-players-see-fingers(Accessed on 20th August 2010, Time8:30 pm)
http://venturebeat.com/2009/06/01/microsoft-shows-off-motion-sensing-game-control-project-natal/(Accessed on 25th August 2010, Time- 7:30 pm)
http://www.tgdaily.com/games-and-entertainment-features/50273-will-microsofts-kinect-appeal-to-core-gamers(Accessed on 30th August 2010, Time- 10:30 am)
http://sandbox.xerox.com/ubicomp/ (Accessed on 10th sept 2010, Time- 2 pm)
    
http://www.byteblocks.com/post/2010/06/22/How-XBox-Kinect-Works.aspx(Accessed on 11th sept 2010, Time- 11:30 pm) http://gizmodo.com/5563047/what-is-xbox-360-kinect(Accessed on 12th sept 2010, Time- 12:30 pm) http://www.usatoday.com/tech/gaming/2010-06-14-vidgame14_ST_N.htm(Accessed on 15th August 2010, Time- 10:30 pm) http://www.geek.com/articles/games/thoughts-halo-reach-for-xbox-360-20100913/(Accessed on 10th August 2010, Time- 10:30 pm) http://gizmodo.com/5633272/where-is-object-scanning-and-controllers-in-xbox-360-kinect(Accessed on 10th August 2010, Time10:30 pm)
http://www.xbox.com/en-GB/news-features/news/Project-Natal-in-detail-050609.htm(Accessed on 14th sept 2010, Time- 10:30 pm)