Pattie Maes + Pranav Mistry: Meet the SixthSense interaction

I've been intrigued by this question of whether we could evolve or develop a sixth sense -- a sense that would give us sea less access and easy access to eta-infor ation or infor ation that ay exist so ewhere that ay be relevant to help us a!e the right decision about whatever it is that we're co ing across" #nd so e of you ay argue$ well$ don't today's cell phones do that already% &ut I would say no" 'hen you eet so eone here at ()* -- and this is the top networ!ing place$ of course$ of the year -- you don't sha!e so ebody's hand and then say$ +,an you hold on for a o ent while I ta!e out y phone and -oogle you%+ .r when you go to the super ar!et and you're standing there in that huge aisle of different types of toilet papers$ you don't ta!e out your cell phone$ and open a browser$ and go to a website to try to decide which of these different toilet papers is the ost ecologically responsible purchase to a!e" So we don't really have easy access to all this relevant infor ation that can /ust help us a!e opti al decisions about what to do next and what actions to ta!e" #nd so y research group at the Media 0ab has been developing a series of inventions to give us access to this infor ation in a sort of easy way$ without requiring that the user changes any of their behavior" #nd I' here to unveil our latest effort$ and ost successful effort so far$which is still very uch a wor! in process" I' actually wearing the device right now and we've sort of cobbled it together with co ponents that are off the shelf -- and that$ by the way$ only cost 123 dollars at this point in ti e" I' wearing a ca era$ /ust a si ple webca $ a portable$ battery-powered pro/ection syste with a little irror" (hese co ponents co unicate to y cell phone in y poc!et which acts as the co unication and co putation device" #nd in the video here we see y student Pranav Mistry$ who's really the genius who's been i ple enting and designing this whole syste " #nd we see how this syste lets hi wal! up to any surface and start using his hands to interact with the infor ation that is pro/ected in front of hi " (he syste trac!s the four significant fingers" In this case$ he's wearing si ple ar!er caps that you ay recogni4e" &ut if you want a ore stylish version you could also paint your nails in different colors" #nd the ca era basically trac!s these four fingers and recogni4es any gestures that he's a!ing so he can /ust go to$ for exa ple$ a ap of 0ong &each$ 4oo in and out$ etc" (he syste also recogni4es iconic gestures such as the +ta!e a picture+ gesture$ and then ta!es a picture of whatever is in front of you" #nd when he then wal!s bac! to the Media 0ab$ he can /ust go up to any wall and pro/ect all the pictures that he's ta!en$ sort through the and organi4e the $ and re-si4e the $ etc"$ again using all natural gestures" So$ so e of you ost li!ely were here two years ago and saw the de o by 5eff 6an or so e of you ay thin!$ +'ell$ doesn't this loo! li!e the Microsoft Surface (able%+ #nd yes$ you also interact using natural gestures$ both hands$ etc" &ut the difference here is that you can use any surface$ you can wal! to up to any surface$ including your hand if nothing else is available and interact with this pro/ected data" (he device is co pletely portable$ and can be """ 7#pplause8 So one i portant difference is that it's totally obile" #nother even ore i portant difference is that in ass production this would not cost ore to orrow than today's cell phones and would actually not sort of be a bigger pac!aging -- could loo! a lot ore stylishthan this version that I' wearing around y nec!" &ut other than letting so e of you live out your fantasy of loo!ing as cool as (o ,ruise in +Minority 9eport$+ the reason why we're really excited about this device is that it really can act as one of these sixth-sense devicesthat gives you relevant infor ation about whatever is in front of you" So we see Pranav here going into the super ar!et and he's shopping for so e paper towels" #nd$ as he pic!s up a product the syste can recogni4e the product that he's pic!ing up$ using either i age recognition or ar!er technology$ and give hi the green light or an orange light" 6e can as! for additional infor ation" So this particular choice here is a particularly good choice$ given his personal criteria" So e of you ay want the toilet paper with the ost bleach in itrather than the ost ecologically-responsible choice"

70aughter8 If he pic!s up a boo! in the boo!store$ he can get an # a4on rating -- it gets pro/ected right on the cover of the boo!" (his is 5uan's boo!$ our previous spea!er$ which gets a great rating$ by the way$ at # a4on" #nd so$ Pranav turns the page of the boo! and can then see additional infor ation about the boo! -- reader co ents$ aybe sort of infor ation by his favorite critic$ etc" If he turns to a particular page he finds an annotation by aybe an expert of a friend of ours that gives hi a little bit of additional infor ation about whatever is on that particular page" 9eading the newspaper -- it never has to be outdated" 70aughter8 :ou can get video annotations of the event that you're reading about :ou can get the latest sports scores etc" (his is a ore controversial one" 70aughter8 #s you interact with so eone at ()*$ aybe you can see a word cloud of the tags$ the words that are associated with that person in their blog and personal web pages" In this case$ the student is interested in ca eras$ etc" .n your way to the airport$ if you pic! up your boarding pass$ it can tell you that your flight is delayed$ that the gate has changed$ etc" #nd$ if you need to !now what the current ti e is it's as si ple as drawing a watch -70aughter8 7#pplause8 on your ar " So that's where we're at so far in developing this sixth sense that would give us sea less access to all this relevant infor ation about the things that we ay co e across" My student Pranav$ who's really$ li!e I said$ the genius behind this" 7#pplause8 7Standing ovation8 6e does deserve a lot of applause because I don't thin! he's slept uch in the last three onths$ actually" #nd his girlfriend is probably not very happy about hi either" &ut it's not perfect yet$ it's very uch a wor! in progress" #nd who !nows$ aybe in another ;3 yearswe'll be here with the ulti ate sixth sense brain i plant" (han! you" 7#pplause8