Perception and Perspective in Robotics

Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group Overview
Goal To build robots that can interact with novel objects and participate in novel activities Challenge Machine perception can be robust for a specific domain such as face detection, but unlike human perception it is not currently adaptable in the face of change (new objects, changed circumstances) Approach Integrate conventional machine perception and machine learning with strategies for opportunistic development Active perception (sensorimotor toil) Interpersonal influences (theft) This work is implemented on a humanoid robot (Cog, see right). The robot uses the structure of familiar activities to learn about novel elements within those activities, and tracks known elements to learn about the unfamiliar activities in which they are used. Perspective
familiar activities (tasks, games, )
segmentation

Toil Example Active Segmentation

Theft Example Search Activity

Robot observes a human searching for objects, and learns to make a connection between the named target of the search and the object successfully found. The robot has no predefined vocabulary or object set. Human says Find says Find Toma Toma (shows cube) (sees cube) No No (shows car) (sees car) No No (shows bottle) (sees bottle) Yes! Yes (shows cube) (sees cube) Say Say Cube (shows bottle) (sees bottle) Say Say Toma Robot

1 2 3

Object boundaries are not always easy to detect visually, so robot Cog sweeps its arm through ambiguous areas This can cause object motion, which makes boundaries much easier to find Then robot can learn to recognize and segment object without further contact

This is a good basis for adaptable object perception:

active probing affordance exploitation
(rolling)

use constraint of familiar activity to discover unfamiliar entity used within it

reveal the structure of unfamiliar activities by tracking familiar entities into and through them

edge catalog object detection, recognition

manipulator detection
(robot, human)

Perception
familiar entities (objects, actors, properties, )

This work is funded by DARPA under contract number DABT 63-00-C-10102, and by the Nippon Telegraph and Telephone Corporation under the NTT/MIT collaboration agreement

Goal To learn how human-level perception is possible, by trying to build it Challenge Machine perception can be robust for a specific domain, but is not adaptable like human perception Approach Integrate conventional machine perception and machine learning with strategies for opportunistic development Active perception (sensorimotor toil) Interpersonal influences (theft) Development If a robot is engaged in a known activity there may be sufficient constraint to identify novel elements within that activity. Similarly, if known elements take part in some unfamiliar activity, tracking those can help characterize that activity. Potentially, perceptual development is an open-ended loop of such discoveries.

Learning a sorting activity Human shows robot where a collection of disparate objects should go, based on some common criterion (color). Robot demonstrates understanding through verbal descriptions, nods towards target locations.

Kismet What is done on Kismet

Novel Perspective leads to Novel Perception

Learning a search activity Human shows robot examples of search activity by speaking. Robot demonstrates understanding by linking name and object. Learning through a search activity Blah blah

Cog What is done on Cog

Perception and Perspective in Robotics

Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group
An Example Active Segmentation Overview Goal To learn how human-level perception is possible, by trying to build it Challenge Machine perception can be robust for a specific domain, but is not adaptable like human perception Approach Integrate conventional machine perception and machine learning with strategies for opportunistic development Active perception (sensorimotor toil) Interpersonal influences (theft) Experimental Platform Expressive active vision head Kismet and upper-torso humanoid robot Cog Open-ended Development If the robot is engaged in a known activity there may be sufficient constraint to identify novel elements within that activity. Similarly, if known elements take part in some unfamiliar activity, tracking those can help characterize that activity. Potentially, perceptual development is an open-ended loop of such discoveries.

1 2 3

Object boundaries are not always easy to detect visually, so robot Cog sweeps its arm through ambiguous areas This can cause object motion, which makes boundaries much easier to find Then robot can learn to recognize and segment object without further contact

familiar activities

use constraint of familiar activity to discover unfamiliar entity used within it

reveal the structure of unfamiliar activities by tracking familiar entities into and through them

familiar entities (objects, actors, properties, )

Kismet What is done on Kismet

Gives opportunity for much development

active probing segmentation affordance exploitation
(rolling)

Sorting activity Human shows robot where a collection of disparate objects should go, based on some common criterion (color). Robot demonstrates understanding through verbal descriptions, nods towards target locations. Search activity Human shows robot examples of search activity by speaking. Robot demonstrates understanding through verbal descriptions, nods towards target locations.

Cog What is done on Cog

edge catalog object detection, recognition

manipulator detection
(robot, human)

Perception and Perspective in Robotics

Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group
familiar activities

Active Perception To foo foo foo Active Segmentation Solve classic problem

use constraint of familiar activity to discover unfamiliar entity used within it

reveal the structure of Objectunfamiliar activities by always boundaries are not tracking familiar entities easy to detect visually (e.g. yellow car oninto and through them yellow table)

Solution: robot Cog sweeps through ambiguous area familiar entities (objects, actors, properties, ) helps Resulting object motion segmentation Robot can learn to recognize and segment object without further Opportunities abound and contact cascade EgoMap short term memory Robot can perform find the toma of objects and their locations so out of sight is not out of mind style tasks Observes search activity Then uses structure of search activity to learn new properties (object names) Searching and sorting

Active Perception Point 1, 2, 3 Motivation Training examples are currently a necessary condition for achieving robust machine perception. Acquiring those examples is properly the role of perception itself. But a human is typically needed to collect those examples. Active Perception To foo foo foo Active Segmentation Solve classic problem

Sorting task Human shows robot where a collection of disparate objects should go, based on some common criterion (color). Robot demonstrates understanding through verbal descriptions, nods towards target locations. Search task Human shows robot examples of search activity by speaking Robot demonstrates understanding through verbal descriptions, nods towards target locations.

Perception and Perspective in Robotics

Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group

Goal To understand perception by trying to build it Approach Extend machine opportuistic deve The grist: The mill: Examples

perception

to

include

Active perception Interpersonal influences Opportunistic development

Object boundaries are not always easy to detect visually (e.g. yellow car on yellow table) Solution: robot Cog sweeps through ambiguous area Resulting object motion helps segmentation Robot can learn to recognize and segment object without further contact

Opportunities abound and cascade Robot can perform find the toma style tasks Observes search activity Then uses structure of search activity to learn new properties (object names) Searching and sorting

familiar activities

use constraint of familiar activity to discover unfamiliar entity used within it

reveal the structure of unfamiliar activities by tracking familiar entities into and through them

familiar entities (objects, actors, properties, )

EgoMap short term memory of objects and their locations so out of sight is not out of mind

Opportunism Standard approach to machine perception is to develop algorithms which, when provided with sufficient training data, can learn to perform some classification or regression task. Can move one step back and develop algorithms which, given physical opportunities, acquire the training data. Need to design system behavior side-by-side with the perceptual code. Opportunistic Development Suppose there is a property P which can normally not be perceived. But there exists a situation S where it can be. Then the robot can try to get into situation S, and observe P, and relate it to other perceptual variables that are observable

Perception and Perspective in Robotics

Paul Fitzpatrick MIT Computer Science and Artificial Intelligence Laboratory Humanoid Robotics Group

Head (7 DOFs)

Training Data
Left arm (6 DOFs)

Task Learning Mechanis

Right arm (6 DOFs)

Instructor Task Modeling

Sequencing Mod

1 2 3
Eyes (3 DOFs)

Torso (3 DOFs)

Stand (0 DOFs)

Task Grounding

State Grounding

Facial (15 DOFs)

poking object segmentation

Neck (3 DOFs)

Perceptual System Demonstrated Task

affordance exploitation
(rolling)

Perceptual Netw

edge catalog
object detection

familiar activities
manipulator detection
(robot, human) (recognition, localization, contact-free segmentation)

use constraint of familiar activity to discover unfamiliar entity used within it

reveal the structure of unfamiliar activities by tracking familiar entities into and through them

familiar entities (objects, actors, properties, )

EgoMap short term memory of objects and their locations so out of sight is not out of mind

(Speech)

Training Data
Instructor

Task Learning Mechanism

Sequencing Model Task Modeling

Understanding perception by trying to build it Machine perception is very fallible. Robots (and humans) need not just particular perceptual competences, but the tools to forge those competences out of raw physical experiences. Three important tools for extending a robots perceptual abilities whose importance have been recognized individually are related and brought together. The first is active perception, where the robot employs motor action to reliably perceive properties of the world that it otherwise could not. The second is development, where experience is used to improve perception. The third is interpersonal influences, where the robots percepts are guided by those of an external agent. Examples are given for object segmentation, object recognition, and orientation sensitivity; initial work on action understanding is also described.
Demonstrated Task Task Grounding State Grounding

Perceptual System

Perceptual Network

 Object boundaries are not always easy to detect visually Solution: Cog sweeps through ambiguous area Resulting object motion helps segmentation Robot can learn to recognize and segment object without further contact

camera image

implicated edges found and grouped

response for each object