0018-9162/07/$25.00 © 2007 IEEE
34
Computer
Published by the IEEE Computer Society
COVER FEATURE
circle differently based on context—a circle can be a pinjoint only if it’s drawn over two bodies that are alreadyoverlapping (in this case the car body and the wheel).We’ve built sketch-understanding systems for a vari-ety of domains, including the simple physics sketchershown in Figure 1, Unified Modeling Language diagrams(Figure 2), analog circuits (Figure 3), and chemical struc-ture sketches
2
(Figure 4).
TERMINOLOGY AND FOCUS
Sketches differ from diagrams. By “diagrams” wemean the more formal, at times draftsman-like figuresthat CAD systems produce, while sketches are thehand-drawn informal figures people create on paper,whiteboards, napkins, and, more recently, tabletcomputers. There is a significant body of work onunderstanding diagrams such as the InternationalConferences on Document Analysis and Recognition,but the task of understanding a sketch is significantlydifferent.Research on sketch understanding also differs fromthe sizable body of work on handwriting understand-ing. Sketch-understanding work proceeds largely froman attempt to recognize the shape of the objects drawn,using the same notion of shape that people use. Hand-writing recognition, on the other hand, doesn’t attemptto recognize each letter by its shape, and has success-fully used machine-learning techniques that derive dis-tinguishing features that might or might not correspondto what people attend to.
Sketches are hand-drawn informal figures often created as a way of thinking about or work-ing through a problem.Sketch-understanding systems let users interact with computers bydrawing naturally,offering a freedom not available with traditional CAD systems.
Randall Davis
Massachusetts Institute of Technology
S
ketching is ubiquitous: We draw as a way of thinking, solving problems, and communicat-ing in a wide variety of fields, for both design(such as sketches of conceptual designs) andanalysis (such as sketches drawn to help puzzlethrough problems in physics or electronic circuits).Unfortunately in today’s technology, sketches aredead—they’re either graphite on slices of dead trees, or,if captured on a PDA or tablet computer, simply pixelsof digitized ink. The Sketch Understanding Group atMIT has been working toward a kind of “magicpaper”—that is, a surface that’s as natural and easy todraw on as paper, yet that understands what you draw.What does it mean for the paper to “understand”?One example, in Figure 1, shows some of our earliestwork. We use Assist (A Shrewd Sketch Interpretationand Simulation Tool) to sketch simple 2D physicaldevices, then watch them behave.
1
Assist understandsthe raw sketch shown in Figure 1a in the sense thatit interprets the ink the same way we do (Figure 1b),that is, as an inclined plane with a wheeled cart. AsFigure 1c shows, it hands this interpretation to a physicssimulator, which animates the device, giving the user theexperience of drawing on intelligent paper.One detail helps illustrate the sense in which the sys-tem understands the sketch in a manner similar to ahuman observer. The wheels (blue circles) are attached tothe car’s body with pin joints (pink circles), yetthe user draws both the wheels and pin joints with thesame geometric shape—a circle. The system interprets a
Magic Paper: Sketch-Understanding Research
Leave a Comment