A Structure from Manipulation for Text-Graphic Objects

Fred H. Lakin *

Stanford Artificial Intelligence Laboratory

Abstract Sections
The general purpose graphics systems of the future will need a Model of User on the Electric Blackboard (handPAM)
simple logic for visual objects - one structure underlying both text Selective Manipulation Leads to Structure
and graphics. As an experiment, perhaps the immediate handling of A Structure from Manipulation for Text-Graphic Objects
visual objects by the user can provide the starting point for ']'he Separation of Space & Tree
developing that structure. This paper describes the PAM graphics
Handling Space & Tree Structure
system, in which the structure of text-graphic objects arises directly
The 4 Levels ofhandPAM
out of manual manipulation. The needs of manual manipulation
determine the text-graphic pattern as the simplest organizing A Typical Blackboard Image
structure for images; PAM stands for PAttern Manipulating. The Ambiguities Resolved by Manipulative Grammar
PAM system is designed for the agile manipulation of text-graphic Relation ofhandPAM to writtenPAM
patterns - first manually, and then, later, programmatically. Manipulative Grammar is Pre-semantic
Starting from this strict 'front-in' viewpoint - where immediate Applications
manipulation (hand powered animation) was to be the primary Phenomenology and Graphic Interface Design
application - a 'manipulative grammar' was evolved to give the user Notes on Illustrations
a simple yet powerful handle on text-graphic images. This grammar Acknowledgements
turned out to be a generalization of LISP syntax from textual References
symbolic expressions to text-graphic forms, structuring such forms as
trees and then offering:
Model of User on the Electric Blackboard (handPAM)
l, spatial GnABBING of objects into attention
tree guided attention shifters like FlaST, nEST, NEXT, and UP
1' spatial & tree manipulations on any object in attention.
The resulting structures also offer surprising computational power
(in a manner direcdy analogous to the way the basic list structures
and functions of LISP give rise to the flexibility and power of a full text-graphic
blown LISP system, McCarthy and Takott [1]), leading finally to
computing with text-graphic forms. Consequendy, a semantic
function is added to supplement the basic manipulative grammar:
1, evaluation of object in attention, result displayed at the cursor.
Evaluation supports facilities like naming (and thus saving) visual
objects, programming, and creation of menus (patterns of evaluatable
function objects).
An experimental version of the PAM system has been
implemented in MACLISP at the Stanford Artificial Intelligence Lab.
Key words and phrases: text-graphic objects, front-in design, hand
powered animation, manipulative grammar, LISP, computing with \ ~ ~.:~°:#~ controls object of
text-graphic forms, phenomenology, interactive computer graphics,
man-machipe interface, graphics command language, graphics
programming language, visual linguistics.
CR Categories: 3.36, 3.41, 4.22, 8.2.

* now at Xerox Palo Alto Research Center, 3333 Coyote Hill Road,
Palo Alto, CA 94304.
Permission t o copy without fee all or part of this material is
g r a n t e d p r o v i d e d that the c o p i e s a r e n o t made or d i s t r i b u t e d
for direct commercial advantage, the ACM copyright notice and
the title of the publication and its date appear, and notice
is given that copying is by permission of the Association for
Computing Machinery. To copy otherwise, or to republish,
requires a fee and/or specific permission.
A Phenomenology of Text-Graphic Manipulation
The human usesthe hand controls to manipulate his object of attention on
the text-graphic display.
Electric Blackboard Implementation
The text-graphic handle monitors the movements oft.he keyset, keyboard
and mouse, and usesthem as commands to manipulate the objatn in bigpat.

©]980 ACH 0-8979]-02]-4/80/0700-0]00 $00.75

Figure 1

100
Selective Manipulation Leads to Structure
Given selective manipulations (drag, rotate, scale, erase) on atomic
visual objects, users will naturally and soon want a way o f
structuring for group manipulation. The tree structure of PAM
patterns is one of the simplest possible grouping devices which clears
up some of the ambiguities in text-graphic manipulation (for
example, as defined by Suthefland [2] and Futrelle [3] in a later
section). Before discussing the powers of PAM's manipulative
grammar, we must first see how it arises.
The user's initial experience of the PAM system is handPAM, a
manual manipulator for text-graphic images which functions as an
'Electric Blackboard'. handPAM is designed to support the
spontaneous, general purpose writing-and-drawing activity which is
often performed on a blackboard. The actual behavior of handPAM
is defined by its top-level function called hand/e Oust as the LISP
editor is defined by edit, and LISP itself by evaO.
Figure 1 shows a user in the handPAM environment. The
sketch depicts only the important features according to the
front-in or phenomenolgical approach, so it's actually a model.
In implementation the object of attention becomes the objatn,
a global variable which is in turrl a member of bigpat, the
largest pattern made up of everything on the display.
The human's purpose in this environment is to make the text and
graphics do what he wants, as ideas come to him. It is within the
organizing effect of this purpose that the Levels of handPAM
progressively unfold themselves (see figure 5).
At Level 1, simple objects (drawlines and textlines) are introduced,
along with spatial manipulations (DRAG,ROTATE,SCALEand ERASE)
which can be applied to any object selected. An object is selected
by spatially pointing it out, thus GRABBING it into attention
(whereupon it becomes the objatn, indicated by thickening); only
when an object is in attention can it be manipulated. And it is the
power of selective manipulations - within the driving purpose of
his desire to perform blackboard type text-graphic activity - which
naturally leads the user to want and acquire the pattern structuring
operations of Levels 2 and 3.
Manipulative grammar, then, is a way of dealing with the forms
and structures of visual objects and their customary arrangement in
simple and complex patterns (so far, paraphrasing Webster's
definition for verbal grammar) which is organized by the in-order.to
of spontaneous manipulation.
A consequence of the design goal of supporting blackboard type
activity is 'visual unobtrusiveness', meaning that no extra objects
appear on the display to distract the viewers from the actual text-
graphic image under manipulation. Instead of using menus or other
graphic devices added-on-just-for-control, handPAM relies on:
straightforward spatial manipulations; simple rules of tree structure
for applying them collectively to groups of objects; and finally a
supple control system connecting the human's hands to the space &
tree manipulations so that within a short time mouse movements and
keyset chording will be second nature.
handPAM is actually a generalization of Warren Teiteiman's LISP
editor to text-graphic patterns on a static display; Teitelman [7]
credits Peter Deutsch for the original idea of a structure editor.
Deutsch first implemented one at the University of California at
Berkeley in 1965.
The non-specific 'hand controls' of the phenomenological
description will be implemented as the mouse and keyset of English,
Engelbart and Berman [5], along with an alphanumeric keyboard.
The version of handPAM currently running has only a keyboard,
using typed commands to simulate mouse and keyseL
I01
A Structure from Manipulation for Text-Graphic Objects
A te×t.graphic object is either a line or a pattern.
A line is a drawline or a character or a textline.
A draw/ine is a line drawn through n o n e or m o r e locations.
A character is one or m o r e drawlines.
L T K ~
A textline is one or m o r e characters.
GEORGE
A pattern is a group o f none or more lines a n d / o r patterns.
El.
•
- -1 T ---~'
]EORGE
Figure 2
The first step in the creation of handPAM was the analysis of
text-graphic performances using more conventional media, including
blackboards, paper and pencil, and overhead projectors. In common
to all these media is the selective creation, moving and erasing of
text-graphic objects. Applying such manipulations selectively
requires some way of structuring complex objects and their parts;
trees are a simple and obvious way to structure visual objects,
e.g. , ,. ~," Complex objects with such a tree structure
O (3
(shown by the thin line network) are called text-graphic patterns.
Figure 2 delineates the basic structures for objects in
handPAM,
Note the three levels of atoms (atomic objects are all called lines
in PAM): drawlines, characters, and textlines. Although this may
seem illogical, it makes sense from a user's ordinary orientation, in
which a line of text and a drawn line are both indivisible atomic
objects for manipulation (later very important when GRABBtNG
objects into attention). For instance, the human in figure 1 has
constructed the pattern george as having seven atomic members,
one of which is the texthne GEORGE, another of which is the
drawline that is the leR ear, another of which is the drawline that is
the outline of the head, and so on. A rough metaphor expressing
this orientation is that there is as much information in each character
in a fine of text as carried by each location in a drawn line. Of
course, other manipulative orientations are possible in which one
would want to edit the locations in a drawline or the characters in a
textline, but these are viewed as secondary 'modes' which one enters
and exits from the basic structures pictured in figure 2.
Note also the ways in which patterns follow LISP exactly: patterns
can be empty, in which case they are represented by the ground-sign
character nitpat; otherwise a pattern is an ordered list of members,
each of which is either a line or a pattern (no recursive patterns are
shown in figure 2).
 The Separation of Space & Tree Handling Space & Tree Structure
And note finally the way in which patterns are displayed, which
can differ radically from the way LISP lists are represented for
human viewing. This difference is the 'separation of space & tree'
which PAM maintains but LISP does not (and can not because order
of members in a printed list must be denoted by spatial ordering).
This separation is not an issue in figure 2 because all the patterns
were carefully constructed to be ordered from left to righL The point
is that PAM patterns can be constructed so that members have other
spatial orderings than left-to-right. In PAM patterns, spatial
structure and tree structure are completely independent; the spatial
arrangement of objects in a pattern does not necessarily express or GEORGE "
imply any particular tree structure.
It is this complete separation of space & tree which requires the
use of additional thin black lines to denote the tree structure of
patterns; i.e. the thin tree networks which show the order of
members in a pattern and at the same time show if any members are tree
patterns themselves. Perhaps not so obviously, it is only the
complete separation of space & tree which permits handPAM to
support non-specific blackboard type text-graphic activity (such as
represented by figure 6, where no simple spatial parsing rule like
left-to-right will group objects in a way natural for human
manipulation).
To illustrate the freedom obtained by divorcing space & tree, a
parallel can be drawn between left-to-right ordering in PAM and Spatial Manipulations are performed by the right hand using the mouse:
alphabetical ordering in LISP. Imagine a LISP input reader which DRAWING a line with the cursor, locating a new line of TEXT,
always rearranged atoms in lists to alphabetical order - clearly this GRABBING the object nearest the cursor into attention; and then
would be useless for most purposes. Instead, atoms in lists can have DRAGGING, SCALING, ROTATINGor ERASING the object in attention.
any order, and then if we want alphabetical we write a sort function.
Likewise, spatial positions of PAM objects necessitate no particular Tree Manipulations are performed by the left hand on the keyset:
tree ordering, and then we can write functions which sort and/or Directing attention along the tree structure to the FIRST object in a
parse depending upon specific applications (for example pattern, or to the RESTof the objects, or UP one level to the pattern of
LeftToRightLayout, DirectionOrder, or the spatial parser for which the current object of attention is a member; changing tree
SIBTRAN in figure 6). structure by COLLECTING new patterns, or INSERTING objects into
existing paUerns or EXTRACTING objects from them.
Figure 3 shows the pattern george with two different tree
orderings and three different spatial arrangements.
Figure 4

a. c.
Figure 4 shows the manipulative grammar in action as the
user is ready to spontaneously modify both space & tree
structure of objects on the display.
The alphanumeric keyboard is not shown to emphasize the clean
separation of functions between the mouse and keyset. The mouse
is used for Level 1 operations (see figure 5), all of which have a
~EDF~E spatial component - DRAWING, locating TEXT, GRABBING, e ~ -- a n d
so it's fit and proper that they should all be operated from the
b. d.
spatial (analog) input device in handPAM. In a sense, the mouse is
Level 1. Tree manipulations, Levels 2 and 3, arc performed on the
keyset.
Manipulative grammar really consists of an objatn in a space &
t ~ED~gE" tree structure; it is not simply trees for visual objects, but rather the
dynamic interplay between a manipulable ob]atn and the use of
space & tree structure for attention shifting (i.e. changing the
EEDF~gE objatn) and object construction.

Figure 3 It is this dynami 9 interplay which characterizes the tree structures

in the handPAM environment. Many graphics systems employ trees
(i.e. Defanti[10]), but the structure of PAM patterns are experiential'
george b has right-to-leR ordering, while the other three have left- trees - the user actively grasps the structures as he uses them to.
to-right ordering. Trees have been shown in all cases, because drive attention around among the objects on the display.
without trees george a and george b both appear the same. Only the
tree structure of george b has been reversed; the spatial location o f
each o f his members is unchanged Therefore the reversal is only
revealed when the trees are shown, or when george b is laid out
spatially by a 'pretty positioner' (georged). george c and george d
have both been laid-out spatially from left to right by that function
- LeftToRightLayout - which uses tree structure to determine
spatial position in the same way that LISP printers always must.

102
The 4 Levels of handPAM commands) and structurally
knowledgeable (including about
structures not directly visible). If these
Line ~.~11" conclusions are correct, then manipulative
Manipulation ~. .~..OIZ~E j~tll- /:.
grammar as here defined may well be a
scarf ro'~,.f~
demonstrative language: hand motions
with the mouse and keyset are used to
determine and manipulate the (not
Simple discernible from passive observation) tree
Pattern structure of patterns.
Manipulation
A similar concept is Futrelle's notion
of deixis : "This is the act of referring to
an item in context without having to
name it. Pointing at an object is such a
-F -F + -;-
deictic reference," Futrelle [3]. This
Recursive certainly links up with Suthefland's ideas,
Pattern
Manipulation as deictic is defined as:
?,~tv~ u9 re~'t gi,'~ 41~t
1. in grammar, pointing out;
demonstrative: as, 'that' is a deictic
-I- °;- + + -F pronoun.
The manipulative grammar used in
handPAM is deictic in both space & tree;
that is, an object can be pointed out in
its spatial context by putting the cursor
on it and grabbing it, or it can be
pointed out in its tree context by guiding
Evaluation
of Names and attention to it along the tree structure o f
Function Calls SETNAME G G ZS NOW A NAME o r ~-~j~s~-~' the larger pattern of which it is a
t
-I- member (using FIRST, N E X T , B A C K , UP
GEORGE and REST).
ooooo0ooO0O
ooeoo t~d~,.o.tz object of ~t:e.,,C,.a~,,., ~eU..,t ~,,.k,t~ ~ ~ J o r More details on the workings of
handPAM - how the Levels
developmentally unfold themselves, etc.
- must be deferred to a future paper.
The purpose of the present paper is
merely to present a tree based
GGoRGE manipulative grammar and show how it
arises from and facilitates manual
e~_tL, o..tT,..bj~t ~ =t~,.-t/.u:,% reX*~ ,,ab,¢ ~'t o,rs~
manipulation.
Likewise, further explanation about the
Figure 5 evaluation of the objatn which can occur
in Level 4 (writtenPAM) will be covered
Figure 5 shows the 4 Levels ofhandPAM. in another paper, Lakin [17] (although a
brief summary of writtenPAM is
Level 1 provides creation of atomic objects, spatial GRABBING of provided in a later section).
objects into attention, and spatial manipulations. Anything which
can be put into attention - made the obiatn - can be However, the possibEity of evaluating text-graphic objects is
important to this paper. It means t_hat writtenPAM possesses the
manipulated. Levels 2 and 3 permit the manipulation of tree
processing power for doing linguistic analysis of text-graphic
structure, and allow the user to 'drive' attention around among the
objects on the display using tree guided attention shifters (FIRST, discourse (ie. the user interaction) in at least the same way that LISP
NEXT, UP, etc.). And in Level 4 evaluation is added to the operations facilitates such computation on textual interactions (Winograd's
that can be performed on the objatn; evaluation permits naming of SHRDLU [15], Martin's DIP [9], etc.). In PAM as in LISP,
an object and then later retrieving of an object by evaluating the processing power means providing atomic objects, ways of
visual entity that is its name. structuring them into complex objects, and equality tests for both
atomic and complex objects.
The point should be repeated that it is the oh]atn along with the
tree structure of PAM objects which together make up the A visual linguistics made possible by such processing could be
'manipulative grammar' that is at the heart of handPAM; tree useful in several ways. First, knowledgeable manipulation assistants
structure alone is some kind of grammar for visual objects, but could be built to increase the speed and dexterity of manual image
having an objatn climb around that tree on a static (non-scrolling) handling; creation/modification of text-graphics at blackboard speeds
interactive display is what gives the grammar its manipulative is a very challenging application and help will be needed. And then,
quality. once good agility has been achieved in the manual handPAM
medium, more complex machine understandings could be attempted
It is interesting to note the likeness between what is here called of discourse taking place in that medium.
manipulative grammar and what Sutherland calls demonstrative
language. In Sketchpad, demonstrative language seems to be ways
Of pointing at objects using the lightpen, both objects actually visible
and objects or parts topologically related to visible objects (for
instance, the end point of a line where only some middle segment of
the line can be seen) Sutherland [8]. A further characterization of
this class of techniques is that it is both non-textual (no typed

103
A Typical Blackboard Image rules of SIBTRAN, the spatial parser for SIBTRAN then proceeds in
three passes to finally return a complex pattern structured like figure
rr--V-I F I --if- . . . . . . . . . . . . "
6. Before parsing can begin, however, it is first necessary to order
all of the objects from the upper left by using the function
"~--~AP'PL~CATION5 DirectionOrder. Then TextGroup, the first pass of parser, collects
L • skNatr'e*
I ~,n~
stacks of textlines into patterns, and also puts underlined textlines
into patterns with the underline first. Enclose, the second pass, is
next; it handles enclosure forms such as Rectangles, HollowArrows
and Ellipses. Enclose collects the inside objects into a pattern with
t-- T - _ • .... ~_ _ -___~
,SENSOr-MOTOR BAS(1)~MAWTIC$
the enclosing graph as first (calling itself on each of the inside
objects as it collects them from up-left). And finally the third pass,
Connectors, constructs patterns in which a connector such as a
SingleArrow, DoubleArrow, HollowArrow or StraightLine is first
and its objects are rest (ordered in the direction of the connector if it
has one or from up-left in the case of DoubleArrows and
i Straightlines). Connectors then calls itself recursively on each
object of a parsed connector.
It was noted earlier that the needs of manipulating blackboard
figures led to the complete separation of space & tree in PAM
patterns. In support, it is claimed that groupings like those
diagrammed in figure 6 are 'natural' to a human who is changing
and moving the objects, and yet cannot be expressed by any simple,
fixed space-to-tree correspondence (hence the need for a parser).

Ambiguities Resolved by Manipulative Grammar

d
I Fourteen years ago Ivan Sutherland wrote an article in which he
outlined 10 unsolved problems in computer graphics [2]. One of
these he characterized as the Structure of Drawings Problem.
I + Sutherland's example is a graphic editing situation in which there are
five similar boxes, each with a dot in it (the situation portrayed at
the top of figure 7). Some pointing device is positioned as shown
Figure 6 and a DELErE command is given. "The correct computer response
depends upon the real underlying structure of the drawing." For
Figure 6 shows a more complex blackboard-type image which instance, the user could mean delete the dot in that box, delete the
might be constructed and manipulated in the handPAM whole box, or perhaps delete the group of two boxes of which that
environment. box is a member!
The graphic devices used in this example (hollow arrows, spirals, Figure 7 presents different structural representations for five
etc) are part of the vocabulary of SIBTRAN. SIBTRAN is a boxes with dots.
simplified version of a visual language invented by David Sibbet
[11], a graphic artist in San Francisco.
Sibbet uses his visual language in the course of drawing on large
sheets of paper to facilitate the problem defining and solving
processes of various groups. The kind of work Sibbet does can be
called 'group graphics', Lakin [12]. This service is based on the fact
that writing and drawing together often help a group think about
something. The term 'explicit group memory' was first coined by
Ball and Gilkey [13] to describe this phenomenon and document its Ambiguous reference

usefulness. But blackboards in seminar rooms and scnools have

been silent testimony to its efficacy for over a hundred years.
Lately, the uses of computer displays for explicit group memory
have begun to be observed in conjunction with computer
teleconferencing (Pferd, Peralta and Prendergast [14]; although there
//
limited to architectural floor plans). And perhaps the general use of
text-graphic display to facilitate cognition was first examined by
Doug Engelbart in his "Augmenting Human Intellect: A Conceptual
Framework" [4]. /
\
mi:
The spontaneous manipulation of images like figure 6 - in order Partial clarification
to facilitate the cognitive activity of a working group - has been the
driving application of the PAM system ever since its very inception
(Lakin [12]). Rigorous scientific appraisal of these media is still in
the future; the goal of the present paper is simply to describe one
underlying logic for the construction of such tools. [ r, F ~r,. ''it"
Finally, before leaving this figure, it should be mentioned (as an
example of linguistic processing of text-graphic objects) that a spatial
parser has been written (but not yet implemented) for SIBTRAN
forms. Spatial parsers take as input a simple pattern of drawlines
and textlines. They then use spatial information and the syntax of
the visual target language to reorganize the lines into a complex handPAM solution

pattern structured so as to facilitate further processing of the form.

Assuming that the input pattern is properly formed according to the Figure 7

104
 Sutherland goes on to say that " . . . we must have languages
powerful enough to explicitly represent the structure of pictures.
Moreover, the languages must represent the structure naturally
enough so that it doesn't get in our way. We must represent the FOOPLEDA~B
structure so users can see it. In [the middle object in figure 7] a
dotted line around the groupings shows the structure, but that would
not be appropriate in every case. We must learn how to represent
structure for human consumption. We must also build languages
which can construct structure."
handPAM offers a way of representing the structure of the boxes
with dots that would be appropriate in this and many other cases.
Yet this structure notation and the need for it arises out of a
different viewpoint than Sutherland's, as evidenced by one other
remark he makes: "In fact, we usually do not even think explicitly
about the underlying structure." Because the user initially
experiences the PAM system as a manual graphics editor providing
selective manipulations on visual objects, the question of structuring
visual objects in order to manipulate them is naturally with the user
from the very first time she wants to handle a group of atomic 5ETO
OUOTE
objects. Structure is not some programmatic issue that comes up
down the road a ways - it arises immediately out of the user's
desire to group and manipulate objects. The user will continually be ×Y
thinking explicitly about the structure of objects.
In handPAM, the user can directly manipulate this underlying
structure of each individual object just as she does its visual
properties, and she can make the structure visible by using the
system function ShowTree. The bottom object in figure 7 shows
how Snowrree would represent the structure indicated by
Sutherland for the group of five boxes. And that part which has
been singled out for manipulation - the obiatn in handPAM - is
indicated by being thickened (although blinking, colored halo or
other devices could be used).
Futrelle [3] raises two important ambiguities which manipulative
grammar seems to resolve. One is the relation of text and graphics:
"It is natural to want to design a 'language' in which the program
text and graphical objects are treated on an equal footing, a language Figure 8
in which the coding is done by typing text and by drawing pictures.
Such a fully integrated language we call an Intrinsically Graphicial Clearing up the relation between text and graphics also partly
Language or IGL. A direct attack on this problem fails because text solves another problem that Futrelle sees, namely the "... careful and
and pictures are of two different modalities so that the relations efficient disambiguation of deictic references .... If a visible object
between them are n o t readily perceptible to a human being." appears on the screen, pointing to it may be an ambiguous reference
By having the three levels of atoms - drawlines, characters and .... Some ambiguities which occur with spatial deictic reference
textlines -- handPAM establishes a readily discernible and useful alone are resolved in handPAM by the three levels of atoms and
relation between text and graphics. Text is just a way of thinking pattern structures. That is, when the cursor is on the crossbar in the
about graphics, with certain rigid rules for laying out combinations E in GEORGE in the pattern george, which object is being pointed
of 26 little pictures (and each of the 26 little pictures is really just a at? Or more concretely, which object will a user get when she does
pattern of drawlines). Experience with handPAM has shown these a GRAB? The answer to these questions is provided first by the
relations quite handy for manual manipulation, providing good convention that textlines are atomic to GRABBING, SO the only
agility as one composes text-graphic images (and sub-atomic editing remaining choice is whether just the atom GEORGE will become the
for drawlines, characters and textlines is provided). Objatn or whether it will be the whole pattern george (ears, head,
These basic structures also provide a means for programmatic name, etc). And that choice is made by the user when she sets the
manipulation of text-graphic images, thus leading to writtenPAM GrabMode, picking either GRABLINE or GRABTOP (the latter grabs
programs which both are text-graphic forms and do computing with the largest pattern of which the result of GRABUNE is a member).
text-graphic forms (see the membership test defining form in figure But finding a way to m/~p graphics onto text only solves some of
8; hence writtenPAM may be an IGL). Put another way, the the ambiguities of deictic reference. "If the programmer points to a
structures in manipulative grammar are a way around the impasse certain part of a picture it is not obvious which attribute, the color,
"...which results because pictorial information does not map onto line width, region, position, etc. is intended, and furthermore
code in any obvious way." And in fact such a mapping must be whether reference is being made to the generic attribute, the value(s)
found if one wants to deal with pictorial symbol systems like bound to it, or what."
Chinese characters, which was one of the defining applications for One way to think of this problem is as the manipulation of
handPAM (the last character example in figure 2). unobservable attributes. That is, in handPAM the location, rotation,
Figure 8 shows various objects which illustrate the relation scale and appearance (vectors in the drawlines) of objects are directly
between text and graphics in PAM. observable and manipulable. Tree structure, on the other hand, is
The top of figure 8 demonstrates the result of trying to show the not directly observable and yet the user is able to knowingly modify
trees for the three different kinds of atomic objects; in the middle is it using tree manipulation operations like INSERT, EXTRACT, and
the textline with its tree shown, possible only because flags have PATTERN COLLECT. He can do so because handPAM supplies him
been reset to permit disclosure of the 'sub-atomic' structures; and on with two ways of observing this 'unobservable' attribute. The first is
the bottom is an example of computing with text-graphic forms, the tree traversing attention shifters - F~RST, NEXT, UP, etc. -- which
presenting a graphic synonym for one definition of the membership offer an experiential way of discovering a pattern's structure by
test in LISP. climbing around it. And the second is the handPAM operation

105
SHOWTREE, which calls the system function ShowTree to draw the
tree of the objatn (or, for atomic objects, label what level atom it is;
see figure 8).
Another unobservable attribute for atomic handPAM objects is
value; the object (if any) that a line is bound to is discovered by
putting it into attention and performing the handPAM operation
EVALUATe. Values can be set by the operation NN~E.
The observation and manipulation of other unobservable attributes
has been handled on an ad hoc basis thanks to the flexibility of the
LISP implementation environment. For instance, the vectors in
objects can have a thickness, but the resolution o f the CRT is not
good enough to show it; so the LISP function call (Thicken 1.0)
thickens the objatn by one pixel on the paper output devices, and
(GetThickness objatn) returns the status o f this attribute.
The ShowTree solution to displaying pattern tree structure is an
example of a classic approach for viewing unobservable attributes.
This approach consists of finding a general graphic representation of
the attribute, and then having the system automatically generate and
display the proper specific representation for the status of a
particular object's attribute whenever the user asks. Sutherland first
used this approach for showing constraints in Sketchpad. Futrelle
uses a similar device he calls a 'Graphic Aid' for visible and not-so-
visible attributes of general classes he calls Objects. In both cases,
the additional wrinkle is offered that the user can then edit the
representation of the attribute and have the attribute itself change.
This wrinkle could be implemented quite easily for object attributes
in handPAM; it would be particularly nice to edit a shown tree and
have the pattern's tree structure change correspondingly. (The point
should be made that in PAM, unlike in Sutherland's and Futrelle's
systems, there are no 'masters' or 'classes' who have 'instances'; in
PAId all objects are unique and individual.)
Figure 9 shows three examples of editable representations for
not-so-observable attributes: Sutherland's constraint
representation in Sketohpad; Futrelle's Graphic Aids; and a
proposed editor for showntrees in handPAM.
CENT
L -~
Constraint representation Graphic Aid ShownTree Editor
Figure 9
In the case of Sutherland's diamond-headed arrow on the left, the
user can manipulate the constraints for this defintion picture through
their displayed forms, including the attacher-point constraints (shown
with code T), non-rotation (code E) and constant size (code F). The
Graphic Aid for Futrelle's Object of class Bow in the middle allows
the user to change the Width, Length or Center (location) of the
Bow by interacting with the textual representation of these qualities.
And on the right, the user is preparing to insert the HollowArrow
(in attention) into the pattern by indicating the place on the pattern's
showntree where the new member is to be inserted.
Relation of handPAM to w r i t t e n P A M
Structuring and Manipulating Text-Graphic Images
PAM
/ \
user entre~ =~. handPAM writtenPAM
Handling Text-Graphic O b j e c t s Computing with Text-Graphic
Forms
PAttern Manipulation includes both manual handling and
computational processing of text-graphic images.
106
handPAMs' 4 Levels are designed to unfold themselves
developmentally according to the logic of text-graphic manipulation.
The 4th Level of handPAM is actually writtenPAM, where text-
graphic patterns are evaluated and the result is displayed on the
screen. By building up his operational experience of pattern
structure and manipulation, handPAM easily leads the user to a
basic understanding of the evaluation function which defines
writtenPAM. In the phenomenolgical view, interface design is seen
as a continuum running from the sensory-motor to the cognitive.
writtenPAM is a generalization of LISP from computing with
textual symbolic expressions to computing with text-graphic forms.
writtenPAM allows people to talk directly about text-graphic objects
and manipulations on them, as for example in setting the value of
constructing patterns
PATCON$ QUOTE
taking them apa~ FIRST~ ' ~'~'
= GEORGE
GEOR&E
and making equality tests I - - ~ - - - - - ~ - r - - ~ ' ~'
E.QUAL? = FALSE
In short, writtenPAM's functions and predicates operate on visual
objects and return them as values. As a matter of fact, this language
may be 'intrinsically graphic' (Futrelle [3]) and only later textual, but
we still speak of writing programs, so we call it writtenPAM for
convenience.
Manipulative Grammar is Pre-semantic
Note the size of the class of possible semantics that can fit onto
the Levels of handPAM in figure 5. The only syntax so far is trees
- and code-objects manipulated by the user could easily have other
syntaxes; and the only semantic commitments are the functions NAME
and EVALUATE -- and the actual rules used by the evaluation
function could vary widely from system to system.
For example, a PASCAL version of handPAM might look the
same to the user in the first 3 Levels (i.e. manual manipulation of
text-graphic objects). But the 4th Level - using text-graphic objects
as instructions to a computer - would have the functions COMPILE
and RUN instead of EVALUATE. Thus an executable object would be
constructed, put into attention, and the COMPILE command given.
The result of compilation would be a text pattern returned at the
cursor consisting of the name of the runable file and/or a list of
error messages. If the compilation was successful, then the name of
the runable file could be put into attention and the RUN command
given, with the result of the execution returned at the cursor.
And an APL version of handPAM would be simply a matter of
having different rules for the interpreter - perhaps APLEVALUATE.
And, actually, there is no reason why writtenPAM, PASCAL and
APL need be confined to separate systems; a user-set system variable
could determine which rule system was invoked when the evaluation
chord on the keyset was struck.
Applications
handPAM is not an application, it is a context for applications -
a general purpose environment for manipulating text and graphics.
And then specialized application programs can be written which live
within this environment; for example, data plotting, program
dynamics diagrammers, circuit design aids, text editors, graphic
programming notations, etc. One advantage of specialized
applications taking place within a general purpose environment is
that all visual objects returned by specialized functions are built as if
they had been constructed by hand and are therefore subject to
further modification by hand. Thus the definable, repeatable parts
of applications can be done automatically, and the rest left to in-
session user intervention. The converse of this feature is that the
programmer can first manually simulate any new specific application,
and then incrementally write programs which take over easily
definable sub-functions in that particular domain.
Another advantage of a general graphic context is that a function
built for one purpose (like LeffToRightLayout, for spacing
characters in textlines) can usually be applied outside its original
application and thus becomes one more general utility function (see
figure 3 for LeftToRightLayout applied to george).
To use an analogy, specialized applications are like trees and
handPAM is a nursery, providing the common ground wherein
different trees can be grown and work together in an
interdependently functioning forest..
At the moment, five different groups of specialized application
functions are being assembled (Lakin [18]):
1) OrawPlan, a function which draws a diagram of the plans
created by Sacerdoti's planning program [6].
2) FinderMap, a group of functions which dynamically display
finder activity as nanoKRL programs execute, Martin [9]).
3) DataPIot, a technical illustration package for experimental
results.
4) FlowCharter, being simply the functions box, connect and
ArrowEdit for boxing objects, connecting them by arrows, and
editing arrows when the automatic routing is unsatisfactory.
5) OUTLINE, a visual synonym system for LISP code using
graphic 'nameshapes' instead of textual function names (see
figure 8).
However, now that the above has been said, the truth is that
handPAM is also itself an application: the Electric Blackboard. A
'smart blackboard' could be useful any time writing and drawing
together would help a group to think more effectively. And because
all the specific application functions live within the general device,
whenever a discussion lead into some specific topic like circuit
design or data analysis, the same medium could still support the
group's thinking wherever it went. Finally, if necessary, new
functions could even be written on the spot to add some new feature
so the discussion could continue more easily.
Phenomenolgy and Graphic Interface Design
Careful description of something without concern for its reality
yields a phenomenology - an exposure and delineation of the logic
of the phenomenon. A phenomenon is literally a 'that which
appears', an appearing to a consciousness; phenomenology provides
a way to give a precise account of such an appearing (Natanson
[16]). Taking manipulation of text-graphic images as a phenomenon,
this viewpoint allows us to describe it while for a time disregarding
any hardware or software 'behind the scenes' which 'really' causes
the existence and behavior of the images. What we have left is the
human, the text-graphics, and his manipulation of t h e m , . , which is
the starting point for this paper.
Designing a graphics system in the method used for handPAM -
starting with data structures and operations which arise from text-
graphic manipulation itself - is thus a phenomenological approach.
The design grows out of the logic of the phenomenon of
manipulation as experienced from the human's point of view.
Another way to think of this approach is as front-in design -
107
starting with the human and the graphics, and then (slowly!) moving
toward the computer. It cannot be emphasized too strongly how
important this phenomenological principle was in creating the PAM
system: the project was born out of it and returned to it many
times for renewal and guidance.
Notes on the Illustrations
Figures 2, 3, 7 and 8 were constructed in the experimental
handPAM environment at SAIL. Figure 3 was printed on a Xerox
Graphics Printer, while figures 2, 7 and 8 were printed double size
on a Varian States and then photographically reduced. The
remaining illustrations are pen and ink sketches. Annotative text in
figures was done on the Xerox Graphics Printer and then pasted up.
Acknowledgements
Special thanks to the people without whom this project could not
have happened. To John McCarthy, for inventing LISP and giving
me access to the computer facilities at SAIL. To Jonathan King,
Allan Hayes and Ben Laws for long discussions on the pro-computer
PAM system. To Jim Collatz for actually learning handPAM before
it was implemented. To Hans Moravec and Rodney Brooks for
providing the basic graphics tools at SAIL: and to Dick Gabriel and
Paul Martin for showing me the finer points of MACLISP.
References
[1] McCarthy, John and Talcott, Carolyn, LISP Programming and
Proving, Class notes CS 206, Computer Science Dept., Stanford
University, 1979.
[2] Suthefland, Ivan E., "Computer Graphics: Ten Unsolved
Problems", Datamation, pages 22-27, May 1966.
[3] Futrelle, R.P. and Barta, G., "Towards the Design of an
Intrinsically Graphical Language", SIGGRAPH 78 Proceedings,
pages 28-32, Aug 1978.
[4] Engelbart, D.C., "Augmenting Human Intellect: A Conceptual
Framework," SRi International, Menlo Park, California, Oct
1962.
[5] English, W.K., Engelbart, D.C., and Berman, M.L., "Display-
Selection Techniques for Text Manipulation", IEEE Trans. o n
Human Factors in Electronics, Vol. HFE-8, No. 1, March 1967.
[6] Sacerdoti, Earl, "Planning in a Hierarchy of Abstraction Spaces",
Adv. Papers 3rd IntL Conf. on Artificial Intelligence, Stanford
University, August 1973.
[7] Teitelman, Warren, InterLIgP Re~rence Manual, Xerox Pain Alto
Research Center, 1978.
[8] Sutherland" Ivan E., "Sketchpad: A Man-Machine Graphical
Communication System", Proceedings-Spring Joint Computer
Conference, pages 329-346, 1963.
[9] Martin, Paul A., DIP: A Program to Understand Diplomacy
Dialogs, PhD thesis, Stanford University, forthcoming.
[10] Defanti, Tom, '"Eae Digital Component of the Circle Graphics
Habitat", Proceedings National Computer Conference, 1976.
[11] Sibbet, David, "Introduction to Group Graphics", T h e
Corospondent, C O R n Foundation North. Cal. Public Affairs
Quarterly, Summer 1976.
[12] Lakin, Fred, Design for a Working Group Display, manuscript
1974.
[13] Ball, Geoffrey H. and Gilkey, James Y., "Facilitation and
Explicit Group Memory -- Their Application in Education",
SRI International IR&D No. 183531-409, Menlo Park,
California, Dec 1971.
[14] Pferd, William, Peralta, L.A. and Predergast, F.X., "Interactive
Graphics Teleconferencing", IEEE Computer, Nov. 1979.
[15] Winograd, Terry, Understanding Natural Language, Academic
Press, 1972.
[16] Natanson, Maurice, Editor, Essays in Phenomenology, Martinus
Nijhoff, The Hague, 1966.
[17] Lakin, Fred, "Computing with Text-Graphic Forms", submitted
to the LISP Conference at Stanford University, August 1980.
[18] Lakin, Fred, "Indigenous Graphics for LISP", submitted to the
LISP Conference at Stanford University, August 1980.