COGNITIVE PSYCHOLOGY 4, 399-424 ( 1973)

Empirical Tests of a Theory of Human Acquisition

of Concepts for Sequential Patterns1

KENNETH KOTOVSKY’ AND HERBERT A. SIMON

Carnegie-Melloft Vnioersity

The paper examines a body of empirical data on S’s performing the

Thurstone Letter Series Completion task, in order to test the theory pro-
posed by the authors in 1963 for explaining behavior on this task. The data
confirm the theory in its main aspects, while indicating the need for some
minor extensions and modifications. In particular the data show that subjects
first discover the periodicity of the letter series, then construct a descrip-
tion of the pattern, and finally use the pattern description to make an ex-
trapolation. Most of the pattern descriptions used by Ss fall within the
pattern language defined in the earIier paper.

In a previous paper (Simon & Kotovsky, 1963), the authors proposed

an information processing theory of human acquisition of concepts for
sequential patterns -specifically, patterns of the sorts used in the Thur-
stone Letter Series Completion Test. The theory was embodied in a
computer program for performing the task, consisting of two parts: a
pattern generator and a sequence generator. The pattern generator takes
a patterned sequence of letters as input and abstracts from it a pattern
description, “concept,” or “rule” of the sequence; the sequence generator
takes the pattern description as input and uses it to extrapolate the
letter sequence.?
Some empirical evidence was presented for accepting the computer
program as a valid description of the processes used by human Ss in per-
forming the same task. The evidence, however, was only aggregative,
showing mainly that there was a substantial correlation between the
difficulty of the several problems for the human Ss (as measured by the
number who failed to solve them) and the difficulty of the same prob-
‘This study was supported by Public Health Service Research Grant MH-07722
from the National Institute of Mental Health, Department of Health, Education and
Welfare.
* Requests for reprints should be sent to H. A. Simon; Department of Psychology,
Carnegie-Mellon University, Pittsburgh, Pennsylvania 15213.
(Accepted September 19, 1972)
‘Additional data and analysis from this experiment can be found in Kotovsky
(1970).
399
Copyright @ I973 by Academic Press, Inc.
All rights of reproduction in any form reserved.
400 XOTOVSKY AND SIMON

lems for the programs (as defined by the number of variants of the
program that failed to solve them), Additional evidence is needed before
we can be satisfied that the theory gives even an approximate explica-
tion of the human processes.
The purpose of the present paper is to present and examine a larger
body of empirical evidence, from the behavior of 14 Ss during the prob-
lem solving process; the order in which they looked at letters in a
sequence, the hypotheses they formed about probable solutions, their
methods for recognizing periodicity in the sequence, the manner in
which they extrapolated it once they had discovered the concept, the
types of errors they made, and so on. On the basis of this evidence, we
can test the detailed hypotheses that are implicit in the organization and
behavior of the computer program.
Since the publication of our previous paper, our theory has been
examined and extended by several other investigators, including Pivar
and Finkelstein ( 1964), WiIIiams ( 1969 f , and KIahr and Wallace ( 1970).
In the following discussion, we will draw upon their investigations as
well as our own data. (See also, ,Restle, 1970; Simon & Sumner, 1968;
Gregg, 1967.)
HYPOTHESES
The basic components of the theory can be summarized in a set of
hypotheses about the characte~stics of the human behavior in this task.
Each hypothesis describes some important characteristic of the computer
program and its variants.

Processes
These hypotheses predict the manner in which Ss proceed through the
problem situation.
AL S first discovers the periodicity in the sequence, then induces the
pattern description (the rule) using this periodicity. (Example:
abmcdm . . . has the period 3-abm, cdm, . . . .)
A2. When extrapolating a sequence, S first initializes the rule (marks
his place on certain lists), then uses the relations of same (I, for
“identical”) next ( N) and buc?cwa7d next (BN), or predecessor,
to extrapolate.3
s We will later discuss a difference between the program and Ss behavior on this
point. The program executes its extrapolation by initiahzing at the bc&Gng of the
letter sequence as presented; Ss usually initialize at some point near the end of the
letter sequence, and extrapolate from there. Thus, the program, given abmcdm . . . ,
would initialize the alphabet at “a,” and produce the extrapolation, abmcdmefmg . . . .
The human S would more often initialize at “c,” say, and produce only cdmefmg . L. .
 ~EQUENTL~LPATTERN &QUISIT~~N 401

A3. S experiences difficulty when he must keep his place on more than
one list in extrapolating. (Information for place-keeping is as-
sumed to be held in short-term memory along with other dynamic
information during extrapolation.)

Mechanisms
These hypotheses are assertions about the mechanisms or abilities that
Ss have available (stored in long-term memory) for solving these kinds
of problems.

Bl. S can recognize the relations of “same,” “next,” and “backward

next” (or “predecessor”) on the alphabet, and breaks in sequences
of such relations.
B2. S can induce a new list (“alphabet”) from a given sequence and
apply the N relation to extrapolate with it. (Example: from
atbataatb . . . induce the list or alphabet b-a, and extrapolate-
b . . a..a.., etc.) Such alphabetic lists are treated circularly, as
in the example just given.
B3. S solves problems by generating and fixating a pattern descrip-
tion based on the relations (I, N, BN) named above.
B4. S can extrapolate a sequence by interpreting the pattern
description.
B5. The language proposed by Simon and Kotovsky ( 1963), based on
the above relations and processes gives an empirically valid ex-
plication of S’s internal representations of patterns (pattern de-
scriptions) and the behavior attendant on their acquisition and
use.

Individual Differences
While our main concern will be with the communalities among Ss,
we will also comment on individual differences, in order to indicate the
relative independence among the various submechanisms.

Cl. The computer program can be varied by rearranging or subtract-

ing small sets of instructions to simulate these individual
differences.

Source of Human Error

Dl. S may be unfamiliar with an alphabet used in the sequence; may
have an inadequate repertoire of relations; may detect, and be
confused by, spurious relations in the sequence (example: the
underlined I relation in urtustuti); or may be unable to or-
 402 KOTOVSKY AND SIMON

ganize and record the relations he detects in a coherent pattern

description.

METHOD
Fourteen Ss were given 15 Thurstone-type letter series completion
problems. The Ss’ thinking-aloud verbalizations were recorded, and the
order in which they observed the letters in the sequence was auto-
matically recorded. The Ss only observed one letter of the sequence at a
time.

Subjects
The Ss were 14 Carnegie Institute of Technology (now Carnegie-
Mellon University) male and female freshmen and sophomore students
who were fulfilling an Introductory Psychology course requirement by
participating in psychology experiments.

Materials and Apparatus

The test problems were presented on a display device consisting of a
board containing 14 vertical sliding wooden slats, each displaying a single
letter of each of the 15 test sequences. The next slat after the last letter
displayed a large dot that signaled the end of the sequence for that prob-
lem. A set of horizontal black pieces of heavy cardboard could be ar-
ranged to cover the entire surface of the display device with the exception
of a narrow horizontal zone located just below the information for the
current problem.
S viewed a letter of the sequence by pressing down on a slat, thus
pushing the letter on that slat into the visible zone. Pressing the slat
activated a switch connected to a keypunch, recording which slat was
moved. Releasing the slat returned it to its original position, thus re-
moving the letter from view.
The problems were of the type used by Thurstone (1941) except that
S was required to extrapolate four letters instead of one. (The problems
are shown in Table 1. )

Procedure
S was seated at a table containing the display device and pencils. A
microphone was placed about his neck and he then read a page of
standard thinking-aloud instructions. Next, he was given a series of prac-
tice problems and if he did not talk much while solving them, was en-
couraged to verbalize more. During the performance of the practice
problems, his questions about the procedures or the problems were an-
swered. At the end he was told the correct answers to the practice prob-
 SEQUENTIAL PA’ITERS ACQUISITIOX 403

lems and asked if he had any questions. S then read the instructions for
use of the display device, after which the operation of the device was
demonstrated to him. The answer and information sheet was then
given to S.
He was told that while he should work through the problems as
rapidly as hc conveniently could, hc was to “work for accuracy and not
speed in performing the problems, since speed is not an important factor
in this experiment.” He was then told to begin the first problem. At the
conclusion of each problem, E a~~nounccd “OK,” to provide a clear indi-
cation on the tape of when the problem ended. The time for solution is
the time between the first punch (first letter looked at) and the “OK”
by E.

Results
A brief samhle of the. data obtained from the card punch records is
presented in Fig. 1. The tape-recorded verbalizations were transcribed,
but analysis had to involve the original tapes as well as the typewritten
versions in order to correlate S’s verbalizations with the particular letter
 Subject l-Problem 13
Punching Sequence Verb& Protocol
WXAXYBYZCZADAB

WXAXYB Ah . . . my anticipation got fouled up.

WXAXYBYZCZADAB WXAX . . YB . . YZ . . AD . . AB. I’ll go back over
WXAXYB these again . . . . the same sequence . . . .
WXAXYBYZCZADAB WXY . . BYZ . CZA DAB
WXAXYBYZCZADA WX . . AXY . . BYZ . . CZA . . DAB. This works its
Z way down through the alphabet. The first
Y B Y Z C Z A 1) A B part confused me because I didn’t know
C Z A 1) A B where it came from . . BYZ . . CZA . . CZA . . DAB
A

Subject 8-Problem 12
Punching Sequence Verbal Protocol
NPAOQAPRAQSA

NPAOQAPRAQSA There seem to be a number of

A’s. Let’s go back and see if we can find

then the third letter of this thing would be

’ f ’ : ; ” i ” ; ‘1 ~~~~~~vI~~~~~~:.~~~~~i~~~~~g~~QS.So

. . . . . .

FIG. 1. Illustrative punching sequences and verbal protocols.

 SEQUENTIAL PATTEHK ACQUISITION 405

he was observing. The tape was also used to identify voice pitch changes,
rhythmic punching, and voice cadences in naming letters-an important
source of information about the processes used by S-and to obtain so-
lution times.
In summary, the data we obtained were: what S said, his correct or
incorrect extrapolations, the order in which he looked at the letters, the
temporal patterns in his naming of letters, and the time it took him to
solve each problem.
The data were organized into six categories: measures of problem dif-
ficulty, the feature or features of each sequence first noticed by S, de-
scriptions of the concepts attained by S, the initialization of the concepts
prior to extrapolation, the method of extrapolation, and S’s errors and
sources of problem difficulty.

Dificulty
Numerous alternative measures of problem difficulty were computed
from the data: median and mean times and numbers of runs through
the sequence for all Ss, for Ss solving the problem, and for Ss not solving
it, and median and mean numbers of punches. The intercorelations
among these were so high (the lowest rank-order corelation was .81, and
most of the corelations were over .9) that they can all be represented
adequately by mean time for all S’s (Table 1) . Table 1 also shows the
number of Ss, from among our 14, solving each problem; the numbers
from the earlier groups of 12 and 67 Ss, respectively, reported in Simon
and Kotovsky (1963); the mean times for the earlier group of 12 Ss and
the list of problems solved and left unsolved by Variant C of the com-
puter program.
Table 2 gives the corelation coefficients (rank-order corelations, and
in one case the point biserial) among the difficulty measures of Table 1.
There are high corelations between the times required by the Ss in the
two previous studies, run under rather different experimental conditions.
The rather lower corelations between the items correct in the present
and previous studies is probably due mainly to the fact that the range
of the data was very small. A difference of four in the number of Ss
missing a problem in the present study meant a difference between first
and tenth rank in difficulty. Similarly, in the group of 12 Ss a difference
of four in the number of subjects obtaining a correct solution could de-
termine whether a problem ranked seventh or fourteenth in difficulty.
We shall see later that the specific conditions under which the Ss
worked affected the difficulty of solving particular problems. The I4 col-
lege Ss of the present study viewed the letters of each sequence singly,
as we have seen. The 67 high school Ss of the one earlier study were
 40s KOTOVSKY AND SIMON

TABLE 2
Intercorrelations of Difficulty Measuresa

(1) (2) (3) (4) (5) (‘3)

Mean time (1) -
(group of 14 Ss)
Number correct (2) .62
(group of 14 Ss)
Number correct (3) .59 .34*
(group of 12 Ss)
Number correct .78
(4) -.48 .87
(group of 67 Ss)
Mean time (5) .88 .45 .75 .87
(group of 12 Ss)
Program C (6) .68 16* .58 .62 .78
(point biserial corelation)

a* - Not significant at .05 level; underline - not significant at .Ol level.

tested as a group, with all the letters and problems visible at once; the
12 SS of the other previous study were tested individually, with
all letters visible, and with the importance of speed emphasized in their
instructions.
Table 2 also shows high corelations among the difficulty measures of
all three studies (except the number-correct measure of the present
study), on the one hand, and the difficulties experienced with the differ-
ent problems by Variant C of the computer program, on the other.
Among the 14 Ss in the present study, the mean number of problems
missed (out of 15) was 3.9, and the range was 0 to 8.

First Feature Noticed

Table 3 summarizes which features of each sequence were noted first
by Ss. Features may be noticed after a smaller or larger part of the se-
quence has been examined. For example, on Pl, S14 noticed the periodic
repetition of the letter C (an (I) feature) after he had looked only at the
letters CDC. By contrast, on P5, S2 noticed the broken sequence (defg/ ),
an N/ feature, on his second run through the entire sequence.
The evidence that S has noticed a feature may be his explicit verbal-
ization of it, or it may be the beginning of rhythmic punching after a
pause. Each noticing act attributed to an S was assigned a confidence
rating (from 1-low-to-&high), based on the quality of the evidence
available. Explicit mention of the feature was assigned a rating of 5;
close rhythmic punching based on the feature, a rating of 4. In the fol-
lowing analysis, only inferences with confidence ratings of 3 (“Mod-
crate”), 4, and 5 (“high”) are included. In 89 of the total of 210 possible
 SEQUENTIAL PATTERN ACQUISITION 407

instances (14 Ss x 15 problems) we were able to determine with con-

fidence rating 3 or greater what features S had noticed first (see Table 3).
Let us see how consistent the data of Table 3 are with the hypotheses
we stated earlier.

TABLE 3
Priorities in Noticing Features”

Number of Ss
Problem Feat’ures present noticing first

cdcdcd (1) 2
N/ -
aaabbbcccdd 3
2 -
atbataatbat (1) 4
W) 1
abmcdmefmghm (1) 4
1
(2 -
5 defgefghfghi (1) 2
7
&( -
6 qxapxbqxa (1) 8
09 1
7 aduacuaeuabuaf (1) 7
(N -
WV -
8 mabmbcmcdm (1) 3
N/ 1
!I urtustuttu (1) 7
(N) -
10 abyabxabwab (1) 4
-
(fi& 2
11 rscdstdetuef (1) -
N/ 3
12 npaoqapraqsa (1) 11
1
(r; -
13 wxaxybyxczadab (1) 3
3
i% 2
14 jkqrklrslmst (1) -
N/ 3
15 pononmnmlmlk (1) 1
BN/ 5
(BN) -

a /-Broken sequence; ( )-periodic recurrence of relation.

408 KOTOVSKY AND SIMON

“Same” relation (I). Each of the 14 Ss noticed in one or more problems

the relation (I), supporting the hypothesis that Ss can recognize the pe-
riodic recurrence of the same letter through a sequence.
Periodic “Nsext” relations (N) and (BN). Only 7 Ss gave evidence of
noticing the relation N when it occurred periodically through a sequence
(Hypothesis BZ), and only 2, the periodic occurrence of BN (Hypothesis
B3). This does not mean, of course, that the remaining Ss were not able
to detect these relations, but only that on no occasion was there evidence
that these were the first relations detected.
These observations lead us to consider whether there was a consistent
priority in noticing order,
Noticing order. The priorities can be pretty well summarized by two
“ceteris paribus” generalizations: ( 1) the I relation is noticed before the
N relations and (2) relations involving broken sequences-I/, N/, and
BN/-are noticed before periodic relations (N), (BN).
Let us consider features in the order: I/, (I), N/, BN/, (N), N*/ (double
next), (BN). In the one problem where I/ occurs, it is the sole feature
that is noticed first (three instances ). In the 14 problems where (I) oc-
curs, it is noticed first 56 times out of 86. Of the remaining 30 instances
where another feature is noticed first, N/ is involved 18 times, BN/ five
times, (N) four times, N2/ once, and (BN) twice. In the 8 problems where
N/ occurs, it is the feature noticed first in 18 instances, yielding to (I)
in 18. BN/ is noticed first five times, in one problem, and gives prece-
dence to (I) once. (N) is noticed first only four times in seven problems,
and yields to other features 59 times. N2/ precedes (I) (in one problem)
only once in 12 instances; and (BN), in three problems, is noticed first
only twice out of 18 instances. Thus, only when the two rules given
above are in conflict (when both (I) and N/ or BN/ are present) is there
much ambiguity as to which feature will be noticed first.
Four problems, of two special types, account for 18 of the 23 instances
where N/ or BN/ take precedence over (I). Two of these, P5 and P15,
begin with a broken sequence (defg/ and ponf, respectively). The
method of presenting the problems undoubtedly contributes to this re-
sult. All Ss (even though they were not required to do so) began almost
all problems by punching the slats in order from left to right.
This interpretation is supported when we compare the ranks of these
problems in their difficulty for the present Ss with their difficulty for the
two previous groups of Ss. It will be recalled that in the previous experi-
ments, all the letters of the sequence were visible to Ss at all times.
Hence, attention was not directed to the initial sequence in P5 and P15
as strongly as it is by the arrangement of the present study, and these
problems should have been relatively more difficult for the Ss in the
 SEQUENTIAL PATTEHK ACQUISITIOK 409

earlier studies. P5 tied for rank 6 in number of solvers in the present

study; in the two previous studies, it ranked fifteenth and ninth. P15
ranked tenth in the present study; in one of the previous studies, it tied
for eleventh rank, in the other it ranked fourteenth (see Table 1). Thus,
the altered arrangements of the present study made these problems
easier.
Pll and P14, two of the other problems in which N/ took precedence
over (I), also share a common characteristic: they consist of two “inter-
twined” sequences, each of period 2. These also proved easier, on the
average, for the present Ss than for the previous groups (see Table 1).
In none of these four problems, P5, P15, Pll, or P14, does a single
letter recur periodically (I) through the entire sequence. However, there
is such a persistent recurrence in the five problems, Pl, P4, P8, PlO and
P12, where (I) has precedence over N/. Hence, most of the inconsistency
in ordering disappears if we make a distinction between an (I) relation
that holds only for symbols in two successive periods and an (I) rela-
tion that persists through the sequence. The distinction would be more
persuasive, of course, if it were not introduced a posteriori.
Nevertheless, the same distinction arises in the processes of the com-
puter programs. A process that first detected the periodicity of the se-
quence, then looked for (I) relations in corresponding positions in succes-
sive periods and checked for consistency throughout the sequence, would
notice the (I) relation in problems Pl, P4, P8, PlO and P12, but not in P5,
Pll, P14, or P15. It will be noted that program Variant C (see Ap-
pendix l), which behaved precisely in this way, succeeded in solving
Pl, P4, P8, and PlO, but not P5, Pll, P14, or P15. (It failed, however,
on P12). We may conclude, that our theory gives us a good under-
standing of the order in which the Ss noticed features of these sequences.
In addition, the above data support hypothesis Bl.

Additional Features
Several features of the sequences were noticed and mentioned by the
SS that were not incorporated in our theory or programs. For example,
at least eight different Ss noticed in one or more problems, that some
items of the sequence were drawn from one end or another of the alpha-
bet. If we represent by B, M, and E letters belonging to the beginnhg,
middle, and end of the alphabet, respectively, then the sequence of Prob-
lem 4 might be abstracted as: BBMBBMBBMBBM, and the sequence of
P6 as MEBMEBMEB.
Another type of relation not hypothesized in the original model is
based on counting. Thus, P12 involves a “double next” relation (labeled
above N’), which can be derived simply by applying N twice. Similarly,
 410 KOTOVSKY AND SIMON

the pattern in P2 is perhaps most easily characterized as “groups of 3,”

an iteration of the I relation. Counting abilities have been included in
other models of sequential concept attainment (Pivar & Finkelstein,
1964; Gregg, 1967; Restle, 1970), and the theory would clearly be im-
proved by incorporating them. This involves no fundamental change in
the structure of the program.
Some Ss went so far as to change parts of the sequences to counting
sequences (substituting numbers for the letters and then employing the
N relation on the alphabet of integers).

Periodicity
After noticing some significant feature or features, most Ss find a
periodicity in the sequence that they then use in constructing a pattern
description. Where they do not find such a period or where they find
the wrong one they almost invariably fail to solve the problem. Explicit
verbalizations indicating the discovery of a periodicity before going on
to form a concept are frequent, occurring in 125 of the 210 problem
protocols (60%). This evidence supports hypothesis Al that “Ss first dis-
cover the periodicity in the sequence and then induce the rule using the
discovered periodicity.” The hypothesis is also supported by the eye
movement studies of Williams ( 1969).
The Ss’ behavior departs in one respect from the model. Periodicity
is determined by noticing I and N relations. In the computer program,
information about relations that are noticed at this stage is not retained,
but is regenerated during the second stage, when the pattern description
is being built up, Ss clearly retain much or all of this information, and
use it while building the pattern description. Thus, the current program
separates the two phases of problem-solving activity-detection of pe-
riod;city and pattern description-more sharply than do the Ss.

Concepts: (Pattern Descriptions)

The principal concepts attained by Ss are described in Table 4 in a
language similar to that used by Simon and Kotovsky (1963), but with
the proviso’that additional elements necessary or useful for the adequate
description of the concepts are introduced as needed. The chunks held in
short-term memory are designated by M,,M,,M,,M,, etc. The N relation
(finding the item next to the current contents of chunk M,) is designated
N ( M, ) ; repetition of the same letter, I, is designated by the name of the
letter; and positions in the period of the sequence are represented by
operations in the pattern description. Using this notation facilitates the
comparison of the present data with the behavior of the program.
 SEQUENTIAL PATTERN ACQUISITION 411

In coding the concepts, however, we did not force them into the Simon
and Kotovsky scheme, but introduced new elements and mechanisms as
dictated by the data, thus permitting an empirical test of how well the
original theory fit the new evidence.
Overall fit of data to model. The data on the concepts generated by
the Ss are summarized in Table 4. In the column headed “Program Con-
cept,” concepts that are identical to those proposed by Simon and Kotov-
sky ( 1963, Table 2, p. 538) are designated by + . For 12 of the 15 prob-
lems, the pattern description most frequently generated by Ss (including
one tie) is identical to that proposed in the earlier paper.
Some 177 concepts are described at a moderate (24) or high (153)
level of certainty (out of a possible total of 210). Of these, 103, or 58%,
were in complete detailed agreement with those proposed by Simon and
Kotovsky ( 1963)) while 74 were different from those proposed in 1963.
Of the latter, 16 were minor variants of those proposed. The remaining
33 concepts which failed to reach a high enough certainty level for in-
clusion in Table 4 do not noticeably differ, to the extent that partial
information is available, from those included in the table.
The general qualitative agreement between the pattern descriptions
attained by the S’s of this experiment and the pattern descriptions pro-
posed by the 1963 model supports hypothesis B3, that Ss possess a
pattern generator that solves patterns by generating and fixating a pat-
tern description associating the relations I, N, BN, with the several posi-
tions within the period of the sequence. This agreement, together with
the fact that Ss were able to use those concepts to extrapolate the se-
quences correctly in 80 of 97 cases, confirms hypothesis B4 that Ss possess
a sequence generator that can interpret such pattern descriptions.
Finally, the fact that most of the concepts (177 out of 210) were de-
scribable at moderate or high certainty levels in the pattern language,
supports hypothesis B5 that the language (in a somewhat expanded
form) proposed by Simon and Kotovsky gives an empirically valid expli-
cation of S’s internal representations of patterns and the behavior attend-
ant on their acquisition and use. Only one of the concepts (S2, P12) was
not easily describable in terms of the expanded language),
Concept 1 on P3, and Concept 2 on P6 confirm, for at least some Ss,
hypothesis B2 that Ss can induce a new alphabet from a given sequence
and apply the N relation to extrapolate it.
The circular next relation. Both model and most subjects treated al-
phabets in a circular fashion. That is to say, on the alphabet: a&, the N
operator applied to a yields b, and applied to .h yields a. The regular
English alphabet is also usually treated circularly, the N of; z being a.
Some Ss encountered difficulties with circular alphabets.
 TABLE 4
Concepts

No. of Ss

Exact Concept Program

Problem concept Variants no. Concept concept” Initialization*

1 9 1 [C, Ul +
2 7 1 1 [3M,, NO%)1 + M, = A, D 8
4 1 2 [MI, M,, MI, N(W,)l + 341 = c, D 4
3 2 1 [A, T, MI, NC.%)1 + M, = B on A, B 9
2 2 2 IA, T, B, A, T, Al 5
1 3 [A, A, T, B, A, Tl *
4 13 1 [MI, NOW, MI, NM), Ml + M, = G, H, E, A
k3
5 8 1 [N(M,), M, = M,, MI, N(M,), + M2 = F, D, E
MI, N(M,), MI, NOWWI B
4 Other
g
6 7 2 1 [Q, X, -4, P, X, Bl +
2 1 2 [MI, NWL), X, Mz, N(MdI + M, = Q on P, Q;
M2= AonB,A
7 5 1 1 [A, Mk, N(Mk), ul Mk = M, = D, F
(MI, N(MdDL, BN(Md)
I I 2 [A, Mt + 1 -+ M,, Mk, N(Mk), N(Md, ul M, = D, C
Mk = (Mt, NUW, MI)
(Mt, BNDW, MI)
Other
 8 11 [MI, NW), MI, Ml + Ml = A, C
9 11 (U, MI, N(W), I-1 + M, = It, S, T
10 13 [A, B, MI, BNUWI M, = W, Y
11 8 WL, NM), MI, Mz, + Ml = T, S, R, E
NCWW M, = E, C, U
2 Other
12 6 1 WI, NWI), Ma N(K), Al + MI = N, P, Q
Mn = P, Q, It, S
4 2 [Ml = M,, M,, N(M,), Mz = MI, nh = Q
NW), MI, Al
Other
13 5 [MI, N(Md, MI, Mt, NW)1 + M, = W, A; Mz = A, D
1 Other
14 10 [Ml, NM), MI, MB, + M, = J, L, T, S
NUW WI M, = Q, S, M, L
2 Other
15 4 1 1 [M,, BN(M,)M,, BN(Mz), M, = P Mz = 0
Ma, BN(&)l MS = N
1 3 2 [Ml, BN(M& M, = MI, Ml,
BN(M,), Ml, Ml = %I
3 Other

a Indicates agreement with the rule proposed by Simon and Kotovsky, 1963.
* Example: M, = A, D; M, = E, T reads: some subjects set memory cell M, at letter A on the Alphabet, some set it at D; somesubjects
set memory cell M2 at E and some set it at T when beginning their ext,rapolation. Information in t.his column is occasionally less certain
than the other information on the Table.
414 KOTOVSKY AND SJMON

The backward alphabet. The first change in the model that is neces-
sary to fit the data is to eliminate the backward alphabet as a separate
entity to which the NEXT operator applies. The behavior of even those
SS who could work backward through the alphabet is more accurately
described by encoding the process by a backward next (BN), or prede-
cessor, operator that is applied to the ordinary alphabet, The possible
need for this alternative method of characterizing backward movement
in the alphabet was mentioned by Simon and Kotovsky (1963, footnote
3). The BN operator appears mainly in P7, PlO, and P15. With this
change in the model, 13 more Ss on PI0 and two more on P15 would be
brought into agreement with it, reducing the number of discrepant cases
from 74 to 59, and increasing the number of positive agreements from
103 to 118 (67%).
The BN operator is not used by all Ss, and even where used, is usually
not used with the same facility as the N operator. In fact, some Ss only
perform “quasi-BN” operations. They pick a point in the alphabet some-
what ahead of the letter they want to perform a BN upon; then moving
down the alphabet, using N, they remember the immediately preceding
letter as well as the current letter; so that when they reach the target let-
ter, they also retain in memory the one before it, i.e., the BN of the target
letter.
Hierarchic patterns. A second change in the original model is necessary
to describe more accurately the organization of the pattern descriptions
used by many Ss on P7, Pll, and P14. A means must be provided for
describing the hierarchic organization of these Ss’ pattern descriptions.
In the pair of similar problems, Pll and P14, the second concept in each
case involves a memory chunk we have labeled Mk, which points to a
position on a two-item list, One item on this list names the subpattern
(Ml, N(M2), M2). Th us, the pattern description for these problems
(rscdstde . . . . . and jkqrklrs . . . . . ) involves a higher level switching be-
tween the two subpatterns-( M,N( Mk))-and the production of dif-
ferent alphabetic sequences by each subpattern-( Ml, N( Ml), Ml)
and (M2, N(M2), M2). Th e introduction of such hierarchic elements
in the descriptions of concepts constituted the only difference between
eight pattern descriptions in this study and those predicted by Simon
and Kotovsky ( 1963).
Counting procedures. In addition to the two changes in the language
for describing pattern descriptions discussed above, a third change was
introduced to describe the pattern descriptions some Ss generated on P7
as well as on P2, P3, and Pl5. On these problems they used counting pro-
cedures of various sorts, Thus in problem two the most frequent rule
was (3M1, N(M1)) w h ic h seemed more appropriate for those seven SS
 SEQUENTIAL PATTERN ACQUISITION 415

than the related (Ml, Ml, Ml, N( Ml) ) used by four Ss and predicted
by Simon and Kotovsky. Other counting procedures involve keeping
tallies in memory (designated M, and M,).
Extensions of the model: summary. The use of a tally combined with
hierarchic pattern descriptions in P7 accounted for the differences be-
tween the concepts of four Ss and those predicted by Simon and Kotov-
sky. These together with the seven concepts using the counting pro-
cedure in P2, the eight concepts using hierarchic pattern descriptions,
and the 15 concepts using BN instead of N on the backward alphabet
account for 34 of the 74 concepts identified in the current study that
differ in any respect from those predicted by Simon and Kotovsky. Thus
the introduction of these three changes substantially improves the pre-
dictive capacity of the model without a great proliferation of new
mechanisms.
Short-term memory. A few of the concepts used by Ss require more
than two chunks, or place-keepers, to be held in short-term memory. We
will have more to say below about the burden this places on short-term
memory.
A different point concerning memory allocation that is only implicit
in the 1963 paper is that a recurring letter (the I relation determining M
in abmcdmefm, for example) is assumed to be held in long-term memory,
hence not to add a chunk to the short-term memory load. Short-term
memory is used only for those portions of the pattern description that
involve N and BN operators associated with continually changing posi-
tions on alphabetical lists. In the computer model also, application of the
I relation involves a simple set of computer operations compared to the
relatively involved operations for applying N or BN. A related distinc-
tion was made by Gregg ( 1967), who differentiated between the direct
and indirect production of items from memory (operations M and
D( M,) ) in the meta-language he used for his sequential patterns.
Whether the M,< chunks postulated for hierarchic patterns impose a
short-term memory load is not easy to determine. The Ss tend to use
auxiliary procedures that have the effect of reducing the load of this in-
formation on short-term memory. Thus rhythmical shifts in voice pitch,
or separate and independent extrapolation of the two sequences (even
though the S has previously achieved a unified pattern description) re-
duce the simultaneous demands on short-term memory. Further experi-
ments will be needed to confirm this conjecture, although in the present
experiment Ss using hierarchic procedures made fewer errors on prob-
lems involving two (P7) or three (Pll and P14) chunks in short-term
memory than did the other Ss. We will look later at the exact process
they used for extrapolating.
416 KOTOVSKY AND SIMON

Initialization
Once S has attained a pattern description from the sequence, he then
uses that pattern description to extrapolate the sequence. Occasionally
this temporal order is not followed strictly. Instead S obtains part of a
pattern description-say, for one position of the period-and extrapolates
it; then obtains another part and extrapolates it; then obtains another
part and extrapolates it.
Generally, in order to use a pattern description to produce an extrap-
olation, S first has to initialize-that is, set “pointers” in short-term mem-
ory to specific letters in alphabets associated with the pattern. Table 5
describes the 147 (of a possible 210) initializations about which informa-
tion was obtained at a moderate (17) or high (130) level of certainty.
The table indicates whether S initialized his memory cells at the be-
ginning (B) of the presented sequence (thus reproducing the sequence
as well as extrapolating it), at the end (E) of the sequence (thus pro-
ducing only the extrapolation), or at some point in the middle ( M) re-
gion of the sequence. Again, the initializations not included on the table
(not discernible at a 3, 4, or 5 certainty level) do not, at rough inspec-
tion, appear different in this respect from those that do appear.
Ss tend to use B or M initializations on harder problems, and E initial-
izations on easier problems. A majority of Ss used E or in one case E

TABLE 5
InitializationU

Problem B M E Other

Pl cdcdcd 4
P2 aaabbbcccdd 1 1 5
P3 atbataatbat 3 4
P4 abmcdmefmghm 1 1 7 1
P5 defgefghfghi 2 1 7
P6 qxapxbqxa 3 4 2
P7 aduacuaeuabuaf 4 1 4 3
P8 mabmbcmcdm 3 2 5
P9 urtustuttu 7 3 1
PI0 abyabxabwab 3 7
PI1 rscdstdetuef 3 7 2
P12 npaoqapraqsa 5 1 4 1
P13 wxaxbybzczadab 5 3 4 2
P14 jkqrklrslmst 7 3 1
P15 pononmnmlmlk 6 3
- - - -
53 14 69 11

QB-Beginning; M-middle; E-end.

 SEQUENTIAL PATTERN ACQUISITION 417

and M initializations on seven of the eight easiest problems, but B or M

initializations on all but one of the seven hardest problems. The ex-
ceptions were easy P14 and hard P3. There is no significant difference
discernible between good and poor Ss in their choice of types of
initializations.

Extrapolation
Table 6 contains information about the last operation performed by
the Ss: the actual extrapolation from the initialized pattern description.
Information is presented in the table for 174 extrapolations (out of a
possible 210), 137 at a high certainty level, and 37 at a moderate cer-
tainty level. Table 6 distinguishes extrapolations by positions, from those
by period, and from mixtures of the two.
In extrapolating by position the letters occupying a specific position
in the period of the answer are initialized and extrapolated separately
from the letters occupying other positions. When a problem is classified
as extrapolated by position, every position for that problem was initial-
ized and extrapolated separately. In contrast other Ss extrapolated the
sequence by period, that is, by iterating through the complete pattern

TABLE 6
Extrapolation

Extrapolation method
Percent
BY Within within-
Problem STM chunks position Mixed period period

PI cdcdcd one 7 100

P2 aaabbbcccdd one 14 100
P3 atbataatbat one 2 1 7 70
P4 abmcdmefmghm one 3 10 77
P5 defgefghfghi several 1 11 92
P6 qzapxbqxa one 2 1 9 75
P7 aduacuaeuabuaf several 7 3 3 23
P8 mabmbcmcdm one 3 9 75
P9 urtustuttu one 1 2 9 75
PlO abyabxabwab one 1 10 91
Pll rscdstdetuef several 10 1 9
P12 npaoqapraqsa several 7 1 4 33
P13 wxaxybyzczadab several 1 4 7 58
P14 j kqrklrslmst several 1 9 3 23
P15 pononunmlmlk several 6 6 50
- -
Total 28 ii 110 62
All problems One chunk 6 10 75 84
Several chunks 22 28 35 41
418 KOTOVSKY AND SIMON

description to generate sequentially a letter for each position, Of the 176

entries on Table 6, 110 are extrapolations by period, 28 extrapolations by
position, and the remaining 38 are mixed.
SS may (and often do) extrapolate a sequence correctly without ever
assembling a complete pattern description, when extrapolating either
totally or partly by position. Even when they obtain a complete pattern
description, they do not in these cases of extrapolation by position, use
the pattern description as a unified memory structure to produce the ex-
trapolation, but rather use it piecemeal. S2’s verbalizations on Pll illus-
trate the point: “Letters are in groups of two by two . . . We start off
with a CD then something like a TU and this alternates. Each little
series advancing by one letter each time,” or Sl4 on Pl4, “. . . . two series
of two, each increasing.” In some more extreme cases S describes part
of the sequence, extrapolates it before generating any sort of total pat-
tern description, and then proceeds to do the same for another part of
the sequence.
In the 66 cases where Ss extrapolate by position, wholly or partly, they
need not store a complete pattern description along with all the short-
term memory pointers and their associated lists. In doing piecemeal ex-
trapolation, the load on memory is therefore much reduced. In addition,
the tasks of assembling and arranging the parts of a total pattern de-
scription in memory are substantially reduced. While the use of piece-
meal extrapolation was suggested by various pencil marks on the prob-
lem sheets administered to the earlier groups of 12 and 67 Ss the data
from the present study constitute the first strong evidence for this
practice.
In all cases where more than two short-term memory cells were used
in a pattern description, the extrapolations were either partially or totally
by position. Thus what seemed to be a violation of the hypothesized limi-
tations on short-term memory (Hypothesis A3) was not, since piecemeal
extrapolations do not require all the pointers to be retained simultane-
ously in memory. The method of extrapolation also affects problem dif-
ficulty, as will be seen in the next section.
A similar question arises of the memory burden imposed by Mt or
M, tallies. There were (see Table 4) seven uses of Mk tallies, three uses
of M, tallies, and four uses of an M, and M, tally within the same pat-
tern description. Initialization information of moderate or high certainty
was obtained for 11 of these 14 cases. In all but one of the 11 cases, the
sequence was extrapolated, in whole or part, by position. In the one re-
maining case ( S6 and P3,) S erred in his extra’ olation. Thus in no case did
P
S use an M, or M, tally in his pattern descrrption to produce a success-
ful extrapolation, except by piecemeal extrapolation. Hence the apparent
 SEQUENTIAL PATTERN ACQUISITION 419

memory load on S in these cases, inferred from the pattern description,

is substantially higher than the actual load. Of course, the causal arrow
may have run in the opposite direction: the high memory load implied
by the description forcing the piecemeal extrapolation.
In the majority of cases (110 out of 176) Ss extrapolate the total pat-
tern description (extrapolation by period) in support of Hypothesis B4;
that Ss in performing the final part of their task, do possess a “sequence
generator” that utilizes the list processes previously discussed.

Errors and Sources of Problem Difkulty

Table 7 presents data on those mistakes made by the Ss that could
be diagnosed at moderate and high certainty levels. Of the 54 mistakes
made by the Ss, 35 were analyzed at a high level of certainty, and eight
with moderate certainty. Eleven additional mistakes contained too little
data for analysis.
Extrapolation errors and concept errors. The most frequent errors were
in extrapolating correct concepts. There were 27 extrapolation errors, 11
errors in concepts, and 5 identifiable errors in both the concept and the
process of extrapolation.
Of the 27 extrapolation errors, 15 occurred when Ss began their ex-

TABLP: 7
Analysis of Diagnosable I<rrors
(by Problem)

Error in

Problem Chunks Concept Extrapolation Both

Pl cdcdcd one
P2 aaabbbcccdd one 3
P3 atbataatbat one 1
P4 abmcdmefmghm one 1
P5 defgefghfghi several 1 1
P6 qxapxbqxa one 2 2
P7 aduacuaeuabuaf several 1 3 1
P8 mabmbcmcdm one 1 2 1
P9 urtustuttu one 7
PlO abyabxabwab one 1
- Pll rscdstdetuef several 2 1
P12 npaoqapraqsa several 1
P13 wxaxybyzczadab several 2 3
P14 jkqrklrslmst several 1 1
P15 pononmnmlmlk several 3 1
11 57 5
220 KOTOVSKY AND SIMON

trapolation at an incorrect part of the sequence. The next most frequent

type of extrapolation error involved Ss who obtained a correct pattern
description, but who were unable to use it to produce a correct extrapola-
tion. Problem 9 caused 7 of the 27 extrapolation errors, because the re-
peated U in position 1 conilicted with the U of the RSTU series in posi-
tion 2. Six Ss initially obtained correct concepts but altered them when
they tried actually to use them in extrapolating. These errors all occurred
on Problems 8, 11, 13, and 14 which are analogous to double-alternation
problems. In these cases, the Ss tended to simplify the patterns, con-
verting them to straight alphabetic sequences.
The remaining major sources of errors involve incorrect concepts, in-
cluding approximations. These approximations all involve selecting cer-
tain parts or features of the presented sequence, ignoring other features
or parts, and producing an extrapolation that fits the selected parts. In
most cases, the problem is reduced to a simpler one by ignoring infor-
mation that would contravene a proposed pattern.
The remaining cases in which Ss did not obtain correct concepts in-
volved failure to find the correct periodicity.
An examination of Hypotheses A3 and Dl, which postulated sources
of problem difficulty, shows good agreement with the data. The only
major source of difficulty not adequately predicted by the hypotheses
is the frequent occurrence of the “double alternation” mistakes.
The relative paucity of instances where an S was unable to apply
BN or use the alphabet correctly accounts for the poor agreement be-
tween the performance of newer variants of the computer program,
which were designed to simulate these specific deficiencies, and the be-
havior of individual Ss. Thus when specific abilities (to recognize and
record N or BN relations either by position or by period) are removed
from the program, or their order of application is changed, the resulting
“crippled” versions of the program miss different problems (Appendix
1) from those most often missed by the Ss. The primary sources of prob-
lem difficulty for human Ss are the presence of spurious relations, the
inability to keep place on initialization, the failure to record a coherent
pattern description that accurately keeps track of positions within the
period, and the difficulty of simultaneous placekeeping on two series;
rather than ignorance of particular alphabets or relations.
Placelceepinig as a source of error. A more detailed examination of
Hypothesis A3, which predicts that Ss will have difficulty utilizing more
than one placekeeper (memory cell) during extrapolation, reveals that
while this difficulty caused some mistakes, it was more frequently seen
( 1) in its contribution to the Ss’ inability to assemble and/or use coherent
pattern descriptions, and (2) in the Ss avoidance of the memory burden
 SEQUENTIAL PATTERN ACQUISITIOK 421

by extrapolating by position. Thus, from Table 6, on Problems Pl, P2, P3,

P4, P6, P8, P9, and PlO, where the Ss almost always used no more than
one placekeeper in short-term memory, they extrapolated partly or wholly
by position 16% of the time, while on the remaining problems (P5, P7,
Pll, P12, P13, P14, and P15) where they usually used more than one
chunk in their pattern descriptions, they extrapolated partly or wholly
by position 59% of the time. This tendency for Ss to extrapolate in piece-
meal fashion when the pattern description involves more than one chunk
in short-term memory, strongly supports Hypothesis A3.
A comparison of the difficulty rankings of the 15 problems for the
present group of Ss with the average of their rankings for the two pre-
vious groups of Ss casts further light on the relation of placekeepers to
difficulty. For the comparison, we use the rankings, in Table 1, by num-
bers of Ss who solved the several problems correctly. The more difficult
problems have the higher rankings.
In the two previous studies, the seven problems calling for multiple
placekeepers in their descriptions ranked among the top eight problems
in difficulty. (Average rank for these seven problems was 11.7; for the
other eight problems, 4.75). In the present study, the same seven prob-
lems ranked only slightly higher than the remaining eight in difficulty.
(Average ranks, from Table 1, 8.4 and 7.6, respectively.)
The main reason for the difference appears to be that, in the present
study, the Ss often avoided the difficulty of retaining two or more place-
keepers by extrapolating by position, instead of by period. Of the three
Ss who used two placekeepers in their description of the concept for
Problem 6, two made errors; only two of the nine who used a single
placekeeper for this problem made errors. Presumably, the anticipation
of these difficulties made Ss avoid procedures which required them to
keep several chunks in short-term memory when they could avoid it.
Summary. Our findings support Hypotheses A3 and Dl. An additional
source of error not previously hypothesized is the tendency for subjects
to convert double alternations to straight alphabet sequences. The ten-
dency of the “poor” Ss occasionally to use three or more placekeepers in
contrast to other subjects solving the same problems is another source
of error for these subjects.

CONCLUSION
The current study attempts to record details of human behavior in the
problem solving process so as to test the hypotheses generated by the
theory and computer model presented in our 1963 paper.
The data on Ss’ behavior were obtained in two main ways: from
verbalizations and from records of a presentation device that required
422 KOTOVSKY AND SIMON

subjects to observe one letter at a time. This sequential presentation can

be expected to influence the concept-forming behavior exhibited by the
SS, especially noticing behavior, on problems involving N/ or BN/ rela-
tions. Thus, the single letter presentation task is a little different from
that used in the earlier study. The general and specific areas of agree-
ment between Ss’ performance in the current task and the previous two
groups’ performances, however, together with the data on Ss’ eye move-
ments during problem solving in which the total sequence was visible
(Williams, 1969)) yield strong evidence for the similarity of most of the
behavior exhibited in the two types of situations. No important differ-
ences were found between the Ss’ behavior in the eye movement study
and that obtained with the more controlled single letter presentation
modes. We conclude that the experimental task in this study is similar
enough to the task used previously to constitute a reasonable test of the
theory derived from Ss’ performances on the earlier task. Some modest
changes in the 1963 model are suggested by the findings of this study,
specifically, an enlargement of the pattern description language both to
handle counting or tally operations and to reflect the hierarchical nature
of certain concepts.
If the model is to simulate more accurately the errors made by the
Ss, plausible means must be found for decreasing its ability to keep its
place in the initialization and iteration of the pattern description, and
its ability to relate positions in the periodicity of the sequence. Another
substantial difference between the error behavior of the model and that
of the Ss is that the computer always initializes its pattern description
at the beginning of the sequence, while the Ss do this only two-fifths of
the time. On problems where no mistakes are made, this difference in
behavior yields no difference in the output. But on problems involving a
mistake, the difference in starting point of the extrapolations often pro-
duces gross differences between the computer’s erroneous extrapolation
and the Ss.
On the other hand, many of the Ss making errors of initialization did
so because of a built-in rigidity somewhat similar to that of the computer.
They tended to start their extrapolations at the beginning of a period
rather than at the end of the presented sequence.
The last change needed in the model to reproduce more adequately
the Ss’ errors is a double alternation mechanism that would cause the
insertion of extra Ns in those concepts that contain such double alterna-
tion sequences.
The testing of the individual hypotheses deduced from the model
against the behavior of the Ss yielded excellent agreement between the
hypothesized behavior and the actual behavior in all respects save those
noted above. In addition, the experiment demonstrates the fruitfulness of
 SEQUENTIAL PATTERN ACQUISITION 423

the pattern description language for describing behavior in tasks of

this type.
The analysis of human behavior presented in the present study demon-
strates that the specific hypotheses incorporated in the computer pro-
gram, the Concept Former, and generated by the behavior of that pro-
gram provide a useful framework for organizing observations of behavior,
are testable, and are (with the exceptions and additions previously
noted) confirmed. In addition, the broad applicability to quite different
tasks (Gregg, 1967; Simon & Sumner, 1968) of the mechanism and rela-
tions that constitute our model of sequential pattern recognizing be-
havior argues for the possibility of using the model to go beyond the
prediction of specific behaviors to the discovery of general mechanisms
that operate over broad ranges of human behavior.

APPENDIX I
Problems Missed by ljifferent Variants of the Computer Model

Variants of the No. of majo

program (var- abilities
iants reported (subroutines)
in previous included in Problems missed
study indicated this version -
by asterisk) of the modela 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

A* 4 x x x x x x x x x x x x
B* 4 x x X xxxxxx
C* 4 x X x x x x x x
I)* 4 X x
1s: 4 x x x x x
F 4 x x X x x
(: 4 x x
H 4 x
I 4 X x x
J 4 X
K 4
I, 3 x x x
M 3 x x X x
N 3 x x
0 3 x x
2 s x X

i 2 x x X
1: 1 x x x x x x x x x
s 1 X x x x x xxxxxxx
T 1 x x x x x x x x x x x x x
U 1 xxxxxxx xxxxxxxx

* Simon and Kotovsky, 1963.

u The order of t,heir use in attacking a problem was independently varied as ww the
degree of refinement of individual subroutines in a few cases.
424 KOTOVSKY AND SIMON

REFERENCES
GREGG, L. W. Internal representations of sequential concepts. In B. Kleinmuntz (Ed. ),
Concepts and the structure of memory. New York, Wiley, 1967.
KLAHR, D., & WALLACE, J. G. An information processing analysis of some Piagetian
experimental tasks. Cognitive Psychology, 1970, 1, 358387.
KOTOVSKY, K. An empirical test of the Simon & Kotovsky “concept former” model
of human letter series sequence extrapolation behavior. Unpublished M. S. thesis,
Carnegie-Mellon University, 1970.
NEWELL, A., & SIMON, H. A. Human problem solving. Englewood Cliffs, N. J.: Pren-
tice-Hall, 1972.
PIVAR, M., & FINKELSTEIN, M. Automation using LISP, of inductive inference on
sequences, 125-216. In E. C. Berkeley & D. Bobrow (Eds.), The programming
language LISP. Cambridge, Mass.: Information International, 1964.
&STLE, F. Theory of serial pattern learning. Psychological Review, 1970 (Novem-
ber), 77, 481495.
SIMON, H. A., & KOTOVSKY, K. Human acquisition of concepts for sequential pat-
terns. Psychological Review, 1963, 70, 534-546.
SIMON, H. A., & NEWELL, A. Information processing in computer and man. American
Scientist, 1964, 52, 281300.
SIMON, H. A., & SUMNER, R. K. Pattern in music. In B. Kleinmuntz (Ed.), Formal
representation of human judgment. New York: Wiley, 1968.
THURSTONE, L. L., & THURSTONE, T. G. Factorial studies of intelligence. Chicago:
University of Chicago Press, 1941.
VAN DE GEER, H. P., & JASPERS, J. F. M. Cognitive function. In P. R. Farnsworth,
0. McNemar & Q. McNemar (Eds.), Ann& Review of Psychology, 1966, 17,
145.
WILLIAMS, D. S. Computer program organization induced by problem examples.
Unpublished Doctoral dissertation, Carnegie-Mellon University, 1969.

(Accepted September 19, 1972)