Welcome to Scribd, the world's digital library. Read, publish, and share books and documents. See more
Download
Standard view
Full view
of .
Look up keyword
Like this
1Activity
0 of .
Results for:
No results containing your search query
P. 1
Elman, Jeffrey, L - Finding Structure in Time

Elman, Jeffrey, L - Finding Structure in Time

Ratings: (0)|Views: 10 |Likes:
Published by henrique_oliv

More info:

Published by: henrique_oliv on May 22, 2011
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

11/16/2013

pdf

text

original

 
COGNITIVE SCIENCE
14,
179-211 (1990)
FindingStructure n Time
JEFFREYL.ELMAN
University of California, San Diego
Time underlies many interesting human behaviors. Thus, the question of how torepresent time in connectionist models is very important. One approach Is to rep-resent time implicitly by its effects on processing rather than explicitly (as in aspatial representation). The current report develops a proposal along these linesfirst described by Jordan (1986) which involves the use of recurrent links in orderto provide networks with a dynamic memory. In this approach, hidden unit pat-terns are fed back to themselves; the internal representations which developthus reflect task demands in the context of prior internal states. A set of simula-tions is reported which range from relatively simple problems (temporal versionof XOR) to discovering syntactic/semantic features for words. The networks areable to learn interesting internal representations which incorporate task demandswith memory demands: indeed, in this approach the notion of memory is inextri-cably bound up with task processing. These representations reveal a rich
struc-
ture, which allows them to be highly context-dependent, while also expressinggeneralizations across classes of items. These representatfons suggest a methodfor representing lexical categories and the type/token distinction.
INTRODUCTION
Time is clearly important in cognition. It is inextricably bound up withmany behaviors (such as language) which express themselves as temporalsequences. Indeed, it is difficult to know how one might deal with such basicproblems as goal-directed behavior, planning, or causation without someway of representing time.The question of how to represent time might seem to arise as a specialproblem unique to parallel-processing models, if only because the parallelnature of computation appears to be at odds with the serial nature of tem-
I would like to thank Jay McClelland, Mike Jordan, Mary Hare, Dave Rumelhart. MikeMozer, Steve Poteet, David Zipser, and Mark Dolson for many stimulating discussions. I thankMcClelland, Jordan, and two anonymous reviewers for helpful critical comments on an earlierdraft of this article.This work was supported by contract NO001 14-85-K-0076 from the Office of Naval Re-search and contract DAAB-O7-87-C-HO27 from Army Avionics, Ft. Monmouth.Correspondence and requests for reprints should be sent to Jeffrey L. Elman, Center forResearch in Language, C-008, University of California, La Jolla, CA 92093.
179
 
180 ELMAN
poral events. However, even within traditional (serial) frameworks, the repre-sentation of serial order and the interaction of a serial input or output withhigher levels of representation presents challenges. For example, in modelsof motor activity, an important issue is whether the action plan is a literalspecification of the output sequence, or whether the plan represents serialorder in a more abstract manner (e.g., Fowler, 1977, 1980; Jordan 8c Rosen-baum, 1988; Kelso, Saltzman, & Tuller, 1986; Lashley, 1951; MacNeilage,1970; Saltzman & Kelso, 1987). Linguistic theoreticians have perhaps tendedto be less concerned with the representation and processing of the temporalaspects to utterances (assuming, for instance, that all the information in anutterance is somehow made available simultaneously in a syntactic tree); butthe research in natural language parsing suggests that the problem is nottrivially solved (e.g., Frazier 8c Fodor, 1978; Marcus, 1980). Thus, what isone of the most elementary facts about much of human activity-that it hastemporal extend-is sometimes ignored and is often problematic.In parallel distributed processing models, the processing of sequentialinputs has been accomplished in several ways. The most common solution isto attempt to “parallelize time” by giving it a spatial representation. How-ever, there are problems with this approach, and it is ultimately not a goodsolution. A better approach would be to represent time implicitly ratherthan explicitly. That is, we represent time by the effect it has on processingand not as an additional dimension of the input.This article describes the results of pursuing this approach, with particu-lar emphasis on problems that are relevant to natural language processing.The approach taken is rather simple, but the results are sometimes complexand unexpected. Indeed, it seems that the solution to the problem of timemay interact with other problems for connectionist architectures, includingthe problem of symbolic representation and how connectionist representa-tions encode structure. The current approach supports the notion outlinedby Van Gelder (in press) (see also, Elman, 1989; Smolensky, 1987, 1988),that connectionist representations may have a functional compositionalitywithout being syntactically compositional.The first section briefly describes some of the problems that arise whentime is represented externally as a spatial dimension. The second sectiondescribes the approach used in this work. The major portion of this articlepresents the results of applying this new architecture to a diverse set of prob-lems. These problems range in complexity from a temporal version of theExclusive-OR function to the discovery of syntactic/semantic categories innatural language data.
THE PROBLEM WITH TIME
One obvious way of dealing with patterns that have a temporal extent is torepresent time explicitly by associating the serial order of the pattern with
 
FINDING STRUCTURE IN TIME
181
the dimensionality of the pattern vector. The first temporal event is repre-sented by the first element in the pattern vector, the second temporal eventis represented by the second position in the pattern vector, and so on. Theentire pattern vector is processed in parallel by the model. This approachhas been used in a variety of models (e.g., Cottrell, Munro, & Zipser, 1987;Elman & Zipser, 1988; Hanson & Kegl, 1987).There are several drawbacks to this approach, which basically uses aspatial metaphor for time. First, it requires that there be some interface withthe world, which buffers the input, so that it can be presented all at once. Itis not clear that biological systems make use of such shift registers. Thereare also logical problems: How should a system know when a buffer’s con-tents should be examined?Second, the shift register imposes a rigid limit on the duration of patterns(since the input layer must provide for the longest possible pattern), andfurthermore, suggests that all input vectors be the same length. These prob-lems are particularly troublesome in domains such as language, where onewould like comparable representations for patterns that are of variablelength. This is as true of the basic units of speech (phonetic segments) as it isof sentences.Finally, and most seriously, such an approach does not easily distinguishrelative temporal position from absolute temporal position. For example,consider the following two vectors.
[0111000001[0001110001
These two vectors appear to be instances of the same basic pattern, but dis-placed in space (or time, if these are given a temporal interpretation). How-ever, as the geometric interpretation of these vectors makes clear, the twopatterns are in fact quite dissimilar and spatially distant.’ PDP models can,of course, be trained to treat these two patterns as similar. But the similarityis a consequence of an external teacher and not of the similarity structure ofthe patterns themselves, and the desired similarity does not generalize tonovel patterns. This shortcoming is serious if one is interested in patterns inwhich the relative temporal structure is preserved in the face of absolutetemporal displacements.What one would like is a representation of time that is richer and doesnot have these problems. In what follows here, a simple architecture isdescribed, which has a number of desirable temporal properties, and hasyielded interesting results.
I The reader may more easily be convinced of this by comparing the locations of the vectors[lo 01, [O 101, and [O 0 l] in 3-space. Although these patterns might be considered “temporallydisplaced” versions of the same basic pattern, the vectors are very different.

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->