Neural Networks, Vol. 3, pp. 535 549, 1990 1~893-t~08()/911$3,110 ÷ .

00
Printed in the USA. All rights reserved. ('opyright ~ I990 Pergamon Press pie

ORIGINAL CONTRIBUTION

Connectionist Nonparametric Regression: Multilayer

Feedforward Networks Can Learn Arbitrary Mappings

HALBERT WHITE
University of California. San Diego
(Received 27 February 1989: revised and accepted 31 January 1990)

Abstract--It has been recently shown (e.g., Hornik, Stinchcombe & White. 1989, 1990) that st(~)ciently comph'x
multilayer feedJbrward networks are capable of representing arbitrarily accurate approximations to arbitra©'
mappings. We show here that these approximations are learnable by proving the consistency (~f a class ()f
eonnectionist nonparametric regression estimators,for arbitrary (square integrable) regression fhnctions. The
consistency property ensures that as network "experience" accumulates (as indexed bv the size (~1 the training
set), the probability of network approximation error exceeding any spec(fied level tends to zero. A k~3, j~'ature
~)[ the demonstration of consistency is the proper control of the growth of network complexity a,s a fimction 01
network experience. We give specific growth rates fi)r network complexity compatible with consistency. We also
consider automatic and semi-automatic data-driven methods for determining network complexi(v in applications.
based on minimization of a cross-validated average squared error measure of network pe@~rmance. We rec-
ommend cross-validated average squared error as a generally applicable criterion/?~r comparing relative per*
f?)rmance of dif[ering network architectures and configurations.

geywords--Learning, Feedforward networks, Nonparametric, Regression, Convergence.

1. I N T R O D U C T I O N erty of consistency for a certain class of learning

Recently, Hornik, Stinchcombe and White (1989,
1990) (HSWa,b) have demonstrated that sufficiently
complex multilayer feedforward networks (e.g., Ru-
melhart, Hinton, & Williams, 1986) are capable of
arbitrarily accurate approximations to arbitrary map-
pings (i.e., measurable or continuous functions). (See
also Carroll and Dickinson. 1989; Cybenko, 1989;
Funahashi, 1989; Hecht-Nielsen, 1989; and Stinch-
combe & White, 1989a). An unresolved issue is that
of "learnability", that is, whether there exist meth-
ods allowing the network weights corresponding to
these approximations to be learned from empirical
observation of such mappings. It is the purpose of
this paper to prove that these approximations are
indeed learnable by establishing the statistical prop-
Acknowledgments: This research was supported by a grant
from the GuggenheimFoundation and by NSF grant SES-8806990.
The author is grateful for helpful conversations with Maxwell
Stinchcombe, Whitney Newey, and James Powell, and for helpful
suggestions by the editor and referees. The author is responsible
for any errors.
Requests for reprints should be sent to Halbert White, De-
partment of Economics, D-008. University of California, San
Diego, La Jolla, CA 92093.
methods for such networks. The consistency prop-
erty ensures that as network experience accumulates
(as indexed by the size of the training set n) the
probability of network approximation error exceed-
ing any specified level tends to zero. This establishes
the nonparametric regression capability of multilayer
feedforward network models.
A key feature of our analysis is the proper control
of network complexity as a function of network ex-
perience n. Arbitrarily accurate approximation to an
arbitrary function will generally require an arbitrarily
complex network. In practice, only networks of finite
complexity can be implemented. Too complex a net-
work (relative to n) learns too much--it memorizes
the training set and therefore generally performs
poorly at generalization tasks. Too simple a network
learns too little--it fails to extract the desired rela-
tionships from the training set and thus also performs
poorly at generalization tasks. Our theory provides
specific growth rates for network complexity that
asymptotically (i.e., as n ~ ~c) avoid the dangers of
both overfitting and underfitting the training set.
Moreover, these rates can be combined with data-
driven methods for determining network complexity
in a manner that guarantees their consistency also.

535
530
The learning methods treated here are extremely
computationally demanding. Thus. they lay no claim
to biological or cognitive plausibility. They do, how-
ever, provide some appreciation for the computa-
tional effort required to train multilayer feedforward
networks to approximate arbitrary mappings arbi-
trarily well. Our methods are feasible in modern I es-
pecially parallel) computing environments and arc
thus of some value to researchers interested in prac-
tical applications. In particular, we recommend gen-
eral application of cross-validated measures of net-
work performance.
To keep this paper to a manageable size. we limit
its scope strictly to the issue of consistency {learna-
bility). This by no means exhausts the relevant issues:
it leaves many relevant questions unaddressed, and
indeed makes possible the formulation of further im-
portant questions. We leave these to other work. and
merely identify them as they arise.
2. HEURISTICS
Consider a sequence {Z,} - ~Z,, t =- 1. 2. . .} ot
identically distributed r a n d o m column vectors. Sup-
pose we are interested in the relationship between
some elements of Y, of Z, and the remaimng elements
X,. We write Z, = ( Y : , X : I ' . where a prime super-
script denotes vector transposition. For example, m
classification or pattern recognition problems Y, is a
binary or multinomial variable designating class
m e m b e r s h i p and X , is a set of variables influencing
the classification. In forecasting problems. Y, is the
set of variables (digital or analog) that we wish to
forecast on the basis of variables X,, which may itself
contain past values of Y,. In image enhancement
problems, 1/, encodes a target image, and X, encodes
a degraded version of the image. In pattern comple-
tion problems, Y, is the missing part of a pattern.
and X, is the supplied part of the pattern (we suppose
the same part is always to be supplied for this ap-
plication).
Regardless of whether a deterministic or stochas-
tic relationship exists between Y, and X,, a natural
object of interest in such situations is the conditional
expectation of Y, given X,, written E ( Y , [ X , ) . (See
White, 1989a,) This can be represented as a regres-
sion function,
O.(X,) = E(Y, X.t.
For example, when 1/, can take on only the values tl
or 1, O,(x) gives the probability that Y, = 1 given
that X, = x. W h e n Y. can assume a continuum of
values. Oo(x) gives the expected value for Y, given
that X, = x. We may also write
Y,- 0,,(~)-~:..
II. U/'hi:c
where ~:. ~ Y, - E ( Y , X , ) is a rm~dom error with
conditional expectation zero given X,. When the re-
lationship between Y, and X, is detcrmimstic. ~ ~
always zero: otherwise, is nonzero with positive
probability. In the situation treated here. the regrc>
sion function tL is assumed m be entirety, unknown.
Our problem is to learn {estimate. approximate ~ lhc
mapping 0. from a realization of the sequence {Z.~
In practice, we observe a realization of ont', :
finite part ot the sequence ~.,:, ' Z ~ • ":aining ~e,t ~-
' s a m p l e " of size n li.e., a realiza*ion of Z" - IZ _
ZT,)'t. Because It is all element o~ a space ol
functions lsay (-)). we have cssenlialtv no hope ot
learning I1 in any complete sense qo~n ,, sample of
fixed finite size. Nevertheless. ~1 ~:< possible to ap-
proximate ft to some degree o~ afCCUlacv USlnE tl
sample ot size n, and to construct increasingly ac-
curate approximations with mcrea>tng n. [n what fol-
lows we will refer to such a procedure interchanee-
abh as learning, estimation, or ~q)proximation A
learning rule is correspondingl,v delined in the weak-
est poss!ble sense as a sequence of mappings, say
1:)-I - 10.- ~l -- I 2 . . . . from the underlying prob-
ability space generating the observed phenomenon
of interest to the space ® in whicl~ u . the object o~
interest describing this p h e n o m e m m , lies. Because
(). is a mapping from a probabiht} ,pace, ~ !~ slo-
chastic
A minimal property for any learning rule is that
ol c o n s i s t e n c y . A stochastic seqt!encc It/,# is consis-
tent for o if the probability that t/ exceeds any ,,,pcc-
ified level of approximation erroT relative to #,. tends
to zero as the sample size n tends ~o infinity. Pro-
cedures that are not consistem will always make
errors in classification, recognitum, forecasting,
enhancement, or pattern completion {forms of gen-
eralization) that are eventually a~.oided b~ a consis-
tent procedure. The only errors ultmmtelv made b~
a consistent procedure are the inherentlx unavoid-
able errors (¢:,) arising from an~ m n d a m e n t a l ran-
domness or fuzziness in the true relation between X,
and Y. Consequently, our focus is on using connec-
tionist networks to obtain learning rules {0,} consis-
tent for an arbitrary regression function fJ,, and we
therefore identify the issue of the "learnability" of
a particular class of mappings with the existence of
a consistent learning rule for that class. This distin-
guishes our focus of attention from such standard
concept-learmtbility issues as treated for example by
Haussier (1989).
We apply the method of sieves, {Grenander- 1981:
G e m a n & Hwang 1982: White & Wooldridge. 1990L
a general nonparametric statistical procedure m
which an object of interest 0~ lying m a general {i.e.,
not necessarily finite-dimensional) space e) is ap-
proximated using a sequence of parametric models
Connectionist Nonparametric Regression
in which the dimensionality of the parameter space
grows along with the sample size. To succeed, the
approximating parametric models must be capable
of arbitrarily accurate approximation to elements of
(-) as the underlying parameter space grows. For this
reason, Fourier series (e.g., Gallant & Nychka, 1987)
and spline functions (e.g., Wahba, 1975; Wahba &
Wold, 1975: Cox, 1984) are commonly used in this
context. Among others, HSWa,b establish that mul-
tilayer feedforward networks also have universal ap-
proximation properties. Without this, attempts at
nonparametric estimation using these network models
would be doomed from the outset.
For concreteness and simplicity, we consider only
single output single hidden layer feedforward net-
works. Our approach generalizes straightforwardly
to the multi-output multi-hidden layer case. Specif-
ically, we write the output of a q hidden unit feed-
forward network given input x as
t"(~, ~") ~ 1~,, + ,~,/;,~,(-f';,,),
w h e r e ff '~ - (flq', ;,,t')' is the p × 1 ( p = q(r
2) + 1) vector of network weights (parameters)
(let []'~ :- ([I,,. [~ . . . . . /~,~)', ;'~ =- (,'[ . . . . . i','¢)'. ,'i
=- (;',~,. . . . . i'i,)' f o r j = 1. . . . . q). qJ is the (given)
hidden unit activation function, and i -= (1, x')'. For
simplicity and without loss of generality, we put no
squashing function at the output unit. Figure 1 de-
picts the corresponding network architecture.
We construct a sequence of approximations to O,,
by letting network complexity q grow with n at an
appropriate rate, and for given n (hence given q)
selecting weights (~,, so that 0,, ~ .fq"(., c~,,) provides
an approximation to the unknown regression func-
tion O,, that is the best possible in an appropriate
sense, given the sample information.
To formulate precisely a solution to the problem
of finding 0,,. we define
r(~,. q. ,5) ~ {0 C e):O(-) .f',(., 3 %
,I -< A, i ' i i < qA ,
the set of output functions of all single hidden layer
feedforward networks with q hidden units having
~ Output
weights
Hidden
Tweights
Input
X~ X2 X3 X4
FIGURE. 1. Single hidden layer feedforward network.
537
activation functions ~u. and with weights satisfying
a particular restriction on their sum norm, indexed
by A. We construct a sequence of "sieves" {®,,(~,)}
by specifying sequences {q,,} and {A} and setting
(.),,(~,) = T(q,,, q,,, A,,), n = 1, 2 . . . . . The sieve
e),,(q/) becomes finer (less escapes) as q,, ~ ~ and
A,,--~ ~c. For given sequences {%} and {A,,}, the "con-
nectionist sieve estimator" O,, is defined as a solution
to the least squares problem (appropriate for learn-
ing E( Y,IX~))
rain n ~2 [Y.. O(X,)]:. n = 1. :."~. . . . . (2.1)
Associated with 0,, is an estimator ~,, of dimension
p, x 1 (p,, =- q,,(r + 2) + 1) such that 0,,(-) = .f'~"(.,
(~,,). Sections 3 and 4 are devoted to specifying precise
conditions on {Z,}, ~u. {q,,}. and {A,,} that ensure the
consistency of 0,, for O,.
Before describing these conditions, we must em-
phasize that obtaining at least a near global solution
to (2.1) is central to this approach. Consequently,
optimization methods that deliver only local optima,
+ such as the method of back-propagation or iterative
Newton methods will be inadequate. Instead, global
optimization methods (as described by Baba, 1989;
or White. 1989a, Section 4.a) are required. This is
one source of the advertised computational burden
of this approach. For simplicity, we suppose for now
that a global solution to (2.1) is available. At the end
of this section, we describe the modifications to our
set-up that permit treatment of cases in which only
approximately optimal solutions are available.
For simplicity throughout we suppose that {Z,} is
a bounded stochastic process, without loss of gen-
erality taking values in the hypercube t'--= x ) ~[0,1].
We also assume that 0,, is a square integrable function
on I', r = v - 1. The activation function q/is assumed
to be bounded and satisfy a Lipschitz condition. The
Lipschitz condition holds whenever q/is continuously
differentiable with bounded derivative, as for the
logistic and hyperbolic tangent squashers commonly
used in applications. Non-sigmoid activation func-
tions are also permitted.
We first consider deterministic methods for net-
work complexity growth. We prove that the appro-
priate choice for &, is such that A,, ~ ~ as n ~ :~
and A,, = o(1ll 4), that is. n ~4 A,, ~ Ill as n ~ ~.
The proper choice for {%} depends on {A,,} and on
the dependence properties of {Z,}. When {Z,} is an
independent identically distributed (i.i.d.) sequence,
Theorem 3.5 below establishes that 0,, is consistent
for 0,. provided that q, ---, ~c as n --~ z and q, A~ log
% A,, = o(n). When {Z,} is a stationary mixing pro-
cess. Theorem 3.5 establishes that 0,, is consistent for
0,, provided that % --~ ~c as n ~ 7- and q,, A~, log q,,
A,, = o(n~-~). Mixing processes are a class of time
538 H. 'i}hi.~c

series processes that can exhibit considerable short- TABLE 1

run d e p e n d e n c e , but display a f o r m of asymptotic Relative Network Complexity for Consistency as a
i n d e p e n d e n c e , in that events involving elements of Function of Sample Size, ~ = .01
{Z,} separated by increasingly greater time intervals n 10 2 10 4 10 6 10 8 10"~
are increasingly closer to i n d e p e n d e n c e . The allow-
n' 1 95.50 9 . 1 2 × 103 8.70 1 0 "~ 8 . 3 2 .- 10 ~
able g r o w t h rate for n e t w o r k complexity is slower in
n" " 1 9,77 95.50 9.33 ,~ t 0 2 9 . t 2 ~ 10 a
the case of mixing processes than for the case ol
i n d e p e n d e n t processes because new information ar-
rives at a slower rate. (See White, 1984, for a further
discussion of mixing processes.) fixed and '&, ~ c log n will m c r c a s e with n. the re-
The allowed g r o w t h rates for network complexity strictions imposed on ~, are quite mild. ( In fact. there
are not immediately obvious from the conditions is even r o o m for letting the n u m b e r of inputs r in-
stated. To gain some insight we consider the allowed crease with n, so that greater a m o u n t s of information
rates in m o r e detail. First, we see a clear trade-off can be exploited as n --, z in attempting to hit the
b e t w e e n the allowed rate of increase of q,, and that target: we leave this to o t h e r work for the sake or
of &,: the slower of the rate of increase of &,. the simplicity.) Because of the mildnes~ of the restric-
greater the permissible rate for %. Because the ap- nons on 7. we do not consider t h e m further.
proximation improves as n e t w o r k complexity in- The force o f our condition is borne bv/~. We ~e-
creases, it is desirable to permit A,, to grow' at only quire that £~" /] -<- c l o g n for some 0 < , ~ ",~
a m o d e s t rate in o r d e r to a c c o m m o d a t e rapid growth With q,, increasing at a rate only slightly slower than
in q,. ,~. this inequality can be satisfied only if an increasing
Accordingly, suppose ,k,, -- O ( l o g n) (i.e.. ~,, ~: proportion of the h','s are close to zero. An elemen--
c log n for some 0 < c <- ~c). This satisfies &, ::: tarv calculation yields
o(n~4), because n ['r4 At, ~ C~'I 14 log n ----" 0 as tt --,
\~ ~I 13 \~ I * ! " t_ll,
~c. For the i n d e p e n d e n t case. we now require % A~
tog q,, A = o(n) (i.e.. n ~ q,, A,, ' ~ log q,,&,--, 0). With
&, = O(log n), it suffices that n ' q,, (log n) 4 log(q,, Now
l o g n ) - + 0. We put % = O(n"), ~ > 0, and seek
admissible choices for a: it now suffices that n I ~ dj- log j]' I" - Iog~q,: 1)----logq.,- --
q,,
n"(log n) 4 (c~ log n + log log n). Because log ~l :-
log log n. we need n"-~(Iog n) 5 --> 0. Because n so that
(logn) 5~0forany~:>0,wecantake~x = 1 - ~:
for any s m a l l t : > 0. Thus. q,, = O(n ~ ~ ) a n d k,, = \~ (t - 1) < log q,, - t ......
O ( l o g n) satisfy the required conditions for indepen-
dent observations. Similarly, for mixing processes W h e n q,, = (, i. we obtain from this inequal-
% : O(nll-~+,2) and A,, = O ( l o g n) suffice. ity that ZY'- (j - l) ' = O(log n). H e n c e . for
These allowed growth rates for n e t w o r k complex- Z'L, /fi] = O(log n) it suffices that {/1, = O(j vl.
ity % are fairly rapid, as Table l illustrates. The W h e n / is large, l/ijl will be decreasing at the rate i
entries are scaled so that unit complexity is achieved under this condition: this is a rather rapid a p p r o a c h
at n = 100. Thus, if five hidden units are appropriate to zero
in the i n d e p e n d e n t case w h e n n - 1 0 0 , 4 7 7 hidden If less stringent restrictions are desired for £ / ~
units can be s u p p o r t e d by a sample of 10,000 and t/fil, they can be achieved by trading off restrictions
45,600 hidden units can be s u p p o r t e d by a sample of on Z'/". for restrictions on q,,. For example, if re-
one million. With mixing processes, allowable growth strictions analogous to those e n c o u n t e r e d for ?, in the
rates are slower. If five hidden units are appropriate previous example are desired, we could set £x. ~ q,
in the mixing case when n = 100. then 49 hidden tog ,l. This choice is compatible with q,, =
units can be s u p p o r t e d by a sample of 10,000. A Oln t'5 "~ for the i n d e p e n d e n t case or q,, -
sample of one million supports 477 hidden units in O(n ~'~-'-) for the mixing case. as some algebra will
this case. E v e n t h o u g h the growth rates in the mixing verify. These are m u c h slower growth rates than in
case a p p e a r dramatically slower than for the inde- the previous example.
p e n d e n t case. they are still rather generous. Despite these results for g r o w t h rates, they pro-
These relatively rapid growth rates for network vide no practical m e t h o d for c h o o s i n g n e t w o r k com-
complexity are achieved by taking &,, to grow at the plexity for a sample o f given size n. O n e appealing
m o d e s t rate of log n. It is helpful to explore the m e t h o d for m a k i n g this choice is the m e t h o d of cross-
implications of this choice. Recall that we require validation (Stone. 1974). The rationale for this method
E q'',=0 ,~J,/¢ <_. A and gT'__'~E ,r ~ tTjit -< q,, zX,,. For the latter is straightforward. First. consider a naive approach.
condition, it suffices that ~-,,:,~ ]~,I " ' <- A,,. Because r is Suppose {A,} is given, and put O,,(~, q) = T(tu. q,
Connectionist Nonpararnetric Regression
&,). For given n, network performance with q hidden
units can be (naively) measured by the smallest at-
tainable average squared error over the training set.
min n ' ~ [Y, - 0(X,)]:.
Unfortunately. this measure is over-optimistic: it is
downward biased (as a measure of minimum risk.
min,E~,,<~,.q~ E([Y, - 0(X,)]2)) and is increasingly
downward biased as q increases. The bias arises be-
cause the observation for (Y;, X ; ) ' is used in arriving
at t);{ (say), the solution to the above minimization
problem. This makes the contribution to squared er-
ror of the tth observation ([Y, - 05{(X,)]e) too small.
Indeed. if network complexity were selected by
choosing q to make average squared error as small
as possible, average squared error could be driven
to zero by choosing q = n, a clearly inappropriate
choice.
The method of cross-validation effectively avoids
these difficulties. The "delete one" cross-validated
average squared error averages an estimate of squared
error for each observation (say observation t) that
uses an estimator for 0,, (say 0~{i,)) that ignores the
information contributed by that (tth) observation.
(For example, a "brute force" choice for ()~,~ can be
obtained as the solution to the problem min0eo,,~,q~
n ~ V~,, [ Y: O(X,)]:.) The cross-validated perform-
ance measure is formally defined as
c,,,(q)= n ' ~ IF, - ",,
0 ..... ( x , ) b
/ I
Because 01{~,, ignores information from the tth ob-
servation. [Y, - I)',I~,I(X,)]: is a measure of out-
of-sample performance and is thus less biased than
[Y, - I);{(X,)] e, an in-sample measure. The cross-
validated average squared error inherits this relative
lack of bias by construction and consequently pro-
vides a measure of network performance superior to
average squared error. Because of its relative un-
biasedness, we recommend its general use.
A completely automatic method for determining
network complexity appropriate for any specific ap-
plication is given by choosing the number of hidden
units c),, to be the smallest solution to the problem
rain C,(q),
~lC ~,tt
where N,, is some appropriate choice set, a subset of
N ---- {1, 2 . . . . }. For the moment, we can suppose
that N,, = {1,2 . . . . . n}. The choice c~,,will be called
the "cross-validated complexity" of the network. The
"cross-validated connectionist sieve estimator" is
then the solution 0,, to the problem
rain ,, ' ~ [ g - 0(X,)]~.
Off e)/~I ~J q,t I t
539
The method of cross-validation has proven widely
successful in other statistical applications; we may
reasonably expect it or some variant (e.g., White,
1989b) to be similarly successful here.
To ensure the consistency of 0,, for 0(,, we can-
not choose {dx,,} and {Nn} arbitrarily. In both the
independent and mixing cases, it is again appro-
priate to take ~,, = o(n~"~). As a crude but expedient
approach we ensure consistency here by taking N,, =
{q. . . . . . . ~,,}. where q,, ~ ~ and ~,, satisfies the rate
conditions for q,, discussed earlier. These condi-
tions control q,, in just the right way to ensure that
O,,(~,, c)n) eventually fills O appropriately. It
may be possible to remove the controls on q,, with-
out affecting consistency, but we leave this to other
work.
In applications it is convenient to pick _q, = ,2@,
for some ,2 < 1. All of the discussion regarding {q,,}
and {din} for the deterministically determined sieves
{O,,(~u)} then applies directly to {~,,} and {4,,}. Ap-
plications entail optimization of C,,(q) over N,, =
[).~,,, ~,,] N N. An advantage of choosing N,, in this
way is that it can greatly reduce the range of values
that must be considered for q. Even so, the range of
N,, may still be large. Exhaustive search over N,, is
thus likely to be expensive; more sophisticated uni-
variate optimization methods are available (e.g.,
Scales, 1982).
A disadvantage of these procedures is that they
are not fully automatic, as we require values for aux-
iliary control parameters A,,, ~,, and either _q,, or ,2..
For this reason, we call these consistent procedures
" s e m i - a u t o m a t i c . " Nevertheless. it is typically
straightforward to pick intuitively plausible values
for these control parameters for a given problem with
given n. Given these, values for all other n can be
readily computed (e.g., using the numbers of Table
1 or numbers similarly constructed).
In practice, full optimization of network perform-
ance over the sample can be difficult or impossible
to attain. Instead, estimated weights will be more or
less optimal depending on the intensity of compu-
tational effort. Results analogous to the consistency
results just described continue to hold when approx-
imate rather than exact optimization is carried out,
although the strength of the results varies with the
assumed degree of computational effort, as one should
expect. Our analysis is conducted by letting the data
determine an appropriate (e.g., approximately op-
timal) degree of network complexity, say gj,,. As be-
fore, we require q,, _< gj,, _< ~,,. For this choice of
complexity, again let the fully optimal solution to the
least squares problem be denoted 0,,, so that (),, now
solves
min n J ~ , [ E - o(X,)] 2.
IJ(I-~,/~.q,,) ,' I
540
Instead o f obtaining 0~, we suppose that we obtain
an a p p r o x i m a t e solution 0,, to this p r o b l e m , char-
acterized by the closeness of n e t w o r k p e r f o r m a n c e
at 0,, to optimal n e t w o r k p e r f o r m a n c e . Specifically,
given a " t o l e r a n c e level" (,, (some positive n u m b e r / ,
we assume that 0,, is chosen so that
i,, ' E
' 1
Iv, - i),,(xj1 e ,, ' Z Iv, - < :
T h e smaller is ~,, the closer is 0,, to being an exact
solution to the n e t w o r k learning problem, and. m
general, the m o r e the c o m p u t a t i o n a l effort required
to obtain {J,,. W h e n ~,, is. large, we can afford to be
m o r e sloppy in finding 0,,.
To obtain our results, we suppose that a sequence
{~,,} of tolerances is specified, t o g e t h e r with a limit
¢o to which the tolerances tend as n ~ ~. T h e limiting
tolerance fro m a y be zero. in which case we d e m a n d
an exact global solution in the limit. In this case. we
can again establish the consistency of 0,, for O. Al-
ternatively, the limiting tolerance m a y be n o n z e r o :
for example, we could require only a fixed tolerance
~,, = ~,, > 0 for all n In this case. consistency of (J,,
for O, c a n n o t be g u a r a n t e e d (so that learning is in-
complete), but n e t w o r k p e r f o r m a n c e m the limit is
adversely affect to a precise extent g o v e r n e d by _.,.
The conditions on {Z,}, ~u. {q,,}. {g/,,}, and {A,,} u n d e r
which these results hold are identical to those pre-
viously described.
To summarize, multilaver f e e d f o r w a r d networks
can learn arbitrarily accurate approximations to un-
k n o w n functions in the sense that they admit a learn-
ing rule possessing the consistency p r o p e r t y To en-
sure consistency, network complexity must be properly
controlled; this control can be data-driven. Exact
optimization for each n is not necessary: a p p r o x v
mate optimization is permitted. H o w e v e r . consist-
ency is g u a r a n t e e d only when exact optimization oc-
curs in the limit.
3. CONSISTENCY OF CONNECTIONIST
SIEVE ESTIMATORS
We now state formal results establishing the con-
sistency of connectionist sieve estimators. We first
treat deterministic choice of connectionist sieves {(9,,},
and then treat the cross-validated connectionist sieve
estimators, assuming exact optimization. We next
give results assuming a p p r o x i m a t e optimization. O u r
results follow f r o m m o r e general statements given in
the next section.
T h e first assumption describes the data generating
process.
ASSUMPTION A.1. L e t (l't. F. P) be a comptete prob-
ability space. The observed data are the realization
kJ. ~Vhll~'
o f a stochastic process {Z,: ~ ..... ~ , v G ? , t -
1, 2 . . . . } on ([L F, P), and P is ~t,'h that either
(i)/Z,} is i.i.d. : or
(ii) {Z,} is a stationary mixing ?:ocus.~ w~th ,,'iehu~
q~,,. ~,, : O. It .- I',, <" l. (Formal definitions ol d~ and
c~ arc given in the next section, !
k'or convenience, we assume that Z, takes values
in the unit cube. W h e n / Z , } is a b o u n d e d sequence.
this can always be ensured b\, appropriate scaling
and shifting. This a s s u m p u o n cn~ttrcs the existence
of E i Y , ) and hence of E ( Y , X , ) . where as belore wc
partition Z, as Z - ( Y,. X; I'. with Y a scalar r a n d o m
variable for simplicity. Let B' B[I'3 denote the
Borel a-field g e n e r a t e d by the ~,pen sets of {'. A n
element A of B' is a subset of t' f~ which the prob
ability P[X, C ,4 1 is defined Define the measure/~
as ,~A~ ~- PlX, ~ A] for each ,~ in B': zL rneasurc~
the relative trequency with which patterns X b e k m g
to the set A. Let L:(I'. It) denot~ the collecuon of
all measurable functions #: i . . . . ;. such that F.{t,I: -
[J't'{O(x) : p ( & ) ] : ~..~ (the ~quarv mtegrablc l u n o
tionsl, and define the metric p:~u . 0:) ~ {~,O, 0:{
for # . rJ, ~ L:(I', ltl. T h e rnelr~c I): measures the
distance between two lunctions u and 0, belonging
to l , : ( / . / z l in terms of weighted roo~ mean squared
error, where the weighting is with respect to ,. Two
functions o and #= diftering on ~, set o f / ~ - m c a s u r c
zero will have IMO,. Oz) - l). Wc droll not distinguish
between such functions: tcchmcallx, tunctions dif-
Iering only o11 a set Ol n-measurt zero arc takur~ ,~
forming an equivalence class. I mqueness of func-
tions of Lq !',/~) is with respect r~ these equivalence
classes.
The space (L-:(I'. p), p_-t ~s , ~omplete scparabic
metric space We take the object of interest to be
the function 0 : / ~ R such that E{ EiX,) = #..{X ~.
and we ~mpose the following structure on .t~
ASSUMPTION A.2. With (('). p) I L (i ~. l~!. pc). M,
~.~ the unique element ol (9 such that E(Y,,X,}
O,,(X,).
This structure is convenient and plausible in many
applications.
Next we formally define the connectiomst s~evc
used to a p p r o x i m a t e elements ¢,t (-)
DEFINITION 3. I. Let q/: R --, R be a given b o u n d e d
function. For any q E N and A ~ R ' . define a subset
T(~v. q, A) o f (9 as
F(~;,. q, ~ ~¢ { 0 C (HI:0(x) = [f,~ \f /;tu(~-';:0,
i ~ II-x')',turatlxm ..
t

('onnectionist Nonparametric Regression
For given ~, a n d s e q u e n c e s {q,,}, {A,,}, define the se-
q u e n c e q [ (single h i d d e n layer) connectionist sieves
{(-),,(~,)} as
(-),,(~) ~ T(~,. q,,, &,). n = l..~". . . . .
This definition implies that when O E (%(~,), we
have O(x) = .f',,(x. (~';,,) in the notation of section 2.
We consider only single hidden layer connectionist
sieves for notational simplicity. T h e results of section
4 also apply to feedforward networks with an arbi-
trary n u m b e r of hidden layers: the results of this
section can be extended to such cases.
To obtain the consistency result, we must place
specific conditions on ~,. and q,, and A,,, A necessary
condition for consistencv is that UT, ~('),,(q.') isp-dense
in (.) (i.e.. U;', : ~('),,(~//) contains an element 0 as close
as we like to any element O* of 6). with distance
measured by p). The universal approximation results
for single hidden layer feedforward networks estab-
lished by HSW a,b make it possible to specify con-
ditions on ~ ensuring the required denseness prop-
erty. To state these conditions, we first define some
terms, We say that q/is a squashing function if t//: R
1. q/(a) --~ (I as a ~ -- z ~,(a) --~ 1 as a --, ~c and
~/is monotonic. (In probability theoretic terms, ~//is
a cumulative distribution function.) We say that ~' is
/-finite if ~/~: R ~ R is continuously differentiable of
order l (0 -< l < ~) and 0 < fID;~u(x)l d x < z , where
D;~// denotes the lth derivative of ~u. (D"~y = ~,: ~u
is assumed continuous when l = 0.) If 0' is a differ-
entiable squashing funclion, then ~ is 1-finite. but
not 0-finite. A continuous nondifferentiable squash-
ing function is not/-finite for any l.
Conditions on ~,. q,, and A,, ensuring the required
denseness are given by the next result.
LEMMA 3.2. L e t { q , ~ N} and {A,, ¢ R +} be increasing
sequences tending to infinity with n. If ~u is a s q u a s h i n g
function or ill q/ is l-finite .[or any integer l. 0 <- l <
~c, then U,; 1(0,,(¢/) is /)-dense (p = P2) in 6) =
L2(~ ', /;).
L e m m a 3.2 ensures that by p r o p e r choice of q/.
q,,, and A,, we can make (0,;(~,) sufficiently "big,"
avoiding underfitting. To avoid overfitting, ®,,(~,)
must be p r e v e n t e d from becoming too big too fast.
This will require restrictions on q, and A,,. The der-
ivation of these restrictions is rather technical, and
is postponed to the next section. A further restriction
on ¢/is also helpful. We say that ¢/satsifies a Lipschitz
condition if I~u(a,) - ~ff(a2)l <- L la, - a21 for all a,,
a2 ~ R and some L ~ R +. For convenience, let L
denote the set of all activation functions ~u: R --> R
such that ~, is b o u n d e d , satisfies a Lipschitz condi-
tion, and is either a squashing function or is/-finite,
0 -< l < zc. The following condition ensures that ®,(~,)
does not permit overfitting in the limit.
541
ASSUMPTION A.3. ®,,(~) = T(~'. q,,, A,,) where ~
L, a n d {q,;} a n d {a,,} are such that q,, and a,, are in-
creasing with n. q,, ~ ~c and A,, ~ ~ as n ---, zc
A,, = o(n l'~) a n d either (i) q,A,4 log q,,A,, = o ( n ) or
(ii) q,,A~, log q,,A,, = o(n~'3). []
We can now state the desired consistency result.
THEOREM 3.3. Given A s s u m p t i o n s A. l(i), A.2 and
A.3(i) or A.l(ii). A.2. a n d A.3(ii), there exists a
m e a s u r a b l e connectionist sieve estimator 0,,: f~ ~ 6)
such that
I' I 2 [}Z __ On(x,} u = [Tlirl " l ~ [}/r ('(X,)] 2,
tl l "~
Further, p(O,,, 0,,) _L~ (t ( i . e . , / b r all ~: > () P[~o E ~:
i,((),,(~,)). 0,,) > ~:] ---, 0 as n --, ~. l~
This theorem delivers the fundamental result that
connectionist network models are capable of consis-
tently estimating (learning) an arbitrary conditional
expectation function 0,, an element of the class of
functions square integrable on I'. The result gives
precise specifications for the growth in network com-
plexity sufficient to achieve consistency, although as
we saw in section 2 these specifications still permit
a wide latitude for choice of network complexity.
The consistency result holds for a wide variety of
activation functions such as the logistic or hyperbolic
tangent squashers, choices c o m m o n in practice.
Consistency of cross-validation-based estimates for
0,, obtains under an appropriate modification of As-
sumption A.3.
ASSUMPTION B.3. Let (~),,(~u) ?-'(~u. q,,. A,,) and
~,; = T(q/. FI,,, A,,) where ~ ~ L. a n d st-oppose that
{q,,}. {~,,} a n d {A,} are such that q,, < F/,,, {q_,,}.{~,,} a n d
{&,} are increasing, q,,--* z a n d A,, ---, ~c as n ~ ~,
~,, = o(n I'4) and either (i) ~j,,zX~ log ~j,,A,, = o ( n ) or
(ii) ~,,A,, log 0,,A,, = o(nJ'Z), t:
The consistency result for the cross-validated sieve
estimator is a straightforward consequence of Theo-
rems 4.2 and 4.1 of the next section, provided that
(2),,(~,) ~_ e),,(tu. 0,,) = T(~u, q,,. A,,) has an analytic
graph. This demonstration is straightforward, and is
relegated to the Appendix. We have the following
result.
THEOREM 3.4. Given A s s u m p t i o n s A. l(i). A.2 a n d
B.3(i) or A.l(ii). A.2 a n d B.3(ii), there exists a mea-
surable cross-validated sieve estimator {),,: ~ --~ O such
that
n ' ~ [Y, - [),,(X,)]: = min n ' ~ [Y, - O(X,)] 2,
i I ncg),/~ i i
n = 1,2 . . . . .
 542
where (a),,(q/) = 0,,(~', q,,) and gl,, is cross-validated
network complexity given in section 2, based on
0;, u, measurabel-FB
/(O_), t = 1. . . . . n, q = ~_t....
q,,, n = 1. 9
Further. p({),,, O,,) ~ O. [:
The data-driven cross-validation procedure for de-
termination of network complexity thus delivers a
consistent estimator for g, (i.e.. a learning procedure
capable of eventually approximating an arbitrary
conditional expectation to any desired level of ac-
curacy with probability arbitrarily close to one).
To obtain results for the case of approximate op-
timization, we impose the following condition.
ASSUMPTION B.4. Let q,,: ~ - - , {q . . . . . . . -q,,} be a
measurable mapping, n = l. 2 . . . . . and with [),,:
{~ --~ -0~ denoting a measurable solution to the prob-
tern mln~,~o,,(~.0,,i
- - " ~,, I [Y, - O(X,)] 2, let 0,,: ~ - - +
n i ~--~t= ..
0,, be a measurable m a p p i n g such that O,,(w) ~ ('),,(qs.
O,,(u))) f o r each ~o in ~ and
!,, ' £ IV, - o,,(X,)l - ,, ' £ IY, - a,,(x,)F):--
' I i
n = 1, 2 . . . . . where {d',,} is a sequence o f positive
real n u m b e r s such that ~,,--, L'< ~ 0 as n --, ~. m
We define the set 0 " ( 0 ~: {0 ~ O: IE([Y, - #(X)J ~)
- E([Y, - o<,(x,)DI = 10 o: z,(0, o,>)
(*'~-}. The desired result is
THEOREM 3.5. Given Assumptions A. l(i). A.2, B.3(i 1.
and B.4 or A.l(ii), A.2, B.3(ii), and B.4. for any
?. > 0 we have P[O,, ~ ®*(~<, + ~)] --+ l as n -+ ~. ! t
in addition C.,, = 0 then p(0,,.
"" 0,,) --->P (i. ~7]
The final conclusion delivers the consistency of i)~
for Oo when optimization becomes exact in the limit
(L, = 0). The first conclusion provides a precise state-
ment of the effects of approximate optimization. In
the limit, the approximate estimator belongs to a
neighborhood of the true mapping Oo with probability
approaching unity. The smaller is ~,, the smaller is
the neighborhood.
4. T H E O R E T I C A L FOUNDATIONS
This section contains theoretical results underlying
those of the previous section. A variety of other net-
work learning results can be obtained from the re-
sults given here. Our results follow as a corollary to
T h e o r e m 2.1 of White and Wooldridge (1990) (WW).
For simplicity, the result here is stated so as to be
easily applied to stationary stochastic processes Its
validity is not limited to this case, however. Where
possible, notation follows that of WW. Other nota-
ti. t,!,'hiu, <
tion and definitions are as in Stinchcombe and White
(1989b) (SW). We write B(.) to denote the Borel c~>
field generated by the open sets of the argument set
gr(.) denotes the graph of the indicated correspon-
dence, and A(-) is the collection o i analytic set~ cd
the indicated a-field.
TtlEOREM 4. I. (a) Let ([t, F. P) b~~a uornplete pr~#>
abifity space and let ((~), p) be a metric space, t.7,
n == [. 2 . . . . . let (~),, be a c(nnpleu, separable Bore/
subset Of 6) and let (}),: [~ --~ 6) bu a corresT)onden~e
with gr (a),, ~ A(F @ B(®,,)) such ,,hat f o r each u, in
~ ("),,(v~) C (%, and the set (~),.( ,~ i,~ n o n e m p t y and
compact Let Q,,: [~ × (.) -~ 7~ be F @ B(O)-mea-
.~urable, and s'uppose that Q,,(~u, } iv lower setnic(m-
tinuom' on (9,1or each ~,J in ~_~ n . I "~
Then f o r each n - 1.2 . . . . tltc~'u .<~tsls a.lumtimt
[),,: [t ~ (% measurable-F/B((%)Ihence-F/B(@))su(:h
that Q,,(u,. b,,(~,~)) = rain ....... Q,:(,,:. tl)/'or all ~
in ~.
(b) In a d d i t i o n , s u p p o s e I~,,} a n d ~ ,,i are increas-
ing sequences o[" compact subsel~ qt (') Such thai
; .... U); ('),, is dense in (~) and tO, ~ (~L(~u) C (7), lbr all
] ,, ,, . . . . .
~,J in [L n = l..~• . . . . . Suppose there exists ,)Urlction
Q: b) ---, :R ~uch that f o r all e ;. !1
PI": sup iCLI,,, nt -- (_)tO)i - ei
a~ ~z • .... (4,11
a n d ) b r 0,, C (~)
inf 0(01 - Q(0. i (k (4.2i
.H0 ,
where #'(0,,. c) ~ 10 ~ O: p(#. (!,,) ~ ~:}, atui Q is
continuous at t).,. ]'hen p([L, 0,3 -L (I :!
In our application. (~l. F. P) is tile space on which
the stochastic process {Z,} is defined. The p r o p e m e s
of P determine whether {Z,} is an independent or a
mixing sequence. The space e) contains the object
of interest {the unknown regression function 0,), and
p is a metric that measures distance in this space
(weighted mean squared error). Q, is the criterion
function (squared error/ optimized to arrive at an
estimator 0,. The set O,(¢u I over which optimization
is carried out may depend on the data through u):
this permits treatment of cross-validation proce-
dures. In some applications it may be natural-to have
Q,, defined only on the graph of ~,(c,~) or on f~ ×
O, instead of on all of F t x O as we assume. L e m m a
2.1 of SW establishes that defining Q, on [) × O
results in no loss of generality, as there generally
exists an appropriate measurable extension to ~) x
O of Q, originally defined on gr (9,, or f/ x (9,,.
Part (a) establishes the existence of a measurable
estimator 0.. Without measurability w e cannot make
probability statements about 0,,. such as statements
Connectionist Nonparametric Regression 543

about consistency. Part (b) establishes consistency of for all 0 in q,,(0") ~ {O E @,,: p(O. 0 °) < d,,(O")}.
(),, for 0,. The object 0, is distinguished by its role as Put ~,, >- sup:ej sup,e~.,,, Is,;(z, O)l and t-~,,
minimizer of Q (condition (4.2)), the limit to which sup:c, supoe,.,,, m , ( z . O). Put ~J,, ~ inf,~:,.,, d,,(O) ~, and
Q,, converges uniformly (condition (4.1)). This uni- suppose ~,, >- d,, ~.
form convergence can be verified in particular sto- Let {Z,: [~ --+ I"} be a stochastic process on ( ~ , F,
chastic contexts; our next result permits this for i.i.d. P). (i) If" {Z,} is an i.i.d, sequence then f o r any c >
and stationary mixing processes. The other notable 0 and Jbr all n sufficiently large
assumption of part (b) is the existence of nonsto-
chastic sets @,, and ~,, bounding ~,,(~). The behavior P sup In I + [,,,(z,, o) - E(.,.,,(z,, 0))11 ~>
;;J1
of (% ensure~s that (~),,(~,o) becomes sufficiently dense
in (4. so that an element of (2),,(c,9) can well approx- <- 2G,,([c/6-~,,]' ')[exp(- (3n/7)
imate an element of @. T h e behavior of @,, ensures
+ cxp(-;::,;/~:~,.lls + 4;:1)].
that (~),,(u)) does not increase too fast and that Q,;
converges to Q uniformly in an appropriate sense. If in addition {@,,} is such that n ~ 7~, --, 0 a s n - - , ~c
The constants q_, and ~,, of the previous section de- and f o r all c > 0
termine e),, and @,, in our application.
We now give a result permitting verification of (,v~/,) H,,([;:,"6~,,]' ' ) ~ II as n--~ ~ (4.3)
condition (4.1), related to Corollary 2 of Haussler then f o r at O, ~: > 0
(1989). Instead of using the concept of V - C dimen-
sion (Vapnik & Chervonenkis. 1971) we use the con-
P [ s u p .... In ' 2 a [s'(Z"
cept of metric entropy ( K o l m o g o r o v & Tihomirov,
1961" see also Lorentz, 1966). The metric entropy of
a closed set K. d e n o t e d H(c), is the logarithm of the E(s,,(Z,, 0))]i > e: J ~ 0 as n ---. ~.
n u m b e r of sets of radius c (with respect to a specified
metric, not necessarily the same as that in T h e o r e m (ii) ~{f {Z,} is" a stationary mixing process" with either
4.1) required to cover K. We also make use of ex- ¢ ( k ) = O,,Pf, or a ( k ) = aopf,, 0~,, ~, > (I, 0 < p,, <
ponential inequalities awfilable for both independent 1. k >- 1, then there exist constants 0 < cj. ce < sc
and d e p e n d e n t stochastic processes. To specify pre- not depending on n such that f o r any ~: > () and all n
ciseh' the allowed d e p e n d e n c e , we utilize mixing sufficiently large
measures of stochastic d e p e n d e n c e , in particular.
uniform ( d ) - ) and strong (ct ) mixing. These are P sup 1,7 ' [s,,(Z, 0) - f(s,,(Z. 0))1[ > ;:
defined as L 0 c (-I / I

c,G,,([;;/6hT,,] ~')[exp( -c4t'-~) + e x p ( - c.¢-n' :/6f,,)l.

rh(k) = sup, sup,, u:;,ch' ~,.,,~ ,,~ iP(BIA) - P(B)[
If'ill addition {@,,I is such that n i 7{,, 3 __+ () as it ~ :c
~t(k) = sup, sup,, ~:.¢,;. ~= !P(A (1 B) - P(A)P(B)]
attd f o r all c > 0
where F~ ~ a(Z, . . . . . Z,) is the a-field generated (L ?n )H,,(IUOm,, 1' ") --, I~ as tz -+ :~ (4.4)
by {Z~ . . . . . Z,}, and F7 =- a ( Z . . . . . ) is the a-field
then .for arty c > 0
generated by {Z,, Z,+, . . . . }. For a discussion of oS(k)
and a ( k ) and the properties of mixing processes {Z,}
(i.e.. processes for which qS(k) ~ 0 or a ( k ) ~ 0 as
k --+ z), we refer to White (1984). For our next result,
P [~ up n ' 2 [s,,(Z,. O) - E(s,,(Z,. 0)) 1 > ;:] ---, II
t c e~ n ~ I

the metric p need not be identical to that of T h e o r e m
4.1.
THEOREM 4.2. Let ( ~), F, P) be a complete probability
space, let (e), p) be a metric space, and let {(%} be a
(nonstochastic) increasing sequence o f separable sub-
sets" o1" @. Let H,,(c) be the metric entropy o f ®,;, and
put G,,(~) ~ exp H,,(c).
Let {s,,: I' x @,,---, R. n = 1, 2 . . . . } a n d { m , , :
]' x @,,---, R +. n = 1,2 . . . . } be sequences o f func-
tions such that s,,(., O) and m,,(., O) are continuous
on 1"for each 0 in ®,,. Suppose there exists a sequence
{d,,: @,, --> R ~} and a constant ). > 0 such that f o r
each z E I' and 0" in @,,
M , ( z , O) - s,,(z, 0")i < m,,Cz, O")p(O, 0");
CIS t l ~ ~c.
Except for the metric entropy conditions (4.3) or
(4.4), the conditions of this result are straightforward
to verify in applications. The metric entropy H,,(c)
is merely a little tedious to determine in most ap-
plications; our next result provides bounds for our
application. Given H,,(¢:), choosing (% appropriately
is fairly direct, allowing verification of condition (4.1)
with Q , , ( . , O) = n i y , , , s , , ( Z , , 0 ) .
A (loose) bound for the metric entropy of single
hidden layer feedforward networks is given by the
following result.
LEMIVlA 4.3. Let q/ C L, q ~ N attd ~ ~ R* be given.
Let p~ denote the tmtform metric, p.(Oj. 0:) =-
.544
sup,ee tO, Ix) - O2(x)], 0,, O: ~ T(~,, q, ~ ) , and let
H(~u, q, A) d e n o t e the m e t r i c e n t r o p y o f T(~u, q, ~)
with respect 1o p.~. Then f o r all ~: > 0 sufficiently s m a l l .
t t , ( ~ u , q , ~ ) :---plogSh: + p l o g l , X -* rL,X] 4 p logt,
wherep~q(r + 2) ~- 1. ¢'.
To apply T h e o r e m 4.2 to connectionist regression.
we must specify Q .... s,, and m,,: from these L, and
n-5,, n e e d e d for (4.3) and (4.4) follow. T h e m e t h o d
of least squares is i m p l e m e n t e d by taking
Q,(.. O} = n ' <" ['}', H(Xr}]'.
I
so that s,,(z, 0) = [y - 0(x)] e. Because (a e -- b:)
= (a + b)(a - b). we have
ls,,(z, O) - s,,(z, o")[
-<~ 12>' - (o(x) + o ' ( . r ) ) l , io(.v)
:: ( sup 2iv - 0(x)[)t~,(0, (Ire) m
so we take m,,(z, 0) = sup0~%(,,,! 2iY - 0Ix)i, ). :- l
and d,,(O) = l. W i t h o u t loss of generality, we m a y
take sup,~xl0(x)] -> 1 (this s u p r e m u m is finite because
the Lipschitz condition of qx ensures the continuity
of 0), so that
sup m,,(z, O) <- sup sup 2[yi + 210(x)i
> sup sup 4[0(.r).
as tY[ -< 1. As ~ is a s s u m e d b o u n d e d , we may take
the b o u n d to be unity, without loss of generality. We
then put ~,, = 4&, -> s u p , e , sup,,~,t, ,,,, 4t0(x)t. Sim-
ilarly,
Iv - O(x)]: ~ v: + 2b'1" 10(x)l + 0Ix): :~ sup 40(.~)-.
and we take L, = 4A~, ~ s u p , ~ , sup(,~_~.,,,~, 40Ix):.
Provided that the metric e n t r o p y conditions (4.3)
and (4.4) hold, we obtain from T h e o r e m 4.2 that
(4:1) holds (i.e.. Q,,(.. O) converges uniformly to
-Q(O) = E ( [ Y , - 0(X,)]2)). Using L e m m a 4.3. we
obtain conditions on {q,} and {A,,} ensuring the va-
lidity of the metric e n t r o p y conditions (4.3) and
(4.4).
]_.EMMA 4.4. L e t {q,, ~ N} and {d~,, E N ' } be s e q u e n c e s
such that {A,,} is increasing a n d & , - - , ~c as n --, ~c.
Put L, = 4A,~, N,, = 4A,,, 2 = 1, a n d f o r given tu
L a n d ~: > 0 p u t H,,(r.) = H,:(~u, q,,, &,).
(i) I f q,, ZX4,,log q,, zX,, = o ( n ) , then f o r any c > 0
(L;'-In)H,,([c/6-~,;] '~) --, I) as iz--, ~c:
(it) I f q,, A~ log % A,, = 0 0 ? ' 2 ) . t h e n f o r a n y
(L,/n~:)H,,([d6-~,,] ~;) --+ 0 as n --+ ,z.
i/ tthih
H e r e we see the growth rates for network complexity
a p p e a r i n g in A s s u m p t i o n s A.3 and B.3. T h e growth
rate for A,,, & -: o(n~4), follows from the require-
ment in T h e o r e m 4.2 that n .s -: ..... u. with ;' set ~o
4,x~.
] ' h e result f o r a p p r o x i m a t e o p t i m i z a t i o n loltows
as a c o r o l l a r y to T h e o r e m 4. t
( ] O R ( ) I A A R Y 4 . 5 . L e t the c o n d i t i o n s a~td d e l t n m o n s
ot T h e o r e m 4.1 hohl. k o r 5 .c: Ip. & ' l i n e (~) I~,~. 2
l0 C e),,(-)): tQ,,(¢o, u) Q,,(-,-~).(~',))l "- ~{ and
~-~ t:~ u, ¢ e): Q ( 0 ) Q(0,.) ~ 2}. l.c, {~ ::
m ?satisl3'~ .-~ >tlasn~ ~ and l e t , d . . ~).----
~r'~ he rt s e q u e n c e o1 measurabh" /)otclicm.s such thal
O,(c,~) C ('),.l~'~. 5,1. ~t - 1_ 2. "fJum for alf :
tl_ P[~) C (*):'(; - £ ) 1 - ~ 1 as ~t - - ' . It in additimt
IL then .~if),,. u , I - G (L
o"(x)i
5. SUMMARY A N D DIRECTIONS FOR
FURTHER RESEARCH
O u r results resolve the learnabiht~ issue for mult>
laver f e e d f o r w a r d n e t w o r k s by establishing the con-
sistencv of connectionist sieve estimators l'o~ an un-
known regression function. These networks thus
possess n o n p a r a m e t r i c regression capabilities Net-
work complexity plays a f u n d a m e n t a l role in our
a n a b s i s . We consider both deterministic and data-
driven m e t h o d s for controlling the growth of net work
complexity as a function of n e t w o r k experience. T h e
data-driven m e t h o d s ate based on ,, cross-validation
m e a s u r e ot n e t w o r k p e r f o r m a n c e , cross-validated
average squared error, that we advocate f o r general
use m evaluating network p e r f o r m a n c e . This is not
the only a p p r o p r i a t e or useful m e a s u r e i t . g . . ~e¢
Barron. 1989~. but it offers a considerable i m p r o v e -
ment over naive methods. We also consider the im-
plications ot a p p r o x i m a t e rathm than exact o p t n n >
zation of n e t w o r k p e r f o r m a n c c .
Consistency ~s a minimal p r o p e r t y tor a learning
rule: the analysis here is inherently limited by our
focus on this property. T h e r e are n u m e r o u s nnpor-
t a m areas for further investigauon. Especially desir
able is further analysis regarding the rate of conx.e"-
gence of connectiomst sieve estimators. A m o n g other
things, this will help m a k e possible m o r e i n f o r m e d
application of the p r o c e d u r e s described here. T h e
results and a p p r o a c h of Severmi and W o n g (1987}
may prove helpful in this investigation. Such results
will permit further evaluation ot the trade-offs be-
tween ~,, and q,, required to ensure c o n v e r g e n c e at
the best possible rate. and will facilitate c o m p a r i s o n s
with other n o n p a r a m e t r i c regression m e t h o d s , such
e >
as kernel (Bierens. 1987) and sptine (Cox, 1984)
methods. Rate of c o n v e r g e n c e results m a y also be
u useful in establishing a s y m p t o t i c optimality p r o p e r -
C(mnectionist Nonparametric Regression
ties (as in Li. 1987) for cross-validation procedures.
A n o t h e r area for further research is greater auto-
mation of consistent and optimal procedures. De-
veloping and i m p l e m e n t i n g well-behaved m e t h o d s
for computing ()',{~,~is especially i m p o r t a n t for prac-
tical application of the cross-validation approach.
Straightforward extensions of our analysis yield
n o n p a r a m e t r i c estimators of conditional functionals
other than the conditional expectations studied here,
such as conditional variance or conditional quantiles.
Thcse can be estimated with p r o p e r choice of Q,,. as
discussed in WW. In fact, it is possible to use multi-
laver f e c d f o r w a r d networks to obtain n o n p a r a m e t r i c
estimates of joint or conditional density, using the
a p p r o a c h described by G e m a n and H w a n g (1982.
section 6). Such multilayer feedforward networks
would be essentially self-organizing, and could be
used in pattern completion p r o b l e m s in which arbi-
trary parts of the pattern are supplied and the ar-
bitrary r e m a i n d e r has to be completed.
Finally. we mention that it is often useful to have
awfilable statistical confidence regions for linear or
n o n l i n e a r functionals of the n o n p a r a m e t r i c esti-
mator. For e x a m p l e , we might want a 95% confi-
dence interval for the conditional expectation of Y,
given X, = x. a specified value. This interval may
be obtainable using m e t h o d s similar to those of An-
drews (1988).
MATHEMATICAL APPENDIX
Unless otherwise noted, all definitions and notations
are as given in the text.
Proo/" o/' L e m m a 3.2. Let E'(~,) -= {g: E,' ---, ~:
/ = 1. . . . . q, q ~ HI. It follows immediately from
T h e o r e m 2.4 of Hornik. Stinchcombe, and White
(1989) or Corollary 3.6 of Hornik, Stinchcombe. and
White (199(i) and T h e o r e m 3.14 of Rudin (1974) that
V'(V/) is/)e-dense in Le(F, I0. Let g: t ' - - , R be an
arbitrary e l e m e n t of E'(,I/), so that for some q ~ IN,
/li ~ R , / = (). . . . . q, 7 / ~ R ' + ' , / = 1 . . . . . q, we
-t
have :,,(x) = /)'<, + W/, ,/:ig/(x ,'i). Because q,, z , ~ N (so T(~,, n, A) E B(("))). ~i
and A,,--. :~. we can always pick n sufficiently large
that V'/ ,, [[:,1 -< A,,, V'/, E: ,, 17,1 -< q,, A,,, and
q =-~ q,,. Thus. for n sufficiently large g belongs to
e),,fu/) = T(~u, q,, A,,) and therefore to U , ; , ("),,(~:).
Because ,g is arbitrary, V ' ( ~ ) C U,; a (9,,(~/~). It fol-
lows from t h e / > - d e n s e n e s s of E'(~u) in L 2 ( F . / 0 that
U,{ , f"),,(u:) is p:-dense in L2(X ' . / 0 . c~
Proof o / Theorem 3.3. We apply T h e o r e m 4. I. By
A s s u m p t i o n A, 1, ([L F. P) is a c o m p l e t e probability
space: ((9, p) = ( L : ( I ' , / 0 , P:) is a c o m p l e t e separable
metric space (e.g., K o l m o g o r o v and Fomin, 1970,
T h e o r e m 37.5, Problem 37.4). Consequently, we may
545
take O,, = (9, n = 1, 2 . . . . . e),,(.) = (a),,(~u) -- T(~u,
q,,, A,,) is n o n e m p t y and c o m p a c t (for all n suffi-
ciently large) given A s s u m p t i o n A.3, and ~,, is an
analytic c o r r e s p o n d e n c e because it is nonstochastic.
Q,,((o, o) n ' N,~,:,[Y,(~o) '¢t
O(X(~o))]2 is F × B
-
(e)) m e a s u r a b l e as a consequence of L e m m a 2.2
of Stinchcombe and White (1989b) (SW) because for
ever}, ~,) in (] Q,,(c,), .) is (/,: - ) continuous and for
every 0 in the separable metric space (9 Q , ( . , 0) is
measurable. Continuity of Q,,(~,), ") implies lower
semi-continuity. The existence of a m e a s u r a b l e con-
nectionist sieve estimator 0,,: ~ ---, (9 now follows
from T h e o r e m 4. l(a).
By A s s u m p t i o n A.3, {(% = (% T(I//, q,,, A,,)}
is an increasing sequence of c o m p a c t subsets of (9.
By L e m m a 3.2 U,~ 1 e),, is dense in (9. A r g u m e n t s
sketched in the text of section 4 and L e m m a s 4.3 and
4.4 apply to establish the existence of Q: (9
x, Q(O) = E ( [ Y , - O(X,)]:) such that P[u):
sup,e_%lQ,,(u), 0) - Q(O)I > ~:] --~ 0 as n --~ ~, as a
consequence of T h e o r e m 4.2, given A s s u m p t i o n s
A . l ( i ) and A.3(i) or A . l ( i i ) and A.3(ii). Now Q(O)
- Q(0,,) = p(O. 0,):, so inf,e,:u,,.,Q(( 0 - Q(O,,) =
inf,~_,,,,,,,~ p(O, 0,,): -> ::: > () and Q(0) = p(0, 0,,) 2
4- Q(O,) is continuous at 0,,. That i~(0,,, O,,) ~ ' 0 now
follows from T h e o r e m 4.1(b). ~i
The p r o o f of T h e o r e m 3.4 m a k e s use of the following
lemmas.
LEMMA A. 1. Let (e), p) = ( L:(~'. lt)./):) and let f ( ~ ,
q, A) be as in Definition 3.1 with ~// E L. Let ( ~ , F)
be a measurable space attd let q: ~ --+ N be measur-
able-FIB(N), where N C N and H is endowed with
the discrete topology. Then ~'or every ~u and A, f(~,.
0, A): ,Q ~ (9 has gr T(~,, c"t, A) C F (~) B((9).
t'roq/~ For each i// and A. T(q/, q, A) is a c o m p a c t
set, as it is the continuous image of the c o m p a c t set
:~ × l ~, B : {/:: ,_:/,, I/q -< ~}. :' - {.,: "~;:, 5:; ,, I;'.[
< q A/. Then gr T(~/, ~. A) _ U,,~((/ '(n) x T ( ~ ,
n, A)) ¢ F @ B(e)), because 0 is m e a s u r a b l e (0 '(n)
C F for all n C N) and T(q/. n, A) is c o m p a c t for all
LEMMA A.2. Suppose Assumption A.1 holds, and
p.: N,, = [q,,.kT,,]n N / o r ~/,,.?,, ¢ i~. ~,, _< ?,,.. =
l, .~. . . . . For each n = 1. . . . . . . t = l ..... n
and q E N,, let O,,t, be measurable-F/B((9). Then for
each n there exists q,,: ~---" N,, measurable-F/B(N,)
such that
4', ~ N,. -= argmin 17 t . [Y. - O,/,,,, (X.)]:,
q \,,, ' I
attd .for all ¢,J in ~ and q,, C N, we have q,,((,J) <
:l,,~u)). E
546 ti. WIute

Proof. We apply Corollary 2.4 of SW. Condition (i) It follows by argument identical to the proof o;
of SW holds given Assumption A. l, and condition Proposition 2.4 of WW that therc exists a nonsto-.
(ii) holds as (N,,, B(N,)) is a complete separable chastic seq_u_ence {0~ ~ O,} such that: 0.~ is a solution
metric space under the discrete topology. For each to inf0~o,, Q(9): for all ~; > ~
q in N,,, Q,(., q) -= r/--1 W~'.:~ [Y, - 0,q,u~(X,)]= is
measurable-F given the measurability of the ele-
lira inl.. { ,,, ,,~,..int~ Q(01 £J{u ~ .~ - 0.
ments of {0%)} and {Z,}. Because N, is a finite set
Q,,: f~ x N, --+ R is measurable-F @ B(N,,), so that
where rr(O;,, ~:1 ~ {0 C O: p(O, 0,,} ~ c}, and p(Oi,
condition (iii) holds. Condition (iv.b) holds as N,
does not depend on o~, so that N,, is an analytic set.
0,,) --, 0 as n --+ ~. Because (% C (~L.(e)), {07,}satisfies
(A.I).
Because N, is a finite set, a minimum exists. Thus.
Because ~/'(0,,. el contains ~',I(#Z ~;I = {fl ~ {aj,,:
by Corollary 2.4 of SW, q,, ~ N~ and 0~ is measurable-
F/B(N,). Further, the arbitrary selection of Corol-
p(O, 0;,) >- c}, it follows that
lary 2.4 can be taken to be such that for all c,) in ~
and q,, ~ N~ we have O,,(co) ~ O,,(o)), I~ lira inf.. { inf Q(0) -0(01) } > u.

Proof of Theorem 3.4. The proof is analogous to that By assumption P[¢u: sup0eo,,iQ,b,u- ol Q(O)! _-. c]
of Theorem 3.3, except for some details. Lemmas -+ 0 as n --, r: for any ~; > o: because O, C_ {4,,, we
A . 1 - A , 2 immediately imply that gr ~),,(~u) ~ F @ also have # E O . It now follows immediately from
B(O) C A ( F @ B(O)). The denseness of U~:~ O,, the argument of Corollary 2.3 of WW that
follows from Lemma 3.2 under Assumption B.3. By
construction, _O, C O,,(~,) C_ O,,, as ~/,, -<q,, s ?t,,. P* ] ~u: ,~,,;,,inl,,:. Q,,(~u.u, Q,,{(u.,,, ,,i-,i
Condition (4.1) holds by Theorem 4.2 for {O,}, using
the same argument as in Theorem 3.3. As the rest , t b ,rt ....

of the argument is unchanged, the conditions of Because O,M,9) C 0,, q,,(O,,, cl contains t/~t#,,, ~:. ;v}
Theorem 4.1 hold and the result follows. ~' so that

Proof of Theorem 3.5. The result follows immedi- c,~: inf Q.,(uJ. tl} - Q.,Ro. u..I - :~I
ately from Theorem 4.5, as the conditions of Theorem
4.1 hold as established in the proof of Theorem 3.4.
~2 c { ,, u ._inf
:,:.~ , Q,
, ,({u.
~ Q,,(ua.o~)>~}.
Proof of Theorem 4.1. (a): The result follows im- It follows immediately from the monotonicitv of P~
mediately from Corollary 2.4 of SW, as we directly that (A.2) holds, so that p(O~, 0~.) ~ 0 bv Theorem
impose their conditions (i)-(iii) and (iv.b) on (fl, F) 2.1 of WW.
and (O,, p). (Completeness and separability of (% Because ~),, is measurable (by ta) above), p(O,.
ensure that it is Souslin, condition (ii).) We also im- 0),) ~ 0. Because p(O~,, 0o) ~ o_ the triangle in-
pose their conditions (v.a) (~),(e~) nonempty and equality yields p(O,,, 0,,) P-%O.
compact) and (vi.b) (Q,(c9, .) lower semi-continuous
on O, for each co), which ensure that {~u: Q,,(co, . )
achieves its infimum in (a),(co)} = D. Proof of Theorem 4.2. We verif.,, the conditions of
(b) We first verify the conditions of Theorem 2. t Theorem 2.5 of WW. The conditions on (~-L F. P~.
of White and Wooldridge (1990) (WW). The con- (O, p) and {O,} are imposed directly. The functions
ditions imposed in (a) suffice for all of their condi- s,,(Z,, .) and m,(Z,, -) correspond to s,, and m,, of
tions through their (2.1) and (2.2), as 0,,(eg) mini- WW. The required measurability and Lipschitz con-
mizes Q , ( ~ , 0) in ~),(co). It remains to verify their ditions for these hold by assumption. We take M, of
(2.3) and (2.4), that is, there exists a nonstochastic WW to be M, = n ~ , .
sequence {0~ ~ O} such that Under conditions (i), ({Z,} i.i.d.) it follows from
the Bernstein inequality (e.g., Proposition 3.2 of
P* [co: 0~, ~ ~)~(o9)] -+ 1 as n --~ ~ (A.1) WW) that
and for all ,: > 0
P[ 2s~(Z,,O)-E(s.(Z,,O))I
1 , :~X
(o: inf
0~:nc(0~.eaol
Q,(o~,0) - Qo(eo, 0;,) > 0

asn~oc,
i 1

(A.2)
where ~/~(0~, e, o~) =- {0 ~ O,(co): p(O, 0~,) >- c} and
P* is the outer measure associated with (fl, IL P).
< 2 exp[-- '~-'/(2a~ - 4~,,&/3)1,
where a~ ~ var(Z%~ sn(Z,, 0)), and we use the fact
that Is,(Z,, O) E(s,(Z,, 0))1 <- 27°. Further. a~ <
n ~ , and we may take ~,, -> I (implying L, ~ ,V~)
Connectionist Nonparametric Regression 547

without loss of generality, so that P r o o f of L e m m a 4.3. We construct an e-net for T(~,

P[ ,~ s,,(Z,, 0 ) - E(s.(Z,. 0 ) ) > A]
-< 2 exp[-A2/f~,(2n + 4A/3)].
Note that this bound is independent of 0, and define
Ff,(A) -= 2 exp[-AZ/?~(2n + 4A/3)]. Application of
the Bernstein inequality with m. in place of s,, leads
to an analogous inequality,
P{ ,~_~m,,(Z,,O) - E(m,,(Z,,O)) >A}
~< 2 exp[-AW(2n~ + 4~,,A/3)].
This bound is also independent of 0; we define
Fm(A) --= 2 e x p [ - a 2 / ( 2 n - ~ + 4~,A13)1.
Theorem 2.5 of WW applies to yield the desired
inequality, provided that n = O(M~ d,). Because
M,, = n ~ , and ~,, -> d , t , we see immediately that
this holds. Consequently, by Theorem 2.5 of WW it
follows that for all e > 0 and all n sufficiently large
q, A). (Recall that T,. = {h . . . . . th} is an e-net for
T if each 0 in T there is at least one element tk of T,:
such that p~(O, tk) <- e.) Then H,(~', q, A) _< log h
(e.g., Lorentz, 1966, chap. 10).
Let r / > 0 be given, and let B, -= {b~ E B, k = 1,
. . . . /}, C,-= {G ~ F, k = 1. . . . . m} be (l~ -
norm) r/ - nets for B = {fl: N[~] -< A} C R q+' and
F = {7: I])'N -< qA} C R <//'+~. (For notational con-
venience ))'ll here denotes the l~-norm in spaces R" of
whatever dimension.) Let D, = B,~ x C,, and put
T,~ = {0 E T(~,, q, A):6 E D,} in an obvious use of
notation. We choose ~l so that 5¢, is an e-net.
Let 0 ~ T(qJ, q, A) be arbitrary, with corre-
sponding parameters d = (fl', 7')'. Then there exists
d = (b', c')' in D,, such that lift - bl[ ~ q and 117' -
cl[ -< r/. Let t be the element of 7~, corresponding to
d. The triangle inequality gives
10(x) - t(x)l = fl,> + ~ fl,~,(~':,,) - l,,,
t I

~,(.i'c,) -< - b. + (fl, - b,)~,(2

r I1
- b,
P sup n ~ _ [G(Z,, O) - E(&(Z,,O))] > e
i Oe I')~ I : I

<_ G,,([eX6"~,,]";)[I'2(2M.) + I';,O;n13)l. + 2 b,(~,(~'i,,)- ~(X-'c,)) .

Now F."(2MD = 2 e x p [ - 6 n / 7 ] and E~(en/3) = 2

For the first term, the fact that ~, is bounded (set the
exp[-e2n/~(18 + 4e)]. Substituting these expres-
bound to unity) and the triangle inequality give
sions into the expression above gives the first result
of part (i).
To obtain the second result (uniform convergence /7,, - b,, + ~] (/7, - b,)~,(.~'~,,)
S I
to zero), it suffices that
G,,([U6~,,]"~)exp[-6n/7] ~ 0 as n --~ ~c 2 Ifl, - b,I = II/S - bll-< ~.
/ II

and
For the second term, the triangle inequality gives
G,,([el6~,,i'~)exp[-c2n/~,(18 + 4e)] ~ 0 as n ~ oc
for any e > 0. Because the force of the condition b,(~,(~ ),) - ~,(~'c,))
j=l
occurs when e is small and because we take ~. -> 1.
the second condition suffices for the first. For this it <- ~2 Ib, I1 ~,(.~'~',) - ~,(x'q)l
suffices that i=1

c'~nl~,(18 + 4r.) - H.([e/6~,,] ''~) ~ zc as n --* oc Ib) ku(2;,,)- ~,(2'c,)l

for any t: > 0. Now
e2n/~,(18 + 4r) - H,,([e/6N,,]l;-) s=l

=(n/~)[c2/(18 + 4~:) - (g~,/n)H.([e/6~o]' ~)]

The Lipschitz condition imposed on ~u and the tri-
>- (n/d~)[e2/2(18 + 4e)].
angle inequality give
where inequality holds for all n sufficiently large be-
/~'(.~'~',) - ~(~'c,)l-< L I~'~', - x-','.,
cause (~/n)H.([e/6-~.] v;-) -+ O. Because n - l ~ ~ 0
we have n/-g~, ~ oc, and the required divergence is
= L ~ £,(;',i- c,i)
verified.
The argument for the mixing case (it) is analogous,
using a mixing analog of the Bernstein inequality
I =l) i tl ,
(Bosq, 1975, Theorem 3.3 of WW). The condition
(~.lnl/'-)H.([el6-~t.] ~/x) --~ 0 can be similarly verified
<- rL ~ 17n - c,il.
to suffice for the uniform convergence result, c~
 54?t f~ ~l h m

Consequently, E';,=~ ]~(.f'i,;) - g/(2'ci) I _< rL E'/=~ P r o o f ot Corollary 4.5. S u p p o s e that for s o m e .:
E";=)!;;;i°' - c;;t -< rLq, so that l) P[O. ~ t')"(L" - ~)]--~ 1. T h e n I'[t:1 = P[,,; {')..(,,.,)
e)"l J, ~ ,j) i.o. ] > 0. Pick ~.~ ~ t" a n d a s u b s e q u c n c c
b;(y;(x'7,) - ~( 2'c,))' ~; rL ~;1. ~;)'} of {n} such that 0,,,(t.o t ~ (9*f:J ' L:). B e c a u s e
sup,,~,, IQ.,(- o) - Q ( O ) ] - L ( ) a n d ;,(~;.. ~).) .... I) ,.~,
It follows that Call w i t h o u t loss of g e n e r a l i t y pick c. ~:_ t. and u Im-
ther s u b s e q u e n c e In"} such that sup., ,., Q,,t<,;. :;)
p,(O, t) ~ sup/0(x) - t(x)[ -<_-d l + rL A) = ;:.
~61 r Q(()I ~ 1) and 140 ,.(t,~l. u.) --->(1 (~'.~.' . T h e o r e m 2.4.4
of L u k a c s . 1975). By the triangle inequality
p r o v i d e d that we c h o o s e q = e/t1 + rL ~). B e c a u s e
0 is a r b i t r a r y a n d t ~ 7~,;. Z ~- "/~:,,, ,;.a) is an c-nel (5l,.(,,)) 0(~;.) --,0(0..,o,)~ .... -~ ~,,,~)
for T(~u, q. &). () ,(<,), u,,,(~,;)) t~ ~ - i;';)~
Let # be the cardinality o p e r a t o r . B e c a u s e +-'(~.{~.'~- ;;.("~)) Q(O.. (<'~)) , ;( ;; ~,',), ?_)(<; )
#7~; = # D , ; a n d D,; = B,; x C , ; , w e h a v e #/~,; =:.
Bv u n i f o r m c o n v e r g e n c e , the facl that p(O.(v?). ~)r ~
( # B , ; ) ( # C , ; ) . It follows f r o m T h e o r e m s IX. X. a n d
-~- (L a n d the c o n t i n u i t y of Q a~ ,~ . u follows that
V of K o i m o g o r o v a n d T i h o m i r o v (1961) a n d ele-
g w e n any ;: "-> t) we have for all n' ~ufficientlv large
m e n t a r y g e o m e t r i c a l a r g u m e n t s that for all c ( h e n c e
that
q) sufficiently small #B,; <-- 2 ( 2 A / q ) ';+ ' a n d #C,; ~:-
2(2q~/q)';I'+'L T h e r e f o r e with ;1 - ;:/(1 + rL ,.k). Q(O,,(.J)) ()(-)
p = q(r + 2) + l, w e h a v e - Q,,(o). U,luJ))

log#T ~ l o g 4 + [q(r + 2) + l]log2~;';/ so that for all n" sufficientl~ lara~

+ q(r + l ) l o g q [ 0 . ((,J. tL.((,Jil (J,.( ~.;11
l o g 4 + p l o g 2 / ~ : + p l o g ( A + rL ~ ) + p l o g l ?.
C)(O,,((,/I) - L)iu ) ;
B e c a u s e H,:(~, q,/.X) = log # T ; a n d log 4 < p log 4 J 3c.
we h a v e
b e c a u s e IJ,, (~,J) ¢ el*(4 c B u t by c o n s t r u c t i o n
H,(~,. q, 20 -< p log 8/;: + p log(~ + rL ~'-) + p log p. of ~}, (c,)). we h a v e IQ,,.(t'>. ;;,.(t,>)) Q,,4t,J. d,.(eJ))
_, ~ _ -- _= 3~: for ~: > i) sufficiently small a n d
all ~l" sufficiently large. ~, c o n t r a d i c t i o n . H e n c e . lor
P r o o f o f L e m m a 4.4. (i) With N,, = 4&,, and ;. = all ~ ~-I). PRO,, C (@~(.L, 411 .... !.
1, for any ~: > 0 W h e n J -= 0. we have that P [ 0 . ~ e)*(d)] --, t a~
H,,([~:/6-~,,] '~') = H,,(;:/24~,, } ;i -~ :'- for all c - {1. It follows from 14.21 that for /
all , > () t~/~ E ;/(p'.,, cl[-~ 1. ~o that p(~/...u )--,
-< p,, log 192 &,/c + p,, log [&,, + rL ',.k~,] + p,, log 17, ().
f r o m L e m m a 4.3. B e c a u s e ~,, --, ~, for any ~: > (I
t h e r e exists N,: sufficiently large that A,, ~ 192/r for REFERENCES
all n > N,. A l s o , for all n sufficiently large ~ , ~ &,
Andrews D.W.K. { 19881. Asyntptott( ~ormaitt. of ser~e.~ esu-.
a n d ~ , > (1 + rL). W i t h o u t loss of generality sup- nlators ffir various nonparametric trod semt-parametric e?ll-
pose this is true for all n > N,:. C o n s e q u e n t l y . mators Cowlcs Foundation Discussion Paper 874L Yale Uni-
versl iv.
p,, log 192 &,/c + p,, logl&, + rL .X~] Baba. N. ( 19891. A new approach tor finding the global mJ m m u m
<- p,, log A~ + p,, log A~,(1 + rL) of crror fu action of neural networks. Neural Networks. 2. 367-
37-4.
= p,,logA~, + p,,logA~ + p, log(l + rL) Barron. A (19S9L Stanstical propcrues ot artiticiat neural net-
-< 6 p,, log A,,. works. In Proceedings of the 28th I E E E (onference on De-
ctsion and Control. Tampa. Florida {pp t:280-2851. New ~t\)rk
SO t h a t H,([e/6N~] ";~) <- 6t),, log p,, ~,. W i t h ~, = IEEE Press.
4 A~, Bierens, H. (19871. Kernel e s n m a t o r s ol regression lunctlon~, in
T. Bewley (Ed.), Advances m econometrtcs--Fifth World ('ot~.-
(~,In)H,,([e./6-~,,]';) <- 96n 'p,, ~ log p,, A,,. gress (Wol. I. pp. 99-144) New York: Cambridge University
Press.
B e c a u s e q,, ~4 log q,, A = o ( n ) , we h a v e p,, A,~ log Bosq, D (1975). ln6galit6 de Bernstein pour les processus sta-
p,, &, = o(n), so t h a t ( ~ / n ) H , , ( [ e / 6 - ~ , ] '/~) ~ 0 as tionnmres et m61angeants. Applications, C.R. Acad. Sc. Paris.
required. S(rie A . 281. 1095-1098.
(ii) Similarly, Carroll, S .M., & Dickinson, B.W. (1989)_ Construction of neural
nets using the radon transform. In Proceedings.of the Inter-
(~,,/n~/2)H,([e./6-~,]'/'-) <- 24n-"2p,, a~ log p,, A . national Joint Conference on Neural Networks, Washington,
D.C. (pp. [:607-6111. New York: I E E E Press,
B e c a u s e q,, ~, log q,, ~ , = o(n';-~), the result fol- Cox. D.D. ( 1984 ). Multivariate s m o o t h i n g splines. S I A M Journal
lows. ~J of Numerical Analysis, 21. 789-813
Connectionist Nonparanletric Regression
Cvbenko. G. (1989). Approximation by superpositions of a sig-
mold function. Mathematics qf Control, Signals and Systems,
2, 3113-314.
Funahashi, K. ( 19891. On the approximate realization of contin-
uous mappings by neural networks. Neural Networks, 2. 183-
192.
Gallant, A.R., Nychka, D,W. 11987). Seminonparametric maxi-
mum likelihood estimation. Econometrica, 55, 363-390.
Geman, S., & ltwang+ C. ( 19821. Nonparametric maximum like-
lihood estimation by the method of sieves. The Annals qf
Statistics, IlL 401 414.
Grenandcr, U. (1981). Abstract inJorence. New York: Wiley.
Haussler, D. 119891. Generalizing the PAC model j o t neural net
and other h'arning applications, (UC Santa Cruz Computer
Research Laboratory Technical Report UCSC-CRL-89-311).
ttechl-Nielscn, R (1989). Theory of the backpropagation neural
network. In Proceedings r?l"the hlternational Joint Conl~'rence
on Neural Net,'orkms, Washington, D.C., (pp. 1:593-606). New
York: 1EEE Press.
Hornik, K., Stinehcombe, M.. & White, H. 11989). Multilayer
feed forward networks arc universal approximators. Neural
Networks, 2. 359-366.
Hornik, K.. Stinchcombe, M.. & White, H. (19911). Universal
approximation of an unknown mapping and its derivatives
using naultilavcr feedforward networks. Neural Networks, 3,
this isstte.
Kohnogorov, A.N., & Fomin, S.V. (197(I). Introductory realanal-
v.si~. New Yl~rk: Dover.
Kolmogorov, A.N,, & Tihomirov, V.M. (1961). ~-cntropy and ~-
capacity of sets in functional spaces. American Mathematical
Society Translations 2, 17, 277-364.
Li, K-C. (1987). Asymptotic optimality for C;,, C;. cross-valida-
tion and generalized cross validation: Discrete index set. The
Anmds' ed'Statistics. 15, 958-975.
Lorentz, G.G. (1966). Approximation qf]tmctions. New York:
Holt, Rinehart and Winston.
Lukacs, E. 11975). Stochastic convergence. New York: Academic
Press.
Rudin, W, (19741. Real ctnd complex analysis. New York: Mc-
(}raw-Hill.
Rumclhart, D.E., tlinton, G.E., & Williams, R.J. (1986). Learn-
549
ing internal representations by error propagation. In D.E.
Rumelhart and J i . McClelland (Eds.), Parallel distributed
processing: Explorations m the microstrucmre of cognition (Voh
1, pp. 318-3621. Cambridge, MA: MIT Press.
Scales, L.E. 11982). Introduction to non-linear oplimization. New
York: Springer-Verlag.
Sevcrini, T.A., & Wong, W.H. (1987). ('onvergcnee rate~ o/max-
imum likelihood and related emStimaws m general parameter
~spaces. (University of Chicago Department of Statistics Work-
ing Paper).
Stinchcombe, M., & White, H. (198%). Universal approximation
using feedforward networks with non-sigmoid hidden layer
activation functions. In ProceedingmS o/'lhe International Joint
Co;([t'rence on Neural Networks, Washington, D.C. (pp. I:613-
6171. New York: 1EEE Press.
Stinehcombc, M., & White, H. (1989b). 3ome measurabililv re-
.sUltLSIOr extrema ql'random /i+nction,~over random ~et.L ( UCSD
Department of Economics Discussion Paper 89-181.
Stone, M. 119741. Cross-wfliditorv choice and assessment of sta-
tistical predictions, Journal O['the Royal Stati,stical Society. Se-
ries B, 36, 111-133.
Vapnik, V.N., & Chervonenkis. A. Ya. 1197I). On the uniform
convergence of relative frequencies ol events to their proba-
bilities. Theory 01' Probability and it.~ Applications, 16, 264-
28O.
Wahba, (}. 119751. Smoothing noisy data with spline functions.
Numerical Mathematics, 24, 383-393.
Wahba, (L, 6: Wold+ S. 11975). A completely automatic French
curve: Fitting spline functions by cross validation. Uommu-
nicalion,~ in Statistics, 4, 1-17.
White, H, 11984). Azwnptolic theo O, /or Econometrician,s. Or-
hmdo: Academic Press.
White. H, (1989a). Learning in artificial neural networks: A sta-
tistical perspective. Neural Computation 1,425 464.
White, H. (1989b). A l?ractical eross-validati+m method jor mahi-
layer/?ed/brward network,s (Internal HNC Report #89-03).
Whitc, H., & Wooldridge, J. (199(I), Some results on sieve csti-
marion with depcndcnt observations. In W. Barnett, J. Powell,
and O. Tauchen (Eds.), Nonparametric and mS'emi-lmrametric
methods in econcm:,,'t,ie.~ attd mStatistiCmS.New York: Cambridge
University Press, to appear.