A Beginner’s Guide to the Mathematics of Neural Networks A.C.C. Coolen Department of Mathematics, King's College London Abstract In this paper T try to describe both the role of mathematis in shap- ing ou understanding of how nearal nesvorks operate, andl the curions new mathematical cancepts generated by our attempts to capture nets ral netwodks in eeuations. My target weaver being the non-espext, Tvl, presenta biased selection of relatively simple ecamplesof neural netwonk tasks, medls anv calculations rathor than ty to give a fill eneyelapedi reviewlce arcount of the muy mathematical developments this fide, Contents 1 Introduction: Neural Information Processing, 2 From Biology to Mathematical Models 21 Fini Biological Nawons to Model N 22. Universality of Model Neurons. 23. Directions and Strategies. . 3 Neural Networks as Associative Memories SAL Recipes for Song Pattems and Pate Seri « 32 Symmetzic Networks: the Fremy Picture. : 3.3. Solving Modek of Noisy Attractor Networks 4 Creating Maps of the Outside World 4AL_ Map Ronnaton Thre Competitive ening 42. Solving Modek cf Map Formation. 5 Learning a Rule From an Expert Perceptrons «+ Multilayer Networ Caloulating what is Achiewable’ | 11 Solving the Dynamics of Leamings for Perceptrons 6 Purzling Mathematics GL Complexity due to Frustration, Disorder and Plasticity . 62 The World of Replica Theory =. . - 7 Further Reading Sear BBs1 Introduction: Neural Information Processing (ur brains perferm sophisticated information processing tasks, using harchvare and operation rues which are quite different from the ones an which convene ‘ional computers are haseel. The processors in the brain, the neurens (e figure 1), are rather noisy elements! which eperate in parallel, They are organise in cense networis, the structure of which can yexy fiom very reghlar 10 almost amerphonus (xe figure 2), ane they commmnicate signals thragh a Inige num- ber of inter-neurom connections (the so-called synapses). "These ocrnections represent the ‘progean? of a network. By continously updating the strengths of the cemections, a netwerk asa whole can modify and optimise its program, “arn? fem experience and adapt to changing circumstances, Figure I: Left: a Purkinje newon in the human oerebdlum. Right: a pyramidal ‘euro of the rabbit cortex. The black blobs are the neurons, the tes Of wines fanning out constitute the input channels (or dendrites) through which signas are receive! which are sent off by other fing neurons. The lines at the bottom, Tifureating only modestly, are the output charmels (Gr axons). Fiom an engineering point of view neuronsare infact rather poor processors, ‘they are low and umneliable (See the table below). In the brain this is overocme yensuring that always very large rmmber of neuronsare inva in any’ Sky any having them operate in parallel, with many connections. Thisisinsharp contrast to ccnventional cempaters, where operations are as a rule performex! sequentially, © that failure of any part of the chain of operations is usally fatal. Furthermore, conventional computers execute a detailed specification of orders, reiting the preggammer to know exactly which data can be expected! and hovr to respond, Subsexquent changes in the actual situation, not foreseen ty the programmers lead to trouble, Neural netwenks on the other hand, can adapé to changing cincumstances. Finally in our brain large munbers of xeurens end their cancers each day umotioed. Compare this to what happens if we randomly cut a few wires in cur workstation. Ty is wermen that dr ona sepals are to some dee sult to rand variations ay exititsovae sonanecas adie whieh appears ek to be eae to heir ‘rosin tak they ae invoke iEe Hs RY eG ASS aes Ee Cana Figure 2 Left: a section of the human cerebellim, Right: a section of the Inne cortex. Note that the staining; method used to produce such pictures ‘colbus anly a reasonably mockst fraction of the newons present, so in reality these netwcrlss are far more dense. Rougiily speaking, conventional computers can be seen as the appropriate tools for performing wellefined and rulesbased infermation processing tasks, in stable and safe envirenments, where all peesble situations aswell 2s howe to respond in every situation, are Inown beforehand. ‘Typical tasks fitting these Giteria ae eg butefone chess playing, wor processing, beeping accounts and mile-based (civil semen) decision making, Nema information systems, on the other hand, are supeticr to comventicnal computers in dealing ‘with realeveonld taskss such as e.g, cemmication (vision, spexch recognition), _moxement coontination (rcboties) and experience-based decision making (ase sification, prediction, system control), where data are often messy, uncertain or perfect softions are for all practical purpares nan-existent 3