Statistical Physics of Spin Glasses and Information Proce: An Introduction HIDETOSHI NISHIMORI Department of Physics ‘Tokyo Institute of Technology CLARENDON PRES 2001 + OXFORDStatistical Physics of Spin Glasses and Information Processing An Introduction Hidetoshi Nishimori, Department of Physics, Tokyo Institute of Technology, Japan © One of the few books in this interdisciplinary area © Rapidly expanding field * Up-to-date presentation of modern analytical techniques Self-contained presentation Spin glasses are magnetic materials, Statistical mechanics has been a powerful tool to theoretically analyse various unique properties of spin glasses. A number of new analytical techniques have been developed to establish a theory of spin glasses. Surprisingly, these techniques have offered new tools and viewpoints for the understanding of information processing problems, including neural networks, error-correcting codes, image restoration, and optimization problems, This book is one of the first publications of the past ten years that provides a broad overview of this interdisciplinary field. Most part of the book is written in a self-contained manner, assuming only a general knowledge of statistical mechanies and basie probability theory. It provides the reader with a sound introduction to the field and to the analytical techniques necessary to follow its most recent developments. Contents: Mean-field theory of phase transitions; Mean-field theory of spin glasses; Replica symmetry breaking: Gauge theory of spin glasses; Error-correcting codes: Image restoration; Associative memory; Learning in perceptron; Optimization problems; A. Eigenvalues of the Hessian; B. Parisi equation; C. Channel coding theorem; D. ‘ibution and free energy of K- Sat; References; Index. International Series of Monographs on Physies No.111, Oxford University Press Paperback, £24.95, 0-19-850941-3 Hardback, £49.50, 0-19-850940-5 August 2001, 285 pages, 58 line figures, 2 halftonesPREFACE ‘The scape of the theory of spin glasses has been expanding well beyond its origi nal goal of explaining the experimental facts of spin glass materials, For the first time in the history of physies we have encountered an explicit example in which the phase space of the system has an extremely complex structure and yet is amenable to rigorous, systematic analyses. Investigations of such systems have ‘opened a new paradigm in statistical physics. Also, the framework of the analyti- cal treatment of these systems has gradually been recognized as an indispensable tool for the study of information processing tasks. One of the principal purposes of this book is to elucidate some of the im- portant recent developments in these interdisciplinary directions, such as error- correcting, codes, image restoration, neural networks, and optimization problems. In particular, 1 would like to provide a unified viewpoint traversing several dif ferent research fields with the replica method as the common language, which emerged from the spin glass theory, One may also notice the close relationship between the arguments using gauge symmetry in spin glasses and the Bayesian ‘method in information processing problems. Accordingly, this book is not neces- sarily written as a comprehensive introduetion to single topics in the conventional Classification of subjects like spin glasses or neural networks. In a certain sense, statistical mechanics and information seiences may have been destined to be directed towards common objectives since Shannon form lated information theory about fifty years ago with the concept of entropy’ as the basic building block. It would, however, have been difficult. to envisage how this actually would happen: that the physies of disordered systems, and spin glass ‘theory in particular, at its maturity naturally encompasses some of the impor tant aspects of information sciences, thus reuniting the two disciplines, It would then reasonably be expected that in the future this cross-disciplinary field will continue to develop rapidly far beyond the current perspective. This is the very purpose for which this book is intended to establish a basis. The book is composed of two parts. The first part concerns the theory of spin glasses. Chapter 1 is an introduction to the general mean-field theory of phase transitions. Basie knowledge of statistical mechanics at undergraduate level is assumed. The standard mean-field theory of spin glasses is developed in Chapters 2 and 3, and Chapter 4 is devoted to symmetry arguments using gauge transformations. These four chapters do not cover everything to do with spin glasses. For example, hotly debated problems like the throe-dimensional spin lass and anomalously slow dynamics are not included here. The reader will find relevant references listed at the end of each chapter to cover these and other topics not treated here.vi PREFACE, ‘The second part deals with statistical-mechanical approaches to information processing problems. Chapter 5 is devoted to error-correcting codes and Chapter 6 to image restoration. Neural networks are discussed in Chapters 7 and 8, and. optimization problems are elucidated in Chapter 9. Most of these topics are Formulated as applications of the statistical mechanics of spin glasses, with a few exceptions. For each topic in this second part, there is of course a long history, and consequently a huge amount of knowledge has been accumulated. ‘The presentation in the second part reflects recent developments in statistical- ‘mechanical approaches and does not necessarily cover all the available materials. Again, the references at the end of each chapter will be helpful in filling the gaps. ‘The policy for listing up the references is, first, to refer explicitly to the original papers for topics discussed in detail in the text, and second, whenever possible, to refer to review articles and books at the end of a chapter in order to avoid an excessively long list of references. I therefore have to apologize to those authors whose papers have only been referred to indirectly via these reviews and books. ‘The reader interested mainly in the second part may skip Chapters 3 and 4 in the first part before proceeding to the second part. Nevertheless it is recom= ‘mended to browse through the introductory sections of these chapters, including replica symmetry breaking (§§3-1 and 3.2) and the main part of gauge theory (§§4.1 to 4.3 and 4.6), for a deeper understanding of the techniques relevant to the second part. It is in particular important for the reader who is interested Chapters 5 and 6 to go through these section ‘The present volume is the English edition of a book written in Japanese by me and published in 1999. I have revised a significant part of the Japanese edition and added new material in this English edition. The Japanese edition emerged from lectures at Tokyo Institute of Technology and several other un versities. I would like to thank those students who made useful comments on the lecture notes. I am also indebted to colleagues and friends for collabora tions, discussions, and comments on the manuscript: in particular, to Jun-ichi Inoue, Yoshiyuki Kabashima, Kazuyuki Tanaka, Tomohiro Sasamoto, Toshiyuki Tanaka, Shigeru Shinomoto, Taro Toyoizumi, Michael Wong, David Saad, Peter Sollich, Ton Coolen, and John Cardy. I am much obliged to David Sherrington for useful comments, collaborations, and a suggestion to publish the present En- lish edition. If this book is useful to the reader, a good part of the credit should be attributed to these outstanding people. Tokyo February 20011 CONTENTS ‘Mean-field theory of phase transitions 1.1 Ising model 1.2. Order parameter and phase transition 1.3 Mean-field theory 1.3.1 Mean-field Hamiltonian 1.3.2 Equation of state 1.3.3. Free energy and the Landau theory 14 Infinite-range model 1.5 Varintional approach ‘Mean-field theory of spin glasses 2.1 Spin glass and the Edwards-Anderson model 2.1.1 Edwards-Anderson model 2.1.2 Quenched system and configurational average 2.1.3 Replica method 2.2 Sherrington-Kirkpatrick model 2.2.1 SK model 2.2.2 Replica average of the partition funetion 2.2.3 Reduction by Gaussian integral 2.24. Stoopest descent 2.2.5. Order parameters 2.3 Replica-symmetrie solution 23.1 Equations of state 23.2 Phase diagram 23.3 Negative entropy mmetry breaking 3.1 Stability of replica-symmetric solution 3.1.1 Hessian 3.1.2 Eigenvalues of the Hessian and the AT line 3.2 Replica symmetry breaking 3.2.1 Parisi solution 3.2.2 First-step RSB 3.2.3 Stability of the first step RSB. 3.3 Fall RSB solution 3.3.1, Physical quantities 3.3.2 Order parameter near the critical point 3.3.3 Vertical phase boundary 34 Physical significance of RSB