Ebook Handbook of Hydroinformatics Volume I Classic Soft Computing Techniques PDF Full Chapter PDF

Handbook of HydroInformatics: Volume
I: Classic Soft-Computing Techniques -

eBook PDF
Visit to download the full and correct content document:
https://ebooksecure.com/download/handbook-of-hydroinformatics-volume-i-classic-so
ft-computing-techniques-ebook-pdf/
Handbook of HydroInformatics
This page intentionally left blank
Handbook
of HydroInformatics
Volume I: Classic Soft-Computing Techniques
Edited by
Saeid Eslamian
Full Professor of Hydrology and Water Resources Sustainability, Department of Water Engineering,
College of Agriculture, Isfahan University of Technology, Iran
Faezeh Eslamian
McGill University, Quebec, Canada
Elsevier
Radarweg 29, PO Box 211, 1000 AE Amsterdam, Netherlands
The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom
50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States
Copyright © 2023 Elsevier Inc. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopying, recording, or any information storage and retrieval system, without
permission in writing from the publisher. Details on how to seek permission, further information about the
Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance
Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by the Publisher
(other than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experience broaden our
understanding, changes in research methods, professional practices, or medical treatment may become
necessary.
Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using
any information, methods, compounds, or experiments described herein. In using such information or methods
they should be mindful of their own safety and the safety of others, including parties for whom they have a
professional responsibility.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability
for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or
from any use or operation of any methods, products, instructions, or ideas contained in the material herein.
ISBN: 978-0-12-821285-1
For information on all Elsevier publications

visit our website at https://www.elsevier.com/books-and-journals
Publisher: Candice Janco

Acquisitions Editor: Maria Elekidou
Editorial Project Manager: Rupinder Heron
Production Project Manager: Bharatwaj Varatharajan
Cover Designer: Greg Harris
Typeset by STRAIVE, India

Dedication
To Late Dr. Mark Twain (American Writer, Humorist, Entrepreneur, Publisher,

and Lecturer, 1835–1910)
“Data is like garbage. You’d better know what you are going to do with it
before you collect it.”
Contents
Contributors xv Elastic net 33

About the editors xvii Training and the cost function 34
Preface xix References 37
1. Advanced machine learning 2. Bat algorithm optimized extreme

techniques: Multivariate learning machine: A new
regression 1 modeling strategy for predicting
Reza Daneshfar, Mohammad Esmaeili,
river water turbidity at the
Mohammad Mohammadi-Khanaposhtani, United States 39
Alireza Baghban, Sajjad Habibzadeh, and
Saeid Eslamian Salim Heddam
1. Introduction 1 1. Introduction 39
2. Linear regression 1 2. Study area and data 40
3. Multivariate linear regression 2 3. Methodology 42
4. Gradient descent method 4 3.1 Feedforward artificial neural
5. Polynomial regression 6 network 42
6. Overfitting and underfitting 9 3.2 Dynamic evolving neural-fuzzy
7. Cross-validation 9 inference system 43
8. Comparison between linear and 3.3 Bat algorithm optimized extreme
polynomial regressions 9 learning machine 43
9. Learning curve 11 3.4 Multiple linear regression 44
10. Regularized linear models 11 3.5 Performance assessment of the
11. The ridge regression 13 models 45
12. The effect of collinearity in the 4. Results and discussion 46
coefficients of an estimator 13 4.1 USGS 1497500 station 46
13. Outliers impact 13 4.2 USGS 11501000 station 47
14. Lasso regression 16 4.3 USGS 14210000 station 49
15. Elastic net 17 4.4 USGS 14211010 station 50
16. Early stopping 17 5. Conclusions 53
17. Logistic regression 18 References 53
18. Estimation of probabilities 18
19. Training and the cost function 19
20. Conclusions 20 3. Bayesian theory: Methods and
Appendix: Python code 20 applications 57
Linear regression 20
Yaser Sabzevari and Saeid Eslamian
Gradient descent method 22
Comparison between linear and polynomial 1. Introduction 57
regressions 23 2. Bayesian inference 57
Learning curve 26 3. Phases 58
The effect of collinearity in the coefficients 4. Estimates 58
of an estimator 30 5. Theorem Bayes 58
Outliers impact 31 5.1 Argument of Bayes 58
Lasso regression 33 5.2 Bayesian estimation theory 59
vii
viii Contents
5.3 Machine learning using Bayesian 5. Cross-validation 89

method 59
Amir Seraj,
5.4 Bayesian theory in machine
Mohammad Mohammadi-Khanaposhtani,
learning 60 Reza Daneshfar, Maryam Naseri,
5.5 Definition of basic concepts 60 Mohammad Esmaeili, Alireza Baghban,
5.6 Bayesian machine learning Sajjad Habibzadeh, and Saeid Eslamian
methods 60
1. Introduction 89
5.7 Optimal Bayes classifier 60
1.1 Importance of validation 89
5.8 Naive Bayes classifier 62
1.2 Validation of the training process 89
6. Bayesian network 63
2. Cross-validation 90
7. History of Bayesian model application in
2.1 Exhaustive and nonexhaustive
water resources 65
cross-validation 90
8. Case study of Bayesian network
2.2 Repeated random subsampling
application in modeling of
cross-validation 90
evapotranspiration of reference plant 66
2.3 Time-series cross-validation 90
9. Conclusions 67
2.4 k-fold cross-validation 90
References 67
2.5 Stratified k-fold cross-validation 91
2.6 Nested 91
4. CFD models 69 3. Computational procedures 91
4. Conclusions 104
Hossien Riahi-Madvar,
Mohammad Mehdi Riyahi, and References 105
Saeid Eslamian
1. Introduction 69 6. Comparative study on the selected
2. Numerical model of one-dimensional node and link-based performance
advection dispersion equation indices to investigate the hydraulic
(1D-ADE) 69 capacity of the water distribution
3. Physically influenced scheme 70 network 107
4. Finite Volume Solution of Saint-Venant
equations for dam-break simulation C.R. Suribabu and P. Sivakumar
using PIS 72 1. Introduction 107
5. Discretization of continuity 2. Resilience of water distribution
equation using PIS 73 network 109
6. Discretization of the momentum 3. Hydraulic uniformity index (HUI) 110
equation using PIS 73 4. Mean excess pressure (MEP) 110
7. Quasi-two-dimensional flow 5. Proposed measure 110
simulation 74 5.1 Energy loss uniformity (ELU) 110
8. Numerical solution of quasi-two- 6. Hanoi network 111
dimensional model 76 7. Results and discussion 112
9. 3D numerical modeling of flow in 8. Conclusions 117
compound channel using turbulence References 117
models 78
10. Three-dimensional numerical
model 79 7. The co-nodal system analysis 119
11. Grid generation and the flow filed
Vladan Kuzmanovi
c
solution 80
12. Comparison of different turbulence 1. Introduction 119
models 80 2. Co-nodal and system analysis 119
13. Three-dimensional pollutant transfer 3. Paleo-hydrology and remote sensing 120
modeling 81 4. Methods 121
14. Results of pollutant transfer 5. Nodes and cyclic confluent system 121
modeling 83 5.1 H-cycloids analysis and fluvial
15. Conclusions 86 dynamics 123
References 86 6. Three Danube phases 124
Contents ix
7. Danubian hypocycles as overlapping 10. Decision tree algorithms 171

phases 127
8. Conclusions 133 Amir Ahmad Dehghani, Neshat Movahedi,
Khalil Ghorbani, and Saeid Eslamian
References 133
Further reading 134 1. Introduction 171
1.1 ID3 algorithm 171
1.2 C4.5 algorithm 172
8. Data assimilation 135 1.3 CART algorithm 173
Mohammad Mahdi Dorafshan, 1.4 CHAID algorithm 173
Mohammad Reza Jabbari, and 1.5 M5 algorithm 173
Saeid Eslamian 1.6 Random forest 173
1. Introduction 135 1.7 Application of DT algorithms in water
2. What is data assimilation? 136 sciences 174
3. Types of data assimilation methods 137 2. M5 model tree 174
3.1 Types of updating procedure 137 2.1 Splitting 174
3.2 Types of updating variable 137 2.2 Pruning 176
4. Optimal filtering methods 140 2.3 Smoothing 176
4.1 Kalman filter 140 3. Data set 176
4.2 Transfer function 143 3.1 Empirical formula for flow
4.3 Extended Kalman filter 144 discharge 177
4.4 Unscented Kalman filter 146 3.2 Model evaluation and
5. Auto-regressive method 147 comparison 178
6. Considerations in using data 4. Modeling and results 179
assimilation 148 4.1 Initial tree 179
7. Conclusions 148 4.2 Pruning 179
References 148 4.3 Comparing M5 model and empirical
formula 184
5. Conclusions 185
9. Data reduction techniques 153 References 185
M. Mehdi Bateni and Saeid Eslamian
11. Entropy and resilience indices 189
1. Introduction 153
2. Principal component analysis 153 Mohammad Ali Olyaei, A.H. Ansari,
Zahra Heydari, and Amin Zeynolabedin
3. Singular spectrum analysis 155
3.1 Univariate singular spectral 1. Introduction 189
analysis 156 2. Water resource and infrastructure
3.2 Multivariate singular spectral performance evaluation 190
analysis 157 3. Entropy 191
4. Canonical correlation analysis 157 3.1 Thermodynamic entropy 191
5. Factor analysis 158 3.2 Statistical-mechanical entropy 192
5.1 Principal axis factoring 158 3.3 Information entropy 192
6. Random projection 160 3.4 Application of entropy in water
7. Isometric mapping 162 resources area 193
8. Self-organizing maps 162 4. Resilience 194
9. Discriminant analysis 163 4.1 Application of resilience in water
10. Piecewise aggregate approximation 165 resources area 195
11. Clustering 165 4.2 Resilience in UWS 196
11.1 k-means clustering 165 4.3 Resilience in urban environments 198
11.2 Hierarchical clustering 166 4.4 Resilience to floods 199
11.3 Density-based clustering 167 4.5 Resilience to drought 201
12. Conclusions 168 5. Conclusions 202
References 169 References 203
x Contents
12. Forecasting volatility in the 14. Gradient-based optimization 243

stock market data using Mohammad Zakwan
GARCH, EGARCH, and GJR
models 207 1. Introduction 243
2. Materials and method 244
Sarbjit Singh, Kulwinder Singh Parmar, 2.1 GRG solver 245
and Jatinder Kaur
3. Results and discussion 245
1. Introduction 207 3.1 Solving nonlinear equations 245
2. Methodology 209 3.2 Application in parameter
2.1 Types of GARCH estimation 246
models 210 3.3 Fitting empirical equations 248
3. Application and results 211 4. Conclusions 249
4. Conclusions 219 References 249
References 219
15. Gray wolf optimization
algorithm 253
13. Gene expression models 221
Mohammad Reza Zaghiyan,
Hossien Riahi-Madvar, Mahsa Gholami, Vahid Shokri Kuchak, and Saeid Eslamian
and Saeid Eslamian
1. Introduction 253
1. Introduction 221 2. Theory of GWO 254
2. Genetic programming 221 3. Mathematical modeling of gray wolf
2.1 The basic steps in GEP optimizer 255
development 222 3.1 Social hierarchy 255
2.2 The basic steps in GEP 3.2 Encircling prey 256
development 222 3.3 Hunting behavior 256
3. Tree-based GEP 223 3.4 Exploitation in GWO-attacking
3.1 Tree depth control 224 prey 258
3.2 Maximum tree depth 224 3.5 Exploration in GWO-search for
3.3 Penalizing the large trees 225 prey 259
3.4 Dynamic maximum-depth 4. Gray wolf optimization example for
technique 226 reservoir operation 259
4. Linear genetic programming 227 5. Conclusions 261
5. Evolutionary polynomial Appendix A: GWO Matlab codes for the
regression 227 reservoir example 262
6. Multigene genetic programming 228 References 265
7. Pareto optimal-multigene genetic
programming 229 16. Kernel-based modeling 267
8. Some applications of GEP-based
models in hydro informatics 230 Kiyoumars Roushangar,
8.1 Derivation of quadric polynomial Roghayeh Ghasempour, and
Saman Shahnazi
function using GEP 230
8.2 Derivation of Colebrook-White 1. Introduction 267
equation using GEP 231 2. Support vector machine 268
8.3 Derivation of the exact form of 2.1 Support vector classification 268
shield’s diagram using GEP 233 2.2 Support vector regression 269
8.4 Extraction of regime river equations 3. Gaussian processes 271
using GEP 234 3.1 Gaussian process regression 271
8.5 Extraction of longitudinal 3.2 Gaussian process classification 273
dispersion coefficient equations 4. Kernel extreme learning machine 274
using GEP 236 5. Kernels type 275
9. Conclusions 237 5.1 Fisher kernel 276
References 239 5.2 Graph kernels 276
Contents xi
5.3 Kernel smoother 276 3.2 Constant heat flux boundary

5.4 Polynomial kernel 276 condition 297
5.5 Radial basis function kernel 276 4. Multicomponent LBM (species
5.6 Pearson kernel 276 transport modeling) 297
5.7 String kernels 277 5. Flow simulation in porous media 298
5.8 Neural tangent kernel 277 6. Dimensionless numbers 299
6. Application of kernel-based 7. Flow chart of the simulation
approaches 277 procedure 300
6.1 Total resistance and form resistance of 8. Multiphase flows 300
movable bed channels 277 8.1 The color-gradient model 301
6.2 Energy losses of rectangular and 8.2 Shan-Chen model 302
circular culverts 277 9. Sample test cases and codes 303
6.3 Lake and reservoir water level 9.1 Free convection in L-cavity 303
prediction 279 9.2 Force convection in a channel 303
6.4 Streamflow forecasting 279 10. Conclusions 304
6.5 Sediment load prediction 279 Appendix A 305
6.6 Pier scour modeling 279 Computer code for free convection in
6.7 Reservoir evaporation prediction 279 L-cavity 305
7. Conclusions 280 Appendix B 311
References 280 Computer code for force convection in a
Further reading 281 channel 311
References 317
17. Large eddy simulation:
Subgrid-scale modeling with 19. Multigene genetic programming
neural network 283 and its various applications 321
Tamas Karches Majid Niazkar
1. Introduction 283 1. Introduction 321

2. LES and traditional subgrid-scale 2. Genetic programming and its variants 321
modeling 284 3. An introduction to multigene genetic
3. Data-driven LES closures 284 programming 322
4. Guidelines for SGS modeling 285 4. Main controlling parameters of
4.1 Simulation project definition 285 MGGP 324
4.2 A priory analysis with DNS 286 5. A review on MGGP applications 325
4.3 Neural network based SGS model 6. Future trends of MGGP applications 327
construction 286 7. A case study of the MGGP
5. Conclusions 287 application 327
References 287 8. Conclusions 329
References 330
18. Lattice Boltzmann method and
its applications 289 20. Ontology-based knowledge
management framework in
Mojtaba Aghajani Delavar and Junye Wang
business organizations and water
1. Introduction 289 users networks in Tanzania 333
2. Lattice Boltzmann equations 289
Neema Penance Kumburu
2.1 BGK approximation 292
2.2 Lattice Boltzmann models 292 1. Introduction 333
2.3 Multirelaxation time lattice 2. Theoretical framework 334
Boltzmann (MRT) 293 3. Empirical literature 336
2.4 Boundary conditions 294 4. Ontology-based knowledge
3. Thermal LBM 296 management framework in business
3.1 Boundary condition with a given organizations: A conceptual
temperature 297 framework 336
xii Contents
5. Ontology-based knowledge 6. Applications of relevance vector

management framework in business machine 377
organizations and water user networks 6.1 Sediment concentration
proposed system 339 estimation 377
6. The practice of knowledge organization 6.2 Drought monitoring 378
and expression 341 6.3 Groundwater quality monitoring 378
6.1 Ontology 341 6.4 Evaporative losses in reservoirs 379
6.2 Knowledge representation and 6.5 Environmental science 380
organization base on ontology 341 7. Conclusions 381
6.3 Knowledge retrieval base ontology 343 References 381
6.4 Knowledge application and
implementation base on ontology 344 23. Stochastic learning algorithms 385
7. Conclusions 347
Amir Hossein Montazeri,
References 347
Sajad Khodambashi Emami,
Mohammad Reza Zaghiyan, and
Saeid Eslamian
21. Parallel chaos search-based
incremental extreme learning 1. Introduction 385
machine 349 2. Gradient descent 386
2.1 Theory of batch gradient descent 386
Salim Heddam 2.2 Theory of SGD 386
1. Introduction 349 3. Perceptron 388
2. Materials and methods 350 3.1 Theory of perceptron 389
2.1 Study area description 350 3.2 Perceptron learning procedure 390
2.2 Modeling approaches 352 4. Adaline 391
2.3 Performance assessment of the 4.1 Theory of Adaline 391
models 353 4.2 Adaline learning procedure 391
3. Results and discussion 355 5. Multilayer network 392
4. Conclusions 361 5.1 Multilayer network learning
References 362 procedure 392
6. Learning vector quantization 393
22. Relevance vector machine 6.1 LVQ learning procedure 395
(RVM) 365 7. K-means clustering 397
7.1 What is clustering? 397
Mohammad Reza Jabbari, 7.2 Theory of K-means 397
Mohammad Mahdi Dorafshan, and 8. Gradient boosting 399
Saeid Eslamian
8.1 What is boosting? 399
1. Introduction 365 8.2 Theory of gradient boosting (GB) 399
2. Machine learning algorithms 365 8.3 Stochastic gradient boosting 400
2.1 Supervised learning 365 9. Conclusions 400
2.2 Unsupervised learning 366 References 401
3. Support vector machine 366 Appendix A 403
4. Relevance vector machine 367 Appendix B 404
4.1 Measurement model Appendix C 406
representation 367 Appendix D 407
4.2 Relevance vector regression 371 Appendix E 409
4.3 Relevance vector classification 372
4.4 Limitations and performance 24. Supporting vector machines 411
analysis 372
Kiyoumars Roushangar
4.5 Multivariate relevance vector
and Roghayeh Ghasempour
machines 373
5. Preprocessing step 375 1. Introduction 411
5.1 Data normalization 375 2. SVMs for classification problems 412
5.2 Data reduction 375 2.1 Linear classifiers 412
5.3 Dataset split ratio 376 2.2 Non-linear classifiers 413
Contents xiii
3. SVMs for regression problems 413 26. Uncertainty-based resiliency

4. Selection of SVM parameters 415 evaluation 435
4.1 Margin 415
4.2 Regularization 415 Hossien Riahi-Madvar,
Mohammad Mehdi Riyahi, and
4.3 Kernels 415 Saeid Eslamian
4.4 Gamma parameter 416
5. Application of support vector 1. Introduction 435
machines 417 2. Uncertainty analysis by the first-order
5.1 Application of support vector method 435
regression in the water recourse 3. Risk and resilience analysis 438
engineering 417 4. Reliability computation by direct
6. Conclusions 421 integration 438
References 421 5. Reliability computation using safety
margin/safety factor 439
6. Safety margin 439
25. Uncertainty analysis using 7. Safety factor 439
fuzzy models in 8. Uncertainty-based hydraulic designs 440
hydroinformatics 423 9. Hydrologic uncertainties 441
10. Hydraulics uncertainties 442
Tayeb Boulmaiz, Mawloud Guermoui, 11. Monte-Carlo uncertainty analysis in
Mohamed Saber, Hamouda Boutaghane,
Habib Abida, and Saeid Eslamian quasi-2D model parameters 442
12. SKM model 443
1. Introduction 423 13. Uncertainty based river flow modeling
2. Fuzzy logic theory 424 with Monte-Carlo simulator 443
2.1 Fuzzification 424 14. Monte-Carlo uncertainty analysis in
2.2 Rule base 425 machine learning techniques 447
2.3 Inference 425 15. Uncertainty evaluation using the
2.4 Defuzzification 425 integrated Bayesian multimodel
3. Concept of fuzzy uncertainty framework 449
analysis 425 16. Copula-based uncertainty analysis 449
4. Uncertainty analysis 17. Uncertainty analysis with Tsallis
applications 426 entropy 450
4.1 Flood forecasting 426 18. Theory of evidence for uncertainty in
4.2 Groundwater modeling 427 hydroinformatics 451
5. Machine learning and fuzzy sets 430 19. Resiliency quantification 451
6. Fuzzy sets and probabilistic 20. Conclusions 452
approach 431 References 452
7. Conclusions 432
References 432 Index 455
Contributors
Habib Abida (423), Laboratory of Modeling of Geological Mohammad Esmaeili (1,89), Department of Petroleum
and Hydrological Systems (GEOMODELE Engineering, Amirkabir University of Technology
(LR16ES17)), Faculty of Sciences, University of Sfax, (Polytechnic of Tehran); Department of Petroleum
Sfax, Tunisia Engineering, Amirkabir University of Technology
A.H. Ansari (189), Department of Agricultural and Bio- (Tehran Polytechnic), Tehran, Iran
logical Engineering, Pennsylvania State University, Roghayeh Ghasempour (267,411), Department of Water
State College, PA, United States Resources Engineering, Faculty of Civil Engineering,
Alireza Baghban (1,89), Chemical Engineering University of Tabriz, Tabriz, Iran
Department, Amirkabir University of Technology Mahsa Gholami (221), Department of Civil Engineering,
(Tehran Polytechnic), Mahshahr Campus, Mahshahr, Faculty of Engineering, Bu-Ali Sina University,
Iran Hamedan, Iran
M. Mehdi Bateni (153), University School for Advanced Khalil Ghorbani (171), Department of Water Engineering,
Studies, Pavia, Italy Gorgan University of Agricultural Sciences & Natural
Tayeb Boulmaiz (423), Materials, Energy Systems Tech- Resources, Gorgan, Iran
nology and Environment Laboratory, University of Mawloud Guermoui (423), Unite de Recherche Appliquee
Ghardaia, Ghardaia, Algeria en Energies Renouvelables, Centre de Developpement
Hamouda Boutaghane (423), Laboratory of Soil and des Energies Renouvelables, Ghardaı̈a, Algeria
Hydraulic, Badji Mokhtar Annaba University, Annaba, Sajjad Habibzadeh (1,89), Chemical Engineering
Algeria Department, Amirkabir University of Technology
Reza Daneshfar (1,89), Department of Petroleum Engi- (Tehran Polytechnic), Mahshahr Campus, Mahshahr;
neering, Ahwaz Faculty of Petroleum Engineering, Surface Reaction and Advanced Energy Materials Lab-
Petroleum University of Technology, Ahwaz, Iran oratory, Chemical Engineering Department, Amirkabir
University of Technology (Tehran Polytechnic),
Amir Ahmad Dehghani (171), Department of Water Engi- Tehran, Iran
neering, Gorgan University of Agricultural Sciences &
Natural Resources, Gorgan, Iran Salim Heddam (39,349), Laboratory of Research in Biodi-
versity Interaction Ecosystem and Biotechnology,
Mojtaba Aghajani Delavar (289), Faculty of Science and Hydraulics Division, Agronomy Department, Faculty
Technology, Athabasca University, Athabasca, AB, of Science, Skikda, Algeria
Canada
Zahra Heydari (189), Department of Civil and Environ-
Mohammad Mahdi Dorafshan (135,365), Department of mental Engineering, University of Illinois at Urbana--
Civil Engineering, Isfahan University of Technology, Champaign, Urbana, IL, United States
Isfahan, Iran
Mohammad Reza Jabbari (135,365), Department of Elec-
Sajad Khodambashi Emami (385), Department of Water trical and Computer Engineering, Isfahan University of
Engineering and Management, Tarbiat Modares Uni- Technology, Isfahan, Iran
versity, Tehran, Iran
Tamas Karches (283), Faculty of Water Science, Uni-
Saeid Eslamian (1,57,69,89,135,153,171,221,253,365, versity of Public Service, Budapest, Hungary
385,423,435), Department of Water Engineering,
College of Agriculture, Isfahan University of Tech- Jatinder Kaur (207), Department of Mathematics,
nology; Center of Excellence in Risk Management I.K. Gujral Punjab Technical University, Kapurthala;
and Natural Hazards, Isfahan University of Technology, Guru Nanak Dev University College, Amritsar, Punjab,
Isfahan, Iran India
xv
xvi Contributors
Vahid Shokri Kuchak (253), Department of Water Engi- Kiyoumars Roushangar (267,411), Department of Water
neering and Management, Tarbiat Modares University, Resources Engineering, Faculty of Civil Engineering;
Tehran, Iran Center of Excellence in Hydroinformatics, University
Neema Penance Kumburu (333), Moshi Co-operative of Tabriz, Tabriz, Iran
University, Moshi, Tanzania Mohamed Saber (423), Disaster Prevention Research
Vladan Kuzmanovic (119), Serbian Hydrological Society, Institute (DPRI), Kyoto University, Kyoto, Japan
International Association of Hydrological Sciences, Yaser Sabzevari (57), Department of Water Engineering,
Belgrade, Serbia College of Agriculture, Isfahan University of Tech-
Mohammad Mohammadi-Khanaposhtani (1), Fouman nology, Isfahan, Iran
Faculty of Engineering, College of Engineering, Uni- Amir Seraj (89), Department of Instrumentation and
versity of Tehran, Tehran, Iran Industrial Automation, Ahwaz Faculty of Petroleum
Amir Hossein Montazeri (385), Department of Water Engineering, Petroleum University of Technology,
Engineering and Management, Tarbiat Modares Uni- Ahwaz, Iran
versity, Tehran, Iran Saman Shahnazi (267), Department of Water Resources
Neshat Movahedi (171), Department of Water Engi- Engineering, Faculty of Civil Engineering, University
neering, Gorgan University of Agricultural Sciences of Tabriz, Tabriz, Iran
& Natural Resources, Gorgan, Iran Sarbjit Singh (207), Guru Nanak Dev University College,
Maryam Naseri (89), Chemical Engineering Department, Pathankot; Department of Mathematics, Guru Nanak
Babol Noshirvani University of Technology, Babol, Dev University, Amritsar, Punjab, India
Iran P. Sivakumar (107), Department of Civil Engineering,
Majid Niazkar (321), Department of Agricultural and North Eastern Regional Institute of Science and Tech-
Environmental Sciences - Production, Landscape, nology, Nirjuli (Itanagar), Arunachal Pradesh, India
Agroenergy, University of Milan, Milan, Italy C.R. Suribabu (107), Centre for Advanced Research in
Mohammad Ali Olyaei (189), Department of Civil Envi- Environment, School of Civil Engineering, SASTRA
ronmental and Geo-Engineering, University of Min- Deemed University, Thanjavur, Tamil Nadu, India
nesota, Minneapolis, MN, United States Junye Wang (289), Faculty of Science and Technology,
Kulwinder Singh Parmar (207), Department of Mathe- Athabasca University, Athabasca, AB, Canada
matics, I.K. Gujral Punjab Technical University, Mohammad Reza Zaghiyan (385), Department of Water
Kapurthala, Punjab, India Engineering and Management, Tarbiat Modares Uni-
Hossien Riahi-Madvar (69,221,435), Department of versity, Tehran, Iran
Water Engineering, Faculty of Agriculture, Vali-e-Asr Mohammad Zakwan (243), School of Technology,
University of Rafsanjan, Rafsanjan, Iran Maulana Azad National Urdu University, Hyderabad,
Mohammad Mehdi Riyahi (69), Department of Civil India
Engineering, Faculty of Civil Engineering and Archi- Amin Zeynolabedin (189), School of Civil Engineering,
tecture, Shahid Chamran University of Ahvaz, Ahvaz, College of Engineering, University of Tehran, Tehran,
Iran Iran
About the editors
Saeid Eslamian has been a Full Professor of Environmental Hydrology and Water
Resources Engineering in the Department of Water Engineering at Isfahan University
of Technology since 1995. His research focuses mainly on statistical and environmental
hydrology in a changing climate. In recent years, he has worked on modeling natural
hazards, including floods, severe storms, wind, drought, and pollution, and on water reuse,
sustainable development and resiliency, etc. Formerly, he was a visiting professor at Prin-
ceton University, New Jersey, and the University of ETH Zurich, Switzerland. On the
research side, he started a research partnership in 2014 with McGill University, Canada.
He has contributed to more than 600 publications in journals, books, and technical reports.
He is the founder and Chief Editor of both the International Journal of Hydrology Science
and Technology (IJHST) and the Journal of Flood Engineering (JFE). Dr. Eslamian is
currently Associate Editor of four important publications: Journal of Hydrology
(Elsevier), Eco-Hydrology and Hydrobiology (Elsevier), Journal of Water Reuse and
Desalination (IWA), and Journal of the Saudi Society of Agricultural Sciences (Elsevier).
Professor Eslamian is the author of approximately 35 books and 180 book chapters.
Dr. Eslamian’s professional experience includes membership on editorial boards, and he is a reviewer of approximately
100 Web of Science (ISI) journals, including the ASCE Journal of Hydrologic Engineering, ASCE Journal of Water
Resources Planning and Management, ASCE Journal of Irrigation and Drainage Engineering, Advances in Water
Resources, Groundwater, Hydrological Processes, Hydrological Sciences Journal, Global Planetary Changes, Water
Resources Management, Water Science and Technology, Eco-Hydrology, Journal of the American Water Resources Asso-
ciation, American Water Works Association Journal, etc. Furthermore, in 2015, UNESCO nominated him for a special
issue of the Eco-Hydrology and Hydrobiology Journal.
Professor Eslamian was selected as an outstanding reviewer for the Journal of Hydrologic Engineering in 2009 and
received the EWRI/ASCE Visiting International Fellowship at the University of Rhode Island (2010). He was also awarded
prizes for outstanding work by the Iranian Hydraulics Association in 2005 and the Iranian petroleum and oil industry in
2011. Professor Eslamian was chosen as a distinguished researcher by Isfahan University of Technology (IUT) and Isfahan
Province in 2012 and 2014, respectively. In 2016, he was a candidate for National Distinguished Researcher in Iran.
Dr. Eslamian has also acted as a referee for many international organizations and universities. Some examples include
the US Civilian Research and Development Foundation (USCRDF), the Swiss Network for International Studies, the His
Majesty’s Trust Fund for Strategic Research of Sultan Qaboos University, Oman, the Royal Jordanian Geography Center
College, and the Research Department of Swinburne University of Technology of Australia. He is also a member of the
following associations: American Society of Civil Engineers (ASCE), International Association of Hydrologic Science
(IAHS), World Conservation Union (IUCN), GC Network for Drylands Research and Development (NDRD), International
Association for Urban Climate (IAUC), International Society for Agricultural Meteorology (ISAM), Association of Water
and Environment Modeling (AWEM), International Hydrological Association (STAHS), and UK Drought National
Center (UKDNC).
Professor Eslamian finished Hakim-Sanaei High School in Isfahan in 1979. After the Islamic Revolution, he was
admitted to Isfahan University of Technology (IUT) to study a BS in water engineering, and he graduated in 1986. He
was subsequently offered a scholarship for a master’s degree program at Tarbiat Modares University, Tehran. He finished
his studies in hydrology and water resources engineering in 1989. In 1991, he was awarded a scholarship for a PhD in civil
engineering at the University of New South Wales, Australia. His supervisor was Professor David H. Pilgrim, who
encouraged Professor Eslamian to work on “Regional Flood Frequency Analysis Using a New Region of Influence
Approach.” He earned a PhD in 1995 and returned to his home country and IUT. He was promoted in 2001 to Associate
xvii
xviii About the editors
Professor and in 2014 to Full Professor. For the past 26 years, he has been nominated for different positions at IUT,
including University President Consultant, Faculty Deputy of Education, and Head of Department. Dr. Eslamian is
now director of the Center of Excellence in Risk Management and Natural Hazards (RiMaNaH).
Professor Eslamian has made three scientific visits, to the United States, Switzerland, and Canada in 2006, 2008, and
2015, respectively. In the first, he was offered the position of visiting professor by Princeton University and worked jointly
with Professor Eric F. Wood at the School of Engineering and Applied Sciences for 1 year. The outcome was a contribution
to hydrological and agricultural drought interaction knowledge through developing multivariate L-moments between soil
moisture and low flows for northeastern US streams.
Recently, Professor Eslamian has written 14 handbooks published by Taylor & Francis (CRC Press): the three-volume
Handbook of Engineering Hydrology (2014), Urban Water Reuse Handbook (2016), Underground Aqueducts Handbook
(2017), the three-volume Handbook of Drought and Water Scarcity (2017), Constructed Wetlands: Hydraulic Design
(2019), Handbook of Irrigation System Selection for Semi-Arid Regions (2020), Urban and Industrial Water Conservation
Methods (2020), and the three-volume Flood Handbook (2022).
An Evaluation of Groundwater Storage Potentials in a Semiarid Climate (2019) and Advances in Hydrogeochemistry
Research (2020) by Nova Science Publishers are also among his book publications. The two-volume Handbook of Water
Harvesting and Conservation (2021, Wiley) and Handbook of Disaster Risk Reduction and Resilience (2021, New Frame-
works for Building Resilience to Disasters) are further Springer publications by Professor Eslamian, as are the Handbook of
Disaster Risk Reduction and Resilience (2022, Disaster Risk Management Strategies) and the two-volume Earth Systems
Protection and Sustainability (2022).
Professor Eslamian was listed among the World’s Top 2% of Researchers by Stanford University, USA, in 2019 and
2020. He has also been a grant assessor, report referee, award jury member, and invited researcher for international orga-
nizations such as the United States Civilian Research and Development Foundation (2006), Intergovernmental Panel on
Climate Change (2012), World Bank Policy and Human Resources Development Fund (2021), and Stockholm Interna-
tional Peace Research Institute (2022), respectively.
Faezeh Eslamian holds a PhD in Bioresource Engineering from McGill University,

Canada. Her research focuses on the development of a novel lime-based product to mit-
igate phosphorus loss from agricultural fields. Dr. Elsamian completed her bachelor and
master’s degrees in Civil and Environmental Engineering at the Isfahan University of
Technology, Iran, where she evaluated natural and low-cost absorbents for the removal
of pollutants such as textile dyes and heavy metals. Furthermore, she has conducted
research on worldwide water quality standards and wastewater reuse guidelines. Dr.
Elsamian is an experienced multidisciplinary researcher with interests in soil and water
quality, environmental remediation, water reuse, and drought management.
Preface
Classic Soft-Computing Techniques is the first volume of three in the Handbook of HydroInformatics series. Through this
comprehensive, 26-chapter work, the contributors explore the difference between traditional computing, also known as
hard computing, and soft computing, which is based on the importance given to issues like precision, certainty, and rigor.
The chapters go on to define fundamental classic soft-computing techniques such as multivariate regressions, bat algorithm
optimized extreme learning machine (Bat-ELM), Bayesian inference, computational fluid dynamics (CFD) models, cross
validation, selected node and link-based performance indices, conodal system analysis, data assimilation, data reduction
techniques, decision tree algorithm, entropy and resilience indices, generalized autorregressive conditional heteroskedas-
ticity (GARCH), exponential general autoregressive conditional heteroskedastic (EGARCH), and Glosten, Jagannathan,
and Runkle (GJR) models, gene expression models, gradient-based optimization, gray wolf optimization (GWO) algo-
rithm, kernel-based modeling, subgrid-scale (SGS) modeling with neural network, lattice Boltzmann method (LBM), mul-
tigene genetic programming (MGGP), ontology-based knowledge management framework, parallel chaos search-based
incremental extreme learning, relevance vector machine (RVM), stochastic learning algorithms, support vector machine,
uncertainty analysis using fuzzy logic models, uncertainty-based resiliency evaluation, etc. It is a fully comprehensive
handbook providing all the information needed regarding classic soft-computing techniques.
This volume is a true interdisciplinary work, and the intended audience includes postgraduates and early-career
researchers interested in computer science, mathematical science, applied science, Earth and geoscience, geography, civil
engineering, engineering, water science, atmospheric science, social science, environment science, natural resources, and
chemical engineering.
The Handbook of HydroInformatics corresponds to courses that could be taught at the following levels: undergraduate,
postgraduate, research students, and short course programs. Typical course names of this type include: HydroInformatics,
Soft Computing, Learning Machine Algorithms, Statistical Hydrology, Artificial Intelligence, Optimization, Advanced
Engineering Statistics, Time Series, Stochastic Processes, Mathematical Modeling, Data Science, Data Mining, etc.
The three-volume Handbook of HydroInformatics is recommended not only for universities and colleges, but also for
research centers, governmental departments, policy makers, engineering consultants, federal emergency management
agencies, and related bodies.
Key features are as follows:
l Contains key insights from global contributors in the fields of data management research, climate change and resilience,
insufficient data problems, etc.
l Offers applied examples and case studies in each chapter, providing the reader with real-world scenarios for comparison
l Introduces classic soft-computing techniques necessary for a range of disciplines
Saeid Eslamian
College of Agriculture, Isfahan University of Technology, Isfahan, Iran
Faezeh Eslamian
McGill University, Montreal, QC, Canada
xix
Chapter 1
Advanced machine learning techniques:

Multivariate regression
Reza Daneshfara, Mohammad Esmaeilib, Mohammad Mohammadi-Khanaposhtanic, Alireza Baghband,
Sajjad Habibzadehd, and Saeid Eslamiane,f
a
Department of Petroleum Engineering, Ahwaz Faculty of Petroleum Engineering, Petroleum University of Technology, Ahwaz, Iran, b Department of
Petroleum Engineering, Amirkabir University of Technology (Polytechnic of Tehran), Tehran, Iran, c Fouman Faculty of Engineering, College of
Engineering, University of Tehran, Tehran, Iran, d Chemical engineering Department, Amirkabir University of Technology (Tehran Polytechnic),
Mahshahr Campus, Mahshahr, Iran, e Department of Water Engineering, College of Agriculture, Isfahan University of Technology, Isfahan, Iran, f Center
of Excellence in Risk Management and Natural Hazards, Isfahan University of Technology, Isfahan, Iran
1. Introduction
Complicated problems in a variety of fields that cannot be solved using conventional techniques are handled using machine
learning (Zeebaree et al., 2019; Bargarai et al., 2020; Dargan et al., 2020). Linear regression is a simple and popular
machine technique employed for prediction purposes. It was introduced by Galton (1894). It is a mathematical approach
in order to analyze and quantify the associations of variables (Akg€un and O€ g€ud€uc€u, 2015; Dehghan et al., 2015; Liu et al.,
2017). To incorporate the outputs of other founders/covariates into a model, one cannot utilize univariate regression—i.e.,
chi-square, Fisher exact test, and analysis of variance (ANOVA). As a result, partial correlation and regression are
employed to identify the association of two variables and evaluate the confusion effect (Zebari et al., 2020; Sulaiman,
2020; Epskamp and Fried, 2018). Mathematical algorithms typically employ linear regression for the purpose of predicted
effect measurement and modeling versus several inputs (Lim, 2019). This data analysis approach linearly relates inde-
pendent and dependent variables, modeling the relationships between the independent and dependent variables based
on model training. The present study conducts a review of recent popular methodologies in the machine learning and linear
regression literature, including databases, performance, accuracy, and algorithms, from 2017 to 2020 (Sarkar et al., 2015).
This chapter is divided into the following sections: The first section focuses on linear regression, this is followed by an
explanation of multivariate linear regression, and then the gradient descent method is described. The polynomial regression
concept is then explained and concepts such as overfitting and under-fitting, cross-validation, and learning curve are
expressed in a clear and fluent manner. Finally, the attractive and practical concepts that are discussed include: regularized
linear models, ridge regression, outliers impact, lasso regression, elastic net, early stopping, and logistic regression.
2. Linear regression
When we know a property or a dependent variable in general depends on several variables but the way of this dependence is
not clear to us, a linear model is the simplest choice to get an insight into this dependence. Although the simplest choice is
not necessarily the best one, linear models can do a lot in the case of algebraic dependency between a function and its
variables. A linear model can provide a reasonable estimation of any function at least in a small neighborhood. Moreover,
some nonlinear dependencies as suggested by theories could be transformed to a linear dependency. For example, consider
the following chemical reaction rate law
r A ¼ kCnA (1)
Handbook of HydroInformatics. https://doi.org/10.1016/B978-0-12-821285-1.00017-8

Copyright © 2023 Elsevier Inc. All rights reserved. 1
2 Handbook of hydroinformatics
In which k and n are constants to be determined from experimental data of reaction rate (rA) versus species concentration
(CA). To apply the favorable linear model, one can transform the above equation taking a natural logarithm from both sides
to have
ln ðr A Þ ¼ ln k + n ln ðCA Þ (2)
Another example that can be suited to multivariate problems is the polynomial regression. This topic will be discussed in a
separate section; however, the linear model can somewhat cover such problems by an interesting trick. Consider the fol-
lowing model
y ¼ a 0 + a1 z + a2 z 2 (3)
In this case, by introducing a new variable, the nonlinear model is transformed into a linear model. Assume that z ¼ x1 and
z2 ¼ x2 then
y ¼ a0 + a1 x 1 + a2 x 2 (4)
Although the values of x2 are not independent of x1, this does not have anything to do with the application of linear
regression algorithm. These two examples demonstrate that linear models for multivariate problems are a fundamental tool
that could not be ignored by practitioners, especially in the field of machine learning or more elegantly artificial intelligence
(Olive, 2017; Matloff, 2017).
In this section, we are going to go through a project called nutrient removal efficiency data and we use a data set con-
taining 7876 data to predict the total phosphorus (TP), ammonium (NH4-N), and total nitrogen (TN) removal efficiency of
an anaerobic anoxic-oxic membrane bioreactor system and the output values are predicted by nine input data given in
Table 1. This dataset was taken from the data reported from an article published by Yaqub et al. (2020).
In this part, we are only using one explanatory variable (e.g., TOC) to explain the output (RE of TN).
The linear regression diagram for this example is shown in Fig. 1. After successful fitting, it is well known that with
increasing TOC, the values of removal efficiency of TN increase.
3. Multivariate linear regression

When y is a function of n variables namely x1 to xn the simplest model for dependency is a linear model which can provide
an estimation yb of the function as
yb ¼ a0 + a1 x1 + a2 x2 +⋯+an xn (5)
TABLE 1 The attribute information of the nutrient removal efficiency project.
Code Input or output Description

TOC Input Total organic contents
TN Input Total nitrogen
TP Input Total phosphorous
COD Input Chemical oxygen demand
NH4-N Input Ammonium

SS Input Suspended solids
DO Input Dissolved oxygen
ORP Input Oxidation-reduction potential
MLSS Input Mixed liquor suspended solids
RE of NH4-N Output Removal efficiency of NH4-N
RE of TN Output Removal efficiency of TN

RE of TP Output Removal efficiency of TP
Advanced machine learning techniques Chapter 1 3
100
80
60
NH4-N-OUT
40
20
4000 6000 8000 10000 12000 14000 16000 18000

TOC
FIG. 1 Linear regression for nutrient removal efficiency project.
Where a0 to an are the model parameters to be determined using available data in combination with a proper linear
regression algorithm (Hackeling, 2017). Matrix notations help provide a compact form of the equations in multivariate
problems. In the matrix form
yb ¼ xT a (6)
Where xT ¼ [1 x1 x2 ⋯xn] indicates the transpose of column matrix x and similarly a shows the column matrix of param-
eters. Note that a new term i.e. x0 ¼ 1 is introduced to make the matrix product possible. Now the problem is reduced to
the determination of elements of matrix a under suitable constraints that eventually provide a system of linear equations
for specifying the model parameters. The first one must note that at each point the model error is defined as ei ¼ yi ybi
that is
ei ¼ yi xTi a (7)
Where xTi can be interpreted as the i0 th row of the matrix XT which includes the values of each variable at different points.
The error vector could be defined as a column matrix as
e ¼ y XT a (8)
Where both e and y have p elements (column vectors with p rows) and XT is an p (n + 1) matrix:
2 3
1 x1, 1 x2,1 xn,1
6 ⋯ 7
6 1 x1, 2 x2,2 xn,2 7
T
X ¼6 6 7 (9)
4 ⋮ ⋱ ⋮ 7 5
1 x1, p x2,p ⋯ xn, p
At the first glance, minimization of the absolute value of the error results in the best values of model parameters. But since
the error is presented by the vector e, one should talk about the minimization of a suitable norm of that. Moreover, the first
norm which adds up the absolute values of the elements of e would bring some problems in terms of differentiation. Thus,
the better choice is the second norm or the Euclidian norm of the error vector:
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
kek2 ¼ e21 + e22 + ⋯ + e2p (10)
And finally, since minimization of the above function is equivalent to the minimization of the summation on the right side,
the Sum of Squares of Errors (SSE) is taken as the target function in linear regression problems:
X
p

SSE ¼ e2i ¼ eT ∗e ¼ yT aT X ∗ y XT a (11)
i¼1
And minimization of this function without any constraint would result in an Ordinary Least Square (OLS) method for deter-
mination of model parameters, i.e., a0 to an. Obviously, this method involves vanishing the first partial derivatives of SSE
concerning the model parameters which provide the required n + 1 equations:
∂ðSSEÞ Xp Xn
¼ 2 yi xk,i + 2 ai XX T ki + ¼ 0, k ¼ 0,1,2,…,n
∂ak i¼1 i¼0
(12)
X
n X
p
; ai XX T ki ¼ yi xk,i
i¼0 i¼1
Or in the matrix form

1
XX T a ¼ Xy ! a ¼ XX T Xy (13)
Of course solution of the system of linear equations could be accomplished by rather low cost calculations. Indeed, the
determination of the inverse matrix for the solution of a linear system of equations is almost always avoided. Instead, a
direct method such as the Gauss-Jordan method of LU-factorization is advised when the system is not too large (say
for n < 100) and indirect methods such as SOR are advised for large systems. Of course, a favorable model, should not
have too many parameters, that is, the number of independent variables is kept low by incorporating the effective terms
and neglecting the variables with minor impact on the output. Therefore, no matter how large the data set, one intends to
solve a linear system of equations with a reasonable number of unknowns.
Fig. 2 illustrates synthetic data for which a random error is involved in the measurement. Here, the data point is deter-
mined by y ¼ 5 + 2.5x + error.
And the linear regression results in a0 ¼ 4.9943 and a1 ¼ 2.4824.
The calculations involve the solution of a linear system of n + 1 equations which involves inverting of the matrix which
generally requires O(np) operations where p lies between 2.373 and three depending on the direct method applied. For
example, Gauss-Jordan requires O(n3) operations; hence if the number of variables (terms) is doubled, the operations
are increased to eight times of the original problem. For many problems, this does not introduce much difficulty but
for problems with a large number of variables, iterative procedures are advised. Among these methods, the gradient descent
method is helpful for both problems with too many variables and problems with a very large set of data (Konishi, 2014;
Izenman, 2008).
4. Gradient descent method

This technique is also known as steepest descent, is a well-known method for the optimization of differentiable functions.
The strategy is based on taking steps proportional to the negative gradient of the function at each point to get closer to the
FIG. 2 Applying linear regression for the synthetic data.
local minimum of the function. For convex function, the global minimum of the function could be determined albeit by a
proper choice of the step size (Harrington, 2012). For a multivariable function F(x), the gradient descent method is put to
action as follows
xn+1 ¼ xn l—Fðxn Þ (14)
Where l is a small positive number which must be small enough to prevent missing the local minimum but not too small to
literally get the process stuck at the neighborhood of our initial guess. Note that l can be updated at each step and under
certain circumstances, values of this parameter could be chosen to guarantee the convergence. As an example, consider the
contour plot shown in Fig. 3.
FIG. 3 Contours of function y ¼ x21/4 + x22 x32/8 + 1: reaching local minimum by GD.
This plot represents the function y ¼ x21/4 + x22 x32/8 + 1 and if one starts searching for the minimum from (x1, x2) ¼ (4, 4),
the direction for the steepest descent is opposite to the gradient of the function at this point, i.e., r y(4,4) ¼ 2i 2j.
Hence, the new point is given by

x1 4 2
¼ l (15)
x2 4 2
Usually, at the first step, values of l are set less than unity and then larger values are examined. Practically, enlarging l is
permitted as long as it provides smaller values of the objective function. For the present problem, the new values of vari-
ables in terms of l could be replaced in function “y” and with straight-forward single variable optimization one arrives at
lopt ¼ 2 and in this way the local minimum is determined by just one shot at (x1, x2) ¼ (0, 0) and ymin ¼ 1. Of course, real-
world problems are not that easy and several steps with proper step size are required. In each step, a single variable opti-
mization might be performed to infer the best value of the step size. Of course, direct searching based on small step size, its
enlargement (usually by 10 times), and comparison of the resulting function values is better suited for sophisticated
functions.
As long as our regression problem is concerned, one has to put it in the form of a minimization problem to apply the
gradient descent method. The objective function is simply the sum of the squares of error

SSE ¼ yT aT X ∗ y XT a (16)
Whose gradient could be simply determined from previous arguments on its derivatives concerning model parameters;
that is

—SSE ¼ 2X∗ XT a y (17)
With an initial guess and a small enough value of step size, one can initiate the algorithm to obtain the right values of the
model’s parameters:
anew ¼ aold l—SSEðaold Þ (18)
There are three approaches to apply the gradient descent method for the training set of the data depending on how the huge
data set is handled by this method. In this respect, if the whole training set is used at every step of the calculations, the
method is addressed as batch gradient descent. For a very large data set, the batch gradient descent might not be economic.
Hence, two other variants are proposed by practitioners: Stochastic gradient descent and mini-batch gradient descent. In
stochastic GD at each step, only a small random set of the training data-set is used to calculate the gradient which makes it
much faster than the original Batch GD; however, there are some issues due to the stochastic nature of the method which
results in a nonmonotonic convergence to the local minimum and hence it needs stopping criteria to prevent bouncing
around the local minimum. On the other hand, the mini-batch variant of GD splits the training data-set into small sets
and computes the gradient on these sets which allows taking the benefit of parallel computation while it resolves the
problem of oscillatory convergence observed in stochastic GD (Shalev-Shwartz and Ben-David, 2014; Brownlee, 2016).
We are going to apply gradient descent to the nutrient removal efficiency project. We do these using parameters of TOC
and the removal efficiency of NH4-N. As can be seen from the Fig. 4, the sum of the squares of error decreases with
increasing epoch to a certain level until the error reaches a minimum value.
5. Polynomial regression
When the data do not depict a linear behavior, still linear regression could be applied by introducing new variables in terms
of the powers of the original variable as discussed in the previous section. Consider Fig. 3 which displays synthetic data
built up by y ¼ 5 + 3x 0.5x2 + noise. To apply linear regression to this problem, consider the following model
y ¼ a0 + a1 x 1 + a2 x 2 (19)
Where x1 ¼ x itself and x2 ¼ x2 so that the values of the new variable are known everywhere. Therefore, the matrix of vari-
ables X is built up as follows
3938.0
3937.5
3937.0
3936.5
3936.0
SSE
3935.5
3935.0
3934.5
3934.0
0 20 40 60 80 100
Epoch
FIG. 4 SSE in terms of epoch for the nutrient removal efficiency project.
2 3 2 3
1 x1,1 x2,1 1 x1,1 x21,1
61 7 6 7
6 x1,2 x2,2 7 6 1 x1,2 x21,2 7
XT ¼ 6
6⋮
7¼6 7 (20)
4 ⋮ ⋮ 7 6
5 4⋮ ⋮ ⋮ 7 5
1 x1,p x2,p 1 x1,p x21,p
And finally, the model parameters are determined as
2 3
5:1129

T 21 6 7
a ¼ XX Xy ¼ 4 3:0234 5 (21)
0:5138
The model prediction is also displayed in Fig. 5.
When there are indeed multiple variables (or features), a true polynomial regression is necessary to capture the rela-
tionship between these features. Mathematically, this relationship is depicted in nonlinear terms which contain a combi-
nation of the variables. For example, the second-order terms are either made by squaring a single variable or multiplying
one feature by another. In this respect, the number of terms of a polynomial of degree m for a problem with n features will be
determined as

n+m ðn + mÞ!
¼ (22)
m n!m!
11
data
model
10
y 8
5
0 1 2 3 4 5
X
FIG. 5 Noisy data and the second-degree least-square polynomial.
FIG. 6 The variance of a too high-degree polynomial which makes it an improper choice.
which includes all possible combinations of variables to construct a multivariate mth degree polynomial. Now, the question
is what degree is the best for a problem. A high-degree polynomial can get closer to more data points but it can easily lose
track by following the inherent noise of the data. In this regard, its predictions might not be relied on. When this is the case,
we say that the model has a high variance. Fig. 6 compares a polynomial of 20th degree with the second-degree polynomial.
Choosing the proper degree of the regression polynomial is a statistical task that considers the trade-off between the high
variance (unacceptable sensitivity of the model with high-degree polynomials) and the bias (underfitting the data with low
degree polynomials) (Shalev-Shwartz and Ben-David, 2014; Raschka, 2015; Ramasubramanian and Singh, 2018). The
issue is discussed in the following section.
FIG. 7 The high-degree polynomial regression.
6. Overfitting and underfitting

Compared to plain linear regression, a high-degree polynomial regression provides a better opportunity for fitting the
training data. As shown in Fig. 7, when a 40-degree polynomial model is applied to the training data of Fig. 7, the data
points are approximated to a great extent but obviously, its trend is not acceptable at both ends that’s why it’s considered
as an overfitting regression polynomial. The linear model neither follows the general trend nor touches the data points
satisfactorily; hence, it underfits the data. However, the quadratic regression satisfactorily follows the general trend
and presents a reasonable approximation as well (Swamynathan, 2019; Burger, 2018).
This is expected since the initial dataset was created through the introduction of some errors in a quadratic function.
Nonetheless, in many practical cases, there is no means to identify the original function behind the dataset. Therefore, there
is a need to determine the level of complexity of a model and to determine whether the model is underfitting or overfitting
the data (Ganesh, 2017).
7. Cross-validation
One of the most common ways to obtain an estimation of the performance of the model in terms of generalization involves
the utilization of cross-validation. It is said that the model is overfitting if it has good performance on the training data, but
provides poor generalization, which is determined by evaluating the cross-validation measures. However, the model is said
to be underfitting if it provides poor performance on both the training data and on the measures of cross-validation. Hence,
this is a satisfactory method for determining whether the model is too complex or too simple.
8. Comparison between linear and polynomial regressions

In this section, we intend to examine polynomial regression for the nutrient removal efficiency project. By drawing the
MLSS on the horizontal axis in terms of removal efficiency of TN data on the y-axis, a nonlinear downward trend is
obtained. The figure related to this example is given in Fig. 8.
After applying linear regression to the data in this example, it becomes clear that this regression cannot be placed cor-
rectly on this data (see Fig. 9).
After applying polynomial regression in the quadratic mode for the data of this example, it becomes clear that this
regression can fit these data better than the linear mode. This issue was also examined numerically and the value of
R2 related to this regression was obtained equal to 0.14, while the value of R2 related to linear regression was equal to
0.10 (see Fig. 10).
Now we change the polynomial features to the degree of 10 and run the lines again and this time we can see from the
following figure that the direction of the graph is changed. The value of R2 is 0.04 and we may seem to have a case of
overfitting (see Fig. 11).
90
80
70
60
50
40
30
20
0 2500 5000 7500 10000 12500 15000 17500 20000

FIG. 8 The values of MLSS in terms of removal efficiency of TN correspond to the nutrient removal efficiency project.
90
80
70
60
50
40
30
20
0 2500 5000 7500 10000 12500 15000 17500 20000

FIG. 9 Using linear regression to predict the values of MLSS and removal efficiency of TN.
90
80
70
60
50
40
30
20
0 2500 5000 7500 10000 12500 15000 17500 20000

FIG. 10 Using polynomial regression (quadratic mode) to predict the values of MLSS and removal efficiency of TN.
9. Learning curve
One of the other available methods is to evaluate the learning curves. Learning curves show the performance of the model
on both training and validation sets as a function of the size of the training set or the training iteration. In order to plot these
curves, the model is trained several times using various subsets of the training set where each subset is of a different size
( Jaber, 2016).
It should be noted that in general, a straight line cannot provide good performance for modeling the data. This is con-
firmed by the fact that the error level reaches a relatively constant level, which is very close to the other curve. It is worth
mentioning that when a model is underfitting, these learning curves are typically observed, i.e., curves that reach constant
error levels, which are close and relatively high. It should be noted that a common method for improving an overfitting
model is to provide it with more training instances until the validation error reaches the training error.
The learning curve for the nutrient removal efficiency project is given in the below diagram. By plotting train and also
test score versus the number of samples, it is clear that these two graphs become closer to each other with increasing the
number of training examples, and from 5000 onwards, the slope and the degree to which the two graphs approach each other
become less intense (see Fig. 12).
10. Regularized linear models

One of the possible ways for decreasing the overfitting phenomenon involves the regularization of the model, which is
another way to say limiting or restricting it. By reducing the degrees of freedom of the model, it will be harder for the
model to overfit the data. One of the east ways to regularize a polynomial model involves decreasing the polynomial degree.
90
80
70
60
50
40
30
20
0 2500 5000 7500 10000 12500 15000 17500 20000

FIG. 11 Using polynomial regression (degree of 10) to predict the values of MLSS and removal efficiency of TN.
Learning Curves (Ridge Regression)

0.006 Training score
Cross-validation score
0.004
0.002
0.000
Score
–0.002
–0.004
–0.006
1000 2000 3000 4000 5000 6000

Training examples
FIG. 12 The learning curve for the nutrient removal efficiency project.
In the case of a linear model, the regularization is generally performed by restricting the weights of the model. In order to
better illustrate the model, the Ridge regression and the Lasso regression models can be evaluated since these models utilize
two distinct methods for restricting the weights (Gori, 2017).
11. The ridge regression

The ridge regression is in fact a regularized or restricted version of the linear regression. In order to regularize the linear
P
n
regression model, a regularization term, i.e., a y2i is introduced into the cost function of the model. Adding this term will
i¼1
make the learning algorithm fit the data, while also minimizing the weights of the model. It is worth mentioning that this
regularization term must only be introduced into the cost function during the training stage. After training the model, the
unregularized performance measure can be used to assess the performance of the model (Saleh et al., 2019; Aldrich and
Auret, 2013).
The extent of the regularization of the model can be controlled using the hyper-parameter a. When a ¼ 0, the Ridge
regression will be the same as the linear regression. However, if a is very large, all the weights will be close to zero,
resulting in a flat line going through the mean values of the data. The cost function of the Ridge regression model is pre-
sented in Eq. (23).
1 Xn 2
J ðyÞ ¼ MSEðyÞ + a y
i¼1 i
(23)
2
2 at i ¼ 1 and not i ¼ 0. If w denotes the
It should be noted that the bias term, denoted by y0 is not regularized and the sum starts
vector of feature weights (y1 to yn), the regularization term will become 12 kwk2 , in which kwk2 signifies the ‘2 norm of
the weight vector. Moreover, for the gradient descent, aw is simply added to the MSE gradient vector (Alpaydin, 2020).
Similar to the linear regression model, the Ridge regression can be performed by calculating a closed-form equation or
by applying the gradient descent. The advantages and disadvantages are the same. The closed-form solution is presented in
Eq. (24). It should be noted that in this equation, A is the (n + 1) (n + 1) identity matrix, with one difference, i.e., the
presence of a 0 in the top-left cell, which corresponds to the bias term.
1
b
y ¼ XT X + aA XT y (24)
12. The effect of collinearity in the coefficients of an estimator

As mentioned earlier, a 0 is a complexity parameter that controls the amount of shrinkage: the larger the value of a, the
greater the amount of shrinkage and thus the coefficients become more robust to collinearity. Each color represents a dif-
ferent feature of the coefficient vector, and this is displayed as a function of the regularization parameter. This example also
shows the usefulness of applying Ridge regression to highly ill-conditioned matrices. For such matrices, a slight change in
the target variable can cause huge variances in the calculated weights. In such cases, it is useful to set a certain regular-
ization (alpha) to reduce this variation (noise) (see Fig. 13).
13. Outliers impact

Before discussing ridge regression, let’s look at an example of the effect of outliers on the slope of the regression line, and
then we will show how ridge regression can reduce these effects. For a data set containing 100 randomly generated points
with a slope of 0.5, after performing linear regression, the slope of the fitted line is equal to 0.47134857. The diagram for
this example is given below (see Fig. 14):
Now to show the effect of outliers on the previous example, we change two points in the data set: we replace the first
point of the chart with 200 and the last point of the chart with +200. After performing linear regression, we see that the
slope of the obtained line is equal to 1.50556072, which is significantly different from the slope of the previous chart and
shows the effect of outliers. The diagram for this example is given below (see Fig. 15).
After applying the ridge regression, we can see that this regression is substantially better than linear regression and it
recovers the original coefficient with a fairly good approximation. The slope of the line obtained in this regression is equal
to 1.00370714. The diagram for this example is given below (see Fig. 16).
200
100
weights
–100
10–3 10–5 10–7 10–9

alpha
FIG. 13 Ridge coefficients as a function of the regularization.
10
–5
–10
–20 –10 0 10 20
FIG. 14 Data set containing 100 randomly generated points and performing linear regression.
200
150
100
50
–50
–100
–150
–200
–20 –10 0 10 20
FIG. 15 Showing the outlier’s effect.
200
150
100
50
–50
–100
–150
–200
–20 –10 0 10 20
FIG. 16 Using ridge regression to offset the impact of outliers.
14. Lasso regression

Another regularized version of the linear regression is the Lasso regression, which comes from Least Absolute Shrinkage
and Selection Operator Regression. Similar to the Ridge regression, this regularized version introduces a regularization
term into the cost function, except that instead of the half of the square of the ‘2 norm, it utilizes the ‘1 norm of the weight
vector. This is expressed in Eq. (25) (Sra et al., 2012; Bali et al., 2016).
Xn
J ðyÞ ¼ MSEðyÞ + a i¼1 yi (25)
One of the differences between this type of regression and the Ridge regression involves the fact that as the parameters get
closer to the global optimum, the gradients get smaller, and the gradient descent becomes slower, increasing the likelihood
of convergence because of the lack of bouncing around. Another difference is that by increasing a, the optimal parameters
gradually get closer to the origin, but they will never reach zero.
It should be noted that at yi ¼ 0 for i ¼ 1, 2, …, n, the cost function of the Lasso regression is not differentiable; however, if
in cases where yi ¼ 0, the subgradient vector g is utilized, the gradient descent will perform well enough. A subgradient vector
equation that can be utilized for gradient descent with the cost function of the Lasso regression is presented by Eq. (26).
0 1
signðy1 Þ
B signðy Þ C
B 2 C
B C 8
B : C
B
B
C
C < 1 if yi < 0
>
gðy, J Þ ¼ ry MSEðyÞ + aB : C where signðyi Þ ¼ 0 if yi ¼ 0 (26)
B : C >
B C :
B C + 1 if yi > 0
B signðyn Þ C
B C
@ A
After applying Lasso regression to the example data related to the ridge regression, it was shown that this regression can fit
the data of this example with a much better approximation than linear one and be less affected by outliers. The slope of the
line related to this regression is 1.06289489. The diagram for this example is given below (see Fig. 17):
200
150
100
50
–50
–100
–150
–200
–20 –10 0 10 20
FIG. 17 Using Lasso regression to offset the impact of outliers.
15. Elastic net

The elastic net is the middle point between the Ridge and the Lasso regression models. In this model, the regularization term
is a combination of the regularization terms from Ridge and Lasso regression models, which is called the mix ratio r. It
should be noted that the elastic net will be equal to the Ridge regression when r is set to 0, while it equals the Lasso
regression when r is set of 1. This is expressed in Eq. (27) (Humphries et al., 2018; Forsyth, 2019).
Xn 1 r Xn
J ðyÞ ¼ MSEðyÞ + ra i¼1 yi + a i¼1 y2i (27)
2
The question is when to use the linear regression without regularization, the Ridge regression, the Lasso regression, or the
Elastic Net. In order to make this decision, it should be noted that a level of regularization is always preferred. Therefore, using
the plain linear regression model must be avoided as much as possible. The Ridge regression model can be a good default
option. However, if there are only a limited number of useful features, it is better to utilize the Lasso regression or the
elastic net, since they usually set the weights of the useless features to zero, as noted earlier. Nonetheless, when the number
of features is larger than the number of training instances, or when there is a strong correlation between several features, it is
better to utilize the elastic net instead of the Lasso regression since the Lasso can have erratic behaviors in such cases.
After applying elastic regression to the example data for ridge and lasso regressions, it was shown that this regression
has a much better approximation than two previous regressions and is less affected by outliers. The slope of the fitted line
for this regression was 0.74724704. The diagram for this example is given below (see Fig. 18):
16. Early stopping

Stopping the training once the validation error is minimized is another distinct way to regularize iterative learning algo-
rithms, including the gradient descent. This method is known as “early stopping.” When applying the early stopping
method, as soon as the validation error is minimized, the training is halted. This is a simple, elegant, and efficient method
for the regularization of iterative learning algorithms (Shukla, 2018).
It should be noted that when using stochastic and mini-batch gradient descent, it is difficult to determine if the error is
minimized or not since the curves are not this smooth. A possible solution is to stop the training after the validation error has
200
150
100
50
–50
–100
–150
–200
–20 –10 0 10 20
FIG. 18 Using elastic net regression to offset the impact of outliers.
stayed above the minimum for a while and the possibility of a better performance by the model is not very high. Afterward,
the parameters of the model can be set at the values they were when the validation error was at the minimum point.
17. Logistic regression

Some algorithms using regression could be applied for classification problems. The possibility of an example belonging to a
particular class can be examined by logistical regression. For example, how likely is it that an email will be spam? Logit
regression is another name for this regression. Whether or not a sample belongs to a class in this model depends on the
probability of more or less than 50%, respectively. In this regression, the probability of more than 50% is called the “pos-
itive class” represented by “1” and the probability of less than 50% is called “negative class” which is indicated by “0.”
Such division is called a binary classification (Mohammed et al., 2016; Lesmeister, 2015).
18. Estimation of probabilities

The question that may be encountered here is how the logistic regression works. A logistic regression model is similar to a
linear regression model in the sense that it computes the weighted sum of the inputs along with a bias term. However, its
main difference with the linear regression model is that it does not provide a direct result; rather, its output is the logistic of
the result. This is expressed in Eq. (28).

b
p ¼ hu ðxÞ ¼ s XT u (28)
It should be noted that the logistic, which is denoted by s(.), is a sigmoid or S-shaped function, whose output ranges from 0
to 1. This function is expressed by Eq. (29) and it is depicted in Fig. 19.
1
s ðt Þ ¼ (29)
1 + exp ðtÞ
After the logistic regression model estimates the probability, i.e., b
p ¼ hy ðxÞ, of an instance x belonging to the positive class,
the model can calculate the prediction ybin a straightforward fashion. This prediction is expressed in Eq. (30) (Lantz, 2019).

0 if bp < 0:5
b
y¼ (30)
1 if bp 0:5
It should be noted that when t < 0, we will have s(t) < 0.5, while when t 0, we will have s(t) 0.5. Accordingly, if xTy is
positive, the logistic regression model’s output as the prediction will be equal to 1; otherwise, the output will be equal to 0.
It is worth mentioning that the score t is usually called the logit. This is because the logit function, which is expressed as
logitðpÞ ¼ log p
1p , is in fact the inverse of the logistic function. Moreover, when calculating the logit of the estimated
probability, i.e., p, the output will be t. It should also be noted that the logit is sometimes called the log-odds because it can
be defined as the logarithm of the ratio of the estimated probability of the positive class to the estimated probability of the
negative class (Harrell, 2015).
1.00
1
σ(t) =
1.75 1 + e–t
0.50
0.25
0.00
–10.0 –7.5 –5.0 –2.5 0.0 2.5 5.0 7.5 10.0

t
FIG. 19 The logistic function.
19. Training and the cost function

Based on the abovementioned considerations, it can be concluded that a logistic regression model is capable of estimating
probabilities and providing predictions. However, the method for training the model must be explained as well. The main goal
of training the model is to find a suitable value for the parameter vector y in a way that the model can provide high probabilities
for the positive instances (y ¼ 1), while it provides low probabilities for the negative instances (y ¼ 0). This notion is expressed
by the cost function, which is presented in Eq. (31) below for a single training instance (x) (Ayyadevara, 2018).

log ðb
pÞ if y ¼ 1
cðuÞ ¼ (31)
log ð1 b
pÞ if y ¼ 0
In order to better understand this cost function, it should be noted that when t ! 0, the value of log(t) will become sig-
nificantly larger. Therefore, if the model provides a probability close to 0 for a positive instance or a probability close to 1
for a negative instance, then the cost will become significantly large. In contrast, as t ! 1, we will have log(t) ! 0. In other
words, if the estimated probability for a negative instance is close to 0 or if the estimated probability for a positive instance
is close to 1, then the cost will be much closer to 0. Incidentally, the latter case is what we are actually seeking (Raschka and
Mirjalili, 2019; Amamra et al., 2018).
On the other hand, the cost function over the entire training set can be an indicator for the average cost over all individual
training instances. This general cost function can be presented as the log loss, which is expressed in Eq. (32).
1 Xm h ðiÞ ðiÞ ðiÞ ð iÞ
i
JðuÞ ¼ y log b
p + 1 y log 1 b
p (32)
m i¼1
Unfortunately, there isn’t any closed-form formula to obtain a value for y in a way that the log loss or the general cost
function can be minimized. In other words, there is no equivalent available for the normal equation. In contrast, the
log loss is a convex function, so using any optimization algorithms, such as the gradient descent, can provide the global
minimum for this function as long as the learning rate is small enough and the algorithm has enough time. Moreover,
Eq. (33) presents the partial derivatives of the log loss based on parameter uj.
∂ 1 Xm
ðiÞ
JðuÞ ¼ s uT XðiÞ yðiÞ xj (33)
∂uj m i¼1
It should be noted that the above equation is similar to partial derivatives of the cost function at the batch gradient descent
equation in that for each of the instances, it provides the prediction error and multiplies it by the value of the jth feature.
Afterward, it calculates the average over all the training instances. When the gradient vector, which includes all the partial
derivatives, is available, it can be utilized for employing the Batch Gradient Descent algorithm. These were the main steps
for training a logistic regression model. It is also worth mentioning that in the stochastic gradient descent algorithm, each
instance is evaluated separately, while in the mini-batch gradient descent algorithm, a mini-batch is utilized each time.
For performing logistic regression by python the following data displayed in Figs. 20 and 21 is used.
Finally, the decision boundary is specified by the violet from Fig. 22.
y=1
1.00 y=0
0.75
0.50
Microchip Test 2
0.25
0.00
–0.25
–0.50
–0.75
–0.75 –0.50 –0.25 0.00 0.25 0.50 0.75 1.00

Microchip Test 1
FIG. 20 Data for logistic regression.
0.70
0.65
0.60
cost function
0.55
0.50
0.45
0 2500 5000 7500 10000 12500 15000 17500 20000

number of iteration
FIG. 21 The cost functions vs increasing of iterations.
y=1
1.00
y=0
0.75
0.50
Microchip Test 2
0.25
0.00
–0.25
–0.50
–0.75
–1.00
–1.00 –0.75 –0.50 –0.25 0.00 0.25 0.50 0.75 1.00
Microchip Test 1
FIG. 22 The decision boundary plot.
20. Conclusions
This chapter describes different types of regressions and how they can be used to solve real problems. To solve our chal-
lenges in different sections, we utilize the Scikit-learn Python library. In this chapter, a complete description of different
regressions such as robust, multiple, regulized (ridge, lasso, and elastic net), polynomial, and logistic is given. Introduction
and practical use of methods such as gradient descent, cross-validation, and the learning curve are illustrated and how to
deal with issues such as outliers, overfitting, and underfitting to have a better approach to regression issues became clear.
Appendix: Python code

Linear regression
The following Python code presents the steps of calculations for this regression:
In [1]:
import numpy as np
import pandas as pd
In [2]:
df = pd.read_csv('nutrient.data', delim_whitespace=True, header=None)
In [3]:
col_name = ['TOC', 'TN' , 'TP', 'COD', 'NH4-N', 'SS', 'DO', 'ORP', 'MLSS', 'NH4-N-OUT'
, 'TN-OUT', 'TP-OUT']
In [4]:
df.columns = col_name
In [5]:
import matplotlib.pyplot as plt

import seaborn as sns
In [6]:
X = df['TOC'].values.reshape(-1,1)
In [7]:
y = df['TN-OUT'].values
In [8]:
from sklearn.linear_model import LinearRegression
In [9]:
model = LinearRegression()
In [10]:
model.fit(X, y)
Out[10]:
LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)
In [11]:
model.coef_
Out[11]:
array([0.02305826])
In [12]:
model.intercept_
Out[12]:
66.88380657331433
In [66]:
plt.figure(figsize=(12,10));
sns.regplot(X, y);
plt.xlabel('TOC')
plt.ylabel("TN-OUT")
plt.show();
Gradient descent method

The following Python code presents the steps of implementing the Gradient Descent method for this example:
In [1]:
import pandas as pd
import numpy as np
sns.set_style("whitegrid")
%matplotlib inline
In [4]:

In [5]:
X = df['TOC'].values.reshape(-1,1)
In [6]:
from sklearn.preprocessing import StandardScaler

sc_x = StandardScaler()
sc_y = StandardScaler()
X_std = sc_x.fit_transform(X)
y_std = sc_y.fit_transform(y.reshape(-1,1)).flatten()
In [7]:
alpha = 0.0001
w_ = np.zeros(1 + X_std.shape[1])
cost_ = []
n_ = 100
for i in range(n_):
y_pred = np.dot(X_std, w_[1:]) + w_[0]
errors = (y_std - y_pred)
w_[1:] += alpha * X_std.T.dot(errors)

w_[0] += alpha * errors.sum()
cost = (errors**2).sum() / 2.0

cost_.append(cost)
In [8]:
plt.figure(figsize=(10,8))
plt.plot(range(1, n_ + 1), cost_);
plt.ylabel('SSE');
plt.xlabel('Epoch');
In [9]:
w_
Out[9]:
array([1.02318154e-15, 3.21037679e-02])
Comparison between linear and polynomial regressions

The code related to this example is given below:
In [1]:
import pandas as pd
import numpy as np
sns.set_style("whitegrid")
%matplotlib inline
In [2]:

In [3]:
X = df['MLSS'].values.reshape(-1,1)
In [4]:
plt.scatter(X, y);
In [5]:
lr = LinearRegression()
lr.fit(X.reshape(-1, 1), y)
model_pred = lr.predict(X.reshape(-1,1))
plt.scatter(X, y);
plt.plot(X, model_pred);
print("R^2 score = {:.2f}".format(r2_score(y, model_pred)))
R^2 score = 0.10
In [6]:
poly_reg = PolynomialFeatures(degree=2)
X_poly_b = poly_reg.fit_transform(X.reshape(-1, 1))
lin_reg_2 = LinearRegression()
In [7]:
lin_reg_2.fit(X_poly_b, y)
Out[7]:
LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)
In [8]:
X_fit = np.arange(X.min(), X.max(), 1)[:, np.newaxis]
In [9]:
X_fit
Out[9]:
array([[ 99.08039698],
[ 100.08039698],
[ 101.08039698],
...,
[19974.08039698],
[19975.08039698],
[19976.08039698]])
Another random document with
no related content on Scribd:
Immediately on his arrival he began to speak and write Greek, to
openly frequent the studios of sculptors and painters, to buy statues
and pictures, and to compose works on the fine arts. The Athenians
were as much delighted as surprised to see one of their conquerors
partake in their most cherished tastes, and thus protest against the
unjust disdain of the rest. Their gratitude, which was always very
noisy, as we know, overwhelmed Atticus with all sorts of flattery.
Decrees in his honour were multiplied, he was offered all the
dignities of the city; they even wished to raise statues to him. Atticus
hastened to refuse everything; but the effect was produced, and the
report of such great popularity did not fail to reach Rome, carried by
those young men of high family who had just finished their education
in Greece. In this manner the reputation of Atticus lost nothing by
his absence; people of taste talked of this enlightened connoisseur of
the arts who had made himself remarked even at Athens; and during
this same time the politicians, no longer seeing him, lost the habit of
classing him with a political party.
This was an important step. There remained a more important one
to take. Atticus had seen betimes that to be rich is the first condition
of independence. This general truth was even more evident at that
time than at any other. How many people were there whose conduct
during the civil wars can only be explained by the state of their
fortunes! Curio had but one motive for serving Caesar, whom he did
not like, namely, the pressure of his creditors; and Cicero himself
puts among the chief reasons that prevented him going to Pompey’s
camp, whither all his sympathies called him, the money that Caesar
had lent him, and which he could not repay. To escape
embarrassments of this kind and gain entire liberty, Atticus resolved
to become rich, and became so. It is of importance, I think, to give
here a few details to show how people got rich at Rome. His father
had left him a rather moderate fortune, two million sesterces
(£16,000). When he left Rome he sold almost all the family property,
that he might leave nothing behind to tempt the proscribers, and
bought an estate in Epirus, in that country of large herds, where the
land brought in so much. It is probable he did not pay much for it.
Mithridates had just ravaged Greece, and, as there was no money,
everything went at a low price. This domain quickly prospered under
skilful management; new lands were bought every year out of the
surplus revenue, and Atticus became one of the great landed
proprietors of the country. But is it likely that his wealth came to him
solely through the good management of his land? He would have
willingly had this believed, in order to resemble somewhat in this
manner Cato and the Romans of the old school. Unluckily for him,
his friend Cicero betrays him. In reading this unreserved
correspondence we are not long in perceiving that Atticus had many
other ways of enriching himself besides the sale of his corn and
herds. This skilful agriculturalist was at the same time a clever
trader, who carried on all businesses successfully. He excelled in
drawing a profit, not only from the follies of others, which is
common, but even from his own pleasures, and his talent consisted
in enriching himself where others ruin themselves. We know for
instance that he was fond of fine books; then, as now, this was a very
costly fancy, but he knew how to make it a source of handsome
profits. He collected in his house a large number of skilful copyists
whom he trained himself; after having made them work for him,
when his passion was satisfied he set them to work for others, and
sold the books they copied to the public very dear. He was thus a
veritable publisher for Cicero, and as his friend’s works sold well it
happened that this friendship, which was full of charm for his heart,
was not without use to his fortune.[144] This commerce might be
avowed, and a friend to letters was not forbidden to become a
bookseller; but Atticus engaged as well in many transactions that
ought to have been more repugnant to him. As he saw the success
that everywhere attended gladiatorial fights, and that no festival took
place without one of these grand butcheries, he thought of raising
gladiators on his estates. He had them carefully instructed in the art
of dying gracefully, and hired them out at a high rate to cities that
wished to amuse themselves.[145] It must be acknowledged that this is
not a suitable trade for a scholar and a philosopher; but the profits
were large, and the philosophy of Atticus was accommodating as
soon as there was a good profit to make. Besides, he was a banker
when the opportunity offered, and lent at a high rate of interest, as
the greatest nobles of Rome did without scruple. Only, he was more
circumspect than others, and took care to appear as little as possible
in the affairs that he conducted, and he had, no doubt, in Italy and
Greece, clever agents who made the most of his capital. His business
relations extended throughout the world; we know of his debtors in
Macedonia, Epirus, Ephesus and Delos, almost everywhere. He lent
to private persons; he lent also to cities, but quite secretly, for this
business was then as little esteemed as it was lucrative, and persons
who took to it were not considered either honest or scrupulous. So
Atticus, who thought as much of his reputation as of his fortune,
would not let any one know that he conducted this sort of business.
He carefully concealed it even from his friend Cicero, and we should
be ignorant of it now if he had not experienced some untoward
accidents in this risky business. Although usually great profits were
gained, some dangers also were run. After having suffered the
Roman domination for two centuries, all the cities, allied and
municipal, and especially those of Asia, were completely ruined.
They all had less revenue than debts, and the proconsuls, combined
with the farmers of the taxes, carried off their resources so
completely that there was nothing left for the creditors to take,
unless they exerted themselves. This is what happened once to
Atticus, notwithstanding his activity. We see that Cicero rallies him
in one of his letters about the siege he is going to lay to Sicyon;[146]
this siege was evidently that of some recalcitrant debtors; Atticus
never made any other campaigns; and, in truth, this one succeeded
badly. While he thus went to war against this unfortunate indebted
town, the senate took pity on it, and protected it by a decree against
its too exacting creditors, so that Atticus, who set out from Epirus as
a conqueror, with flying banners, was reduced, says Cicero, when he
had arrived under the walls, to extract from the Sicyonians a few
poor crowns (nummulorum aliquid) by means of prayers and
flatteries.[147] We must, however, suppose that Atticus was usually
more lucky in the investment of his funds, and by his well-known
prudence we are assured that he knew how to choose more solvent
debtors. All this business that he carried on would certainly soon
have made him very rich; but he had no need to take so much
trouble, for while he was working so skilfully to make his fortune it
came to him ready made from another quarter. He had an uncle, Q.
Caecilius, who passed for the most terrible usurer of Rome, where
there were so many, and who only consented to lend to his nearest
relations, and as a special favour, at the rate of one per cent. per
month. He was a hard, inflexible man, who had rendered himself so
hateful to everybody that the people could not be prevented from
outraging his corpse on the day of his funeral. Atticus was the only
person who had been able to get on with him. Caecilius adopted him
by will, and left him the greater part of his property, ten million
sesterces, a little more than £80,000. Henceforth his fortune was
made, he was independent of everybody, and free to follow his own
inclinations.
But was it not to be feared, that when he was back in Rome, the
resolution that he took to shun all ties would have a bad look? He
could not decently pretend indifference or fear as a reason for
keeping aloof from parties; he had to find a more honourable motive
and one that he might proclaim; a school of philosophy furnished
him with it. The Epicureans, sacrificing everything to the
conveniences of life, said that it was good to abstain from public
employments to avoid the worry they brought. “Do not engage in
politics,” was their favourite maxim. Atticus professed to be an
Epicurean; henceforth his abstention had a plausible pretext, fidelity
to the opinions of his sect, and if he was blamed, the blame fell upon
the whole school, which always makes the share of each individual
very light. Was Atticus in reality a veritable and complete Epicurean?
This is a question that the learned discuss, and that the character of
this personage easily permits us to solve. To suppose that in anything
whatever he attached himself scrupulously to a school, and pledged
himself to be a faithful disciple of it, would be to know him ill. He
had studied them all for the pleasure that this study gave to his
inquisitive mind, but he was determined not to be a slave to their
systems. He had found a principle in the Epicurean morals that
suited him, and seized it in order to justify his political conduct. As to
Epicurus himself and his doctrine, he cared very little about them,
and was ready to abandon them on the first pretext. Cicero shows
this very pleasantly in a passage of the De Legibus. He represents
himself in this work chatting with Atticus on the banks of the
Fibrenus, under the delightful shades of Arpinum. As he wishes to
trace back the origin of laws to the gods, it is necessary for him first
to lay down that the gods concern themselves with men, which the
Epicureans denied. He turns then to his friend, and says: “Do you
admit, Pomponius, that the power of the immortal gods, their
reason, their wisdom, or, if you like it better, their providence, rule
the universe? If you do not admit it I must begin by demonstrating it.
—Well then, replies Atticus, I admit it, if you like, for thanks to these
birds that are singing, and to the murmuring of these brooks, I have
no fear that any of my fellow-disciples may hear me.”[148] Here is a
very accommodating philosopher, and the school will not get very
much good from an adept who abandons it as soon as he is sure that
it will not be known. The character of Atticus is here well seen. To
embrace an opinion resolutely is to pledge oneself to defend it, and
to expose oneself to the necessity of fighting for it. Now,
philosophical quarrels, although they be not bloody, are no less
desperate than others; this is war all the same, and Atticus wishes for
peace in all things, at least for himself. It is amusing to examine the
part that Cicero gives him in the philosophical dialogues into which
he introduces him. In general he does not discuss, he incites to
discussion. Inquisitive and insatiable, he asks, he interrogates
continually; he compels a reply, he raises objections, he animates the
combatants, and during this time he quietly enjoys the fight without
ever taking part in it. We shall see, by and by, that this was exactly
the part he took in politics.
Atticus remained twenty-three years away from Rome, only
visiting it at long intervals and usually remaining but a very short
time. When he thought that, by his long absence, he was quite free
from the ties that attached him to the political parties, when he had
gained independence with wealth, when he had secured himself
against all the reproaches that might be made him on his conduct by
giving his prudence the appearance of a philosophical conviction, he
thought of returning definitively to Rome and there resuming his
interrupted course of life. He chose a moment for returning when all
was calm, and, as if to break entirely with his past, he came back with
a new surname, by which people soon learnt to call him. This name
of Atticus, which he brought back from Athens, seemed to indicate
clearly that he would only live henceforth for the study of letters and
the enjoyment of the arts.
From this moment he divided his time between residence in Rome
and in his country houses. He quietly wound up his banking affairs,
some of which were still standing over, and took measures to hide
from the public the sources of his wealth. He kept only his estates in
Epirus and his houses in Rome, which brought him in a good deal,
and the profits of which he could acknowledge. His property
continued to increase, thanks to the way in which he managed it.
Besides, he had none of those weaknesses which might have
endangered it; he did not care about buying or building, he did not
possess any of those splendid villas at the gates of Rome or at the
sea-side, the keeping up of which ruined Cicero. He still sometimes
lent money, but, as it appears, rather to oblige than to enrich himself.
He was careful, besides, to choose safe persons, and showed himself
without pity when debts fell due. This he did, he said, in the interest
of his debtors, for, in tolerating their negligence he would encourage
them to ruin themselves. But he did not stand upon ceremony in
dismissing those with whom his money would have run some risk,
even if they were his nearest relations. Cicero, relating to him one
day that their common nephew, the young Quintus, had come to him
and tried to move him by the picture of his poverty, added: “I took
then something of your eloquence; I answered nothing.” It was a
good contrivance, and Atticus must have employed it more than once
with regard to his brother-in-law and his nephew, who were always
without money. He had learnt how to make for himself a high social
position at small cost. He lived in his house on the Quirinal—which
was more spacious and commodious within than handsome without,
and which he repaired as little as possible—among the works of art
that he had selected in Greece, and the lettered slaves whom he had
carefully trained himself, and whom everybody envied him. He often
assembled the cultivated people of Rome at feasts where there was a
great display of learning. His hospitality did not cost much, if it is
true, as Cornelius Nepos, who had seen his accounts, asserts, that he
only spent 3000 asses (£6) a month on his table.[149] Cicero, always
indiscreet, relates that Atticus often served to his guests very
common vegetables on very costly dishes;[150] but what did it matter?
every one considered himself fortunate in taking part in these select
parties, where they heard Atticus talk and Cicero’s finest works read
before they were published, and it may be said that all the most
distinguished persons of that great period held it an honour to
frequent that house on the Quirinal.
II.
Of all the advantages of Atticus, one is most tempted to envy him
his good fortune in attaching to himself so many friends. He took
much trouble to do so. From his arrival at Rome we see him busied
in putting himself on good terms with everybody, and using every
means to please men of all parties. His birth, his wealth, and the
manner in which he had acquired it, drew him towards the knights;
these rich farmers of the taxes were his natural friends, and he soon
enjoyed a great reputation among them; but he was not less
connected with the patricians, usually so disdainful of all who were
not of their caste. He had taken the surest means to conciliate them,
which was to flatter their vanity. He took advantage of his historical
knowledge to manufacture for them agreeable genealogies, in which
he made himself partaker in a good many lies, and supported their
most fanciful pretensions by his learning. This example shows at
once his knowledge of the world, and the advantage he drew from it
when he wished to gain the friendship of anybody. We can see what a
close observer he must have been, and the talent that he had for
seizing and profiting by the weak side of people, merely by
considering the nature of the services that he rendered to each
person. He had proposed to Cato to undertake the management of
his affairs at Rome during his absence, and Cato hastened to accept
this: a steward of such capacity was not to be despised by a man who
cared so much for his wealth. He had gratified the vain Pompey, by
busying himself in selecting in Greece some fine statues to ornament
the theatre he was building.[151] As he well knew that Caesar was not
accessible to the same kind of flattery, and that, to attract him, more
real services were necessary, he lent him money.[152] Naturally, he
attached himself by preference to the heads of parties; but he did not
neglect others when he could serve them. He carefully cultivated
Balbus and Theophanes, the confidants of Caesar and Pompey; he
even went sometimes to visit Clodius and his sister Clodia, as well as
other people of doubtful reputation. Having neither rigid scruples
like Cato, nor violent aversions like Cicero, he accommodated
himself to everybody; his good-nature lent itself to everything; he
suited all ages as well as all characters. Cornelius Nepos remarks
with admiration, that while yet very young he charmed the old Sulla,
and that when very old he could please the young Brutus. Atticus
formed a common link between all these men who were so different
in temper, rank, opinions, and age. He went continually from one to
the other, as a sort of pacific ambassador, trying to bring them
together and unite them, for it was his habit, says Cicero, to form
friendships between others.[153] He removed the suspicions and
prejudices which prevented them knowing one another; he inspired
them with the desire to see each other and become intimate, and if,
later, any differences arose between them, he became their
intermediary, and brought about explanations which made them
friends again. His masterpiece in this line is to have succeeded in
reconciling Hortensius and Cicero, and making them live amicably
together notwithstanding the violent jealousy that separated them.
What trouble must he not have had to calm their irritable vanity,
which was always ready to fly out, and which fate seemed to take
pleasure in exciting still more, by putting them in constant rivalry!
All these acquaintanceships of Atticus were certainly not real
friendships. He visited many of these personages only for the
advantage that his safety or his wealth might draw from them; but
there are a great number of others who were really his friends. To
confine ourselves to the most important, Cicero loved no one so
much as he did him; Brutus showed him an unreserved confidence to
the last, and on the eve of Philippi wrote him his last confidences.
There remain too many striking proofs of these two illustrious
friendships for them to be called in question, and we must admit that
he was able to inspire a lively affection in two of the noblest minds of
that time. At first we are very much surprised at this. His prudent
reserve, that openly avowed determination to keep clear of all
entanglements in order to escape all danger, ought, as it seems, to
have kept aloof from him men of conviction who sacrificed fortune
and life for their opinions. By what merit was he nevertheless able to
attach them to himself? How was a man so taken up with himself
and so careful of his interests able to enjoy so fully the pleasures of
friendship, which seem at first sight, to exact devotedness and self-
forgetfulness? How did he succeed in making the moralists, who
assert that egotism is the death of true affection, belie themselves?
[154]
This is still one of those problems of which the life of Atticus is full,
and it is the most difficult to solve. Seen from a distance, even
through the praises of Cicero, Atticus does not seem attractive, and
one would not be tempted to choose him for a friend. And yet it is
certain that those who lived with him did not judge him as we do.
They loved him, and felt themselves from the first inclined to love
him. That general good-will that he inspired, that determination of
every one to pardon or not to see his defects, those lively friendships
that he called forth, are evidences that it is impossible to resist,
whatever surprise they may cause us. There was, then, about this
personage something else than we see; he must have possessed a
kind of attraction that is inexplicable to us, which was personal to
him, and which has disappeared with him. For this reason it is no
longer possible for us to understand thoroughly that strange
attraction that he exercised at first sight on all his contemporaries.
We can, however, form some idea of it, and the writers who knew
him, especially Cicero, give a glimpse of some of those brilliant or
solid qualities by which he gained over those who approached him. I
shall enumerate them according to their testimony, and if they still
do not seem sufficient to justify altogether the number of his
friendships and their ardour, we must join to them in thought that
personal charm that it is impossible now to define or recover because
it vanished with himself.
Firstly, he had a good deal of cultivation, everybody agrees about
that, and a sort of cultivation especially agreeable to the society that
he frequented. He was not solely one of those pleasant triflers who
charm for a moment on a passing acquaintance, but who have not
the qualifications for a longer connection. He was a person of many
attainments and solid knowledge; not that he was a man of deep
learning, this title is not a great recommendation in the intercourse
of society; Cicero thought that people like Varro, who are perfect
mines of knowledge, are not always amusing, and relates that when
the latter came to see him at Tusculum he did not tear his mantle in
trying to retain him.[155] But, without being really a scholar, Atticus
had touched on everything in his studies, the fine arts, poetry,
grammar, philosophy, and history. Upon all these subjects he
possessed just and sometimes original ideas; he could discuss
matters with learned men without too great disadvantage, and he
always had some curious detail to tell those who were not so. Pascal
would have called him a cultivated gentleman (honnête homme); in
everything he was an intelligent and enlightened amateur. Now, for
several reasons, the knowledge that an amateur acquires is of the
kind most current in society. Firstly, as he does not study according
to rule, he interests himself above all in curiosities; he learns by
preference racy and novel details, and it is precisely these that people
of society want to know. Besides, the very multiplicity of the studies
which tempt him, prevents him exhausting any; his caprice always
carries him off elsewhere before he has thoroughly examined
anything. The result is that he knows a great many things, and
always within the limits in which it pleases men of the world to know
them. In fact, the characteristic of the amateur is to do everything,
even what he only does for a moment, with enthusiasm. As it is a
personal taste that draws him to his studies, and as he only continues
them as long as they interest him, his language is more lively when
he speaks of them, his tone freer and more original, and
consequently more agreeable, than that of scholars by profession.
Such is the notion we must form of the learning of Atticus. It was too
extensive for his conversation ever to become monotonous; it was
not deep enough to run the risk of being tedious; it was, in fine,
living, for when things are done with enthusiasm it is natural to
speak of them with interest. This is what made his conversation so
attractive, and this is how he charmed the most fastidious and least
favourably disposed minds. He was still quite young when the aged
Sulla, who had no reason to like him, met him at Athens. He took so
much pleasure in hearing him read Greek and Latin verses and talk
about literature, that he would not leave him, and wished by all
means to take him back with him to Rome. Long after, Augustus felt
the same charm; he was never tired of hearing Atticus talk, and when
he could not go to see him, he wrote to him every day simply to
receive his answers, and thus to continue, in some sort, those long
conversations with which he was so delighted.
We can imagine, then, that the first time people met this
accomplished man they felt themselves drawn towards him by the
charm of his conversation. In proportion as he was better known,
other and more solid qualities were discovered, which retained those
whom his culture had attracted. In the first place there was a great
security in his intercourse. Although he was connected with people
holding very diverse opinions, and though, through them, he had the
secrets of all parties, he was never reproached with having betrayed
these to anybody. We cannot see that he ever furnished a serious
cause to any of his friends to keep aloof from him, or that any of his
connections were broken otherwise than by death. This intercourse,
so secure, was at the same time very easy. No one was ever more
indulgent and accommodating. He took care not to weary by his
demands or to repulse by bluntness. Those storms which so often
troubled the friendship of Cicero and Brutus were not to be feared in
his. It was rather one of those calm and uniform intimacies which
grow stronger from day to day by their regular continuance. It was
this especially that must have charmed those politicians who were
oppressed and fatigued by that bustling activity which used up their
lives. On coming out of this whirlwind of business, they were happy
to find, at a few paces from the Forum, that peaceful house on the
Quirinal into which outside quarrels did not enter, and to go and
chat for a moment with that even-tempered and accomplished man
who always received them with the same smile, and in whose good-
will they had such a tranquil confidence.
But nothing, assuredly, could have won him so many friends as his
readiness to oblige them. This was inexhaustible, and it could not be
asserted that it was interested, since, contrary to custom, he gave
much and demanded nothing. Here again is one of the reasons why
his friendships were so lasting, for it is always this sort of
interchange that we think we have a right to demand, and the
comparisons that we make, in spite of ourselves, between good
offices which we render and those which we receive, which in the end
disturb the most firm friendships. Atticus, who knew this well, had
so contrived as to have need of nobody. He was rich, he never had
law-suits, he did not seek public employments, so that a friend who
was determined to recompense the services he had received could
never find the opportunity,[156] and remained always under
obligations to him, and his debt continued to increase, for Atticus
never wearied of being useful. We have an easy means of
appreciating the extent of this serviceableness, to see it close, and, so
to say at work, namely, to rapidly recall the services of all sorts that
he had rendered Cicero during their long intimacy. Cicero had much
need of a friend like Atticus. He was one of those clever men who
cannot reckon; when his account-books were presented to him he
would gladly have said, like his pupil, Pliny the Younger, that he was
used to another sort of literature: aliis sum chartis, aliis litteris
initiatus. Atticus became his man of business; we know his talent for
this profession. He leased Cicero’s property very dear, saved as much
as he could out of the income and paid the most pressing debts.
When he discovered new ones, he dared to scold his friend, who
hastened to reply very humbly that he would be more careful for the
future. Atticus, who did not much believe this, set to work to make
up the deficit. He went to see the wealthy Balbus or the other great
bankers of Rome with whom he had business relations. If the
calamities of the times made it difficult to get credit, he did not
hesitate to dip into his own purse. Those who know him will not
think this generosity without merit. When Cicero wished to buy some
estate, Atticus at first would get angry; but if his friend did not give
way, he quickly went to visit it and discuss the price. If it was a
question of building some elegant villa, Atticus lent his architect,
corrected the plans, and overlooked the work. When the house was
built, it had to be adorned, and Atticus would send to Greece for
statues. He excelled in selecting them, and Cicero was inexhaustible
in his praises of the Hermathenae in Pentelican marble that he had
procured for him. In a villa of Cicero, we can well understand that
the library was not forgotten, and it was from Atticus again that the
books came. He traded in them, and kept the handsomest for his
friend. The books being bought, it was necessary to arrange them, so
Atticus sent his librarian Tyrannion with his workmen, who painted
the shelves, pasted together the detached leaves of papyrus, put the
labels on the rolls, and arranged the whole in such good order that
Cicero, enchanted, wrote: “Since Tyrannion has arranged my books
one would say that the house has a soul.”[157]
But Atticus did not stop at these services, which we might call
external; he penetrated into the home, he knew all its secrets. Cicero
kept nothing from him, and confided to him unreservedly all his
domestic griefs. He tells him about the violent temper of his brother
and the follies of his nephew; he consults him on the vexations that
his wife and son cause him. When Tullia was of an age to marry, it
was Atticus who sought her a husband. The one he proposed was the
son of a rich and well-conducted knight. “Return,” he said sagely to
Cicero, “return to your old flock.” Unfortunately he was not listened
to. They preferred to the rich financier a broken-down nobleman,
who squandered Tullia’s dowry and forced her to leave him. When
Tullia was dead, of grief perhaps, Atticus went to the nurse’s to visit
the little child she had left, and took care that it wanted for nothing.
At the same time Cicero gave him plenty of occupation with his two
divorces. After he had divorced his first wife, Terentia, it was Atticus
whom he charged to get her to make a will in his favour. It was to
him also that he gave the disagreeable commission to remove the
second, Publilia, when she was determined to forcibly re-enter the
home of her husband, who would have nothing more to do with her.
These are doubtless great services; he rendered others still more
delicate, still more appreciated. It was to him that Cicero entrusted
what was most dear to him in the world, his literary glory. He
communicated his works to him as soon as he had written them, he
took his advice in making corrections, and waited for his decision to
publish them. Thus he treated him as a friend with whom one feels at
home, and to whom one unbosoms oneself completely. Although he
was eager that his eloquence should be taken seriously, when he was
sure of being heard by Atticus only, he made no scruple of joking
about himself and his works. He introduced him without reserve to
all the secrets of the craft, and showed him the receipts for his most
popular effects. “This time, said he gaily, I employed the whole scent-
box of Isocrates, and all the caskets of the disciples.”[158] Nothing can
be more curious than the way in which he related to him one day,
one of his greatest oratorical successes. It was a question of
celebrating the fame of the great consulship, a subject upon which, as
we know, he was inexhaustible. That day he had a reason for
speaking with more brilliancy than usual. Pompey was present; now
Pompey had the weakness to be jealous of Cicero’s glory. It was a
good opportunity to enrage him, and Cicero took care not to neglect
it. “When my turn came to speak, he writes to Atticus, immortal
gods! what rein I gave myself! What pleasure I took in loading myself
with praises in the presence of Pompey, who had never heard me
extol my consulship! If I ever called to my aid periods, enthymemes,
metaphors, and all the other figures of rhetoric, it was then. I did not
speak, I shouted, for it was a question of my stock subjects, the
wisdom of the senate, the good-will of the knights, the union of all
Italy, the smothered remains of the conspiracy, peace and plenty re-
established, etc. You know my thunders when I speak of these
subjects. They were so fine that day that I have no need to tell you
more about them; you may often have heard the like at Athens!”[159]
It is impossible to quiz oneself with greater lightheartedness. Atticus
repaid these confidences by the trouble he took for the success of his
friend’s works. As he had seen their birth, and had busied himself
with them before they were known to the public, he almost regarded
himself as their parent. It was he who took upon himself to start
them in the world and make them succeed. Cicero says that he was
admirably well skilled in this, and it does not surprise us. The means
he most frequently employed to create a good opinion of them, was
to have the finest passages read by his best readers to the clever men
whom he assembled round his table. Cicero, who knew the usual
frugality of his repasts, begged him to deviate from it a little on these
occasions. “Have a care, he writes to him, to treat your guests well,
for if they are in a bad humour with you, they will vent it on me.”[160]
It was natural that Cicero should be extremely grateful for all these
services; but it would be judging him ill to suppose that he was only
attached to him for the benefits he received from him. He really
loved him, and all his letters are full of evidences of the most sincere
affection. He was always happy with him; he was never tired of
associating with him; he had scarcely left him than he ardently
wished to see him again. “May I die, he wrote to him, if either my
house at Tusculum, where I feel so comfortable, or the Isles of the
Blest could please me without you!”[161] Whatever pleasure he
experienced at being fêted, applauded, flattered, at having around
him an obsequious and admiring multitude, in the midst of this
crowd and noise he always turned with regret towards his absent
friend. “With all these people, he tells him, I feel myself more alone
than if I had only you.”[162] All these people, in fact, are composed of
political friends who change with circumstances, whom a common
interest brings to you, and a rival ambition takes away again; with
them Cicero is obliged to be reserved and careful, which is a torture
for such an open-hearted nature. On the other hand, he can tell
Atticus everything, and confide in him without restraint. So he
hastens to demand his presence when the least annoyance happens
to him. “I want you, he writes to him, I have need of you, I am
waiting for you. I have a thousand things that disturb and vex me,
and a single walk with you will relieve me.”[163] We should never end
if we were to collect all those charming expressions of which the
correspondence is full, and in which his heart plainly speaks. They
leave no doubt about Cicero’s feelings; they prove that he regarded
Atticus not only as one of those steadfast and serious friends on
whose support he could count, but also, which is more surprising, as
a sensitive and tender soul: “You take your share,” he tells him, “in
all the troubles of others.”[164]
Here is something far removed from the notion we usually have of
him, and yet we cannot resist such clear testimony. How can we
contend that he had only a doubtful affection for his friends when we
see all his friends contented with it? Are we to be more exacting than
they, and would it not be wronging men like Brutus and Cicero to
suppose that they had been dupes so long without perceiving it? On
the other hand, how can we explain the fact that posterity, which
only judges by the documents that the friends of Atticus have
furnished it, draws from these very documents an opinion quite the
reverse of that held by them? Evidently it is because posterity and
contemporaries do not judge men from the same stand-point. We
have seen that Atticus, who had made a rule not to engage in public
affairs, did not think himself obliged to partake the dangers that his
friends might run, through having taken part in them. He left them
both the honours and the perils. Sensitive, obliging, devoted to them
in the ordinary business of life, when a great political crisis occurred
that compromised them, he stood aside, and left them to expose
themselves alone. Now, when we look at the facts from a distance,
and are separated from them, as we are, by several centuries, we only
perceive the most important events, and especially the political
revolutions, that is to say, precisely those circumstances with which
the friendship of Atticus had nothing to do. Hence the severe
judgment we pronounce upon it. But his contemporaries judged
otherwise. Those great crises are, after all, but rare and passing
exceptions; without doubt contemporaries are much struck by them,
but they are still more impressed by those numberless small
incidents which make up every-day life, and which posterity does not
perceive. They judge of a man’s friendship by those services which
are rendered every moment, and which are important by their mere
number, much more than by any exceptional service which may be
given on one of these great and rare occasions. This accounts for the
fact that they had an opinion of Atticus so different from ours.
It is, beyond doubt, one of the characteristic traits of this person,
that it was a necessity to him to have many friends, and that he took
trouble to attract and retain them. We may refuse to admit, if we will,
that this need was, with him, the effect of a generous and
sympathetic nature, that it came from what Cicero admirably calls
“the impulse of the soul that desires to love;” but, even supposing
that he only thought of occupying and filling up his life, we must
acknowledge that to fill it up in this manner is not a mark of a vulgar
nature. This refined Epicurean, this master in the art of living at
ease, knew “that life is no longer life if we cannot repose on the
affection of a friend.”[165] He had given up the excitement of political
strife, the triumphs of eloquence, the joys of satisfied ambition, but,
as a compensation, he was determined to enjoy all the pleasures of
private life. The more he confined and limited himself to it, the more
particular and refined he became with regard to the pleasures it
could give; as he had only left himself these, he wished to enjoy them
fully, to relish them, to live on them. He needed friends, and among
them the greatest minds, the noblest souls of his time. He expended
all that energy which he did not employ in anything else, in
procuring for himself those pleasures of society that Bossuet calls the
greatest good of human life. Atticus enjoyed this good even beyond
his desires, and friendship generously repaid him for all the trouble
he had taken for it. It was his single passion; he was able to satisfy it
completely, and friendship, after having adorned his life, has shed a
lustre on his name.
III.
Atticus appears in a favourable light in private life. He is less
fortunate when we study the course he followed in public affairs. On
this point he has not been spared blame, and it is not easy to defend
him.
We should not however be very unfavourable to him if we judged
his conduct entirely according to the ideas of our days. Opinion has
become less severe now on those who openly make profession of
living apart from politics. So many men aspire to govern their
country, and it has become so difficult to make choice among this
multitude, that we are tempted to look kindly upon those who have
not this ambition. Far from being blamed, they are called moderate
and wise; they form an exception which is encouraged in order to
lessen the number of aspirants. At Rome they thought otherwise, and
it is not difficult to find reasons for this difference. There, what we
may call the political body was in reality very circumscribed. Besides
the slaves, who did not count, and the common people, who
contented themselves with giving or rather selling their votes in the
elections, and whose greatest privilege it was to be entertained at the
expense of the candidates, and fed at the expense of the public
treasury, there remained only a few families of ancient lineage or
more recent celebrity who divided all public employments among
themselves. The aristocracy of birth and of fortune was not very
numerous, and scarcely sufficed to furnish the required number of
officials of all sorts to govern the world. It was necessary therefore
that no one should refuse to take his part, and to live in retirement
was considered a desertion. It is not the same in our democracy. As
all offices are open to everybody, and as, thanks to the diffusion of
education, men worthy to occupy them may arise in all ranks, we
need no longer fear lest the absence of a few quiet people, friends of
peace and repose, will make a sensible and regrettable gap in the
serried ranks of those who struggle from all quarters for power.
Moreover, we think now that there are many other ways of serving
one’s country besides public life. Romans of high birth knew no
other; they looked upon commerce as a not very honourable
means[166] that a private man might employ to make his fortune, and
did not see what the state might gain by it; literature seemed an
agreeable but trivial pastime, and they did not understand its social
importance. It follows that among them, a man of a certain rank
could only find one honourable mode of employing his activity and
being useful to his country, namely, to fill political offices.[167] To do
anything else was, according to their ideas, to do nothing; they gave
the name of idlers to the most laborious scholars, and it did not come
into their heads that there was anything worth the trouble of
occupying a citizen’s time beyond the service of the state. All the
ancient Romans thought thus, and they would have experienced a
strange surprise if they had seen any one claim the right, as Atticus
did, not to serve his country within the limits of his powers and
talents. Assuredly Cato, who never rested, and who, at ninety years of
age, bravely quitted his villa at Tusculum to go and accuse Servius
Galba, the butcher of the Lusitanians, would have thought that to
remain in his house on the Quirinal, or on his estate in Epirus in the
midst of his books and statues, while the fate of Rome was being
decided in the Forum or at Pharsalia, was to commit the same crime
as to remain in his tent on the day of battle.
This systematic abstention of Atticus was not, then, a Roman
custom; he had it from the Greeks. In those small ungovernable
republics of Greece, where they knew no repose, and which passed
constantly and without warning from the sternest tyranny to the
most unbridled licence, we can understand that quiet and studious
men should have grown weary of all this sterile agitation, and ceased
to desire public employments which were only obtained by flattering
the capricious multitude, and only kept on condition of obeying it.
Moreover, what value could this power, so hardly acquired, so
seldom preserved, have, when it was necessary to share it with the
most obscure demagogues? was it really worth while to take so much
trouble in order to become the successor or the colleague of Cleon?
At the same time that weariness and disgust kept honourable men
aloof from these paltry struggles, philosophy, more studied every
day, communicated to its disciples a sort of pride which led them to
the same result. Men who passed their time in meditating upon God
and the world, and who endeavoured to understand the laws that
govern the universe, did not deign to descend from these heights to
govern states a few leagues square. Thus they constantly discussed in
the schools, whether a man should occupy himself with public
affairs, whether the sage ought to seek public office, and whether the
active or the contemplative life was the better. A few philosophers
hesitatingly gave the preference to active life, the greater number
sustained the opposite opinion, and under cover of these discussions
many men thought themselves authorized to create a sort of elegant
indolence in voluptuous retreats embellished by letters and the arts,
where they lived happily while Greece was perishing.
Atticus followed their example. Importing this custom from Greece
into Rome, he openly announced his resolution not to take part in
political discussions. He began by adroitly keeping aloof during all
those quarrels that continually agitated Rome from the time of
Cicero’s consulship to the civil wars. At the very moment when these
struggles were most active he frequented all parties, he had friends
on all sides, and found in these widespread friendships a new pretext
for remaining neutral. Atticus was more than sixty years old when
Caesar passed the Rubicon, an age when the obligation of military
service ceased among the Romans. This was another reason for
remaining quiet, and he did not fail to use it. “I have taken my
discharge,”[168] he replied to those who wished to enrol him. He held
the same course, and with the same success, after the death of
Caesar; but he then disappointed public opinion still more. He was
so well known to be the friend of Brutus that it was thought he would
not hesitate to take his side this time. Cicero himself, who ought to
have known him, reckoned upon it; but Atticus was not inconsistent
with himself, and took advantage of an important occasion to let the
public know that he would not be drawn in against his will. While
Brutus was raising an army in Greece, some knights, his friends,
started the idea of raising a subscription among the richest men of
Rome to give him the means of maintaining his soldiers. They
applied at first to Atticus, whose name they wished to put at the head
of the list. Atticus bluntly refused to subscribe. He answered that his
fortune was at the service of Brutus, if he had need of it, and asked
him as a friend, but he declared at the same time that he would take
no part in a political manifestation, and his refusal caused the failure
of the subscription. At the same time, true to his habit of flattering all
parties, he welcomed Fulvia, Antony’s wife, as well as Volumnius the
superintendent of his workmen, and, sure of having friends
everywhere, he waited for the result of the struggle without much
fear.
The strangest thing is that this man, while so persistent in
remaining neutral, was not indifferent. His biographer gives him this
praise, that he always belonged to the best party,[169] and that is true;
only he made it a rule not to serve his party; he was contented with
giving it his good wishes. But these good wishes were the warmest
imaginable. He had, though we should scarcely believe it, political
passions which he dared to express in private with incredible vigour.
He hated Caesar so much that he went as far as to blame Brutus for
having permitted his interment.[170] He would have wished, no doubt,
as the most furious demanded, that his corpse should be thrown into
the Tiber. Thus he did not abstain from having preferences, and
showing them to his most intimate friends. His reserve only began
when it was necessary to act. He never consented to take part in the
struggle; but if he did not share its danger he felt at least all its
excitement. We smile at seeing him become animated and excited as
if he were a real combatant; he takes his share in all successes and all
reverses, he congratulates the energetic, he entreats the lukewarm,
and even scolds the faltering, and permits himself to advise and
reprimand those who seem to him, who did not act at all, to act too
languidly. It is amusing to hear the reproaches he addresses to Cicero
when he sees him hesitating to go and join Pompey; he adopts the
most pathetic tone, he reminds him of his actions and his words, he
entreats him in the name of his glory, he quotes his own words to
him to persuade him.[171] This excess of audacity into which he allows
himself to be drawn for others, sometimes produces rather comic
incidents. At the moment when Pompey had just shut himself up in
Brundusium, Atticus, moved by the most lively grief, wished for
some attempts to be made to save him, and went so far as to ask
Cicero to do some striking action before leaving. “It only requires a
banner,” said he, “every one will flock to it.”[172] The worthy Cicero
felt himself quite excited by these lively exhortations of his friend,
and there were times when he was tempted to be bold, and when he
only demanded the opportunity to strike a heavy blow. The
opportunity came, and he relates in the following words how he took
advantage of it. “As I arrived at my house at Pompeii, your friend
Ninnius came to tell me that the centurions of three cohorts who
were there, asked to see me the next day, as they wished to deliver up

Ebook Handbook of Hydroinformatics Volume I Classic Soft Computing Techniques PDF Full Chapter PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ebook Handbook of Hydroinformatics Volume I Classic Soft Computing Techniques PDF Full Chapter PDF

Uploaded by

Copyright:

Available Formats

Handbook of HydroInformatics: Volume

I: Classic Soft-Computing Techniques -

Copyright © 2023 Elsevier Inc. All rights reserved.

For information on all Elsevier publications

Publisher: Candice Janco

Typeset by STRAIVE, India

To Late Dr. Mark Twain (American Writer, Humorist, Entrepreneur, Publisher,

Contributors xv Elastic net 33

1. Advanced machine learning 2. Bat algorithm optimized extreme

5.3 Machine learning using Bayesian 5. Cross-validation 89

7. Danubian hypocycles as overlapping 10. Decision tree algorithms 171

12. Forecasting volatility in the 14. Gradient-based optimization 243

5.3 Kernel smoother 276 3.2 Constant heat flux boundary

1. Introduction 283 1. Introduction 321

5. Ontology-based knowledge 6. Applications of relevance vector

3. SVMs for regression problems 413 26. Uncertainty-based resiliency

Faezeh Eslamian holds a PhD in Bioresource Engineering from McGill University,

Advanced machine learning techniques:

Handbook of HydroInformatics. https://doi.org/10.1016/B978-0-12-821285-1.00017-8

3. Multivariate linear regression

TABLE 1 The attribute information of the nutrient removal efficiency project.

Code Input or output Description

NH4-N Input Ammonium

RE of TN Output Removal efficiency of TN

4000 6000 8000 10000 12000 14000 16000 18000

Or in the matrix form

4. Gradient descent method

FIG. 2 Applying linear regression for the synthetic data.

anew ¼ aold l—SSEðaold Þ (18)

FIG. 7 The high-degree polynomial regression.

6. Overfitting and underfitting

8. Comparison between linear and polynomial regressions

0 2500 5000 7500 10000 12500 15000 17500 20000

0 2500 5000 7500 10000 12500 15000 17500 20000

0 2500 5000 7500 10000 12500 15000 17500 20000

10. Regularized linear models

0 2500 5000 7500 10000 12500 15000 17500 20000

Learning Curves (Ridge Regression)

1000 2000 3000 4000 5000 6000

11. The ridge regression

12. The effect of collinearity in the coefficients of an estimator

13. Outliers impact

10–3 10–5 10–7 10–9

14. Lasso regression

15. Elastic net

16. Early stopping

17. Logistic regression

18. Estimation of probabilities

–10.0 –7.5 –5.0 –2.5 0.0 2.5 5.0 7.5 10.0

19. Training and the cost function

–0.75 –0.50 –0.25 0.00 0.25 0.50 0.75 1.00

0 2500 5000 7500 10000 12500 15000 17500 20000

Appendix: Python code

df = pd.read_csv('nutrient.data', delim_whitespace=True, header=None)

import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression

Gradient descent method

df = pd.read_csv('nutrient.data', delim_whitespace=True, header=None)

from sklearn.preprocessing import StandardScaler

w_[1:] += alpha * X_std.T.dot(errors)

cost = (errors**2).sum() / 2.0

Comparison between linear and polynomial regressions

df = pd.read_csv('nutrient.data', delim_whitespace=True, header=None)