Textbook Fundamentals of Linear Algebra and Optimization Gallier J Ebook All Chapter PDF

Fundamentals of linear algebra and
optimization Gallier J.
Visit to download the full and correct content document:
https://textbookfull.com/product/fundamentals-of-linear-algebra-and-optimization-galli
er-j/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Fundamentals of Linear Algebra and Optimization Jean

Gallier And Jocelyn Quaintance
https://textbookfull.com/product/fundamentals-of-linear-algebra-
and-optimization-jean-gallier-and-jocelyn-quaintance/
Fundamentals of optimization theory with applications

to machine learning Gallier J.
https://textbookfull.com/product/fundamentals-of-optimization-
theory-with-applications-to-machine-learning-gallier-j/
Fundamentals of optimization theory with applications

to machine learning Gallier J.
https://textbookfull.com/product/fundamentals-of-optimization-
theory-with-applications-to-machine-learning-gallier-j-2/
Basics of algebra topology and differential calculus

Gallier J
https://textbookfull.com/product/basics-of-algebra-topology-and-
differential-calculus-gallier-j/
Algebra Topology Differential Calculus and Optimization
Theory For Computer Science and Engineering Jean
Gallier
https://textbookfull.com/product/algebra-topology-differential-
calculus-and-optimization-theory-for-computer-science-and-
engineering-jean-gallier/
Fundamentals of Linear Algebra Textbooks in Mathematics

1st Edition J.S. Chahal
https://textbookfull.com/product/fundamentals-of-linear-algebra-
textbooks-in-mathematics-1st-edition-j-s-chahal/
Linear Algebra and Optimization for Machine Learning: A

Textbook Charu C. Aggarwal
https://textbookfull.com/product/linear-algebra-and-optimization-
for-machine-learning-a-textbook-charu-c-aggarwal/
Applied Linear Algebra Second Edition Peter J. Olver
https://textbookfull.com/product/applied-linear-algebra-second-
edition-peter-j-olver/
Applied Linear Algebra Second Edition Peter J. Olver
https://textbookfull.com/product/applied-linear-algebra-second-
edition-peter-j-olver-2/
Fundamentals of Linear Algebra
and Optimization
Jean Gallier and Jocelyn Quaintance

Department of Computer and Information Science
University of Pennsylvania
Philadelphia, PA 19104, USA
e-mail: jean@cis.upenn.edu
c Jean Gallier
February 23, 2017

2
Figure 1: Cover page from Bourbaki, Fascicule VI, Livre II, Algèbre, 1962
3
Figure 2: Page 156 from Bourbaki, Fascicule VI, Livre II, Algèbre, 1962
4
Contents
1 Vector Spaces, Bases, Linear Maps 11

1.1 Motivations: Linear Combinations, Linear Independence, Rank . . . . . . . 11
1.2 Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.3 Linear Independence, Subspaces . . . . . . . . . . . . . . . . . . . . . . . . 27
1.4 Bases of a Vector Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.5 Linear Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2 Matrices and Linear Maps 45

2.1 Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.2 Haar Basis Vectors and a Glimpse at Wavelets . . . . . . . . . . . . . . . . 61
2.3 The Effect of a Change of Bases on Matrices . . . . . . . . . . . . . . . . . 78
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3 Direct Sums, Affine Maps, The Dual Space, Duality 83

3.1 Direct Products, Sums, and Direct Sums . . . . . . . . . . . . . . . . . . . . 83
3.2 Affine Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
3.3 The Dual Space E and Linear Forms . . . . . . . . . . . . . . . . . . . . . 100
3.4 Hyperplanes and Linear Forms . . . . . . . . . . . . . . . . . . . . . . . . . 118
3.5 Transpose of a Linear Map and of a Matrix . . . . . . . . . . . . . . . . . . 119
3.6 The Four Fundamental Subspaces . . . . . . . . . . . . . . . . . . . . . . . 126
3.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
4 Gaussian Elimination, LU, Cholesky, Echelon Form 131

4.1 Motivating Example: Curve Interpolation . . . . . . . . . . . . . . . . . . . 131
4.2 Gaussian Elimination and LU -Factorization . . . . . . . . . . . . . . . . . . 135
4.3 Gaussian Elimination of Tridiagonal Matrices . . . . . . . . . . . . . . . . . 161
4.4 SPD Matrices and the Cholesky Decomposition . . . . . . . . . . . . . . . . 164
4.5 Reduced Row Echelon Form . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
4.6 Transvections and Dilatations . . . . . . . . . . . . . . . . . . . . . . . . . . 186
4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
5 Determinants 193
5.1 Permutations, Signature of a Permutation . . . . . . . . . . . . . . . . . . . 193
5
6 CONTENTS
5.2 Alternating Multilinear Maps . . . . . . . . . . . . . . . . . . . . . . . . . . 197

5.3 Definition of a Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
5.4 Inverse Matrices and Determinants . . . . . . . . . . . . . . . . . . . . . . . 208
5.5 Systems of Linear Equations and Determinants . . . . . . . . . . . . . . . . 211
5.6 Determinant of a Linear Map . . . . . . . . . . . . . . . . . . . . . . . . . . 212
5.7 The Cayley–Hamilton Theorem . . . . . . . . . . . . . . . . . . . . . . . . . 213
5.8 Permanents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
5.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
5.10 Further Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
6 Vector Norms and Matrix Norms 223

6.1 Normed Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
6.2 Matrix Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
6.3 Condition Numbers of Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 242
6.4 An Application of Norms: Inconsistent Linear Systems . . . . . . . . . . . . 251
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
7 Eigenvectors and Eigenvalues 255

7.1 Eigenvectors and Eigenvalues of a Linear Map . . . . . . . . . . . . . . . . . 255
7.2 Reduction to Upper Triangular Form . . . . . . . . . . . . . . . . . . . . . . 262
7.3 Location of Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
8 Iterative Methods for Solving Linear Systems 271

8.1 Convergence of Sequences of Vectors and Matrices . . . . . . . . . . . . . . 271
8.2 Convergence of Iterative Methods . . . . . . . . . . . . . . . . . . . . . . . . 274
8.3 Methods of Jacobi, Gauss-Seidel, and Relaxation . . . . . . . . . . . . . . . 276
8.4 Convergence of the Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
8.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288
9 Euclidean Spaces 289

9.1 Inner Products, Euclidean Spaces . . . . . . . . . . . . . . . . . . . . . . . . 289
9.2 Orthogonality, Duality, Adjoint of a Linear Map . . . . . . . . . . . . . . . 297
9.3 Linear Isometries (Orthogonal Transformations) . . . . . . . . . . . . . . . . 309
9.4 The Orthogonal Group, Orthogonal Matrices . . . . . . . . . . . . . . . . . 312
9.5 QR-Decomposition for Invertible Matrices . . . . . . . . . . . . . . . . . . . 314
9.6 Some Applications of Euclidean Geometry . . . . . . . . . . . . . . . . . . . 318
9.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
10 QR-Decomposition for Arbitrary Matrices 321

10.1 Orthogonal Reflections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
10.2 QR-Decomposition Using Householder Matrices . . . . . . . . . . . . . . . . 324
10.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
CONTENTS 7
11 Hermitian Spaces 331

11.1 Hermitian Spaces, Pre-Hilbert Spaces . . . . . . . . . . . . . . . . . . . . . 331
11.2 Orthogonality, Duality, Adjoint of a Linear Map . . . . . . . . . . . . . . . 340
11.3 Linear Isometries (Also Called Unitary Transformations) . . . . . . . . . . . 345
11.4 The Unitary Group, Unitary Matrices . . . . . . . . . . . . . . . . . . . . . 347
11.5 Orthogonal Projections and Involutions . . . . . . . . . . . . . . . . . . . . 350
11.6 Dual Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
11.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356
12 Spectral Theorems 359

12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
12.2 Normal Linear Maps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
12.3 Self-Adjoint and Other Special Linear Maps . . . . . . . . . . . . . . . . . . 368
12.4 Normal and Other Special Matrices . . . . . . . . . . . . . . . . . . . . . . . 375
12.5 Conditioning of Eigenvalue Problems . . . . . . . . . . . . . . . . . . . . . . 378
12.6 Rayleigh Ratios and the Courant-Fischer Theorem . . . . . . . . . . . . . . 381
12.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
13 Introduction to The Finite Elements Method 391

13.1 A One-Dimensional Problem: Bending of a Beam . . . . . . . . . . . . . . . 391
13.2 A Two-Dimensional Problem: An Elastic Membrane . . . . . . . . . . . . . 402
13.3 Time-Dependent Boundary Problems . . . . . . . . . . . . . . . . . . . . . . 405
14 Singular Value Decomposition and Polar Form 413

14.1 Singular Value Decomposition for Square Matrices . . . . . . . . . . . . . . 413
14.2 Singular Value Decomposition for Rectangular Matrices . . . . . . . . . . . 421
14.3 Ky Fan Norms and Schatten Norms . . . . . . . . . . . . . . . . . . . . . . 424
14.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
15 Applications of SVD and Pseudo-Inverses 427

15.1 Least Squares Problems and the Pseudo-Inverse . . . . . . . . . . . . . . . . 427
15.2 Properties of the Pseudo-Inverse . . . . . . . . . . . . . . . . . . . . . . . . 432
15.3 Data Compression and SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . 437
15.4 Principal Components Analysis (PCA) . . . . . . . . . . . . . . . . . . . . . 438
15.5 Best Affine Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
15.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
16 Annihilating Polynomials; Primary Decomposition 451

16.1 Annihilating Polynomials and the Minimal Polynomial . . . . . . . . . . . . 451
16.2 Minimal Polynomials of Diagonalizable Linear Maps . . . . . . . . . . . . . 457
16.3 The Primary Decomposition Theorem . . . . . . . . . . . . . . . . . . . . . 463
16.4 Nilpotent Linear Maps and Jordan Form . . . . . . . . . . . . . . . . . . . . 469
8 CONTENTS
17 Topology 475
17.1 Metric Spaces and Normed Vector Spaces . . . . . . . . . . . . . . . . . . . 475
17.2 Topological Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
17.3 Continuous Functions, Limits . . . . . . . . . . . . . . . . . . . . . . . . . . 490
17.4 Continuous Linear and Multilinear Maps . . . . . . . . . . . . . . . . . . . . 497
17.5 The Contraction Mapping Theorem . . . . . . . . . . . . . . . . . . . . . . 502
17.6 Futher Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
17.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
18 Differential Calculus 505

18.1 Directional Derivatives, Total Derivatives . . . . . . . . . . . . . . . . . . . 505
18.2 Jacobian Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
18.3 The Implicit and The Inverse Function Theorems . . . . . . . . . . . . . . . 526
18.4 Second-Order and Higher-Order Derivatives . . . . . . . . . . . . . . . . . . 530
18.5 Taylor’s Formula, Faà di Bruno’s Formula . . . . . . . . . . . . . . . . . . . 535
18.6 Futher Readings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
18.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 539
19 Quadratic Optimization Problems 541

19.1 Quadratic Optimization: The Positive Definite Case . . . . . . . . . . . . . 541
19.2 Quadratic Optimization: The General Case . . . . . . . . . . . . . . . . . . 549
19.3 Maximizing a Quadratic Function on the Unit Sphere . . . . . . . . . . . . 553
19.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 558
20 Schur Complements and Applications 561

20.1 Schur Complements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561
20.2 SPD Matrices and Schur Complements . . . . . . . . . . . . . . . . . . . . . 563
20.3 SP Semidefinite Matrices and Schur Complements . . . . . . . . . . . . . . 565
21 Convex Sets, Cones, H-Polyhedra 567

21.1 What is Linear Programming? . . . . . . . . . . . . . . . . . . . . . . . . . 567
21.2 Affine Subsets, Convex Sets, Hyperplanes, Half-Spaces . . . . . . . . . . . . 569
21.3 Cones, Polyhedral Cones, and H-Polyhedra . . . . . . . . . . . . . . . . . . 572
22 Linear Programs 579

22.1 Linear Programs, Feasible Solutions, Optimal Solutions . . . . . . . . . . . 579
22.2 Basic Feasible Solutions and Vertices . . . . . . . . . . . . . . . . . . . . . . 585
23 The Simplex Algorithm 593

23.1 The Idea Behind the Simplex Algorithm . . . . . . . . . . . . . . . . . . . . 593
23.2 The Simplex Algorithm in General . . . . . . . . . . . . . . . . . . . . . . . 602
23.3 How Perform a Pivoting Step Efficiently . . . . . . . . . . . . . . . . . . . . 609
23.4 The Simplex Algorithm Using Tableaux . . . . . . . . . . . . . . . . . . . . 612
CONTENTS 9
23.5 Computational Efficiency of the Simplex Method . . . . . . . . . . . . . . . 622
24 Linear Programming and Duality 625

24.1 Variants of the Farkas Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . 625
24.2 The Duality Theorem in Linear Programming . . . . . . . . . . . . . . . . . 630
24.3 Complementary Slackness Conditions . . . . . . . . . . . . . . . . . . . . . 639
24.4 Duality for Linear Programs in Standard Form . . . . . . . . . . . . . . . . 640
24.5 The Dual Simplex Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 643
24.6 The Primal-Dual Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 649
25 Extrema of Real-Valued Functions 661

25.1 Local Extrema and Lagrange Multipliers . . . . . . . . . . . . . . . . . . . . 661
25.2 Using Second Derivatives to Find Extrema . . . . . . . . . . . . . . . . . . . 671
25.3 Using Convexity to Find Extrema . . . . . . . . . . . . . . . . . . . . . . . 674
25.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684
26 Newton’s Method and Its Generalizations 685

26.1 Newton’s Method for Real Functions of a Real Argument . . . . . . . . . . 685
26.2 Generalizations of Newton’s Method . . . . . . . . . . . . . . . . . . . . . . 686
26.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 692
27 Basics of Hilbert Spaces 693

27.1 The Projection Lemma, Duality . . . . . . . . . . . . . . . . . . . . . . . . 693
27.2 Farkas–Minkowski Lemma in Hilbert Spaces . . . . . . . . . . . . . . . . . . 710
28 General Results of Optimization Theory 713

28.1 Existence of Solutions of an Optimization Problem . . . . . . . . . . . . . . 713
28.2 Gradient Descent Methods for Unconstrained Problems . . . . . . . . . . . 727
28.3 Conjugate Gradient Methods for Unconstrained Problems . . . . . . . . . . 743
28.4 Gradient Projection for Constrained Optimization . . . . . . . . . . . . . . 753
28.5 Penalty Methods for Constrained Optimization . . . . . . . . . . . . . . . . 756
28.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 758
29 Introduction to Nonlinear Optimization 759

29.1 The Cone of Feasible Directions . . . . . . . . . . . . . . . . . . . . . . . . . 759
29.2 The Karush–Kuhn–Tucker Conditions . . . . . . . . . . . . . . . . . . . . . 772
29.3 Hard Margin Support Vector Machine . . . . . . . . . . . . . . . . . . . . . 784
29.4 Lagrangian Duality and Saddle Points . . . . . . . . . . . . . . . . . . . . . 794
29.5 Uzawa’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 810
29.6 Handling Equality Constraints Explicitly . . . . . . . . . . . . . . . . . . . . 816
29.7 Conjugate Function and Legendre Dual Function . . . . . . . . . . . . . . . 823
29.8 Some Techniques to Obtain a More Useful Dual Program . . . . . . . . . . 833
29.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 842
10 CONTENTS
30 Soft Margin Support Vector Machines 845

30.1 Soft Margin Support Vector Machines; (SVMs1 ) . . . . . . . . . . . . . . . . 846
30.3 Soft Margin Support Vector Machines; (SVMs20 ) . . . . . . . . . . . . . . . 861
30.4 Soft Margin SVM; (SVMs3 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . 875
30.6 Soft Margin SVM; (SVMs5 ) . . . . . . . . . . . . . . . . . . . . . . . . . . . 885
30.7 Summary and Comparison of the SVM Methods . . . . . . . . . . . . . . . 887
31 Total Orthogonal Families in Hilbert Spaces 899

31.1 Total Orthogonal Families, Fourier Coefficients . . . . . . . . . . . . . . . . 899
31.2 The Hilbert Space l2 (K) and the Riesz-Fischer Theorem . . . . . . . . . . . 907
Bibliography 916
Chapter 1
Vector Spaces, Bases, Linear Maps
1.1 Motivations: Linear Combinations, Linear Inde-

pendence and Rank
Consider the problem of solving the following system of three linear equations in the three
variables x1 , x2 , x3 2 R:
x1 + 2x2 x3 = 1
2x1 + x2 + x3 = 2
x1 2x2 2x3 = 3.
One way to approach this problem is introduce the “vectors” u, v, w, and b, given by
0 1 0 1 0 1 0 1
1 2 1 1
u = @2A v=@ 1 A w=@ 1 A b = @2A
1 2 2 3
and to write our linear system as
x1 u + x2 v + x3 w = b.
In the above equation, we used implicitly the fact that a vector z can be multiplied by a
scalar λ 2 R, where 0 1 0 1
z1 λz1
λz = λ z2 = λz2 A ,
@ A @
z3 λz3
and two vectors y and and z can be added, where
0 1 0 1 0 1
y1 z1 y1 + z1
y + z = @y2 A + @z2 A = @y2 + z2 A .
y3 z3 y3 + z3
11
12 CHAPTER 1. VECTOR SPACES, BASES, LINEAR MAPS
The set of all vectors with three components is denoted by R3 1 . The reason for using
the notation R3 1 rather than the more conventional notation R3 is that the elements of
R3 1 are column vectors; they consist of three rows and a single column, which explains the
superscript 3 1. On the other hand, R3 = R R R consists of all triples of the form
(x1 , x2 , x3 ), with x1 , x2 , x3 2 R, and these are row vectors. However, there is an obvious
bijection between R3 1 and R3 and they are usually identifed. For the sake of clarity, in this
introduction, we will denote the set of column vectors with n components by Rn 1 .
An expression such as
x1 u + x2 v + x3 w
where u, v, w are vectors and the xi s are scalars (in R) is called a linear combination. Using
this notion, the problem of solving our linear system
x1 u + x2 v + x3 w = b.
is equivalent to determining whether b can be expressed as a linear combination of u, v, w.

Now, if the vectors u, v, w are linearly independent, which means that there is no triple
(x1 , x2 , x3 ) 6= (0, 0, 0) such that
x1 u + x2 v + x3 w = 03 ,
it can be shown that every vector in R3 1

can be written as a linear combination of u, v, w.
Here, 03 is the zero vector 0 1
0
03 = 0A .
@
0
It is customary to abuse notation and to write 0 instead of 03 . This rarely causes a problem
because in most cases, whether 0 denotes the scalar zero or the zero vector can be inferred
from the context.
In fact, every vector z 2 R3 1
can be written in a unique way as a linear combination
z = x1 u + x2 v + x3 w.
This is because if
z = x1 u + x2 v + x3 w = y1 u + y2 v + y3 w,
then by using our (linear!) operations on vectors, we get
(y1 x1 )u + (y2 x2 )v + (y3 x3 )w = 0,
which implies that

y1 x1 = y 2 x2 = y 3 x3 = 0,
by linear independence. Thus,
y 1 = x1 , y2 = x2 , y3 = x3 ,
1.1. MOTIVATIONS: LINEAR COMBINATIONS, LINEAR INDEPENDENCE, RANK13
which shows that z has a unique expression as a linear combination, as claimed. Then, our
equation
x1 u + x2 v + x3 w = b
has a unique solution, and indeed, we can check that
x1 = 1.4
x2 = 0.4
x3 = 0.4
is the solution.
But then, how do we determine that some vectors are linearly independent?
One answer is to compute the determinant det(u, v, w), and to check that it is nonzero.
In our case,
1 2 1
det(u, v, w) = 2 1 1 = 15,
1 2 2
which confirms that u, v, w are linearly independent.
Other methods consist of computing an LU-decomposition or a QR-decomposition, or an
SVD of the matrix consisting of the three columns u, v, w,
0 1
1 2 1
A = u v w = @2 1 1 A.
1 2 2
If we form the vector of unknowns 0 1

x1
x = x2 A ,
@
x3
then our linear combination x1 u + x2 v + x3 w can be written in matrix form as
0 10 1
1 2 1 x1
x1 u + x2 v + x3 w = 2
@ 1 1 A @ x2 A ,
1 2 2 x3
so our linear system is expressed by

0 10 1 0 1
1 2 1 x1 1
@2 1 1 A @ x2 = 2A ,
A @
1 2 2 x3 3
or more concisely as
Ax = b.
Now, what if the vectors u, v, w are linearly dependent? For example, if we consider the
vectors
0 1 0 1 0 1
1 2 1
u= 2 @ A v= @ 1 A w= @ 1 A,
1 1 2
we see that
u v = w,
a nontrivial linear dependence. It can be verified that u and v are still linearly independent.
Now, for our problem
x1 u + x2 v + x3 w = b
to have a solution, it must be the case that b can be expressed as linear combination of u and
v. However, it turns out that u, v, b are linearly independent (because det(u, v, b) = 6),
so b cannot be expressed as a linear combination of u and v and thus, our system has no
solution.
If we change the vector b to 0 1
3
b = 3A ,
@
0
then
b = u + v,
and so the system
x1 u + x2 v + x3 w = b
has the solution
x1 = 1, x2 = 1, x3 = 0.
Actually, since w = u v, the above system is equivalent to
(x1 + x3 )u + (x2 x3 )v = b,
and because u and v are linearly independent, the unique solution in x1 + x3 and x2 x3 is
x1 + x3 = 1
x2 x3 = 1,
which yields an infinite number of solutions parameterized by x3 , namely
x1 = 1 x 3
x2 = 1 + x3 .
In summary, a 3 3 linear system may have a unique solution, no solution, or an infinite

number of solutions, depending on the linear independence (and dependence) or the vectors
1.1. MOTIVATIONS: LINEAR COMBINATIONS, LINEAR INDEPENDENCE, RANK15
u, v, w, b. This situation can be generalized to any n n system, and even to any n m

system (n equations in m variables), as we will see later.
The point of view where our linear system is expressed in matrix form as Ax = b stresses
the fact that the map x 7! Ax is a linear transformation. This means that
A(λx) = λ(Ax)
for all x 2 R3 1
and all λ 2 R and that
A(u + v) = Au + Av,
for all u, v 2 R3 1 . We can view the matrix A as a way of expressing a linear map from R3 1
to R3 1 and solving the system Ax = b amounts to determining whether b belongs to the
image of this linear map.
Yet another fruitful way of interpreting the resolution of the system Ax = b is to view
this problem as an intersection problem. Indeed, each of the equations
x1 + 2x2 x3 = 1
2x1 + x2 + x3 = 2
x1 2x2 2x3 = 3
defines a subset of R3 which is actually a plane. The first equation
x1 + 2x2 x3 = 1
defines the plane H1 passing through the three points (1, 0, 0), (0, 1/2, 0), (0, 0, 1), on the
coordinate axes, the second equation
2x1 + x2 + x3 = 2
defines the plane H2 passing through the three points (1, 0, 0), (0, 2, 0), (0, 0, 2), on the coor-
dinate axes, and the third equation
x1 2x2 2x3 = 3
defines the plane H3 passing through the three points (3, 0, 0), (0, 3/2, 0), (0, 0, 3/2), on
the coordinate axes. The intersection Hi \ Hj of any two distinct planes Hi and Hj is
a line, and the intersection H1 \ H2 \ H3 of the three planes consists of the single point
(1.4, 0.4, 0.4). Under this interpretation, observe that we are focusing on the rows of the
matrix A, rather than on its columns, as in the previous interpretations.
Another great example of a real-world problem where linear algebra proves to be very
effective is the problem of data compression, that is, of representing a very large data set
using a much smaller amount of storage.
Typically the data set is represented as an m n matrix A where each row corresponds
to an n-dimensional data point and typically, m n. In most applications, the data are not
independent so the rank of A is a lot smaller than minfm, ng, and the the goal of low-rank
decomposition is to factor A as the product of two matrices B and C, where B is a m k
matrix and C is a k n matrix, with k minfm, ng (here, means “much smaller than”):
0 1 0 1
B C B C
B C B C0 1
B C B C
B C B C
B
B A C=B
C B B C@
C C A
B
B m n C B m k C
C B C k n
@ A @ A
Now, it is generally too costly to find an exact factorization as above, so we look for a
low-rank matrix A0 which is a “good” approximation of A. In order to make this statement
precise, we need to define a mechanism to determine how close two matrices are. This can
be done using matrix norms, a notion discussed in Chapter 6. The norm of a matrix A is a
nonnegative real number kAk which behaves a lot like the absolute value jxj of a real number
x. Then, our goal is to find some low-rank matrix A0 that minimizes the norm
2
kA A0 k ,
over all matrices A0 of rank at most k, for some given k minfm, ng.
Some advantages of a low-rank approximation are:
1. Fewer elements are required to represent A; namely, k(m + n) instead of mn. Thus
less storage and fewer operations are needed to reconstruct A.
2. Often, the process for obtaining the decomposition exposes the underlying structure of
the data. Thus, it may turn out that “most” of the significant data are concentrated
along some directions called principal directions.
Low-rank decompositions of a set of data have a multitude of applications in engineering,

including computer science (especially computer vision), statistics, and machine learning.
As we will see later in Chapter 15, the singular value decomposition (SVD) provides a very
satisfactory solution to the low-rank approximation problem. Still, in many cases, the data
sets are so large that another ingredient is needed: randomization. However, as a first step,
linear algebra often yields a good initial solution.
We will now be more precise as to what kinds of operations are allowed on vectors. In
the early 1900, the notion of a vector space emerged as a convenient and unifying framework
for working with “linear” objects and we will discuss this notion in the next few sections.
1.2. VECTOR SPACES 17
1.2 Vector Spaces

A (real) vector space is a set E together with two operations, + : E E ! E and : R E !
E, called addition and scalar multiplication, that satisfy some simple properties. First of all,
E under addition has to be a commutative (or abelian) group, a notion that we review next.
However, keep in mind that vector spaces are not just algebraic
objects; they are also geometric objects.
Definition 1.1. A group is a set G equipped with a binary operation : G G ! G that
associates an element a b 2 G to every pair of elements a, b 2 G, and having the following
properties: is associative, has an identity element e 2 G, and every element in G is invertible
(w.r.t. ). More explicitly, this means that the following equations hold for all a, b, c 2 G:
(G1) a (b c) = (a b) c. (associativity);
(G2) a e = e a = a. (identity);
1 1 1
(G3) For every a 2 G, there is some a 2 G such that a a =a a = e. (inverse).
A group G is abelian (or commutative) if
a b = b a for all a, b 2 G.
A set M together with an operation : M M ! M and an element e satisfying only

conditions (G1) and (G2) is called a monoid . For example, the set N = f0, 1, . . . , n, . . .g of
natural numbers is a (commutative) monoid under addition. However, it is not a group.
Some examples of groups are given below.
Example 1.1.
1. The set Z = f. . . , n, . . . , 1, 0, 1, . . . , n, . . .g of integers is a group under addition,
with identity element 0. However, Z = Z f0g is not a group under multiplication.
2. The set Q of rational numbers (fractions p/q with p, q 2 Z and q 6= 0) is a group
under addition, with identity element 0. The set Q = Q f0g is also a group under
multiplication, with identity element 1.
3. Similarly, the sets R of real numbers and C of complex numbers are groups under
addition (with identity element 0), and R = R f0g and C = C f0g are groups
under multiplication (with identity element 1).
4. The sets Rn and Cn of n-tuples of real or complex numbers are groups under compo-
nentwise addition:
(x1 , . . . , xn ) + (y1 , . . . , yn ) = (x1 + y1 , . . . , xn + yn ),
with identity element (0, . . . , 0). All these groups are abelian.
5. Given any nonempty set S, the set of bijections f : S ! S, also called permutations
of S, is a group under function composition (i.e., the multiplication of f and g is the
composition g f ), with identity element the identity function idS . This group is not
abelian as soon as S has more than two elements.
6. The set of n n matrices with real (or complex) coefficients is a group under addition
of matrices, with identity element the null matrix. It is denoted by Mn (R) (or Mn (C)).
7. The set R[X] of all polynomials in one variable with real coefficients is a group under
addition of polynomials.
8. The set of n n invertible matrices with real (or complex) coefficients is a group under
matrix multiplication, with identity element the identity matrix In . This group is
called the general linear group and is usually denoted by GL(n, R) (or GL(n, C)).
9. The set of n n invertible matrices with real (or complex) coefficients and determinant
+1 is a group under matrix multiplication, with identity element the identity matrix
In . This group is called the special linear group and is usually denoted by SL(n, R)
(or SL(n, C)).
10. The set of n n invertible matrices with real coefficients such that RR> = In and of
determinant +1 is a group called the special orthogonal group and is usually denoted
by SO(n) (where R> is the transpose of the matrix R, i.e., the rows of R> are the
columns of R). It corresponds to the rotations in Rn .
11. Given an open interval ]a, b[, the set C(]a, b[) of continuous functions f : ]a, b[! R is a
group under the operation f + g defined such that
(f + g)(x) = f (x) + g(x)
for all x 2]a, b[.
It is customary to denote the operation of an abelian group G by +, in which case the

inverse a 1 of an element a 2 G is denoted by a.
The identity element of a group is unique. In fact, we can prove a more general fact:
Fact 1. If a binary operation : M M ! M is associative and if e0 2 M is a left identity
and e00 2 M is a right identity, which means that
e0 a = a for all a 2 M (G2l)
and
a e00 = a for all a 2 M, (G2r)
then e0 = e00 .
Proof. If we let a = e00 in equation (G2l), we get
e0 e00 = e00 ,
and if we let a = e0 in equation (G2r), we get
e0 e00 = e0 ,
and thus
e0 = e0 e00 = e00 ,
as claimed.
Fact 1 implies that the identity element of a monoid is unique, and since every group is
a monoid, the identity element of a group is unique. Furthermore, every element in a group
has a unique inverse. This is a consequence of a slightly more general fact:
Fact 2. In a monoid M with identity element e, if some element a 2 M has some left inverse
a0 2 M and some right inverse a00 2 M , which means that
a0 a = e (G3l)
and
a a00 = e, (G3r)
then a0 = a00 .
Proof. Using (G3l) and the fact that e is an identity element, we have
(a0 a) a00 = e a00 = a00 .
Similarly, Using (G3r) and the fact that e is an identity element, we have
a0 (a a00 ) = a0 e = a0 .
However, since M is monoid, the operation is associative, so
a0 = a0 (a a00 ) = (a0 a) a00 = a00 ,
as claimed.
Remark: Axioms (G2) and (G3) can be weakened a bit by requiring only (G2r) (the exis-
tence of a right identity) and (G3r) (the existence of a right inverse for every element) (or
(G2l) and (G3l)). It is a good exercise to prove that the group axioms (G2) and (G3) follow
from (G2r) and (G3r).
Before defining vector spaces, we need to discuss a strategic choice which, depending
how it is settled, may reduce or increase headackes in dealing with notions such as linear
combinations and linear dependence (or independence). The issue has to do with using sets
of vectors versus sequences of vectors.
Our experience tells us that it is preferable to use sequences of vectors; even better,
indexed families of vectors. (We are not alone in having opted for sequences over sets, and
we are in good company; for example, Artin [6], Axler [8], and Lang [60] use sequences.
Nevertheless, some prominent authors such as Lax [64] use sets. We leave it to the reader
to conduct a survey on this issue.)
Given a set A, recall that a sequence is an ordered n-tuple (a1 , . . . , an ) 2 An of elements
from A, for some natural number n. The elements of a sequence need not be distinct and
the order is important. For example, (a1 , a2 , a1 ) and (a2 , a1 , a1 ) are two distinct sequences
in A3 . Their underlying set is fa1 , a2 g.
What we just defined are finite sequences, which can also be viewed as functions from
f1, 2, . . . , ng to the set A; the ith element of the sequence (a1 , . . . , an ) is the image of i under
the function. This viewpoint is fruitful, because it allows us to define (countably) infinite
sequences as functions s : N ! A. But then, why limit ourselves to ordered sets such as
f1, . . . , ng or N as index sets?
The main role of the index set is to tag each element uniquely, and the order of the
tags is not crucial, although convenient. Thus, it is natural to define an I-indexed family of
elements of A, for short a family, as a function a : I ! A where I is any set viewed as an
index set. Since the function a is determined by its graph
f(i, a(i)) j i 2 Ig,
the family a can be viewed as the set of pairs a = f(i, a(i)) j i 2 Ig. For notational
simplicity, we write ai instead of a(i), and denote the family a = f(i, a(i)) j i 2 Ig by (ai )i2I .
For example, if I = fr, g, b, yg and A = N, the set of pairs
a = f(r, 2), (g, 3), (b, 2), (y, 11)g
is an indexed family. The element 2 appears twice in the family with the two distinct tags
r and b.
When the indexed set I is totally ordered, a family (ai )i2I often called an I-sequence.
Interestingly, sets can be viewed as special cases of families. Indeed, a set A can be viewed
as the A-indexed family f(a, a) j a 2 Ig corresponding to the identity function.
Remark: An indexed family should not be confused with a multiset. Given any set A, a
multiset is a similar to a set, except that elements of A may occur more than once. For
example, if A = fa, b, c, dg, then fa, a, a, b, c, c, d, dg is a multiset. Each element appears
with a certain multiplicity, but the order of the elements does not matter. For example, a
has multiplicity 3. Formally, a multiset is a function s : A ! N, or equivalently a set of pairs
f(a, i) j a 2 Ag. Thus, a multiset is an A-indexed family of elements from N, but not a
N-indexed family, since distinct elements may have the same multiplicity (such as c an d in
the example above). An indexed family is a generalization of a sequence, but a multiset is a
generalization of a set.
WePalso need to take care of an annoying technicality, which is to define sums of the
form i2I ai , where I is any finite index set and (ai )i2I is a family of elements in some set
A equiped with a binary operation + : A A ! A which is associative (axiom (G1)) and
commutative. This will come up when we define linear combinations.
The issue is that the binary operation + only tells us how to compute a1 + a2 for two
elements of A, but it does not tell us what is the sum of three of more elements. For example,
how should a1 + a2 + a3 be defined?
What we have to do is to define a1 +a2 +a3 by using a sequence of steps each involving two
elements, and there are two possible ways to do this: a1 + (a2 + a3 ) and (a1 + a2 ) + a3 . If our
operation + is not associative, these are different values. If it associative, then a1 +(a2 +a3 ) =
(a1 + a2 ) + a3 , but then there are still six possible permutations of the indices 1, 2, 3, and if
+ is not commutative, these values are generally different. If our operation is commutative,
then all six permutations have the same value. P Thus, if + is associative and commutative,
it seems intuitively clear that a sum of the form i2I ai does not depend on the order of the
operations used to compute it.
This is indeed the case, but a rigorous proof requires induction, and such a proof is
surprisingly
P involved. Readers may accept without proof the fact that sums of the form
i2I ai are indeed well defined, and jump directly to Definition 1.2. For those who want to
see the gory details, here we go.
P
First, we define sums i2I ai , where I is a finite sequence of distinct natural numbers,
say I = (i1 , . . . , im ). If I = (i1 , . . . , im ) with m 2, we denote the sequence (i2 , . . . , im ) by
I fi1 g. We proceed by induction on the size m of I. Let
X
ai = ai 1 , if m = 1,
i2I
X X
ai = ai 1 + ai , if m > 1.
i2I i2I fi1 g
For example, if I = (1, 2, 3, 4), we have

X
ai = a1 + (a2 + (a3 + a4 )).
i2I
If the operation + is not associative, the grouping of the terms matters. For instance, in
general
a1 + (a2 + (a3 + a4 )) 6= (a1 + a2 ) + (a3 + a4 ).
P
However, if the operation + is associative, the sum i2I ai should not depend on the grouping
of the elements in I, as long as their order is preserved. For example, if I = (1, 2, 3, 4, 5),
J1 = (1, 2), and J2 = (3, 4, 5), we expect that
X X X
ai = aj + aj .
i2I j2J1 j2J2
This indeed the case, as we have the following proposition.
Proposition 1.1. Given any nonempty set A equipped with an associative binary operation
+ : A A ! A, for any nonempty finite sequence I of distinct natural numbers and for
any partition of I into p nonempty sequences Ik1 , . . . , Ikp , for some nonempty sequence K =
(k1 , . . . , kp ) of distinct natural numbers such that ki < kj implies that α < β for all α 2 Iki
and all β 2 Ikj , for every sequence (ai )i2I of elements in A, we have
X X X
aα = aα .
α2I k2K α2Ik
Proof. We proceed by induction on the size n of I.

If n = 1, then we must have p = 1 and Ik1 = I, so the proposition holds trivially.
Next, assume n > 1. If p = 1, then Ik1 = I and the formula is trivial, so assume that
p 2 and write J = (k2 , . . . , kp ). There are two cases.
Case 1. The sequence Ik1 has a single element, say β, which is the first element of I.
In this case, write C for the sequence obtained from I by deleting its first element β. By
definition, X
X
aα = aβ + aα ,
α2I α2C
and
X X X X
aα = aβ + aα .
k2K α2Ik j2J α2Ij
Since jCj = n 1, by the induction hypothesis, we have

X X X
aα = aα ,
α2C j2J α2Ij
which yields our identity.

Case 2. The sequence Ik1 has at least two elements. In this case, let β be the first element
of I (and thus of Ik1 ), let I 0 be the sequence obtained from I by deleting its first element β,
let Ik0 1 be the sequence obtained from Ik1 by deleting its first element β, and let Ik0 i = Iki for
i = 2, . . . , p. Recall that J = (k2 , . . . , kp ) and K = (k1 , . . . , kp ). The sequence I 0 has n 1

elements, so by the induction hypothesis applied to I 0 and the Ik0 i , we get
X X X X X X
aα = aα = aα + aα .
α2I 0 k2K α2Ik0 α2Ik0 j2J α2Ij
1
If we add the lefthand side to aβ , by definition we get

X
aα .
α2I
If we add the righthand side to aβ , using associativity and the definition of an indexed sum,
we get
X X X X X X
aβ + aα + aα = aβ + aα + aα
α2Ik0 j2J α2Ij α2Ik0 j2J α2Ij
1 1
X X X
= aα + aα
α2Ik1 j2J α2Ij
X X
= aα ,
k2K α2Ik
as claimed.
Pn P
If I = (1, . . . , n), we also write i=1 a i instead of i2I ai . Since + is associative, Propo-
sition 1.1 shows that the sum ni=1 ai is independent of the grouping of its elements, which
P
justifies the use the notation a1 + + an (without any parentheses).
If we also assume that
P our associative binary operation on A is commutative, then we
can show that the sum i2I ai does not depend on the ordering of the index set I.
Proposition 1.2. Given any nonempty set A equipped with an associative and commutative
binary operation + : A A ! A, for any two nonempty finite sequences I and J of distinct
natural numbers such that J is a permutation of I (in other words, the unlerlying sets of I
and J are identical), for every sequence (ai )i2I of elements in A, we have
X X
aα = aα .
α2I α2J
Proof. We proceed by induction on the number p of elements in I. If p = 1, we have I = J

and the proposition holds trivially.
If p > 1, to simplify notation, assume that I = (1, . . . , p) and that J is a permutation
(i1 , . . . , ip ) of I. First, assume that 2 i1 p 1, let J 0 be the sequence obtained from J by
deleting i1 , I 0 be the sequence obtained from I by deleting i1 , and let P = (1, 2, . . . , i1 1) and
Q = (i1 + 1, . . . , p 1, p). Observe that the sequence I 0 is the concatenation of the sequences
P and Q. By the induction hypothesis applied to J 0 and I 0 , and then by Proposition 1.1
applied to I 0 and its partition (P, Q), we have
iX1 1 X p
X X
aα = aα = ai + ai .
α2J 0 α2I 0 i=1 i=i1 +1
If we add the lefthand side to ai1 , by definition we get

X
aα .
α2J
If we add the righthand side to ai1 , we get

iX
1 1 p
X
ai 1 + ai + ai .
i=1 i=i1 +1
Using associativity, we get

iX1 1 p
X iX
1 1 p
X
ai 1 + ai + ai = ai 1 + ai + ai ,
i=1 i=i1 +1 i=1 i=i1 +1
then using associativity and commutativity several times (more rigorously, using induction
on i1 1), we get
iX
1 1 Xp iX 1 1 Xp
ai 1 + ai + ai = ai + ai 1 + ai
i=1 i=i1 +1 i=1 i=i1 +1
p
X
= ai ,
i=1
as claimed.
The cases where i1 = 1 or i1 = p are treated similarly, but in a simpler manner since
either P = () or Q = () (where () denotes the empty sequence).
P
Having done all this, we can now make sense of sums of the form i2I ai , for any finite
indexed set I and any family a = (ai )i2I of elements in A, where A is a set equipped with a
binary operation + which is associative and commutative.
Indeed, since I is finite, it is in bijection with the set f1, . . . , ng for some n 2 N, and any
total ordering on I corresponds to a permutation I of f1, . . . , ng (where P we identify a
permutation with its image). For any total ordering on I, we define i2I, ai as
X X
ai = aj .
i2I, j2I
0
Then, for any other total ordering on I, we have
X X
ai = aj ,
i2I, 0 j2I0
and since I and I 0 are different permutations of f1, . . . , ng, by Proposition 1.2, we have
X X
aj = aj .
j2I j2I0
P
Therefore,
P the sum i2I, ai does
P not depend on the total ordering on I. We define the sum
i2I ai as the common value i2I, ai for all total orderings of I.
Vector spaces are defined as follows.
Definition 1.2. A real vector space is a set E (of vectors) together with two operations
+ : E E ! E (called vector addition)1 and : R E ! E (called scalar multiplication)
satisfying the following conditions for all α, β 2 R and all u, v 2 E;
(V0) E is an abelian group w.r.t. +, with identity element 0;2
(V1) α (u + v) = (α u) + (α v);
(V2) (α + β) u = (α u) + (β u);
(V3) (α β) u = α (β u);
(V4) 1 u = u.
In (V3), denotes multiplication in R.
Given α 2 R and v 2 E, the element α v is also denoted by αv. The field R is often
called the field of scalars.
In definition 1.2, the field R may be replaced by the field of complex numbers C, in which
case we have a complex vector space. It is even possible to replace R by the field of rational
numbers Q or by any other field K (for example Z/pZ, where p is a prime number), in which
case we have a K-vector space (in (V3), denotes multiplication in the field K). In most
cases, the field K will be the field R of reals.
From (V0), a vector space always contains the null vector 0, and thus is nonempty.
From (V1), we get α 0 = 0, and α ( v) = (α v). From (V2), we get 0 v = 0, and
( α) v = (α v).
1
The symbol + is overloaded, since it denotes both addition in the field R and addition of vectors in E.
It is usually clear from the context which + is intended.
2
The symbol 0 is also overloaded, since it represents both the zero in R (a scalar) and the identity element
of E (the zero vector). Confusion rarely arises, but one may prefer using 0 for the zero vector.
Another important consequence of the axioms is the following fact: For any u 2 E and
any λ 2 R, if λ 6= 0 and λ u = 0, then u = 0.
Indeed, since λ 6= 0, it has a multiplicative inverse λ 1 , so from λ u = 0, we get
1 1
λ (λ u) = λ 0.
1
However, we just observed that λ 0 = 0, and from (V3) and (V4), we have
1
λ (λ u) = (λ 1 λ) u = 1 u = u,
and we deduce that u = 0.
Remark: One may wonder whether axiom (V4) is really needed. Could it be derived from
the other axioms? The answer is no. For example, one can take E = Rn and define
: R Rn ! Rn by
λ (x1 , . . . , xn ) = (0, . . . , 0)
for all (x1 , . . . , xn ) 2 Rn and all λ 2 R. Axioms (V0)–(V3) are all satisfied, but (V4) fails.
Less trivial examples can be given using the notion of a basis, which has not been defined
yet.
The field R itself can be viewed as a vector space over itself, addition of vectors being
addition in the field, and multiplication by a scalar being multiplication in the field.
Example 1.2.
1. The fields R and C are vector spaces over R.
2. The groups Rn and Cn are vector spaces over R, and Cn is a vector space over C.
3. The ring R[X]n of polynomials of degree at most n with real coefficients is a vector
space over R, and the ring C[X]n of polynomials of degree at most n with complex
coefficients is a vector space over C.
4. The ring R[X] of all polynomials with real coefficients is a vector space over R, and
the ring C[X] of all polynomials with complex coefficients is a vector space over C.
5. The ring of n n matrices Mn (R) is a vector space over R.
6. The ring of m n matrices Mm,n (R) is a vector space over R.
7. The ring C(]a, b[) of continuous functions f : ]a, b[! R is a vector space over R.
Let E be a vector space. We would like to define the important notions of linear combi-
nation and linear independence. These notions can be defined for sets of vectors in E, but
it will turn out to be more convenient (in fact, necessary) to define them for families (vi )i2I ,
where I is any arbitrary index set.
1.3. LINEAR INDEPENDENCE, SUBSPACES 27
1.3 Linear Independence, Subspaces

One of the most useful properties of vector spaces is that there possess bases. What this
means is that in every vector space, E, there is some set of vectors, fe1 , . . . , en g, such that
every vector v 2 E can be written as a linear combination,
v = λ1 e1 + + λn e n ,
of the ei , for some scalars, λ1 , . . . , λn 2 R. Furthermore, the n-tuple, (λ1 , . . . , λn ), as above
is unique.
This description is fine when E has a finite basis, fe1 , . . . , en g, but this is not always the
case! For example, the vector space of real polynomials, R[X], does not have a finite basis
but instead it has an infinite basis, namely
1, X, X 2 , . . . , X n , . . .
For simplicity, in this chapter, we will restrict our attention to vector spaces that have a
finite basis (we say that they are finite-dimensional ).
Given a set A, recall that an I-indexed family (ai )i2I of elements of A (for short, a family)
is a function a : I ! A, or equivalently a set of pairs f(i, ai ) j i 2 Ig. We agree that when
I = ;, (ai )i2I = ;. A family (ai )i2I is finite if I is finite.
Remark: When considering a family (ai )i2I , there is no reason to assume that I is ordered.
The crucial point is that every element of the family is uniquely indexed by an element of
I. Thus, unless specified otherwise, we do not assume that the elements of an index set are
ordered.
Given two disjoint sets I and J, the union of two families (ui )i2I and (vj )j2J , denoted as
(ui )i2I [ (vj )j2J , is the family (wk )k2(I[J) defined such that wk = uk if k 2 I, and wk = vk
if k 2 J. Given a family (ui )i2I and any element v, we denote by (ui )i2I [k (v) the family
(wi )i2I[fkg defined such that, wi = ui if i 2 I, and wk = v, where k is any index such that
k2 / I. Given a family (ui )i2I , a subfamily of (ui )i2I is a family (uj )j2J where J is any subset
of I.
In this chapter, unless specified otherwise, it is assumed that all families of scalars are
finite (i.e., their index set is finite).
Definition 1.3. Let E be a vector space. A vector v 2 E is a linear combination of a family
(ui )i2I of elements of E iff there is a family (λi )i2I of scalars in R such that
X
v= λi ui .
i2I
P
When I = ;, we stipulate that v = 0. (By proposition 1.2, sums of the form i2I λi ui are
well defined.) We say that a family (ui )i2I is linearly independent iff for every family (λi )i2I
of scalars in R, X
λi ui = 0 implies that λi = 0 for all i 2 I.
i2I
Equivalently, a family (ui )i2I is linearly dependent iff there is some family (λi )i2I of scalars
in R such that X
λi ui = 0 and λj 6= 0 for some j 2 I.
i2I
We agree that when I = ;, the family ; is linearly independent.
Observe that defining linear combinations for families of vectors rather than for sets of
vectors has the advantage that the vectors being combined need not be distinct. For example,
for I = f1, 2, 3g and the families (u, v, u) and (λ1 , λ2 , λ1 ), the linear combination
X
λi ui = λ1 u + λ2 v + λ1 u
i2I
makes sense. Using sets of vectors in the definition of a linear combination does not allow
such linear combinations; this is too restrictive.
Unravelling Definition 1.3, a family (ui )i2I is linearly dependent iff either I consists of a
single element, say i, and ui = 0, or jIj 2 and some uj in the family can be expressed as
a linear combination of the other vectors in the family. Indeed, in the second case, there is
some family (λi )i2I of scalars in R such that
X
λi ui = 0 and λj 6= 0 for some j 2 I,
i2I
and since jIj 2, the set I fjg is nonempty and we get

X
uj = λj 1 λi ui .
i2(I fjg)
Observe that one of the reasons for defining linear dependence for families of vectors
rather than for sets of vectors is that our definition allows multiple occurrences of a vector.
This is important because a matrix may contain identical columns, and we would like to say
that these columns are linearly dependent. The definition of linear dependence for sets does
not allow us to do that.
The above also shows that a family (ui )i2I is linearly independent iff either I = ;, or I
consists of a single element i and ui 6= 0, or jIj 2 and no vector uj in the family can be
expressed as a linear combination of the other vectors in the family.
When I is nonempty, if the family (ui )i2I is linearly independent, note that ui 6= 0 for
all i 2 I. Otherwise, if ui = 0 for some i 2 I, then we get a nontrivial linear dependence
P
i2I λi ui = 0 by picking any nonzero λi and letting λk = 0 for all k 2 I with k 6= i, since
λi 0 = 0. If jIj 2, we must also have ui 6= uj for all i, j 2 I with i 6= j, since otherwise we
get a nontrivial linear dependence by picking λi = λ and λj = λ for any nonzero λ, and
letting λk = 0 for all k 2 I with k 6= i, j.
1.3. LINEAR INDEPENDENCE, SUBSPACES 29
Thus, the definition of linear independence implies that a nontrivial linearly independent
family is actually a set. This explains why certain authors choose to define linear indepen-
dence for sets of vectors. The problem with this approach is that linear dependence, which
is the logical negation of linear independence, is then only defined for sets of vectors. How-
ever, as we pointed out earlier, it is really desirable to define linear dependence for families
allowing multiple occurrences of the same vector.
Example 1.3.
1. Any two distinct scalars λ, µ 6= 0 in R are linearly dependent.
2. In R3 , the vectors (1, 0, 0), (0, 1, 0), and (0, 0, 1) are linearly independent.
3. In R4 , the vectors (1, 1, 1, 1), (0, 1, 1, 1), (0, 0, 1, 1), and (0, 0, 0, 1) are linearly indepen-
dent.
4. In R2 , the vectors u = (1, 1), v = (0, 1) and w = (2, 3) are linearly dependent, since
w = 2u + v.
When I is finite, we often assume that it is the set I = f1, 2, . . . , ng. In this case, we
denote the family (ui )i2I as (u1 , . . . , un ).
The notion of a subspace of a vector space is defined as follows.
Definition 1.4. Given a vector space E, a subset F of E is a linear subspace (or subspace)
of E iff F is nonempty and λu + µv 2 F for all u, v 2 F , and all λ, µ 2 R.
It is easy to see that a subspace F of E is indeed a vector space, since the restriction
of + : E E ! E to F F is indeed a function + : F F ! F , and the restriction of
: R E ! E to R F is indeed a function : R F ! F .
It is also easy to see that any intersection of subspaces is a subspace.
Since F is nonempty, if we pick any vector u 2 F and if we let λ = µ = 0, then
λu + µu = 0u + 0u = 0, so every subspace contains the vector 0. For any nonempty finite
index set I, one can show by induction on the cardinalityPof I that if (ui )i2I is any family of
vectors ui 2 F and (λi )i2I is any family of scalars, then i2I λi ui 2 F .
The subspace f0g will be denoted by (0), or even 0 (with a mild abuse of notation).
Example 1.4.
1. In R2 , the set of vectors u = (x, y) such that
x+y =0
is a subspace.
Another random document with
no related content on Scribd:
"I'm sorry John couldn't come with us," Pritchard commented between
puffs of his pipe as he swung the car rapidly from the bluestone drive onto
the macadam road. "He sticks too close to the grind. A chap needs some
sport over the week-end. I'd pass out cold if I didn't get in my eighteen holes
Sundays."
Prichard was evidently well known and well liked at the Greenwich
Country Club. He had no difficulty in making up a foursome from among
the crowd clustered about the first tee. Rodrigo was introduced to a Mr.
Bryon and a Mr. Sisson, men of about Pritchard's own age and standing. The
latter and his guest teamed against the two other men at a dollar a hole.
Rodrigo was quite aware that the eyes of the other three players were
critically upon him as he mounted the tee. He made a special effort to drive
his first ball as well as possible. He had learned golf at Oxford and was a
good player. But he had not hit a ball for months and was uncertain how the
lay-off and the strange clubs he was using would affect his game. However,
he got off a very respectable drive straight down the fairway and was
rewarded by the approbation of his mates.
After the first few holes, in which Rodrigo more than held his own, the
other developed a more friendly and natural attitude toward the titled
foreigner. Rodrigo, due to his English training, his predilection for
Americans like Terhune at Oxford, and his previous visit to the States,
together with his unaffectedness and adaptability, had few of the marked
unfamiliar characteristics of the Latin. Soon he was accepted on a free and
easy footing with the others. He laughed and chaffed with them and had a
very good time indeed.
Warren Pritchard took golf too seriously to derive much diversion out of
it. The money involved did not mean anything to him, but he was the sort of
intensely ambitious young American who always strove his utmost to do
even the most trivial things well. He whooped with childish joy at
extraordinary good shots by either himself or Rodrigo. At the end of the
match, which the Dorning representatives won by a substantial margin, he
congratulated the Italian heartily and uttered an enthusiastic tribute to his
game. Pritchard seemed more at home with average, go-getting Americans
like Bryon and Sisson than he had with the Dornings, Rodrigo thought. On
the way back from the links, they post-mortemed the match gayly. Warren
Pritchard, who had been inclined to look a little askance at first at his
brother-in-law's rather exotic acquaintance, was now ready to concede
Rodrigo was very much all right.
Having taken a shower and changed his clothes, Rodrigo came down and
pulled up a chair beside Henry Dorning on the front piazza. Alice had at the
last moment joined John in his ride over to the Fernalds, it seemed, and
Warren was down at the stables talking with the caretaker of the estate.
Henry Dorning remarked pleasantly that John and Alice had not returned
as yet but would doubtless be back any moment. "I am somewhat worried
about John," the elderly man continued. "He is not so very strong, you know,
and he applies himself altogether too steadily to business. He tells me that
you are rapidly taking hold and are of great assistance to him already." He
looked intently at Rodrigo, as if debating with himself whether or not to
make a confidant of him. Then he asked quietly, "You like my son very
much, do you not?"
"Very much," Rodrigo said promptly.
"He is a young man of honor and of considerable artistic and business

ability besides," said John's father. "Sometimes though, I wonder if he is not
missing something in life. For a man of his age, he is singularly ignorant of
some things. Of the world outside of his own business and family, for
instance. I feel that I can speak freely to you, Rodrigo—if you will permit
me to call you that upon such short acquaintance. He admires you very
much, and I think you are destined to be even closer friends than you are
now."
"I hope so," acknowledged Rodrigo.
"You are a man of the world. You can see for yourself that John's
development has been—well, rather one-sided. It is largely my own fault, I
admit. He has been reared upon Dorning and Son from the cradle. But there
are other things in life. He has no predilection whatever, for instance, for
feminine society. Oh, he adores his sister and he mingles with women and
girls we know. But he takes no especial interest in any of them except Alice.
That is wrong. Women can do a lot toward developing a man. They can do a
lot of harm to a man, too, but that has to be risked. A man has not reached
real maturity until he has been violently in love at least once. He does not
acquire the ability to look upon life as a whole until he has been through
that. Of that I am quite convinced."
Had John told his father of Rodrigo's former career of philandering? The
Italian wondered. Then he decided that John was no tale-bearer. Henry
Dorning must have deduced from his guest's general air of sophistication and
his aristocratic extraction that he was worldly wise.
The elder Dorning went on, "I have sometimes wondered what will
happen to John when he has his first love affair. Because sooner or later it
will happen, and it will be all the more violent because of its long
postponement. And the girl is quite likely to be of the wrong sort. I can
imagine an unscrupulous, clever woman setting out deliberately to ensnare
my son for his money and succeeding very handily. He is utterly
inexperienced with that type of woman. He believes they are all angels.
That's how much he knows about them. He is so much the soul of honor
himself that, though he has developed a certain shrewdness in business
matters, in the affairs of the heart he is an amateur.
"John is such a sensitive, high-strung boy. It is quite conceivable that an

unfortunate love affair would ruin his whole life. He would be without the
emotional resiliency to recover from such a catastrophe that the average man
possesses. I am boring you with all this, Rodrigo, because I believe you can
help him. Without in any way appointing yourself either his chaperon or his
guide to worldly things, I think you can gradually draw him a little out of his
present narrow way of life. You are a very attractive man, and John is not
exactly unpleasing to the feminine eye. Together you could meet people who
are engaged upon the lighter things of life. Frivolous, pleasure-loving
people. People of Broadway. Enter into New York's night life. Go to
Greenwich Village, Palm Beach, Newport. Loaf and play. It will do you both
good.
"Of course I am very selfish in this as far as you are concerned. I am

thinking primarily of my son and his future. As soon as he told me about
you, I secretly rejoiced that he had made such a friend—a cosmopolitan, a
man who presumably knew the world. I had hoped that my son-in-law,
Warren, might prove such a companion for John. But Warren is too much in
love with his wife and too engrossed in his business. In the matter of taking
time to play, he is almost as bad as John."
Rodrigo smiled rather dourly to himself. He appreciated that Henry

Dorning's diagnosis of John was correct. He was sensible of the honor paid
him by the elderly man's confidence and request. But it impressed him as
ironical that he should now be urged by John's father to resume his former
mode of life, and to resume it to aid the very man for whom he had forsaken
it.
Nevertheless, he was about to indicate his willingness to conspire with

Mr. Dorning for the education of his son when the object of their discussion,
accompanied by Alice, was whirled up the drive in the limousine. John
joined the two men on the porch and Alice, with the object of speeding
dinner, disappeared into the house.
With a significant and quite unnecessary glance at Rodrigo, Mr. Dorning

changed the subject. John offered some laughing comment upon the
eccentric ideas of his friend, Edward Fernard, as to interior decoration and
inquired about Rodrigo's golf. The conversation lulled a bit and then Henry
Dorning, as if recalling something that had for the time being escaped his
mind, said, "Mark Rosner is back from Europe. He was up to see me the
other day."
"Yes, I told you he crossed with us," John replied. "I understand he has
bought a building on Forty-Seventh Street, a converted brown-stone front
and intends opening up an antique shop very soon."
"That's what he came to see me about," Mr. Dorning commented dryly.

"He wanted me to take a mortgage on the property, so that he could buy it."
"Did you do it?"
"Yes. Fifteen thousand dollars."
John frowned. "I wish he hadn't bothered you about that. He is such a
nervous, irritating little man. He could just as well have come to me, and you
wouldn't have been annoyed."
"I didn't mind. And you needn't either, John. I got in touch with Bates
and he is taking care of the whole matter. We can both dismiss it from our
minds." Emerson Bates was the Dornings' very efficient and very expensive
lawyer. Mr. Dorning smiled reminiscently. "Rosner was always such a fretty,
worried type, as you say. I tried diplomatically to dissuade him from
attempting a big undertaking such as he is in for. He hasn't the temperament
or the business ability to swing it. If anything goes wrong, he is liable to
suffer a nervous breakdown or worse. This failure in London nearly did for
him for a while, I understand. And he tells me he married over there, and
they have two small children. Such men should be kept out of large business
undertakings. They aren't built for it."
"And yet you advanced him fifteen thousand dollars," John smiled
affectionately at his father. He knew this white-haired man's weakness for
helping others. He had inherited it himself.
"Well, Rosner was with me quite a while at the shop. He is getting along
in years now, and he is fearfully anxious to make a success. We old chaps
have to stick together, you know."
As Alice appeared in the broad doorway, announcing dinner, John

Dorning put a tender arm about his father to assist him from his chair. There
was something touching and ennobling in the scene to Rodrigo, watching
them, and something a little pathetic too.
CHAPTER VII
When Rodrigo reached his office the next morning, his exasperatingly
efficient spinster secretary had long since opened his mail and had the
letters, neatly denuded of their envelopes, upon his desk. That is, all but one.
She had evidently decided that this one was of too private a nature for her to
tamper with. The envelope was pale pink and exuded a faint feminine scent.
It was addressed in the scrawly, infantile hand of Sophie Binner and was
postmarked Montreal. Rodrigo fished it out of the pile of business
communications, among which it stood out like a chorus girl at a Quaker
meeting, and, breaking the seal, read it:
Dearest Rod,
Why the elusiveness, dear boy? I called you up three times. I

hope it was accidental that I couldn't reach you, though it
looks bad for poor Sophie, since you never tried to get in
touch with me as you promised. Or did you?
Well, I'm here with the show in Montreal. They decided to

get us ready up here among our own land before springing us
upon the Yankees. But it's so lonesome. Christy is such a
bore.
We open in New York a week from to-night. Times Square

Theatre. How about a party after the show? I can get some of
the other girls if you like. But would prefer just us two. You
know—like the good old days in London. I miss you
dreadfully, dear boy. Do drive my blues away as soon as I get
back to the U.S.A. Be nice to me. And write.
Your loving
SOPHIE.
Rodrigo smiled wryly as he folded up the letter and slipped it into his
pocket. He had received scores of such communications from Sophie. He
had been used to replying to them in kind. He had seldom been temperate in
his letters to her. He rather prided himself upon the amount of nonsense he
was able to inject into plain black ink. That had been the trouble in the case
of his billets doux to Rosa Minardi.
But he was not thinking of Rosa at the present moment. It had occurred
to him that some use might be made of the invitation in the pink letter in
connection with the promise he had made to Henry Dorning to broaden
John's horizon. By Jove, he would take up Sophie's suggestion for a party on
the night of the New York opening of the Christy Revue. He would invite
John and another of Sophie's kind to accompany them. Pretty, thrill-seeking
Sophie—she was certainly a great little horizon-broadener. And he would
leave it to her to pick from the Christy company another coryphee of similar
lightsomeness.
He resolved to set the ball rolling at once and, the rest of his mail unread,
rose and started into the neighboring office. Opening the door of John's
sanctum, he stopped for a moment to view the tableau inside.
Two blond heads were bent absorbedly over a letter on John's desk, a
man's and a woman's. They were talking in low voices, and Mary Drake's
pencil was rapidly underscoring certain lines in the letter. She was advancing
an argument in her soft, rapid voice, evidently as to how the letter should be
answered. John was frowning and shaking his head.
Rodrigo, standing watching them, wondered why they were not in love
with each other. Here was the sort of woman John needed for a wife. Though
he could not catch her exact words, he gathered that she was trying to
influence him to answer this letter in much more decided fashion than he had
intended. That was Mary Drake all over. Thoroughly business-like,
aggressive, looking after John's interests, bucking him up at every turn. That
was the trouble as far as love was concerned. John regarded her as a very
efficient cog in the office machinery rather than as a woman. And yet she
was very much of a woman. Underneath the veneer of almost brusqueness,
there was a tender stratum, as Rodrigo thought he had discovered in her
unguarded moments. Love could be awakened in Mary Drake by the right
man, and it would be a very wonderful sort of love.
Rodrigo asked himself if he really wanted John Dorning to be the

awakener. Something in his own heart seemed to protest. Watching her, a
feeling of tenderness for her swept over him. He had never again sought
jauntily to flirt with her as he had attempted to do that first day he met her. A
deeper feeling for her, such as he had never experienced before for any
woman, was being slowly kindled within him. And this feeling was steadily
growing deeper as she began admitting him to her friendship on much the
same status that John Dorning enjoyed.
She glanced up and saw Rodrigo. Smiling good-morning to him and
quickly gathering up John's letters, including the one under debate, and her
stenographic notebook, she made a movement to retire to her own office.
"Don't let me drive you away, Mary," Rodrigo said in a genial voice.
"You're not. I was just going anyway." She turned to Dorning. "Then I'll
write Mr. Cunningham we cannot take care of him until he pays for the other
consignment?"
John hesitated, then he nodded affirmatively. "You're absolutely ruthless,

Mary," he protested ruefully, "and you may lose us a good customer, as well
as the money he owes us. But perhaps you know best. Go ahead—write him
as you like."
She enjoyed her little triumph. "Don't worry, John. I know Mr.
Cunningham, and he's no person to be treated with silk gloves on." And she
hurried into her office and closed the door behind her. In an instant they
heard the hurried clack of her typewriter.
"John, I can't tell you how much I enjoyed that little visit with your
folks," Rodrigo began sincerely.
John beamed. "That's fine. And I can tell you they liked you too."
Rodrigo continued, "Maybe I'm to have the chance soon to repay you in
some small measure. Do you remember Sophie Binner, the English actress
we met on the ship coming over? The pretty blonde we walked around the
deck with?" After a slight pause, John concluded he did.
Rodrigo produced the little pink missive from his pocket and flourished
it. "Well, Sophie has invited you and me to a party the night her show opens
here in town. A week from to-night. It will be a nice, lively time. You'll like
it. Shall I answer her it's a date?"
John shot a questioning glance at Rodrigo. The latter wondered uneasily

if his friend was interpreting the invitation as a sign Rodrigo was back-
sliding a bit. "She particularly wants to see you," Rodrigo hastened to lie.
Then, impulsively, "Oh, let's go, John. We both need a change, a little tonic.
I know you don't care for Sophie's kind of people or entertainment usually.
Neither do I—any more. But, for one night, I think it would be a lot of fun.
We could go to some night club, see the sights, dance around a little, leave
them at their hotels, and go on home. What do you say?"
Perhaps John agreed with him. Perhaps it was merely the eagerness in
Rodrigo's voice that swung him. At least he finally concluded, "You're right.
We have been sticking pretty close. I'll be glad to come along, though the
girls will probably find me a bit slow."
"Nonsense," cried Rodrigo, and slapped his friend lustily on the back.
"That's fine," he added. "I'll write Sophie directly."
Falling into an old habit, he started the letter "Dearest Sophie" almost
subconsciously and he used rather intimate language, without paying much
heed to what he was doing. He would rather like to see Sophie again and
bask in her effulgence for a few hours. But as she would be merely the
means of carrying out his and Henry Dorning's purpose, he excused himself.
There would be none of the old thrill in flattering her in ink, he feared, as he
sat down to write her. Yet he surprised himself with the warmth he worked
up in the letter to her.
"COME ON OUTSIDE AND I'LL SHOW YOU HOW MUCH OF A
SHEIK YOU ARE," SNARLED HIS ANTAGONIST.
He received an immediate reply from her. She was tickled as pink as her
note-paper, he gathered. He wrote her two more notes, even more
affectionate than the first—one had to pretend to be mad over Sophie or she
would lose interest at once—and was rewarded with many long, scrawled
pages telling of joy over their coming meeting, the selection of one Betty
Brewster as "a great sport and a neat little trick" as the fourth member of the
party, complaints about Christy and the neutral reception the show had
received in Canada.
John Dorning's coming-out party was assuming the proportions of a

festive affair.
John himself made no further mention of it. Rodrigo did not remind him,
having a feeling that his friend might shy off if he gave the matter much
thought. Then, on the morning of the Christy Revue opening, Rodrigo as off-
handedly as possible spoke of their engagement that evening. And John,
looking blankly, and then confusedly, said, "Why, Rodrigo, I thought I told
you. I'm leaving for Philadelphia this afternoon to attend the dinner of the
Rand Library trustees. You knew we'd put in a bid to furnish the fresco work
for the new building."
Rodrigo's face fell. But his first feeling of irritation and disappointment
passed quickly. John was so frankly mortified. He had so completely
forgotten all about Sophie. It was almost funny. Rodrigo said, "Can't you put
off your trip? Sophie will be very much disappointed."
"You know I can't postpone it," John faltered. "The dinner at

Philadelphia was arranged especially for me. I'll have to go."
Rodrigo shrugged. "Well, I dare say I can patch it up with Sophie. We'll
make it some other time. I'll give her a ring later and call it off for to-night."
"Rodrigo, I hope I haven't caused you any inconvenience. I'll be glad to

go out with your friends any other time you say," John pleaded.
"Oh, don't worry, old boy. I'll fix it up. You just go right ahead down to
Philadelphia, and bring home that contract. Business before pleasure, you
know."
But, around six o'clock, Rodrigo wondered if that were such an excellent
motto after all. He had been too busy all day to call Sophie. Dorning and Son
closed at five o'clock, and he was all alone there now in the deserted quasi-
mausoleum. Mary Drake, who was usually a late worker, had left in the
middle of the afternoon, because her mother was not feeling well. Now that
the party with Sophie was definitely off and he had nothing but a long
lonesome evening to look forward to, Rodrigo had a feeling of
disappointment. He had been working hard and faithfully for three months,
and he had been looking forward to this evening of pleasure. He deserved it,
by Jove.
On an impulse, he located Bill Terhune's telephone number and picked
up the instrument. Waiting while the bell buzzed, he told himself that
Terhune had probably long since left his office. He half guiltily hoped the
former Oxonian had. But Terhune's familiar voice smote his ear with a bull-
like "Hullo!"
This was followed by a roar of joyous surprise as Rodrigo identified

himself. Agitated questions and replies. Rodrigo broached the proposition of
appointing his delighted listener a substitute for John Dorning on the Sophie
Binner junket.
"Fine! Great!" fairly shouted Terhune. "I'll call my wife up and tell her
I've dropped dead or something."
"Bill—you're married?" questioned Rodrigo.
"Sure. All architects have to get married. It gives them the necessary
standing of respectability that gets the business. I even live in Jersey. Think
of that, eh? Don't worry about my wife. I can fix it up. She's used to having
me stay in town over-night, and has gotten tired of asking questions. I'll
bring the liquor, too. What's that? Oh, sure—we need liquor. This Binner
baby's a regular blotter, if I remember her rightly. I've got a stock right here
in the office. Good stuff too. I'll meet you in the lobby of the Envoy. I'll take
a room there for the night. What's that? Oh, no—couldn't think of staying at
your place. You know me, Rod—what would your cultured neighbors say,
eh? Don't forget now—lobby of the Envoy at six-thirty. I'll dash right around
there now and book a room."
Bill Terhune had already registered at the plush-lined Hotel Envoy and
was waiting at the desk, key in one hand and a suitcase in the other, when
Rodrigo walked in. Terhune was bigger, especially around the waistline, and
more red-faced than ever, Rodrigo saw at a glance. The waiting man
greeting the Italian with a lusty roar, bred on the broad Dakota prairies, that
could be heard all around the decorous, palm-decorated lobby.
"Well, well," Bill rumbled, "who would have thought the Count would
have come to this, eh? But say, boy, I'm sure glad to see you. Come up and
have a drink. Hey, bellboy! Grab that bag, will you, and be very careful with
it too. It contains valuable glassware."
Up in the twelfth floor room which Bill had hired for the night at a
fabulous stipend, the American at once dispatched the bellboy for ice,
glasses, and White Rock. Then he disrobed, sputtered in the shower-bath for
a few minutes, rubbed himself a healthy pink and dressed in his dinner
clothes, which he had brought along in his bag.
"Always keep them at the office," he chuckled. "I can't tell when I might
have an emergency call." He poured bootleg Scotch into the glasses and
rocked the ice around with a spoon.
"How do you get away with it, Bill?" Rodrigo asked, smiling. "I thought
American wives were regular tyrants."
"That's how much you foreigners know," scoffed Bill. "All women love
my type. You can always keep their love by keeping them wondering. That's
my system—I keep my wife wondering whether I'm coming home or not."
He handed Rodrigo a full glass with a flourish. "To good old Oxford," he
toasted with mock reverence. Rodrigo echoed the toast.
The Italian refused another drink a few minutes later, though his action
did not discourage Terhune from tossing off another. In fact, the genial Bill
had three more before he agreed that they had better eat dinner if they
wished to make the Christy Revue by the time the curtain rose. Rodrigo did
not fancy Bill's taking on an alcoholic cargo that early in the evening. Bill
was a nice fellow, but he was the sort of chronic drinker who, though long
habit should have made him almost impervious to the effects of liquor,
nevertheless always developed a mad desire to fight the whole world after
about the fifth imbibing.
They descended in the elevator, Bill chattering all the while about his
pleasure at seeing his old friend again and about the extreme hazards of the
architect business in New York. A small concern like his didn't have a
chance, according to Bill. The business was all in the hands of large
organizations who specialized in specific branches of construction, like
hotels, residences, restaurants and churches, and made money by starving
their help.
After dinner the two men made jerky, halting taxicab progress through
the maelstrom of theatre-bound traffic and reached their seats at the Times
Square Theatre over half an hour late. The house was filled with the usual
first-night audience of friends of the company, critics, movie stars, society
people, chronic first-nighters, men and women about town, and
stenographers admitted on complimentary tickets given them by their bosses.
It was a well-dressed, lively crowd, and one that was anxious to be very kind
to the show. In spite of this, Rodrigo was quite sure by the middle of the first
act that the revue wouldn't do. It was doomed to the storehouse, he feared.
The girls were of the colorless English type, comparing not at all with the
hilariously healthy specimens one found in the American musical comedies.
Christy had skimped on the costumes and scenery, both of which items were
decidedly second rate. The humor had too Londonish a flavor, and the ideas
behind the sketches were banal in the extreme.
However, when Sophie Binner came on quite late in the act, Rodrigo sat
up and admitted that the sight of her again gave him decided exhilaration.
She was alluring in her costume of pale blue and gold, a costume which
exposed the famous Binner legs to full advantage and without the
encumbrance of stockings. The audience liked her also. She was the prettiest
woman the footlights had revealed thus far, and she had a pleasing, though
not robust voice. Coupled with this was an intimate, sprightly personality
that caught on at once. She responded to two encores and finally disappeared
amid enthusiastic applause.
Rodrigo turned to comment upon her success to Bill Terhune, and

discovered that the Dakotan had fallen fast asleep.
During the intermission, Rodrigo left his somnolent seat-mate and,

buttonholing an usher, sent him back-stage with his card. In a few minutes,
he followed the card to the dressing room of Sophie, where, in contrast to the
noisy confusion outside, he was permitted to gaze upon her gold-and-tinsel
liveliness at close range. She was sitting at her dressing-table, a filmy wrap
thrown carelessly about the costume she has worn in the first act. Her slim,
white body looked very girlish. Her wise, laughing blue eyes welcomed him.
With a swift look at the closed door, she invited, "Kiss me, Rodrigo, and say
you're glad to see me."
He obeyed, not altogether because it is always polite to accommodate a

pretty lady who asks to be kissed. He wanted to kiss her. He would have
done it without the invitation. He did it very expertly too. Sophie waved her
hatchet-faced English maid out of the room. But that gesture was
unnecessary. Rodrigo explained that he could only stay a minute. He had left
the other male member of their contemplated foursome, sleeping. They
laughed merrily over that. Sophie said she would be overjoyed to see Bill
Terhune again. "I was afraid you were going to bring that sober-faced
business partner of yours," she interjected. Rodrigo stiffened a little, but
decided that this was neither the time nor the place to start an impassioned
defence of John Dorning. The principal thing, he said, was to be sure Sophie
and her companion were set for the festivities after the show. They were, she
cried. She and Betty Brewster would meet them at the stage door fifteen
minutes after the final curtain.
CHAPTER VIII
For an enormous bribe, the head waiter at the Quartier Latin removed the
"Reserved" sign from a cozy table very near the dance floor and assisted the
two ladies in draping their cloaks about their chairs. The "club" was crowded
with the usual midnight-to-dawn merry-makers—brokers, theatrical
celebrities, society juveniles of both sexes, sweet sugar daddies and other
grades of daddies, bored girls, chattering girls, and plain flappers.
The Quartier Latin, Bill Terhune, awake, loudly proclaimed, was

Broadway's latest night club rage. Well protected by the police.
Powdered white cheeks matched laundered white shirt-fronts as their

owners "charlestoned" in each other's arms to the nervous, shuffling, muffled
rhythm of the world's greatest jazz band. The air was full of talk, laughing,
smoke, the discreet popping of corks and the resultant gurgle. The walls of
the Quartier Latin were splashed with futurist paintings of stage and screen
stars. The Frenchy waitresses wore short velvety black skirts, shiny silk
stockings and artists' tams. They carried trays shaped like palettes. The
tables were jammed so close together that one little false move would land
one in one's neighbor's lap. Which would probably not have annoyed one's
neighbor in the least, such was the spirit of the place. Everybody seemed to
be working at top speed to have a good time as quickly as possible. It was
rowdy, upsetting, exciting.
With the orchestra in action, one had to almost shout across the table to
be heard above the din. Bill Terhune shouted at once to the waitress for
glasses and the non-spiritous ingredients of highballs. They arrived, were
flavored with libations from Bill's hip, and were consumed with approval.
Then they danced, Rodrigo with Sophie and Bill with Betty Brewster. The
latter was older than Sophie and much less vivacious and attractive. There
were suggestions of hollows in her neck, her hair was that dead blond that
comes from an excessive use of artificial coloring, and her eyes had a lack-
lustre gleam. She was a typical show-girl who is nearing the declining period
of her career. Next year one would find her on the variety stage, the
following in a small-time burlesque production, then God knows where. To
Rodrigo, there was, at first glance, something a little pathetic about her. He
had expected that Sophie would invite a girl somewhat less radiant than
herself. It is the habit with beauties to eliminate as much competition as
possible of their own sex in their engagements with men.
But Rodrigo had little time to think about Betty. The highball, the
disarmingly close presence of Sophie, and the general hilarious laxity of his
surroundings were lulling his feelings. Sophie snuggled more closely to him.
He breathed the faint, sweet perfume of her hair. The throbbing jungle music
beat. The close atmosphere scented with cigarettes and cosmetics, the faces
of dancing couples near him smothered thoughts of Dorning and Son. For
the time being, he was the old Rodrigo.
"Boy, you can dance," breathed Sophie, slowly disengaging herself from
his embrace as the music stopped.
He looked at her. "You're a witch, Sophie, a soft, white witch," he
whispered.
They had another round of highballs. Bill Terhune, fast attaining a

fighting edge, began abusing the waitress. In his growing quarrelsomeness,
he noticed that Betty Brewster was not to be compared in pulchritude to
Sophie. He breathed alcoholically upon the latter and demanded with
unnecessary peremptoriness that she dance next with him. With a little
grimace of annoyance at Rodrigo, she turned smilingly to Bill and
acquiesced.
After the next dance, Terhune again produced his enormous flask, whose
contents seemed capable of flowing endlessly, like Tennyson's brook.
Rodrigo suggested mildly that they had all had enough. But the motion was
overruled, three to one. Bill's watery and roving eye caught the equally
itinerant optics of a sleek, dark girl two feet from him, at the next table. She
smiled veiledly, and he elaborately offered her a drink. Rodrigo was not
pleased with this by-play. He had been watching the girl's escort, a florid
chubby stock-broker type who had also been drinking copiously and who
now eyed Bill Terhune with a decidedly disapproving frown. With a defiant
toss of her shiny bobbed head at her middle-aged table-mate, the dark girl
accepted the glass and bent her ear to hear Bill's blurred invitation to dance
that accompanied it. The tom-toms and saxophones commenced their lilting
cadence, and Bill's new conquest and Bill arose simultaneously to dance. So
did the fat man. He seized Bill's wrist, which was around the girl.
Rodrigo was to his feet in a flash. He knew Bill Terhune. He caught the
Dakotan's wrist as, eluding the jealous sugar daddy's grip, it was whipped
back and started on its swift devastating journey to the corpulent one's jaw.
"No rough stuff, Bill," Rodrigo cautioned rapidly in a low voice. Bill turned
angrily upon his friend, but the Italian held his wrist like a vise. The eyes of
all three girls were popping with excitement. They were in the mood to
enjoy the sight of embattled males.
"Come on outside and I'll show you how much of a sheik you are,"
snarled Bill's red-faced antagonist.
Bill was keen to comply, and Rodrigo, welcoming the chance at least to
transfer the impending brawl to a less conspicuous battleground, loosed him.
The two champions set off for the lobby, picking their way unsteadily
through the staring dancers, Rodrigo by Bill's side, endeavoring to talk him
into a less belligerent mood, hopeless as the task was. Once in the wide open
spaces of the lobby, Bill suddenly eluded Rodrigo's arm upon his shoulder,
leaped toward his adversary, and smote him cleanly upon the jaw. The fat
man crashed against a fantastic wall painting of Gilda Grey and remained
huddled quietly where he had landed. All the fight had been knocked out of
him by this one sledge-hammer blow. Bill, his honor vindicated, was
contented also. All that remained was for Rodrigo to soothe the feelings of
the worried manager, who arrived on the run, and two husky bouncers, now
standing by to toss the embroiled patrons out upon the sidewalk.
Rodrigo did his task of diplomacy very nicely. The manager cooperated,
being anxious to avoid trouble. Cold water was administered to the fallen
gladiator. The girl who had caused all the trouble was summoned. Contrite at
the sight of her escort's damaged countenance, she readily agreed to take him
home, and the two were bundled into a taxicab.
Then the manager turned to Rodrigo and insisted firmly that the other
brawler should leave also. He could not afford further disturbances, which
might involve the police, however loathe the bluecoats might be to interfere
with the licensed Quartier Latin. Bill began to see red all over again at this
edict. But there were two husky bouncers at his elbow, and Rodrigo
supported the manager. Betty Brewster was paged, and Bill, muttering and
defiant to the last, followed in another taxi in the wake of his enemy.
Having banished Bill Terhune to the cool night air, Rodrigo turned to
hasten back to Sophie, who, he was afraid, would be furious at him for
leaving her sitting alone for such a long time.
"Good evening, Count Torriani," said a melting feminine voice at his

elbow. He stopped and turned to confront Mrs. Porter Palmer, who seemed
gushingly delighted to see him. He bowed and saw that, accompanying Mrs.
Palmer, was a young woman of such striking appearance as to arrest his eye
at once and hold it. Jet black hair caught tight to the head set off the waxen
pallor of her face. Her dark eyes were slightly almond-shaped and singularly
bright. She was dressed in a shimmering black satin evening gown that
displayed the graceful lines of her slim, svelte body and the creamy
whiteness of her shoulders. She was American, but not in appearance. In
Paris and Monte Carlo, Rodrigo had met beauties like this, but never in
America. She looked exactly like the type of woman who, in the old days,
had been irresistible to him. But that first swift impression, he told himself,
was nonsense. She was probably the soul of modesty.
"I want you to meet my niece, Elise Van Zile," said Mrs. Palmer.
He bent and kissed the glamorous lady's hand and was aware of her
languid eyes upon him. A moment later, he was introduced to Mr. Porter
Palmer, the twittering bald-headed little man who had been disposing of his
ladies' wraps.
"Elise has just come on from San Francisco for a few weeks, and we are
showing her the sights," explained Mrs. Palmer, and then to her husband. "It
seems terribly crowded and noisy in there, Edward. Do you think it's quite
respectable?" Mr. Palmer waved his hands in the air, deprecating his wife's
fastidiousness. She turned to Rodrigo, "Won't you join us at our table, Count
Torriani?"
"Thanks, really, but the lady I am with and I are just leaving," he made
haste to reply, immediately afterward wondering why he had invented this
falsehood. He glanced at the coolly beautiful Miss Van Zile, on whom his
refusal had apparently made no impression. Was he foolish in sensing, at his
very first glimpse of this girl from the West, something that warned him?
"But you will come to the tea I am giving for Elise next Saturday
afternoon at the Plaza, will you not, Count Torriani?" Mrs Palmer insisted.
He hesitated, then accepted. He again kissed the hand of Elise Van Zile,
and he raised his eyes to find her looking enigmatically at him. Somehow he
was reminded of the Mona Lisa, in whose dark eyes are painted all the
wisdom and intrigues of the world.
Rodrigo returned to a petulant Sophie. Both her white elbows were on

the table, and she was impatiently fingering the blazing diamond pendant at
her throat. It was a magnificent bauble, set in clusters of sapphires and
platinum. Her position revealed also her gorgeous diamond bracelets and the
large dazzling assortment of rings upon her fingers. Sophie was an assiduous
collector of jewelry, and, in the absence of something more interesting to do,
she was offering an exhibition of her arsenal to the crowd about her.
"Where have you been, Rodrigo?" she fretted as he sat down. "At least
you might have come back as soon as you made Betty leave me. I have felt a
perfect fool—sitting here alone, with everybody in the place staring at me."
He apologized profusely. She was right. People were staring at her. He

stared back so intently at the two young men with too-slicked hair and ill-
fitting evening clothes who had taken the table vacated by Bill Terhune's
antagonist, that they dropped their bold eyes.
"In that case," he answered her complaint, "let's leave. We can go to

some other place."
"I've a very pretty little apartment on the Drive," she suggested

demurely.
In the shadowy depths of the taxi tonneau a few moments later, she made
herself comfortable against his shoulder. It was long after midnight. Save for
machines bound on errands similar to theirs, the streets were deserted. The
car sped westward toward the river. Sophie broke a long silence by
murmuring, "You write the most wonderful letters, Rodrigo. I've saved them
all. Though I don't suppose you mean a word you say in them."
Rodrigo laughed contentedly. Close to him thus, Sophie was again

stirring his senses.
"Do you love me, Rodrigo—more than you ever did in London?" she
asked suddenly.
"You are lovelier than you ever were in London, Sophie," he quibbled.
"You are the loveliest girl I have ever known." But the image of Elise Van
Zile obtruded itself and rather spoiled this bit of flattery.

Textbook Fundamentals of Linear Algebra and Optimization Gallier J Ebook All Chapter PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Textbook Fundamentals of Linear Algebra and Optimization Gallier J Ebook All Chapter PDF

Uploaded by

Copyright:

Available Formats

Fundamentals of linear algebra and

Fundamentals of Linear Algebra and Optimization Jean

Fundamentals of optimization theory with applications

Fundamentals of optimization theory with applications

Basics of algebra topology and differential calculus

Fundamentals of Linear Algebra Textbooks in Mathematics

Linear Algebra and Optimization for Machine Learning: A

Applied Linear Algebra Second Edition Peter J. Olver

Applied Linear Algebra Second Edition Peter J. Olver

Jean Gallier and Jocelyn Quaintance

February 23, 2017

1 Vector Spaces, Bases, Linear Maps 11

2 Matrices and Linear Maps 45

3 Direct Sums, Affine Maps, The Dual Space, Duality 83

4 Gaussian Elimination, LU, Cholesky, Echelon Form 131

5.2 Alternating Multilinear Maps . . . . . . . . . . . . . . . . . . . . . . . . . . 197

6 Vector Norms and Matrix Norms 223

7 Eigenvectors and Eigenvalues 255

8 Iterative Methods for Solving Linear Systems 271

9 Euclidean Spaces 289

10 QR-Decomposition for Arbitrary Matrices 321

11 Hermitian Spaces 331

12 Spectral Theorems 359

13 Introduction to The Finite Elements Method 391

14 Singular Value Decomposition and Polar Form 413

15 Applications of SVD and Pseudo-Inverses 427

16 Annihilating Polynomials; Primary Decomposition 451

18 Differential Calculus 505

19 Quadratic Optimization Problems 541

20 Schur Complements and Applications 561

21 Convex Sets, Cones, H-Polyhedra 567

22 Linear Programs 579

23 The Simplex Algorithm 593

23.5 Computational Efficiency of the Simplex Method . . . . . . . . . . . . . . . 622

24 Linear Programming and Duality 625

25 Extrema of Real-Valued Functions 661

26 Newton’s Method and Its Generalizations 685

27 Basics of Hilbert Spaces 693

28 General Results of Optimization Theory 713

29 Introduction to Nonlinear Optimization 759

30 Soft Margin Support Vector Machines 845

31 Total Orthogonal Families in Hilbert Spaces 899

Vector Spaces, Bases, Linear Maps

1.1 Motivations: Linear Combinations, Linear Inde-

and to write our linear system as

is equivalent to determining whether b can be expressed as a linear combination of u, v, w.

it can be shown that every vector in R3 1

(y1 x1 )u + (y2 x2 )v + (y3 x3 )w = 0,

which implies that

If we form the vector of unknowns 0 1

so our linear system is expressed by

which yields an infinite number of solutions parameterized by x3 , namely

In summary, a 3 3 linear system may have a unique solution, no solution, or an infinite

u, v, w, b. This situation can be generalized to any n n system, and even to any n m

defines a subset of R3 which is actually a plane. The first equation

Low-rank decompositions of a set of data have a multitude of applications in engineering,

1.2 Vector Spaces

A set M together with an operation : M M ! M and an element e satisfying only

(f + g)(x) = f (x) + g(x)

for all x 2]a, b[.

It is customary to denote the operation of an abelian group G by +, in which case the

e0 a = a for all a 2 M (G2l)

Proof. If we let a = e00 in equation (G2l), we get