Linear Algebra Review and Reference Zico Kolter (updated by Chuong Do) October 7.2008 Contents 1 Basic Concepts and Notation 2 l.1 Basic Notation...···.········· 2 2 Matrix Multiplication 3 2.1 Vector-Vector Products... 4 2.2 Matrix-Vector Products 4 2.3 Matrix-Matrix Products 5 3 Operations and Properties 个 3.1 The Identity Matrix and Diagonal Matrices 8 3.2 The Transpose·········· 8 3.3 Symmetric Matrices..·..·. 8 3.4 The Trace..········· 9 3.5 Norms.····· 10 3.6 Linear Independence and Rank 11 3.7 The Inverse·.:..······· 11 3.8 Orthogonal Matrices...................·.··.. 12 3.9 Range and Nullspace of a Matrix 12 3.l0 The Determinant.·.·. 14 3.11 Quadratic Forms and Positive Semidefinite Matrices.......··· 17 3.12 Eigenvalues and Eigenvectors... 18 3.13 Eigenvalues and Eigenvectors of Symmetric Matrices 19 4 Matrix Calculus 20 4.1 The Gradient 20 4.2 The Hessian.. 2 4.3 Gradients and Hessians of Quadratic and Linear Functions 23 4.4 Least☒Squares...............······· 5 4.5 Gradients of the Determinant 25 4.6 Eigenvalues as Optimization ... 26Linear Algebra Review and Reference Zico Kolter (updated by Chuong Do) October 7, 2008 Contents 1 Basic Concepts and Notation 2 1.1 Basic Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 Matrix Multiplication 3 2.1 Vector-Vector Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Matrix-Vector Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Matrix-Matrix Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3 Operations and Properties 7 3.1 The Identity Matrix and Diagonal Matrices . . . . . . . . . . . . . . . . . . 8 3.2 The Transpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.3 Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.4 The Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.5 Norms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.6 Linear Independence and Rank . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.7 The Inverse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.8 Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.9 Range and Nullspace of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . 12 3.10 The Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.11 Quadratic Forms and Positive Semidefinite Matrices . . . . . . . . . . . . . . 17 3.12 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.13 Eigenvalues and Eigenvectors of Symmetric Matrices . . . . . . . . . . . . . 19 4 Matrix Calculus 20 4.1 The Gradient . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.2 The Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4.3 Gradients and Hessians of Quadratic and Linear Functions . . . . . . . . . . 23 4.4 Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.5 Gradients of the Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.6 Eigenvalues as Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 1