Ch. 1 Matrix Algebra 1 Some Terminology A matrix is a rectangular array of numbers, denoted 1 ai1 ai2 aik where a subscribed element of a matrix is always read as arou, column. Here we confine the element to be real number a vector is a matrix with one row or one column. Therefore a row vector is Alxk and a column vector is AixI and commonly denoted as ak and ai,respec- tively. In the followings of this course, we follow conventional custom to say that a vector is a columnvector except for particular mention The dimension of a matrix is the numbers of rows and columns it contained If i equals to k, then A is a square matrix. Several particular types of square matrices occur in econometrics (1).A symmetric matrix, a is one in which aik=ak: for all i and k. (e2).A diagonal matrix is a square matrix whose nonzero elements appears on the main diagonal, moving from upper left to lower right (3). A scalar matrix is a diagonal matrix with the same values in all diagonal elements (4). An identity matrix is a scalar matrix with ones on the diagonal. This matrix is always denoted as I. A subscript is sometimes included to indicate its size. for example 010 001 (5). A triangular matrix is one that has only zeros either above or below the main diagonal
Ch. 1 Matrix Algebra 1 Some Terminology A matrix is a rectangular array of numbers, denoted Ai×k = [aik] = a11 a12 . . . a1k a21 a22 . . . a2k . . . . . . . . . . . . . . . . . . ai1 ai2 . . . aik , where a subscribed element of a matrix is always read as arow,column. Here we confine the element to be real number. A vector is a matrix with one row or one column. Therefore a row vector is A1×k and a column vector is Ai×1 and commonly denoted as a ′ k and ai , respectively. In the followings of this course, we follow conventional custom to say that a vector is a columnvector except for particular mention. The dimension of a matrix is the numbers of rows and columns it contained. If i equals to k, then A is a square matrix. Several particular types of square matrices occur in econometrics. (1). A symmetric matrix, A is one in which aik = aki for all i and k. (2). A diagonal matrix is a square matrix whose nonzero elements appears on the main diagonal, moving from upper left to lower right. (3). A scalar matrix is a diagonal matrix with the same values in all diagonal elements (4). An identity matrix is a scalar matrix with ones on the diagonal. This matrix is always denoted as I. A subscript is sometimes included to indicate its size. for example, I3 = 1 0 0 0 1 0 0 0 1 . (5). A triangular matrix is one that has only zeros either above or below the main diagonal. 1
2 Algebraic Manipulation of Matrices 2.1 Equality of matrices Matrices A and B are equal if and only if they have the same dimensions and each elements of A equal the corresponding element of B A=B if and only if aik= bik for all i and k 2.2 Transposition The transpose of a matrix A, denoted A, is obtained by creating the matrix whose kth row is the kth column of the original matrix. If A is i x k, a is k x For example 1563 A 645 A 2141 314 3554 If A is symmetric, A=A. It is also apparent that for any matrix A, (A)=A Finally,the transpose of a column vector, a, is a row vector: 2.3 Matrix Addition Matrices cannot be added unless they have the same dimension. The operation of addition is extended to matrices by definin A+B=aik + bil We also extend the operation of subtraction to matrices precisely as if they were scalars by performing the operation element by element. Thus It follows that matrix addition is commutative A+b=b+A and associative
2 Algebraic Manipulation of Matrices 2.1 Equality of Matrices Matrices A and B are equal if and only if they have the same dimensions and each elements of A equal the corresponding element of B. A=B if and only if aik = bik for all i and k. 2.2 Transposition The transpose of a matrix A, denoted A′ , is obtained by creating the matrix whose kth row is the kth column of the original matrix. If A is i × k, A′ is k × i. For example, A = 1 2 3 5 1 5 6 4 5 3 1 4 , A′ = 1 5 6 3 2 1 4 1 3 5 5 4 . If A is symmetric, A=A′ . It is also apparent that for any matrix A, (A′ ) ′ = A. Finally, the transpose of a column vector, ai is a row vector: a ′ i = a1 a2 . . . ai . 2.3 Matrix Addition Matrices cannot be added unless they have the same dimension. The operation of addition is extended to matrices by defining C = A + B=[aik + bik]. We also extend the operation of subtraction to matrices precisely as if they were scalars by performing the operation element by element. Thus, C = A − B=[aik − bik]. It follows that matrix addition is commutative, A + B = B + A, and associative, 2
(A+B)+C and that (A+B=A+B 2.4 Matrix Multiplication Matrices are multiplied by using the inner product. The inner product of two vectors. a and b, is a scalar and is written a'b=anb+a2b2 +.+anbn= ba For an n x k matrix A and a k x T matrix B, the product matrix C= AB is an n x T matrix whose ith element is the inner product of row i of A and column k of B. generally, AB+BA The product of a matrix and a vector is a vector and is written as C= Ab ba1+ b2a+.+ bak where 6, are ith element of vector b and a are ith column of matrix A. here we see that the right-hand side is a linear combination of the columns of the matrix where the coefficients are the elements of the vector In the calculation of a matrix product C= Anxk BkxT, it can be written as aB where b: are ith column of matrix B Some general rules for matrix multiplication are as follow Associate law:(ABC=A(BC) Distributive law: A(B+C)=AB+AC Transpose of a product:(AB)=BA Scalar multiplication: aa=[aaik]fo
(A + B) + C = A + (B + C), and that (A + B) ′ = A′ + B′ . 2.4 Matrix Multiplication Matrices are multiplied by using the inner product. The inner product of two vectors, an and bn, is a scalar and is written a ′b = a1b1 + a2b2 + ... + anbn = b ′a. For an n × k matrix A and a k × T matrix B, the product matrix, C = AB, is an n × T matrix whose ikth element is the inner product of row i of A and column k of B. Generally, AB 6= BA. The product of a matrix and a vector is a vector and is written as c = Ab = b1a1 + b2a2 + ... + bkak, where bi are ith element of vector b and ai are ith column of matrix A. Here we see that the right-hand side is a linear combination of the columns of the matrix where the coefficients are the elements of the vector. In the calculation of a matrix product C = An×kBk×T , it can be written as C = AB = [Ab1 Ab2 AbT], where bi are ith column of matrix B. Some general rules for matrix multiplication are as follow: Associate law: (AB)C = A(BC). Distributive law: A(B + C) = AB + AC. Transpose of a product: (AB) ′ = B′A′ . Scalar multiplication: αA = [αaik] for a scalar α. 3
2.5 Matrix Inversion Definition A square matrix A is said to be nonsingular or invertible if there exist a unique matrix(square) B such that AB= BA The matrix b is said to be a multiplicative inverse of A We will refer to the multiplicative inverse of a nonsingular matrix a as simply the inverse of A and denote it by a Some computational results involving inverse are A (AB)-1 when both inverse matrices exist. Finally, if A is symmetric, then a-1is also symmetric Suppose that a, b and a+b are all m x m nonsingular matrices. Then (A+B)1=A-1-A-1(B-1+A-1)- 2.6 A useful idempotent matrix Definition An idempotent matrix is the one that is equal to its square, that is M2=MM M A useful idempotent matrix we will often face is the matrix Mo=I-ii
2.5 Matrix Inversion Definition: A square matrix A is said to be nonsingular or invertible if there exist a unique matrix (square) B such that AB = BA = I. The matrix B is said to be a multiplicative inverse of A. We will refer to the multiplicative inverse of a nonsingular matrix A as simply the inverse of A and denote it by A−1 . Some computational results involving inverse are |A−1 | = 1 |A| , (A−1 ) −1 = A, (A−1 ) ′ = (A′ ) −1 (AB) −1 = B −1A−1 , when both inverse matrices exist. Finally, if A is symmetric, then A−1 is also symmetric. Lemma: Suppose that A, B and A + B are all m × m nonsingular matrices. Then (A + B) −1 = A−1 − A−1 (B −1 + A−1 ) −1A−1 . 2.6 A useful idempotent matrix Definition: An idempotent matrix is the one that is equal to its square, that is M2 = MM = M. A useful idempotent matrix we will often face is the matrix M0 = I − 1 n ii′ , 4
such that Mox where i is a column of ones' s vector, x=[ 1, 2,,n]and i=i2iai Proof As definition Mox=(I--iix=x--iix Using the idempotent matrix Mo to calculate 2=G-j)2, where j=200 2=IG) from gauss 2. 7 Trace of matrix The trace of a square k x k matrix is the sums of its diagonal elements tr(A Some useful results are tr(ca)=ctr(a)) (A)=tr(A'), tr(A+B)=tr(B)+tr(A), tr(ABCD)= tr(BCDa)=tr(CDaB)=tr(DAbC)
such that M0x = x1 − x¯ x2 − x¯ . . . xn − x¯ , where i is a column of ones’s vector, x = [x1, x2, ..., xn] ′ and ¯x = 1 n Pn i=1 xi . Proof. As definition, M0x = (I − 1 n ii′ )x = x − 1 n ii′x = x − ix. ¯ Exercise: Using the idempotent matrix M0 to calculate P200 j=1(j−¯j) 2 , where ¯j = 1 200 P200 j=1(j) from Gauss. 2.7 Trace of Matrix The trace of a square k × k matrix is the sums of its diagonal elements: tr(A)=Pk i=1 aii. Some useful results are: 1. tr(cA) = c(tr(A)), 2. tr(A) = tr(A′ ), 3. tr(A + B) = tr(B) + tr(A), 4. tr(ABCD) = tr(BCDA) = tr(CDAB) = tr(DABC). 5
3 Geometry of matrices Matrix algebra is extremely useful in the formulation and solution of sets of linear equations. At the same time, the algebraic results have a geometrical basis that is very helpful in understanding the calculation. It is helpful to digress to a geometric treatment of matrices and vectors 3.1 Vector Spaces 3.1.1 Euclidean Vector Space Perhaps the most elementary vector are the Euclidean vector space R", n 1, 2,.... For simplicity, let us consider first R2. Non zero vector in R2 can be represented geometrically by directed line segments. Given a nonzero vector we can associate it with the line segment in the plane from(0, 0) to (1, 12). If we equate line segment that have the same length and direction, x can be represented by any line segment from(a, b)to(a+a1, b+r2). For example, the vector x I in R2 could be represented by the directed line segment from(2,2)to(4,3), or fron(-1,-1)to(1,0) We can think of the euclidean length of a vector x= as the length of any directed line segment representing x. The length of the segment from(0, 0) Two basic operations are defined for vectors, scalar multiplication and ad dition. The geometric representation will help us to visualize how the operation of scalar multiplication and addition work in R2 (1). Scalar multiplication: For each vector x 1 d scalar a. the product ax is defined by a t1 For example, the set of all possible scalar multiple of x is the line through 0 and x. Any scalar multiple of x is a segment of this line Example
3 Geometry of Matrices Matrix algebra is extremely useful in the formulation and solution of sets of linear equations. At the same time, the algebraic results have a geometrical basis that is very helpful in understanding the calculation. It is helpful to digress to a geometric treatment of matrices and vectors. 3.1 Vector Spaces 3.1.1 Euclidean Vector Space Perhaps the most elementary vector are the Euclidean vector space R n , n = 1, 2, .... For simplicity, let us consider first R 2 . Non zero vector in R 2 can be represented geometrically by directed line segments. Given a nonzero vector x = x1 x2 , we can associate it with the line segment in the plane from (0, 0) to (x1, x2). If we equate line segment that have the same length and direction, x can be represented by any line segment from (a, b) to (a + x1, b + x2). For example, the vector x = 2 1 in R 2 could be represented by the directed line segment from (2, 2) to (4, 3), or from (−1, −1) to (1, 0). We can think of the Euclidean length of a vector x = x1 x2 as the length of any directed line segment representing x. The length of the segment from (0, 0) to (x1, x2) is p x 2 1 + x 2 2 . Two basic operations are defined for vectors, scalar multiplication and addition. The geometric representation will help us to visualize how the operation of scalar multiplication and addition work in R 2 . (1). Scalar multiplication: For each vector x = x1 x2 and each scalar α, the product αx is defined by αx = αx1 αx2 . For example, the set of all possible scalar multiple of x is the line through 0 and x. Any scalar multiple of x is a segment of this line. Example: 6
2 The vector x*(= 2x) is in the same direction as x, but its length is two that of x. The vector x**(=-x) has half of length as x but its point in in the opposite direction (2). Addition: The sum of two vectors a and b is a third vector whose coordinates are the sums of the corresponding coordinates of a and b. For example +b Geometrically, c is obtained by moving in the distance and direction defined by b from the tip of a or, because addition is commutative from the tip of b in the distance and direction of a In a similar manner, vectors in R can be represented by directed line segments in a 3-space. Vector in Rn can be views as the coordinates of a point in a n- dimensional space or as the definition of the line segment connecting the origin and this point In general, scalar multiplication and addition in Rn are defined by and + 2+y2 for any x and y e Rn and any scalar a
x = 1 2 x ∗ = 2x = 2 4 x ∗∗ = − 1 2 x = − 1 2 −1 . The vector x ∗ (= 2x) is in the same direction as x, but its length is two times that of x. The vector x ∗∗(= − 1 2 x) has half of length as x but its point in the opposite direction. (2). Addition: The sum of two vectors a and b is a third vector whose coordinates are the sums of the corresponding coordinates of a and b. For example , c = a + b = 1 2 + 2 1 = 3 3 . Geometrically, c is obtained by moving in the distance and direction defined by b from the tip of a or, because addition is commutative, from the tip of b in the distance and direction of a. In a similar manner, vectors in R 3 can be represented by directed line segments in a 3-space. Vector in R n can be views as the coordinates of a point in a ndimensional space or as the definition of the line segment connecting the origin and this point. In general, scalar multiplication and addition in R n are defined by αx = αx1 αx2 . . . αxn and x + y = x1 + y1 x2 + y2 . . . xn + yn for any x and y ∈ R n and any scalar α. 7
3.1.2 Vector Space Axiom Definiti Let v be a set on which the operations of addition and scalar multiplication are (1).Ifx,y∈, then x+y∈, (2).Ifx∈ V and a is a scalar, then ax∈v The set v together with the operations of addition and scalar multiplication said to form a vector space if the following axioms are satisfied (a) x+y=y+x fo (b).(x+y)+z=x+(y+z) (c). There exist an element 0 in V such that x+0=x for each E V (d). For each x∈ 0 (e). a(x+y)=ax+ay for each real number a and any x and y in v (f).(a+ B)x=ax+ Bx for any real number a and B and any xE V (g).(aB)x=a(Bx)for any real number a and B and any xE V (h).1 The two-dimensional plane is the set of all vectors with two real-valued coordi nates. We label this set R2. It has two important properties (1).R is closed under scalar multiplication; every scalar multiple of a vector in the plane is also in the plane (2). R is closed under addition; the sum of any two vectors is always a vector in the plane 3.2 Linear Combination of vectors and basis vectors Definition A set of vectors in a vector is a basis for that vector space if any vector in the vector space can be written as a linear combination of them Example Any pair of two dimensional vectors that point in different directions will form
3.1.2 Vector Space Axiom Definition: Let V be a set on which the operations of addition and scalar multiplication are defined. By this we mean that (1). If x, y ∈ V, then x + y ∈ V, (2). If x ∈ V and α is a scalar, then αx ∈ V. The set V together with the operations of addition and scalar multiplication is said to form a vector space if the following axioms are satisfied: (a). x + y = y + x for any x and y in V. (b). (x + y) + z = x + (y + z). (c). There exist an element 0 in V such that x + 0 = x for each x ∈ V. (d). For each x ∈ V, there exist an element −x ∈ V such that x + (−x) = 0. (e). α(x + y) = αx + αy for each real number α and any x and y in V. (f). (α + β)x = αx + βx for any real number α and β and any x ∈ V. (g). (αβ)x = α(βx) for any real number α and β and any x ∈ V. (h). 1 · x = x for all x ∈ V. Example: The two-dimensional plane is the set of all vectors with two real-valued coordinates. We label this set R 2 . It has two important properties. (1). R 2 is closed under scalar multiplication; every scalar multiple of a vector in the plane is also in the plane. (2). R 2 is closed under addition; the sum of any two vectors is always a vector in the plane. 3.2 Linear Combination of Vectors and Basis Vectors Definition: A set of vectors in a vector is a basis for that vector space if any vector in the vector space can be written as a linear combination of them. Example: Any pair of two dimensional vectors that point in different directions will form 8
a basis for r2 Consider an arbitrary set of vectors in R, a, b, and c. If a and b are a basis, we can find numbers a and a% such that c=ara+a2b. Let b b2 C2 Then C1=a1a1+a2b1, C2=a1a2+ The solutions to this pair of equations are a1 C2-a2C1 b2- b1a2 This gives a unique solution unless(a1b2-b1a2)=0. If(a1b2-b1a2)=0, then a1/a2=61/b2, which means that b is just a multiple of a. This returns us to our original condition, that a and b point in different direction. The implication is that if a and b are any pair of vectors for which the denominator in o is not zero, then any other vector c can be formed as a unique linear combination of and b. The basis of a vector space is not unique, since any set of vectors that satisfy the definition will do. But for any particular basis, there is only one linear combination of them that will produce another particular vector in the vector space. 3.3 Linear Dependence As the preceding should suggest, k vectors are required to form a basis for R However it is not every set of k vectors will suffices. As we see, to form a basis we require that this k vectors to be linearly independent Definition: A sets of vectors is linearly dependent if any one of the vectors in the set can be
a basis for R 2 . Proof: Consider an arbitrary set of vectors in R 2 , a, b, and c. If a and b are a basis, we can find numbers α1 and α2 such that c = α1a + α2b. Let a = a1 a2 , b = b1 b2 , and c = c1 c2 . Then c1 = α1a1 + α2b1, c2 = α1a2 + α2b2. The solutions to this pair of equations are α1 = b2c1 − b1c2 a1b2 − b1a2 , (1) α2 = a1c2 − a2c1 a1b2 − b1a2 . (2) This gives a unique solution unless (a1b2 − b1a2) = 0. If (a1b2 − b1a2) = 0, then a1/a2 = b1/b2, which means that b is just a multiple of a. This returns us to our original condition , that a and b point in different direction. The implication is that if a and b are any pair of vectors for which the denominator in () is not zero, then any other vector c can be formed as a unique linear combination of a and b. The basis of a vector space is not unique, since any set of vectors that satisfy the definition will do. But for any particular basis, there is only one linear combination of them that will produce another particular vector in the vector space. 3.3 Linear Dependence As the preceding should suggest, k vectors are required to form a basis for R k . However it is not every set of k vectors will suffices. As we see, to form a basis we require that this k vectors to be linearly independent. Definition: A sets of vectors is linearly dependent if any one of the vectors in the set can be 9
written as a linear combination of the others Definition The vector V1, V2, . Vn in a vector space V are said to be linearly independent if and only if the solution to C1V1+C2V2+ The vector d 2 are linear independent, since if C +c2 0 C1+2c2=0 and the only solution to this system is 3.4 Subspace Definition The set of all linear combinations of a set of vectors is the vector space that is spanned by those vectors Example R=Spn(v1…,k) for a basis(v1,…,vk) We now consider what happens to the vector space that is spanned by linearly dependent vectors
written as a linear combination of the others. Definition: The vector v1, v2, ..., vn in a vector space V are said to be linearly independent if and only if the solution to c1v1 + c2v2 + ... + cnvn = 0 is c1 = c2 = ... = cn = 0. Example: The vector 1 1 and 1 2 are linear independent, since if c1 1 1 + c2 1 2 = 0 0 , then c1 + c2 = 0 c1 + 2c2 = 0 and the only solution to this system is c1 = c2 = 0. 3.4 Subspace Definition: The set of all linear combinations of a set of vectors is the vector space that is spanned by those vectors. Example: R k = Span(v1, ..., vk) for a basis (v1, ..., vk). We now consider what happens to the vector space that is spanned by linearly dependent vectors. 10