Springer Texts in StatisticsAdvisors:George Casella Stephen Fienberg Ingram OlkinNew YorkBerlinHeidelbergBarcelonaHongKongLondonMilanParisSingaporeTokyo
SpringerTexts in StatisticsAlfred:Elements of Statistics for the Life and Social SciencesBerger:An Introduction toProbability and Stochastic ProcessesBilodeauand Brenner:Theory of Multivariate StatisticsBlom:Probability and Statistics:Theory andApplicationsBrockwell and Davis: An Introduction to Times Series and ForecastingChow and Teicher:Probability Theory:Independence,Interchangeability,Martingales,ThirdEditionChristensen:Plane Answers to Complex Questions: The Theory of LinearModels,SecondEditionChristensen: Linear Models for Multivariate, Time Series, and Spatial DataChristensen:Log-Linear Models and Logistic Regression, Second EditionCreighton: A First Course in Probability Models and Statistical InferenceDeanandVoss:Design andAnalysis ofExperimentsdu Toit, Steyn,and Stumpf: Graphical ExploratoryData AnalysisDurrett:Essentials of StochasticProcessesEdwards:IntroductiontoGraphical ModellingFinkelstein and Levin: Statistics for LawyersFlury:AFirst Course inMultivariate StatisticsJobson:Applied Multivariate Data Analysis,Volume I:Regression andExperimentalDesignJobson: Applied Multivariate Data Analysis,Volume II: Categorical andMultivariate MethodsKalbfleisch:Probability and Statistical Inference,Volume I:ProbabilitySecondEditionKalbfleisch:Probability and Statistical Inference,Volume I:StatisticalInference,SecondEditionKarr:ProbabilityKeyfitz:Applied Mathematical Demography,Second EditionKiefer:Introduction to Statistical InferenceKokoska and Nevison: Statistical Tables and FormulaeKulkarni:Modeling,Analysis,Design,andControlofStochasticSystemsLehmann:Elements of Large-Sample TheoryLehmann:Testing Statistical Hypotheses,Second EditionLehmann and Casella: Theory of Point Estimation, Second EditionLindman:Analysis of Variance in Experimental DesignLindsey:Applying Generalized LinearModelsMadansky:Prescriptions forWorking StatisticiansMcPherson: Statistics in Scientific Investigation: Its Basis, Application, andInterpretationMueller:Basic Principles of Structural Equation ModelingNguyen and Rogers:Fundamentals of Mathematical Statistics: VolumeI:ProbabilityforStatisticsNguyen and Rogers: Fundamentals of Mathematical Statistics: Volume II:Statistical Inference
Martin Bilodeau David BrennerTheory ofMultivariate StatisticsSpringer
David BrennerMartin BilodeauDepartment of StatisticsUniversite de MontrealUniversity of TorontoFaculte des Arts et des ScicncesOntarioM5S3G4Departement de Mathematiques et de StatistiqueCanadaC.P. 6128, Succursale Centre-villebrenner@utstat.toronto.eduMontreal (Quebec) H3C 3J7Canadabilodeau@dms.umontreai.caEditorialBoardIngramOlkinGeorge CasellaStephen FienbergDepartment of StatisticsBiometrics UnitDepartment of StatisticsStanford UniversityCormell UniversityCarnegie Mellon UniversityStanford, CA 94305Pittsburgh,PA 15213-3890Ithaca, NY 14853-7801USAUSAUSAWith9 illustrationsLibrary of Congress Cataloging-in-Publication DataBilodeau,Martin,1961-Theory of multivariate statistics / Martin Bilodeau, DavidBrenner.cm, - (Springer texts in statistics)p.Includes bibliographical references and indexes.ISBN 0-387-98739-8 (hardcover : alk.paper)II.Title1.Multivariate analysis,I.Brenner,David.III. Series.1999QA278.B5599-26378519.5*35dc211999 Springer-Verlag New York, Inc.All rights reserved. This work may not be translated or copied in whole or in part without thewritten permission of the publisher(Springer-Verlag New York,Inc., 175 Fifth Avenue,New York,NY 10010, USA),except for brief excerpts in connection with reviews or scholarly analysis.Usein connection with any form of information storage and retrieval, electronic adaptation, computersoftware, or by similar or dissimilar methodology now known or hereafter developed is forbidden.The use of general descriptive names, trade names, trademarks, etc., in this publication, even if theformer are not especially identified, is not to be taken as a sign that such names, as understood bythe Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.ISBN0-387-98739-8Springer-VerlagNewYorkBerlinHeidelbergSPIN10709151
A la memoire de mon pere, Arthur, a ma mere, Annette, et a Kahina.M.BilodeauToRebeccaand Deena.D.Brenner
A la m´emoire de mon p`ere, Arthur, `a ma m`ere, Annette, et `a Kahina. M. Bilodeau To Rebecca and Deena. D. Brenner
PrefaceOur object in writing this book is to present the main results of the mod-ern theory of multivariate statistics to an audience of advanced studentswho would appreciate a concise and mathematically rigorous treatment ofthat material. It is intended for use as a textbook by students taking afirst graduate course in the subject, as well as for the general reference ofinterested research workers who will find, in a readable form, developmentsfrom recently published work on certain broad topics not otherwise easilyaccessible, as,for instance, robust inference (using adjusted likelihood ratiotests)and the use of the bootstrap in a multivariatesetting.Thereferencescontains over 150 entries post-1982.The main development of the text issupplemented by over 135 problems, most of which are original with theauthors.A minimum background expected of the reader would include at leasttwo courses in mathematical statistics, and certainly some exposure to thecalculusof several variables together with thedescriptivegeometry of linearalgebra. Our book is, nevertheless, in most respects entirely self-contained,although a definite need forgenuine fluency in general mathematics shouldnot be underestimated. The pace is brisk and demanding, requiring an in-tense level of active participation in every discussion. The emphasis is onrigorous proof and derivation.The interested reader would profit greatly, ofcourse,from previous exposure to a wide variety of statistically motivatingmaterial as well, and a solid background in statistics at the undergraduatelevel would obviously contribute enormously to a general sense of famil-iarity and provide some extra degree of comfort in dealing with the kindsof challenges and difficulties to be faced in the relatively advanced work
Preface Our object in writing this book is to present the main results of the modern theory of multivariate statistics to an audience of advanced students who would appreciate a concise and mathematically rigorous treatment of that material. It is intended for use as a textbook by students taking a first graduate course in the subject, as well as for the general reference of interested research workers who will find, in a readable form, developments from recently published work on certain broad topics not otherwise easily accessible, as, for instance, robust inference (using adjusted likelihood ratio tests) and the use of the bootstrap in a multivariate setting. The references contains over 150 entries post-1982. The main development of the text is supplemented by over 135 problems, most of which are original with the authors. A minimum background expected of the reader would include at least two courses in mathematical statistics, and certainly some exposure to the calculus of several variables together with the descriptive geometry of linear algebra. Our book is, nevertheless, in most respects entirely self-contained, although a definite need for genuine fluency in general mathematics should not be underestimated. The pace is brisk and demanding, requiring an intense level of active participation in every discussion. The emphasis is on rigorous proof and derivation. The interested reader would profit greatly, of course, from previous exposure to a wide variety of statistically motivating material as well, and a solid background in statistics at the undergraduate level would obviously contribute enormously to a general sense of familiarity and provide some extra degree of comfort in dealing with the kinds of challenges and difficulties to be faced in the relatively advanced work
viliPrefaceof the sort with which our book deals.In this connection, a specific intro-duction offering comprehensive overviews of the fundamental multivariatestructures and techniques would be well advised. The textbook A FirstCourse inMultivariate Statistics byFlury(1997),published bySpringer-Verlag, provides such background insight and general description withoutgetting much involved in the “nasty" details of analysis and construction.This would constitute an excellent supplementary source. Our book is inmost ways thoroughly orthodox, but in several ways novel and unique.In Chapter 1we offer abrief account of the prerequisite linear algebraas it will be applied in the subsequent development. Some of the treatmentis peculiar to the usages of multivariate statistics and to this extent mayseemunfamiliar.Chapter 2 presents in review, the requisite concepts, structures, anddevices from probability theory that will be used in the sequel. The ap-proach taken in the following chapters rests heavily on the assumption thatthis basic material is well understood, particularly that which deals withequality-in-distribution and the Cramer-Wold theorem, to be used withunprecedented vigor in the derivation of the main distributional results inChapters 4 through 8.In this way,our approach to multivariate theoryis much more structural and directly algebraic than is perhaps traditional,tied in this fashion much more immediately to the way in which the variousdistributions arise either in nature or may be generated in simulation. Wehope that readers will find the approach refreshing, and perhaps even a bitliberating,particularly those saturated in a lifetime of matrix derivativesand jacobians.As a textbook, the first eight chapters should provide a more than ade-quate amount of material for coverage in one semester (13 weeks). Theseeight chapters, proceeding from a thorough discussion of the normal dis-tribution and multivariate sampling in general, deal in random matrices,Wishart's distribution, and Hotelling's T?, to culminate in the standardtheory of estimation and the testing of means and variances.The remaining six chapters treat of more specialized topics than it mightperhaps be wise to attempt in a simple introduction, but would easily beaccessible to those already versed in the basics.With such an audienceinmind, we have included detailed chapters on multivariate regression, prin-cipal components, and canonical correlations, each of which should be ofinterest to anyone pursuing further study.The last three chapters, dealing,in turn,with asymptoticexpansion,robustness,and thebootstrap,discussconcepts that are of current interest for active research and take the reader(gently) into territory not altogether perfectly charted. This should servetodrawone (gracefully)intotheliterature.Theauthors would like to express their most heartfelt thanks to everyonewho has helped with feedback, criticism, comment, and discussion in thepreparation of this manuscript. The first author would like especially toconvey his deepest respect and gratitude to his teachers, Muni Srivastava
viii Preface of the sort with which our book deals. In this connection, a specific introduction offering comprehensive overviews of the fundamental multivariate structures and techniques would be well advised. The textbook A First Course in Multivariate Statistics by Flury (1997), published by SpringerVerlag, provides such background insight and general description without getting much involved in the “nasty” details of analysis and construction. This would constitute an excellent supplementary source. Our book is in most ways thoroughly orthodox, but in several ways novel and unique. In Chapter 1 we offer a brief account of the prerequisite linear algebra as it will be applied in the subsequent development. Some of the treatment is peculiar to the usages of multivariate statistics and to this extent may seem unfamiliar. Chapter 2 presents in review, the requisite concepts, structures, and devices from probability theory that will be used in the sequel. The approach taken in the following chapters rests heavily on the assumption that this basic material is well understood, particularly that which deals with equality-in-distribution and the Cram´er-Wold theorem, to be used with unprecedented vigor in the derivation of the main distributional results in Chapters 4 through 8. In this way, our approach to multivariate theory is much more structural and directly algebraic than is perhaps traditional, tied in this fashion much more immediately to the way in which the various distributions arise either in nature or may be generated in simulation. We hope that readers will find the approach refreshing, and perhaps even a bit liberating, particularly those saturated in a lifetime of matrix derivatives and jacobians. As a textbook, the first eight chapters should provide a more than adequate amount of material for coverage in one semester (13 weeks). These eight chapters, proceeding from a thorough discussion of the normal distribution and multivariate sampling in general, deal in random matrices, Wishart’s distribution, and Hotelling’s T2, to culminate in the standard theory of estimation and the testing of means and variances. The remaining six chapters treat of more specialized topics than it might perhaps be wise to attempt in a simple introduction, but would easily be accessible to those already versed in the basics. With such an audience in mind, we have included detailed chapters on multivariate regression, principal components, and canonical correlations, each of which should be of interest to anyone pursuing further study. The last three chapters, dealing, in turn, with asymptotic expansion, robustness, and the bootstrap, discuss concepts that are of current interest for active research and take the reader (gently) into territory not altogether perfectly charted. This should serve to draw one (gracefully) into the literature. The authors would like to express their most heartfelt thanks to everyone who has helped with feedback, criticism, comment, and discussion in the preparation of this manuscript. The first author would like especially to convey his deepest respect and gratitude to his teachers, Muni Srivastava
ixPrefaceof the University of Toronto and Takeaki Kariya of Hitotsubashi University,who gave their unstinting support and encouragement during and after hisgraduate studies. The second author is very grateful for many discussionswith Philip McDunnough of the University of Toronto. We are indebtedto Nariaki Sugiura for his kind help concerning the application of Sug-iura's Lemma and to Rudy Beran for insightful comments, which helpedto improve the presentation. Eric Marchand pointed out some errors inthe literature about the asymptotic moments in Section 8.4.1. We wouldlike to thank the graduate students at McGill University and Universitede Montreal, Gulhan Alpargu, Diego Clonda, Isabelle Marchand, PhilippeSt-Jean, Gueye N'deye Rokhaya, Thomas Tolnai and Hassan Younes, whohelped improve the presentation by their careful reading and problem solv-ing. Special thanks go to Pierre Duchesne who, as part of his MasterMemoir, wrote and tested the S-Plus function for the calculation of therobust S estimate in Appendix C.M.BilodeaulD.Brenner
Preface ix of the University of Toronto and Takeaki Kariya of Hitotsubashi University, who gave their unstinting support and encouragement during and after his graduate studies. The second author is very grateful for many discussions with Philip McDunnough of the University of Toronto. We are indebted to Nariaki Sugiura for his kind help concerning the application of Sugiura’s Lemma and to Rudy Beran for insightful comments, which helped to improve the presentation. Eric Marchand pointed out some errors in the literature about the asymptotic moments in Section 8.4.1. We would like to thank the graduate students at McGill University and Universit´e de Montr´eal, Gulhan Alpargu, Diego Clonda, Isabelle Marchand, Philippe St-Jean, Gueye N’deye Rokhaya, Thomas Tolnai and Hassan Younes, who helped improve the presentation by their careful reading and problem solving. Special thanks go to Pierre Duchesne who, as part of his Master Memoir, wrote and tested the S-Plus function for the calculation of the robust S estimate in Appendix C. M. Bilodeau D. Brenner
ContentsPrefaceviiList of TablesxVxviiList of Figures11 Linear algebra11.1Introduction11.2Vectors and matrices31.3Image spaceand kernel41.4Nonsingular matrices and determinants51.5Eigenvaluesand eigenvectors91.6Orthogonalprojections1.710Matrix decompositions1.811Problems142Randomvectors2.114Introduction2.214Distribution functions162.3Equals-in-distribution2.416Discrete distributions2.517Expected values2.618Mean and variance2.721Characteristic functions2.822Absolutely continuous distributions2.924Uniform distributions
Contents Preface vii List of Tables xv List of Figures xvii 1 Linear algebra 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Vectors and matrices . . . . . . . . . . . . . . . . . . . . 1 1.3 Image space and kernel . . . . . . . . . . . . . . . . . . . 3 1.4 Nonsingular matrices and determinants . . . . . . . . . . 4 1.5 Eigenvalues and eigenvectors . . . . . . . . . . . . . . . . 5 1.6 Orthogonal projections . 9 1.7 Matrix decompositions . . . . . . . . . . . . . . . . . . . 10 1.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2 Random vectors 14 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2 Distribution functions . . . . . . . . . . . . . . . . . . . . 14 2.3 Equals-in-distribution . . . . . . . . . . . . . . . . . . . . 16 2.4 Discrete distributions . . . . . . . . . . . . . . . . . . . . 16 2.5 Expected values . . . . . . . . . . . . . . . . . . . . . . . 17 2.6 Mean and variance . . . . . . . . . . . . . . . . . . . . . 18 2.7 Characteristic functions . . . . . . . . . . . . . . . . . . . 21 2.8 Absolutely continuous distributions . . . . . . . . . . . . 22 2.9 Uniform distributions . . . . . . . . . . . . . . . . . . . . 24
xiiContents252.10Joints and marginals272.11Independence282.12Change of variables2.1330Jacobians.2.1433Problems363Gamma, Dirichlet, and F distributions363.1Introduction3.236Gamma distributions3.338Dirichlet distributions3.442F distributions423.5Problems434Invariance434.1Introduction4.243Reflection symmetry444.3Univariate normal and related distributions474.4Permutation invariance4.548Orthogonal invariance4.652Problems55Multivariate normal5555.1Introduction5.255Definitionand elementaryproperties5.358Nonsingular normal5.462Singular normal625.5Conditional normal5.664Elementary applications645.6.1Sampling the univariatenormal655.6.2Linearestimation675.6.3Simplecorrelation5.769Problems73Multivariatesampling6736.1Introduction736.2Random matricesand multivariate sample786.3Asymptotic distributions6.481Problems857Wishart distributions857.1Introduction857.2Joint distribution of xand S7.387Properties ofWishart distributions7.494Box-Cox transformations7.596Problems
xii Contents 2.10 Joints and marginals . . . . . . . . . . . . . . . . . . . . 25 2.11 Independence . . . . . . . . . . . . . . . . . . . . . . . . 27 2.12 Change of variables . . . . . . . . . . . . . . . . . . . . . 28 2.13 Jacobians . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.14 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3 Gamma, Dirichlet, and F distributions 36 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.2 Gamma distributions . . . . . . . . . . . . . . . . . . . . 36 3.3 Dirichlet distributions . . . . . . . . . . . . . . . . . . . . 38 3.4 F distributions . . . . . . . . . . . . . . . . . . . . . . . 42 3.5 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 4 Invariance 43 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.2 Reflection symmetry . . . . . . . . . . . . . . . . . . . . 43 4.3 Univariate normal and related distributions . . . . . . . 44 4.4 Permutation invariance . . . . . . . . . . . . . . . . . . . 47 4.5 Orthogonal invariance . . . . . . . . . . . . . . . . . . . . 48 4.6 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5 Multivariate normal 55 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.2 Definition and elementary properties . . . . . . . . . . . 55 5.3 Nonsingular normal . . . . . . . . . . . . . . . . . . . . . 58 5.4 Singular normal . . . . . . . . . . . . . . . . . . . . . . . 62 5.5 Conditional normal . . . . . . . . . . . . . . . . . . . . . 62 5.6 Elementary applications . . . . . . . . . . . . . . . . . . 64 5.6.1 Sampling the univariate normal . . . . . . . . . . 64 5.6.2 Linear estimation . . . . . . . . . . . . . . . . . . 65 5.6.3 Simple correlation . . . . . . . . . . . . . . . . . . 67 5.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 6 Multivariate sampling 73 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.2 Random matrices and multivariate sample . . . . . . . . 73 6.3 Asymptotic distributions . . . . . . . . . . . . . . . . . . 78 6.4 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 7 Wishart distributions 85 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 85 7.2 Joint distribution of x¯ and S . . . . . . . . . . . . . . . . 85 7.3 Properties of Wishart distributions . . . . . . . . . . . . 87 7.4 Box-Cox transformations . . . . . . . . . . . . . . . . . . 94 7.5 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 96