15.1 Least Squares as a Maximum Likelihood Estimator 657 should provide (i)parameters,(ii)error estimates on the parameters,and(iii)a statistical measure of goodness-of-fit.When the third item suggests that the model is an unlikely match to the data,then items (i)and (ii)are probably worthless. Unfortunately,many practitioners of parameter estimation never proceed beyond item(i).They deem a fit acceptable if a graph of data and model "looks good."This approach is known as chi-by-eve.Luckily,its practitioners get what they deserve. CITED REFERENCES AND FURTHER READING: Bevington,P.R.1969,Data Reduction and Error Analysis for the Physical Sciences (New York: McGraw-Hill). Brownlee,K.A.1965,Statistical Theory and Methodology,2nd ed.(New York:Wiley). Martin,B.R.1971,Statistics for Physicists(New York:Academic Press). von Mises,R.1964,Mathematical Theory of Probability and Statistics (New York:Academic Press),Chapter X. ICAL Korn,G.A.,and Korn,T.M.1968,Mathematical Handbook for Scientists and Engineers,2nd ed. (New York:McGraw-Hill),Chapters 18-19. RECIPES 15.1 Least Squares as a Maximum Likelihood gF2” 9 Estimator Suppose that we are fitting N data points (i,yi)i=1,...,N,to a model that has M adjustable parameters aj,j=1,...,M.The model predicts a functional IENTIFIC relationship between the measured independent and dependent variables. 6 y(x)=y(x;a1...aM) (15.1.1) where the dependence on the parameters is indicated explicitly on the right-hand side. What,exactly,do we want to minimize to get fitted values for the a;'s?The first thing that comes to mind is the familiar least-squares fit, Numerica 10.621 N 43126 minimize over a1...aM (15.1.2) i=1 But where does this come from?What general principles is it based on?The answer North to these questions takes us into the subject of maximm likelihood estimators. Given a particular data set of i's and yi's,we have the intuitive feeling that some parameter sets a1...aM are very unlikely-those for which the model function y()looks nothing like the data-while others may be very likely-those that closely resemble the data.How can we quantify this intuitive feeling?How can we select fitted parameters that are"most likely"to be correct?It is not meaningful to ask the question,"What is the probability that a particular set of fitted parameters a1...aM is correct?"The reason is that there is no statistical universe of models from which the parameters are drawn.There is just one model,the correct one,and a statistical universe of data sets that are drawn from it!15.1 Least Squares as a Maximum Likelihood Estimator 657 Permission is granted for internet users to make one paper copy for their own personal use. Further reproduction, or any copyin Copyright (C) 1988-1992 by Cambridge University Press. Programs Copyright (C) 1988-1992 by Numerical Recipes Software. Sample page from NUMERICAL RECIPES IN C: THE ART OF SCIENTIFIC COMPUTING (ISBN 0-521-43108-5) g of machine￾readable files (including this one) to any server computer, is strictly prohibited. To order Numerical Recipes books or CDROMs, visit website http://www.nr.com or call 1-800-872-7423 (North America only), or send email to directcustserv@cambridge.org (outside North America). should provide (i) parameters, (ii) error estimates on the parameters, and (iii) a statistical measure of goodness-of-fit. When the third item suggests that the model is an unlikely match to the data, then items (i) and (ii) are probably worthless. Unfortunately, many practitioners of parameter estimation never proceed beyond item (i). They deem a fit acceptable if a graph of data and model “looks good.” This approach is known as chi-by-eye. Luckily, its practitioners get what they deserve. CITED REFERENCES AND FURTHER READING: Bevington, P.R. 1969, Data Reduction and Error Analysis for the Physical Sciences (New York: McGraw-Hill). Brownlee, K.A. 1965, Statistical Theory and Methodology, 2nd ed. (New York: Wiley). Martin, B.R. 1971, Statistics for Physicists (New York: Academic Press). von Mises, R. 1964, Mathematical Theory of Probability and Statistics (New York: Academic Press), Chapter X. Korn, G.A., and Korn, T.M. 1968, Mathematical Handbook for Scientists and Engineers, 2nd ed. (New York: McGraw-Hill), Chapters 18–19. 15.1 Least Squares as a Maximum Likelihood Estimator Suppose that we are fitting N data points (xi, yi) i = 1,...,N, to a model that has M adjustable parameters aj, j = 1,...,M. The model predicts a functional relationship between the measured independent and dependent variables, y(x) = y(x; a1 ...aM ) (15.1.1) where the dependence on the parameters is indicated explicitly on the right-hand side. What, exactly, do we want to minimize to get fitted values for the aj ’s? The first thing that comes to mind is the familiar least-squares fit, minimize over a1 ...aM :
N i=1 [yi − y(xi; a1 ...aM)]2 (15.1.2) But where does this come from? What general principles is it based on? The answer to these questions takes us into the subject of maximum likelihood estimators. Given a particular data set of xi’s and yi’s, we have the intuitive feeling that some parameter sets a1 ...aM are very unlikely — those for which the model function y(x) looks nothing like the data — while others may be very likely — those that closely resemble the data. How can we quantify this intuitive feeling? How can we select fitted parameters that are “most likely” to be correct? It is not meaningful to ask the question, “What is the probability that a particular set of fitted parameters a1 ...aM is correct?” The reason is that there is no statistical universe of models from which the parameters are drawn. There is just one model, the correct one, and a statistical universe of data sets that are drawn from it!