Contents Preface vij Mathematical notation xi Contents xiii 1 Introduction 1 1.1 Example:Polynomial Curve Fitting 4 1.2 Probability Theory 12 1.2.1 Probability densities 7 1.2.2 Expectations and covariances 19 1.2.3 Bayesian probabilities 21 1.2.4 The Gaussian distribution 24 1.2.5 Curve fitting re-visited 2 1.2.6 Bayesian curve fitting 30 1.3 Model Selection...,......。·.··.·········· 1.4 The Curse of Dimensionality.··.················: 3 1.5 Decision Theory························· 8 1.5.1 Minimizing the misclassification rate 39 1.5.2 Minimizing the expected loss........ 41 l.5.3 The reject option...···.·· 1.5.4 Inference and decision.............. 44。 42 1.5.5 Loss functions for regression....... 46 1.6 Information Theory....·..·····. 1.6.1 Relative entropy and mutual information Exercises..............·..·.·...·.· 85568 xiiiContents Preface vii Mathematical notation xi Contents xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting . . . . ............. 4 1.2 Probability Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.2.1 Probability densities . . . . . . . . . . . . . . . . . . . . . 17 1.2.2 Expectations and covariances . . . . . . . . . . . . . . . . 19 1.2.3 Bayesian probabilities . . . . . . . . . . . . . . . . . . . . 21 1.2.4 The Gaussian distribution . . . . . . . . . . . . . . . . . . 24 1.2.5 Curve fitting re-visited . . . . . . . . . . . . . . . . . . . . 28 1.2.6 Bayesian curve fitting . . . . . . . . . . . . . . . . . . . . 30 1.3 Model Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 1.4 The Curse of Dimensionality . . . . . . . . . . . . . . . . . . . . . 33 1.5 Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 1.5.1 Minimizing the misclassification rate . . . . . . . . . . . . 39 1.5.2 Minimizing the expected loss . . . . . . . . . . . . . . . . 41 1.5.3 The reject option . . . . . . . . . . . . . . . . . . . . . . . 42 1.5.4 Inference and decision . . . . . . . . . . . . . . . . . . . . 42 1.5.5 Loss functions for regression . . . . . . . . . . . . . . . . . 46 1.6 Information Theory . . . . . . . . . . . . . . . . . . . . . . . . . . 48 1.6.1 Relative entropy and mutual information . . . . . . . . . . 55 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 xiii