20 STATISTICAL LEARNING METHODS In which we view learning as a form of uncertain reasoning from observations Part V pointed out the prevalence of uncertainty in real environments.Agents can handle uncertainty by using the methods of probability and decision theory,but first they must learn their probabilistic theories of the world from experience.This chapter explains how they can do that.We will see how to formulate the leamning task itself as a process of probabilistic inference(Section 20.1).We will see that a Bayesian view of learning is extremely powerful,providing general solutions to the problems of noise,overfitting,and optimal prediction.It also takes into account the fact that a less-than-omniscient agent can never be certain about which theory of the world is correct,yet must still make decisions by using some theory of the world. We describe methods for learning probability models-primarily Bavesian networks- in Sections 20.2 and 20.3.Section 20.4 looks at learning methods that store and recall specific nstances.Section 20.5 covers neural network learning and Section 206 introduces kernel ofthe mat rial in this chanter is fairl mathe atical (requiring a basic un erstanding of multivariate calculus).although the general lessons can be understood without unging into the details It may benefit the Ch eader at this point to review the material in ndix A 20.1 STATISTICAL LEARNING The key concepts in this chapter,just as in Chapter 18,are data and hypotheses.Here,the data are evidence-that is,instantiations of some or all of the random variables describing the domain.The hypotheses are probabilistic theories of how the domain works,including logical theories as a special case. Let us consider a very simple example.Our favorite Surprise candy comes in two flavors:cherry (yum)and lime(ugh).The candy manufacturer has a peculiar sense of humor and wraps each piece of candy in the same opaque wrapper,regardless of flavor.The candy is sold in very large bags,of which there are known to be five kinds-again,indistinguishable from the outside: 712 20 STATISTICAL LEARNING METHODS In which we view learning as a form of uncertain reasoning from observations. Part V pointed out the prevalence of uncertainty in real environments. Agents can handle uncertainty by using the methods of probability and decision theory, but first they must learn their probabilistic theories of the world from experience. This chapter explains how they can do that. We will see how to formulate the learning task itself as a process of probabilistic inference (Section 20.1). We will see that a Bayesian view of learning is extremely powerful, providing general solutions to the problems of noise, overfitting, and optimal prediction. It also takes into account the fact that a less-than-omniscient agent can never be certain about which theory of the world is correct, yet must still make decisions by using some theory of the world. We describe methods for learning probability models—primarily Bayesian networks— in Sections 20.2 and 20.3. Section 20.4 looks at learning methods that store and recallspecific instances. Section 20.5 covers neural network learning and Section 20.6 introduces kernel machines. Some of the material in this chapter is fairly mathematical (requiring a basic un￾derstanding of multivariate calculus), although the general lessons can be understood without plunging into the details. It may benefit the reader at this point to review the material in Chapters 13 and 14 and to peek at the mathematical background in Appendix A. 20.1 STATISTICAL LEARNING The key concepts in this chapter, just as in Chapter 18, are data and hypotheses. Here, the data are evidence—that is, instantiations of some or all of the random variables describing the domain. The hypotheses are probabilistic theories of how the domain works, including logical theories as a special case. Let us consider a very simple example. Our favorite Surprise candy comes in two flavors: cherry (yum) and lime (ugh). The candy manufacturer has a peculiar sense of humor and wraps each piece of candy in the same opaque wrapper, regardless of flavor. The candy is sold in very large bags, of which there are known to be five kinds—again, indistinguishable from the outside: 712