Contents ix 7 Properties of Expectation 297 7.1 Introduction 297 7.2 ahePobablstiewetogectation 311 7.2.2 The Maximum-Minimums Identity 313 Moments of the Number of Events that Occur.,·,·,315 7.5 7.5.1 Definitions 7.5.2 Computing Expectations by Conditioning Comby oditionn 34 ati Moment Generating Functions. 354 7.7.1 Joint Moment Generating Functions 36 7.8 Additional Properties of Norma Random Variables The Joint Distribution of the sampis Mean 36 782 and sample variance 36 7.9 General Definition of Expectation. 369 380 Self-Test Problems and Exercises 8.2 Chebyshev's Inequality and the Weak Law of Large Numbers 388 83 301 Other Ineg arge 403 8.6 Bounding the Error Probability When Approximating a Sum of Problems 412 ···414 415 9 Additional Topics in Probability 417 9.1 The Poisson Process. 417 Markov Chains 94 428 Summary Problems and Theoretical Exercises. 435 ell-lest Problems and Exercises.43o References·········· Contents ix 7 Properties of Expectation 297 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 7.2 Expectation of Sums of Random Variables . . . . . . . . . . . . . . . . 298 7.2.1 Obtaining Bounds from Expectations via the Probabilistic Method . . . . . . . . . . . . . . . . . . . . 311 7.2.2 The Maximum–Minimums Identity . . . . . . . . . . . . . . . . 313 7.3 Moments of the Number of Events that Occur . . . . . . . . . . . . . . 315 7.4 Covariance, Variance of Sums, and Correlations . . . . . . . . . . . . . 322 7.5 Conditional Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . 331 7.5.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 7.5.2 Computing Expectations by Conditioning . . . . . . . . . . . . 333 7.5.3 Computing Probabilities by Conditioning . . . . . . . . . . . . 344 7.5.4 Conditional Variance . . . . . . . . . . . . . . . . . . . . . . . . 347 7.6 Conditional Expectation and Prediction . . . . . . . . . . . . . . . . . 349 7.7 Moment Generating Functions . . . . . . . . . . . . . . . . . . . . . . . 354 7.7.1 Joint Moment Generating Functions . . . . . . . . . . . . . . . 363 7.8 Additional Properties of Normal Random Variables . . . . . . . . . . 365 7.8.1 The Multivariate Normal Distribution . . . . . . . . . . . . . . 365 7.8.2 The Joint Distribution of the Sample Mean and Sample Variance . . . . . . . . . . . . . . . . . . . . . . . . 367 7.9 General Definition of Expectation . . . . . . . . . . . . . . . . . . . . . 369 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 Theoretical Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 380 Self-Test Problems and Exercises . . . . . . . . . . . . . . . . . . . . . 384 8 Limit Theorems 388 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388 8.2 Chebyshev’s Inequality and the Weak Law of Large Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388 8.3 The Central Limit Theorem . . . . . . . . . . . . . . . . . . . . . . . . 391 8.4 The Strong Law of Large Numbers . . . . . . . . . . . . . . . . . . . . 400 8.5 Other Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 8.6 Bounding the Error Probability When Approximating a Sum of Independent Bernoulli Random Variables by a Poisson Random Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412 Theoretical Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414 Self-Test Problems and Exercises . . . . . . . . . . . . . . . . . . . . . 415 9 Additional Topics in Probability 417 9.1 The Poisson Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 9.2 Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 9.3 Surprise, Uncertainty, and Entropy . . . . . . . . . . . . . . . . . . . . 425 9.4 Coding Theory and Entropy . . . . . . . . . . . . . . . . . . . . . . . . 428 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434 Problems and Theoretical Exercises . . . . . . . . . . . . . . . . . . . . 435 Self-Test Problems and Exercises . . . . . . . . . . . . . . . . . . . . . 436 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436