16. 322 Stochastic Estimation and Control, Fall 2004 Prof vander velde Lecture 6 Example: Sun of two independent randoin variable Z=X+Y f(=xh=P(a<Z≤b) =P(a<X+Y≤b P(a-X<Y≤b-X) =im∑P(x<X≤x+d)P(a-x<y≤b-x) =im∑f(x)dxf(y f(x)x」f(y)小y We can reach this same point by just integrating the joint probability density function for X and y over the region for which the event is true In the interior strip, the event a <=s bis true. Page 1 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde Lecture 6 Example: Sum of two independent random variables Z=X+Y b f z dz () ( = Pa < Z ≤ b) ∫ z a = P a( < X +Y ≤ b) = P a( − X < Y ≤ b − X ) = lim ∑P x( < X ≤ x + dx P a ) ( − x < Y ≤ b − x) dx→0 x b x − = lim ∑ f ( x dx f y dy ) ( ) dx→0 x ∫ y x a x − ∞ b x − () y f x dx ( ) ∫ = f y dy ∫ x −∞ a x − We can reach this same point by just integrating the joint probability density function for X and Y over the region for which the event is true. In the interior strip, the event a z < ≤ b is true. Page 1 of 10
16. 322 Stochastic Estimation and Control, Fall 2004 Prof. VanderⅤelde P(a<zsb)=aJJ、(xyx df(x)∫(y Let z=x+y, d=d 」女f(x)f(2-x)d f,(x),(z-x)dxd= This is true for all a b. Therefore f (=)=f(x)f,(z-x)dx =f((=-y)dy This result can readily be generalized to the sum of more independent random variables z=X1+X2+…+Xn f()=∫∫-Jdn2(x)(x)J(x)(-x-x Also, if w=Y-X, for X, Y independent f(w)=f(x),(w+x)dx ∫()/(-m Direct determination of the joint probability density of several functions o several ra andom variables Suppose we have the joint probability density function of several random variables x, Y, Z, and we wish the joint density of several other random variables defined as functions xyz U=u(X,Y, z V=v(X,Y,Z) W=(X,Y,2Z) Page 2 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde ∞ b x − ( < ≤ b) = dx f , P a Z (x y dy ∫ ∫ xy , ) −∞ a x − ∞ b x − dx f x () f y dy ∫ = ∫ x y ( ) −∞ ax − Let zx = + y dz , = dy ∞ b dx f x ∫ = () f z ( ) − x dz ∫ x y −∞ a b ∞ ⎡ ⎤ ⎢ f () x f z − x dx dz y ( ) = ∫ ∫ x ⎥ a ⎣−∞ ⎦ This is true for all a,b. Therefore: ∞ f () z = f x f z − x dx ∫ x ( ) y ( ) z −∞ ∞ f () ( ) y = y f z − y dy ∫ x −∞ This result can readily be generalized to the sum of more independent random variables. Z = X1 + X2 + ... + Xn ∞ ∞ ∞ f (z) = dx1 dx2... ∫ dxn−1 f (x f (x )... f (xn−1) f xn (z − x − x2 − ... − x ) x1 xn−1 z ∫ ∫ 1 1 n−1 ) x2 2 −∞ −∞ −∞ Also, if W Y = − X , for X,Y independent: ∞ f () w = f x f w + x dx ∫ x ( ) y ( ) w −∞ ∞ f () ( ) y = y f y − w dy ∫ x −∞ Direct determination of the joint probability density of several functions of several random variables Suppose we have the joint probability density function of several random variables X,Y,Z, and we wish the joint density of several other random variables defined as functions X,Y,Z. = U u X Y Z ( ,, ) V v X Y Z = ) ( ,, = i W w X Y Z ( ,, ) Page 2 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. VanderⅤelde If fw(x, y, = is finite evervwhere, the density fux.(u, v, w)can be found directly by the following steps 1. Evaluate the jacobian of the transformation from X, Y, Z to U,V,w. ou(x,y, =) au(x,y, 2) au(,y, = J(x,y,=) o(x,y2)∂(x,y2)∂(x,y2) aw(x,y,=av(x,y, =)av(x,y,z 2. For every value of u, v, w, solve the transformation equations for x, y, =. If there is more than one solution get all of then i(X, Y, Z x, (u, v, w) f(X,Y,Z)="}→{y(xw) W(X, Y, z)=w 二(l,1. 3. Then fm(u,v,w)= ∑ Jx2(x,y,=) with x;.];, =i given in terms of u,v,w This approach can be applied to the determination of the density function for m variable which are defined to be functions of n variables(n>m) by adding some simple auxiliary variables such as x, y, etc to the list of m so as to total n variables Then apply this procedure and finally integrate out the unwanted auxiliary variables Example: Product u=XY To illustrate this procedure, suppose we are given (x, y) and wish to find the probability density function for the product U=Xr First, define a second random variable; for simplicity, choose V=X. Then use the given 3 step procedure 1. Evaluate the jacobian: (x,y) 2. Solve the transformation equations x=1 Xv=u y=x=7 3. Then find f(u,v)=f(v
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde If f ( , , ,, xyz , , xyz) is finite everywhere, the density f uvw (,, uvw) can be found directly by the following steps: 1. Evaluate the Jacobian of the transformation from X,Y,Z to U,V,W. ∂uxyz ( , , ) ∂x ∂vxyz ( , , ) J xyz ( , , ) = ∂x ∂wxyz ( , , ) ∂x ∂uxyz ( , , ) ∂y ∂vxyz ( , , ) ∂y ∂wxyz ( , , ) ∂y ∂uxyz ( , , ) ∂z ∂vxyz ( , , ) ∂z ∂wxyz ( , , ) ∂z 2. For every value of u,v,w, solve the transformation equations for x,y,z. If there is more than one solution, get all of them. ( ,, ⎪ i u X Y Z ) = u ⎫ ⎧ x u v w (,, ) v X Y Z ( , , (,, ) = v ⎬ ⎪ → ⎨ y u v w ) i ( ,, ⎩ w X Y Z ) = w z u v w ) ⎪ ⎭ ⎪ (,, i 3. Then ( , , x y z ) f (,, ,, i i uvw ) = ∑ f xyz i uvw , , J x y z ( , , ) i i i k with xi,yi,zi given in terms of u,v,w. This approach can be applied to the determination of the density function for m variable which are defined to be functions of n variables (n>m) by adding some simple auxiliary variables such as x,y,etc. to the list of m so as to total n variables. Then apply this procedure and finally integrate out the unwanted auxiliary variables. Example: Product U=XY To illustrate this procedure, suppose we are given f x y ( , x y) and wish to find , the probability density function for the product U = XY . First, define a second random variable; for simplicity, choose V = X . Then use the given 3 step procedure. 1. Evaluate the Jacobian: y x Jxy ( , ) = = −x 10 2. Solve the transformation equations: ⎧ x = v xy = u⎫ ⎪ ⎬ ⇒ ⎨ u u v x = ⎭ y = = ⎪ ⎩ x v 3. Then find: 1 f ( , uv ) = f xy (v, u ) uv , , v v Page 3 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. VanderⅤelde To get the density for the product only, integrate out with respect to v. f(u)=言f,(v,=)a If X and Y are independent this becomes (0-J f() f(x)f The Uniform distribution In our problems we have been using the uniform distribution without having concisely defined it. This is a continuous distribution in which the probability density function is uniform(constant) over some finite interval F(x) 1/(b-a) Thus a random variable having a uniform distribution takes values only over some finite interval(a, b) and has uniform probability density over that interval In what situation does it arise? Examples include part tolerances, quantization error, limit cycles. Often you do not know anything more than that the unknown value lies between known bounds Page 4 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde To get the density for the product only, integrate out with respect to v. ∞ () = ∫ 1 f u f x y (v, u )dv u , u v −∞ If X and Y are independent this becomes ∞ 1 u fu () = ∫ ( ) ⎛ ⎞ f v f y ⎜ ⎟dv, or u x v ⎝ ⎠ v −∞ ∞ 1 u f () u = ∫ ( ) ⎛ ⎞ f x f y ⎜ ⎟dx u x x ⎝ ⎠ x −∞ The Uniform Distribution In our problems we have been using the uniform distribution without having concisely defined it. This is a continuous distribution in which the probability density function is uniform (constant) over some finite interval. Thus a random variable having a uniform distribution takes values only over some finite interval (a,b) and has uniform probability density over that interval. In what situation does it arise? Examples include part tolerances, quantization error, limit cycles. Often you do not know anything more than that the unknown value lies between known bounds. Page 4 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof vander velde X=x d x 2 d x b3-a3 3 6 (a2+ab+b2) x2=(a2+ab+b2)-(a2+2ab+b2) (a2-2ab+b2) 12 O 12 The binomial distribution Outl 1. Definition of the distribution 2. Determination of the binomial coefficient and binomial distribution 3. Useful relations in dealing with binomial coefficients and factorials 4. The mean, mean square and variance of the binomial distribution 1. Definition of the distribution Consider an experiment in which we identify two outcomes: one of which we call success and the other failure. The conduct of this experiment and the observation of the outcome may be called a simple trial. If the trial is then repeated under such circumstances that we consider the outcome on any trial to be independent of the outcomes on all other trials, we have a process frequently called bernoulli Trials after the man who first studied at length the results of such a process The number of successes in n bernoulli trials is a random discrete variable whose distribution is known as the binomial distribution ote that the binomial distribution need not refer p s observing the number of heads in n tosses of a coin. An experiment may have a great many simple outcomes- the outcomes may even be continuously Page 5 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde 1 2 − 2 b X = ∫ x 1 dx = 2 (b a ) ba − b a − a 1 = (a b + ) 2 b X 2 1 1 3 = ∫ x 2 1 dx = (b a 3 − ) ba − − 3 b a a 2 = 1 (a ab b 2 + + ) 3 2 X 2 2 σ = − = 1 (a ab b 2 ) 2ab b 2 X ) 2 + + − 1 (a 2 + + 3 4 = 1 (a 2 − 2ab b + 2 ) 12 = 1 (b a) 2 − 12 1 σ = (b a − ) 12 The Binomial Distribution Outline: 1. Definition of the distribution 2. Determination of the binomial coefficient and binomial distribution 3. Useful relations in dealing with binomial coefficients and factorials 4. The mean, mean square, and variance of the binomial distribution 1. Definition of the distribution Consider an experiment in which we identify two outcomes: one of which we call success and the other failure. The conduct of this experiment and the observation of the outcome may be called a simple trial. If the trial is then repeated under such circumstances that we consider the outcome on any trial to be independent of the outcomes on all other trials, we have a process frequently called Bernoulli Trials after the man who first studied at length the results of such a process. The number of successes in n Bernoulli trials is a random discrete variable whose distribution is known as the Binomial Distribution. Note that the binomial distribution need not refer only to such simple situations as observing the number of heads in n tosses of a coin. An experiment may have a great many simple outcomes – the outcomes may even be continuously Page 5 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. VanderⅤelde distributed -but for some purposes we may choose to regard some collection of he simple events as success, and employ the binomial distribution to study such ple, in the man re of transits al is drawn. The resistivity of a crystal may b regarded as a random variable taking values over a continuous range, but for the purposes of quality control in the manufacturing process, the occurance of any resistivity value within the specified tolerance may be regarded as success and all other values as failure. From this point of view, the experiment of measuring e resistivity of a crystal has only two outcomes How many units must be produced to yield P=0.95(for example)that 100 will be within specifications. Or, for a given yield, how many units must be manufactured to achieve a P=0.95 of getting 100"in-spec"items(not sequentially evaluated) 2. Determination of the binomial coefficient and binomial distribution The probability of any specified arrangement of k successes and n-k failures in n independent trials is p q where p is the probability of success on any one trial and q=l-p is the probability of failure. The probability of exactly k successes in n trials is then p q"- times the number of arrangements in which exactly k successes occur. This is due to the fact that the different arrangements are mutually exclusive so their probabilities add The probability of exactly k successes in n trials where p is the probability of success on each individual trial is P(k)=p(1-p)y- q The number s called the binomial coefficient since it is exactly the coefficient of the k" term in the binomial expansion of(a+b) (a+b) 3. Useful relations in dealing with binomial coefficients and factorials Use the convention that for integral n, =0 for kn. Also use the conⅴ entions that Page 6 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde distributed – but for some purposes we may choose to regard some collection of the simple events as success, and employ the binomial distribution to study such successes. For example, in the manufacture of transistors, the resistivity of each crystal is measured after the crystal is drawn. The resistivity of a crystal may be regarded as a random variable taking values over a continuous range, but for the purposes of quality control in the manufacturing process, the occurance of any resistivity value within the specified tolerance may be regarded as success and all other values as failure. From this point of view, the experiment of measuring the resistivity of a crystal has only two outcomes. How many units must be produced to yield P=0.95 (for example) that 100 will be within specifications. Or, for a given yield, how many units must be manufactured to achieve a P=0.95 of getting 100 “in-spec” items (not sequentially evaluated). 2. Determination of the binomial coefficient and binomial distribution The probability of any specified arrangement of k successes and n-k failures in n k nk independent trials is p q − where p is the probability of success on any one trial and q=1-p is the probability of failure. The probability of exactly k successes in n k n k trials is then p q − times the number of arrangements in which exactly k successes occur. This is due to the fact that the different arrangements are mutually exclusive so their probabilities add. The probability of exactly k successes in n trials where p is the probability of success on each individual trial is ⎛ ⎞ n n ( ) = ⎜ ⎟ p k (1− p) n−k = ⎛ ⎞ k n−k k Pk p q k ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ ⎛ ⎞ n The number ⎜ ⎟is called the binomial coefficient since it is exactly the coefficient k⎝ ⎠ of the k th term in the binomial expansion of (a b) n + . n n n ⎛ ⎞ k n−k (a b + ) = ∑⎜ ⎟a b k =0 k⎝ ⎠ 3. Useful relations in dealing with binomial coefficients and factorials ⎛ ⎞ n Use the convention that for integral n, ⎜ ⎟ = 0 for kn. Also use the k⎝ ⎠ conventions that Page 6 of 10
16. 322 Stochastic Estimation and Control, Fall 2004 Prof vander velde n Useful relations involving sums of binomial coefficients n +.+=2", n a positive integer 0)(1 +…±=0, n a positive integer n 4. The mean, mean square and variance of the binomial distribution s)=∑Ps ∑|(msyq (ps+q The moments can be derived from this using n(ps+qp ds n(n-D)(ps+q dG dgl dG n(n X2-F2 =n npq Page 7 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde ⎛ ⎞ n ⎜ ⎟ = 1 ⎝ ⎠ 0 ⎛ ⎞ n ⎜ ⎟ = 1 ⎝ ⎠ n Useful relations involving sums of binomial coefficients: ⎛ n ⎞ ⎛ ⎞ n ⎛ n +1⎞ + = ⎜ ⎟ ⎜ ⎟ ⎜ ⎟, n any number k ⎝ k −1⎠ ⎝ ⎠ ⎝ k ⎠ ⎛ ⎞ ⎛ ⎞ ⎛ ⎞ n n n n ⎜ ⎟ ⎜ ⎟ + + ⎜ ⎟ + ... = 2 , n a positive integer ⎝ ⎠ 0 1 ⎝ ⎠ ⎝ ⎠ n ⎛ ⎞ n n n ⎛ ⎞ ⎛ ⎞ ⎜ ⎟ ⎜ ⎟ + ± ⎜ ⎟ − ... = 0, n a positive integer ⎝ ⎠ 0 1 ⎝ ⎠ ⎝ ⎠ n 4. The mean, mean square, and variance of the binomial distribution Gs k k () = ∑ p s k n ⎛ ⎞ n k n−k k = ∑⎜ ⎟p q s k =0 k⎝ ⎠ n n ⎛ ⎞ k n−k k = ∑ ps q ⎜ ⎟() k =0 ⎝ ⎠ n = ( ps + q) The moments can be derived from this using dG = n ps ( + q) n−1 p ds 2 dG ( 2 = nn −1)( ps + q) n−2 p ds2 dG X = = np ds s=1 2 X 2 dG dG = + ds2 ds s=1 s=1 = nn ( −1) p 2 + np 2 2 σ = X − X 2 2 2 2 2 = n p − np 2 + np − n p = npq Page 7 of 10
16. 322 Stochastic Estimation and Control, Fall 2004 Prof. VanderⅤelde The poisson distribution Outline 1 Discussion of the significance of the distribution 2) Summarize the derivation of the distribution 3)The Poisson approximation to the Binomial distribution 1. Discussion of the significance of the distribution The Posson distribution is a discrete integer-valued distribution which is of great importance in practical problems. It is employed in the study of telephone traffi through a switchboard, of the emission of B rays from radioactive material, of the emission of electrons from heated filaments and photo-sensitive surfaces, of line surges in a power transmission system due to the throwing of switches, of demands for service in a store or stockroom, and others. Using the statistics of the Poisson distribution, one answers questions related to the number of telephone operators, or stock clerks, or turnstiles in the subway which are required to yield a certain quality of service -where the quality of service is defined in terms of the probability that a customer will have to wait, or the expectation of the time he must wait, or the like. In control system analysis, the Poisson distribution may well lead to an excellent statistical description of the inputs seen by a system, or the occurrence of disturbances, etc. This distribution applies-in rough terms-to situations in which events occur at random over an interval I and we wish to know the probability P(k, 1, n)of the occurrence of exactly k events in the subinterval Il, of I where a is the average rate of occurrence of the events- which may possibly vary over the interval Very often the interval is time: thus we ask the probability that exactly k electrons be emitted in a given length of time However, other dimensions may equally well apply -such as intervals of distance. They may even be multidimensional-units of surface area or volume An example of an interval of volume is given by the statistics of the number of bacteria in volume samples of blood. In the"aerodynamics" of very high altitudes we may very well employ the Poisson distribution to give the probability of occurrence of k atoms in a specified volume The conditions under which the Poisson distribution may be expected to apply are given by the assumptions under which the distribution is derived Page 8 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde The Poisson Distribution Outline: 1) Discussion of the significance of the distribution 2) Summarize the derivation of the distribution 3) The Poisson approximation to the Binomial distribution 1. Discussion of the significance of the distribution The Posson distribution is a discrete integer-valued distribution which is of great importance in practical problems. It is employed in the study of telephone traffic through a switchboard, of the emission of B rays from radioactive material, of the emission of electrons from heated filaments and photo-sensitive surfaces, of line surges in a power transmission system due to the throwing of switches, of demands for service in a store or stockroom, and others. Using the statistics of the Poisson distribution, one answers questions related to the number of telephone operators, or stock clerks, or turnstiles in the subway which are required to yield a certain quality of service – where the quality of service is defined in terms of the probability that a customer will have to wait, or the expectation of the time he must wait, or the like. In control system analysis, the Poisson distribution may well lead to an excellent statistical description of the inputs seen by a system, or the occurrence of disturbances, etc. This distribution applies – in rough terms – to situations in which events occur at random over an interval I and we wish to know the probability P k I (,, λ) of the occurrence of exactly k events in the subinterval I1, of I where λ is the average rate of occurrence of the events – which may possibly vary over the interval. Very often the interval is time: thus we ask the probability that exactly k electrons be emitted in a given length of time. However, other dimensions may equally well apply – such as intervals of distance. They may even be multidimensional – units of surface area or volume. An example of an interval of volume is given by the statistics of the number of bacteria in volume samples of blood. In the “aerodynamics” of very high altitudes we may very well employ the Poisson distribution to give the probability of occurrence of k atoms in a specified volume. The conditions under which the Poisson distribution may be expected to apply are given by the assumptions under which the distribution is derived. Page 8 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. VanderⅤelde Example: Telephone calls throughout a business day 9am 12pm 5pm where a is the average number of phone calls per minute P(exactly k events in an interval (ti, L))=P(k)=ue- where u=2(r)dr 2. Summarize the derivation of the distribution In an analytic derivation of the distribution, the following assumptions are made a. The occurrence of events in all non-overlapping intervals are independent. b. The probability of the occurrence of 1 event in an infinitesimal interval dI is an infinitesimal of the order of dl; the probability of occurrence of more than one event in that interval is an infinitesimal of higher order than dI These assumptions permit the derivation of the distribution; thus if the circumstances of a given situation seem to meet these conditions, the Poisson distribution may be expected to apply The derivation proceeds by determining the probabilities for 0 and 1 event which establish an apparent functional form for the distribution, then by proving the form by showing in general that the form holds for 0 and 1 and that if it holds for n, it holds also for n+I The derivation is developed in adequate detail in the text, and leads to the general form of the Poisson distribution P(h)=ue, where u=a(r)dr This is the probability of exactly k events in the interval (t, t2) in which i(t)is the of events over the interval Page 9 of 10
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde Example: Telephone calls throughout a business day where λ is the average number of phone calls per minute. () = 1 µk e−µ P(exactly k events in an interval (t , t )) = P k 12 k ! t2 where µ = ( )d ∫ λ τ τ . t 1 2. Summarize the derivation of the distribution In an analytic derivation of the distribution, the following assumptions are made: a. The occurrence of events in all non-overlapping intervals are independent. b. The probability of the occurrence of 1 event in an infinitesimal interval dI is an infinitesimal of the order of dI; the probability of occurrence of more than one event in that interval is an infinitesimal of higher order than dI. These assumptions permit the derivation of the distribution; thus if the circumstances of a given situation seem to meet these conditions, the Poisson distribution may be expected to apply. The derivation proceeds by determining the probabilities for 0 and 1 event which establish an apparent functional form for the distribution, then by proving the form by showing in general that the form holds for 0 and 1 and that if it holds for n, it holds also for n+1. The derivation is developed in adequate detail in the text, and leads to the general form of the Poisson distribution: () = 1 µ e−µ , where µ = ∫ t2 Pk λ τ τ k ( )d k ! t 1 This is the probability of exactly k events in the interval (t1,t2) in which λ( )t is the average rate of occurrence of events over the interval. Page 9 of 10
16. 322 Stochastic Estimation and Control. Fall 2004 Prof. VanderⅤelde If the average rate is the constant 2, and the interval length is written as t, the probability of k events in the interval t becomes P(k)=(r)e- G()=∑P e
16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde If the average rate is the constant λ , and the interval length is written as t, the probability of k events in the interval t becomes t e −λt ( ) = 1 (λ ) k Pk k ! k Gs k ( ) = ∑ ps k ∞ k −µ k = ∑ 1 µ es k=0 k ! −µ ∞ ( )k = e ∑ µs k=0 k ! −µ µs = ee s− = eµ( 1) Page 10 of 10