18 CHAPTER 2.ENTROPY AND MUTUAL INFORMATION H(XY) HX) HOY) Figure 2.3:mutual information and conditional entropy 2.4.3 Conditional mutual information Definition 2.4.3.The conditional mutual information between the random variables X and Y,given the r.v.Z,is defined by I(X:YZ)=H(Z)-H(YZ) =∑∑∑P,u)o()P同 P(,) =s (2.18) It follows from the definition that I(X;Y1Z)=IY;XZ)≥0 with ec statistically ind P(,)=P()P()for each element in the joint sample space for which P()>0. We can visualize the situation in which I(X:YZ)=0 as a pair of channels in cascade as shown in Fig.2.4.We assume that the output of the 2nd channel depends statistically only on the input to the 2nd channel,i.e.p(ylz)=p(ylz,r),all r,y,z with p2,x)>0. I(x:YIZ)= X Channel 1 Z Y y=f(Z) Figure 2.4:Channels in cascade .An important property: I(X:YZ)=I(YZ:X) =I(x:Y)+I(x:ZY) =I(X:Z)+I(X;YZ) (2.19) 18 CHAPTER 2. ENTROPY AND MUTUAL INFORMATION H(X) H(Y) H(XY) H(X|Y) I(X;Y) H(Y|X) Figure 2.3: mutual information and conditional entropy 2.4.3 Conditional mutual information Definition 2.4.3. The conditional mutual information between the random variables X and Y , given the r.v. Z, is defined by I(X; Y |Z) = H(X|Z) − H(X|Y Z) = ∑ x ∑ y ∑ z P(x, y, z) log P(x, y|z) P(x|z)P(y|z) = Ep(x,y,z) [ log P(x|yz) P(x|z) ] (2.18) It follows from the definition that I(X; Y |Z) = I(Y ; X|Z) ≥ 0 with equality iff conditional on each Z, X and Y are statistically independent; i.e., P(x, y|z) = P(x|z)P(y|z) for each element in the joint sample space for which P(z) > 0. We can visualize the situation in which I(X; Y |Z) = 0 as a pair of channels in cascade as shown in Fig. 2.4. We assume that the output of the 2nd channel depends statistically only on the input to the 2nd channel,i.e. p(y|z) = p(y|z, x), all x, y, z with p(z, x) > 0. Mutiplying both sides by P(x|z), we obtain P(x, y|z) = P(x|z)P(y|z), so that I(X; Y |Z) = 0. Channel y f = (z) X Z Y Figure 2.4: Channels in cascade • An important property: I(X; Y Z) = I(Y Z; X) = I(X; Y ) + I(X;Z|Y ) = I(X;Z) + I(X; Y |Z) (2.19)