2.2.JOINT ENTROPY AND CONDITIONAL ENTROPY 2.2 Joint entropy and conditional entropy (X,Y)can be considered to be a single vector-valued random variable H(XY)≌-logP(X,Y] =P(,y)log P(r,y) (2.4) (z.v)ESxr Definition 2.2.2.The conditional entropy of the diserete random variable X,given that the event Y=y occurs,is defined as H(XIY=)=->P(xly)log P(ly) =E[-l0g P(XIY)IY =y] (2.5) Definition 2.2.3.If (X,Y)~P(),then the conditional entropy of the discrete random variable X,given the discrete random variable Y,is defined as HxIY=∑Pw)I(XIY=) =-∑P(y)∑P(xly)logP(l) =- =E-logP(XIY (2.6) Notice that H(YX)(XY). Theorem 2.2.1.H(XY)=H(X)+H(YX)=H(Y)+(XIY) Proof. H(XY)=-∑P(红，ogP(z,) =-∑∑P,osP国+o P() =-∑P()logP(m)-∑∑P(z,y)logP(l =H(X)+H(YX) (2.7) Corollary 2.2.2.H(XY|Z)=H(XZ)+H(YXZ)2.2. JOINT ENTROPY AND CONDITIONAL ENTROPY 11 2.2 Joint entropy and conditional entropy We now extend the definition of the entropy of a single random variable to a pair of random variables. (X, Y ) can be considered to be a single vector-valued random variable. Definition 2.2.1. The joint entropy H(XY ) of a pair of discrete random variables (X, Y ) with a joint distribution P(x, y) is defined as H(XY ) , E[− log P(X, Y )] = − ∑ x∈X ∑ y∈Y P(x, y) log P(x, y) = − ∑ (x,y)∈SXY P(x, y) log P(x, y) (2.4) Definition 2.2.2. The conditional entropy of the discrete random variable X, given that the event Y = y occurs, is defined as H(X|Y = y) = − ∑ x∈X P(x|y) log P(x|y) = E [− log P(X|Y )|Y = y] (2.5) Definition 2.2.3. If (X, Y ) ∼ P(x, y), then the conditional entropy of the discrete random variable X, given the discrete random variable Y , is defined as H(X|Y ) = ∑ y∈Y P(y)H(X|Y = y) = − ∑ y∈Y P(y) ∑ x∈X P(x|y) log P(x|y) = − ∑ x∈X ∑ y∈Y P(x, y) log P(x|y) = E[− log P(X|Y )] (2.6) Notice that H(Y |X) ̸= H(X|Y ). Theorem 2.2.1. H(XY ) = H(X) + H(Y |X) = H(Y ) + H(X|Y ) Proof. H(XY ) = − ∑ x ∑ y P(x, y) log P(x, y) = − ∑ x ∑ y P(x, y) [log P(x) + log P(y|x)] = − ∑ x P(x) log P(x) − ∑ x ∑ y P(x, y) log P(y|x) = H(X) + H(Y |X) (2.7) Corollary 2.2.2. H(XY |Z) = H(X|Z) + H(Y |XZ)