Sheet (1)

Question # 1
Given a DMS, where a source transmits binary input, p(0) = 4/7 , p(1) = 3/7 find :
1- Source entropy
2- Destination entropy 3/5
0 0
3- Mutual information
4- Conditional entropy
5- System entropy
1 2/5 1

Solution
1- H(X) = - Σ PK log PK = - 4/7 log 4/7 – 3/7 log 3/7 = 0.985 bits/symbol
2- o/p probability P(YJ) = Σ P(Yj/Xi) P(Xi)
P(Y0) = Σ P(Y0/Xi) P(Xi) = (3/5 * 4/7) + (1/3 * 3/7) = 17/35
P(Y1) = Σ P(Y1/Xi) P(Xi) = (2/5 * 4/7) + (2/3 * 3/7) = 18/35

H(X) = - Σ P(YJ) log P(YJ) = - 17/35 log 17/35 – 18/35 log 18/35 = 0.999 bits/symbol
P (Y 0/ X 0) 3/5
3- I(X0,Y0) = log = log = 0.3 bits/symbol
P(Y 0) 17/35
P (Y 0/ X 1) 1/3
I(X1,Y0) = log = log = - 0.54 bits/symbol
P(Y 0) 17/35
P (Y 1 / X 0) 2/5
I(X0,Y1) = log = log = - 0.36 bits/symbol
P(Y 1) 18/35
P (Y 1 / X 1) 2/3
I(X1,Y1) = log = log = 0.37 bits/symbol
P(Y 1) 18/35
P (Y j/ X i) P( Xi)
4- H(X/Y) = - Σ Σ P(Yj/Xi) P(Xi) log P(Xi/Yj) = - Σ Σ P(Yj/Xi) P(Xi) log
P(Y j)
P (Y 0/ X 0)P( X 0) P (Y 1 / X 0) P(X 0)
H(X/Y) = - P(Y0/X0)P(X0) log - P(Y1/X0)P(X0) log -
P(Y 0) P(Y 1)
P (Y 0/ X 1) P(X 1) P (Y 1 / X 1)P( X 1)
P(Y0/X1)P(X1) log - P(Y1/X1)P(X1) log = - 3/5 *
P(Y 0) P(Y 1)
3 2 1 2
∗4 ∗4 ∗3 ∗3
5 5 3 3
4/7 log – 2/5 * 4/7 log – 1/3 * 3/7 log – 2/3 * 3/7 log = 0.93
7 7 7 7
17/35 18/35 17/35 18/35
bits/symbol
5- H(X,Y) = - Σ Σ P(Yj,Xi) log P(Xi,Yj) = - Σ Σ P(Yj/Xi)P(Xi) log P(Xi/Yj) P(Xi)
 H (X,Y) = - P(Y0/X0)P(X0) log P(Y0/X0) P(X0) - P(Y0/X1)P(X1) log P(Y0/X1) P(X1) -
P(Y1/X0)P(X0) log P(Y1/X0) P(X0) - P(Y1/X1)P(X1) log P(Y1/X1) P(X1)
= - (3/5 * 4/7 log 3/5 * 4/7) – ( 1/3 * 3/7 log 1/3 * 3/7) – (2/5 * 4/7 log 2/5 * 4/7) – (2/3
* 3/7 log 2/3 * 3/7) = 1.93 bits/symbol

Question # 2
Given the following forward channel matrix P(a1) = 0.6, P(a2) = 0.2
0.6 0.25 0.15
0.15 0.7 0.15
0.25 0.05 0.7
1- Draw the forward channel matrix
2- Find the backward channel matrix
3- Find the system entropy
4- Find channel mutual information
5- If b2 is received, how much uncertainty of a1 was sent
6- Find the entropy of second order zero memory source? Show your work
Solution
1- Forward channel matrix A1 0.150.6 0.25 b1

0.15
A2 0.7 b2
0.15
0.25
0.05
A3 0.7 b3

2- Backward channel matrix

o/p probability  P(b) = Σ P(