&

email: chrnikol@otenet.gr
http://users.otenet.gr/~chrnikol

2004


, 1948,
C.E. Shannon
A Mathematical Theory of Telecommunications.


20 . ,

(, ).
Shannon
.


.
,
.
,

,

.
.
,
, ,


.
, ,

.
!
.

 ,
, ,

. .
,
& .
, ,
. .

2004

1:

1.1

1.2

11

12

2:

2.1

13

2.2

14

2.3

16

2.4

19

2.5

19

2.6

22

2.7

24

26

28

3:

3.1

29

3.2

29

3.3

35

3.4

36

3.5

38

42

46

4:

4.1

49

4.2

Shannon-Fano

53

4.3

57

4.4

59

4.5

Shannon

61

4.6

Huffman

63

67

70

5:

5.1

Shannon

76

5.2

78

5.3

Bit

80

5.4

81

5.5

84

5.6

88

5.7

Hamming, H(3)

89

92

95

99

B :

101

103

ii

107

109

114

125

143

153

iii

iv

1.

1.1

.

- .
, ,
. 10 (
!). :

200 365 ,
10 !
,
,
10 .
.

.

,
.
:

200
.

= 0

= 1

00

01

10

10

110

11

111

1 -
-
,
. , , .
, , ,
.
,
.

1. ;
;

. .

. ,
, .

!
5
. ,
,
.

1

.

.
,
,
.

.

2.

; ;

, .

.
, ,

.
,

, ,

,
.

. , .
.
1 2, .
, ,
(
).
(

).
, ,
(
). ,
,

(
).
1 2 Shannon, o ,
,
(
).

1.2

,
.

, , !

,
.
.
, ,
.
:

.
.

.
. .
{0,1}
, Morse
{ , , }.
.
.

.

.
,

, ).
. ,
, ,
, .
1 : Morse

()
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z

()

, ( ) ( ),

.
.

, ,
.
,
{ , , }.
, SOS,
( / )
/

,
SOS
.

. Q
Y .

( )
.

.
2
,
.

1 =

0 =

, .
5

0 00000

1 11111
, ..
:

11111

11101

11111

( )

, /
3 : ASCII

/ 1968 ASCII (American
Standard Code for Information Interchange)
7 bits. 128 .
, :

A
B
C
D
E
F

1000001
1000010
1000011
1000100
1000101
1000110

a
b
c
d
e
f

1100001
1100010
1100011
1100100
1100101
1100110

!
$
@
1
2
3

0100001
0100100
1000000
0110001
0110010
0110011

bit.
0,
128 1 (
extended ASCII). ,
. ASCII
.
128
128 .

ASCII
1 (

bits
).
1-1 .

.

1948,
C.E. Shannon,
A Mathematical Theory of Telecommunications.


20 . ,

(, ).
Shannon
.


.
, ,

.

00100001

01000001

01100001

11000001

11100001

"

00100010

01000010

01100010

11000010

11100010

00100011

01000011

01100011

11000011

11100011

00100100

01000100

01100100

11000100

11100100

00100101

01000101

01100101

11000101

11100101

&

00100110

01000110

01100110

11000110

11100110

'

00100111

01000111

01100111

11000111

11100111

00101000

01001000

01101000

11001000

11101000

00101001

01001001

01101001

11001001

11101001

00101001

01001010

01101010

11001010

11101010

00101011

01001011

01101011

11001011

11101011

00101100

01001100

01101100

11001100

11101100

00101101

01001101

01101101

11001101

11101101

00101110

01001110

01101110

11001110

11101110

00101111

01001111

01101111

11001111

11101111

00110000

01010000

01110000

11010000

11110000

00110001

01010001

01110001

11010001

11110001

00110010

01010010

01110010

11110010

00110011

01010011

01110011

11010011

11110011

00110100

01010100

01110100

11010100

11110100

00110101

01010101

01110101

11010101

11110101

00110110

01010110

01110110

11010110

11110110

00110111

01010111

01110111

11010111

11110111

00111000

01011000

01111000

11011000

11111000

00111001

01011001

01111001

11011001

11111001

00111010

01011010

01111010

00111011

01011011

01111011

<

00111100

01011100

01111100

00111101

01011101

01111101

>

00111110

01011110

01111110

00111111

01011111

DEL

01111111

01000000

01100000

1. Extended ASCII (8-bits) .

10

1.1

200 , 100 , 50
15 .

;
; ;
1.2 7
.
;
1.3 (. )
(. ) , -20
+43. ;
, ()
;
;
1.4

Morse. ;
1.5
Extended ASCII 1;
24 bit .
;
ASCII ;

11

1.1

1: 3650 ,

10

2: 2300 ,

6,3

1.2

3 .

1.3

={-20,-19,-18, ... , 42, 43} (64 ), = 6

1.4

1.5

12

2.

2.1

.

( !)

.
, 1,
100%. (
, !)
6,
. 1,2,3,4 5.
1 .

,
. ,

1
13.983.816
(
!)
11 2001,
,
,
!. , (
!), ( 0).

13

.
,
.

=0

=1

.
.

2.2

()

:
:

p(A) p :

() :

( )

1: ()

() p

I ( AB) = I ( A) + I ( B)

.

14

: ( )

I ( A) = log 2 p log 2

1
p

bit.
2:
2. ,
.

log 2

bit

log e

nat

log10

hartley

1 .
() p
p q ,

I ( AB) = log 2 ( pq ) = log 2 p log1 q = I ( A) + I ( B)

1. ,
0 1 , p (0) = 1

p (1) = 1 .

1
I (1) = log 2 = log 2 2 = 1 bit
2

1
I (0) = log 2 = log 2 2 = 1 bit
2
{0,1}.
1000 bits
1000 0 1.
:

log10 2 = 0.301 Hartley

15

log e 2 = ln 2 = 0.693 nat

2. 256
,

1
.
256

1
I = log 2
= log 2 256 = 8 bits
256
, , 256
8 .

2.3

:
()

a1 , a2 , a3 ,K, an

p`1 , p2 , p3 ,K, pn
.

( A, ) ,

A = {a1 , a2 ,K, an }
= [ p1 p 2 K p n ] 1.

,

H ( A) = p1I (a1 ) + p2 I (a2 ) + L + pn I (an )

bits/

H ( A) = p1 log 2

1
1
1
+ p 2 log 2
+ L + p n log 2
p1
p2
pn

( H ( A) =

pi log 2

1
)
pi

( A) = [ p (a1 ) p (a 2 ) K

16

p (a n )]

bits/

: H ( A) 0

H (a1 , a2 ,K, an ) = H (an ,K, a2 , a1 ) = H (a2 , a1 , a3 , a4 ,K, an ) = L

( )

p1 = p2 = L = pn =

1
n

( (p1,p2,pn) )
,

H ( A) = p1 log 2
=

1
1
1
+ p 2 log 2
+ L + p n log 2
p1
p2
pn

1
1
1
n
log 2 n + log 2 n + L + log 2 n = log 2 n
n
n
n
n

H ( A) = log 2 n

3.

bits/

( A, ) A = {0,1} .

I)

, = [

II)

=[

III)

1
999
=[
]
1000 1000

IV)

= [0 1] ( 0 0 ,

1 3
]
4 4

( = [

1
2

1
]
2

3 1
])
4 4

1 1, )
V)

= [1 0] ( 0, 1 )

VI)

= [ p

1 p]

:
I)

H ( A) = H (0,1) =

1
1
log 2 2 + log 2 2 = log 2 2 = 1 bit/
2
2

, 3 .

17


II)

H ( A) = H (0,1) =
=

III)

3
4 1
3
1
log 2 4 + log 2 = 2 + (log 2 4 log 2 3)
4
3 4
4
4

1 3 3
+ log 2 3 0,81
2 2 4

H ( A) = H (0,1) =

bits/

1000
1
999
1

10 + 0
log 2
log 2 1000 +
999 1000
1000
1000

= 0,01 bits/ ( log 2

1000
log 2 1 = 0 )
999

IV)

H ( A) = H (0,1) = 0 I (0) + 1 I (1) = 1 log 2 1 = 0 bits/1

V)

VI)

H ( A) = H (0,1) = p log 2

1
1
+ (1 p) log 2
p
1 p

h( p) = p log 2

1
1
+ (1 p) log 2
p
1 p

(0,1) Shannon.
p =

1
, 0 1.
2

h(p)

1
h = 1 ,
2

1
3
h = h = 0.81 ,
4
4

0 I (0) = 0 log 2

1
0

1
999
h
= h
= 0.01 ,
1000
1000

lim+ p log2

p0

18

1
= lim
p p0+

log2
1
p

1
p

h(0) = h(1) = 0

= lim

x+

log2 x
=0
x


2.4

3 .

.
IV) 1.

0.
V).
0 III)
, 1.
I)
. 1,
0. ,
. .
,

.

, .

2.5

( A, ( A)) ( B, ( B)) ,

A = {a1 , a2 ,K, an } ,

( A) = [ p(a1 ) p (a 2 ) K p(a n )]

B = {b1 , b2 ,K, bm } ,

( B) = [ p (b1 ) p(b2 ) K p (bm )]

19

( AB, ( AB))

AB = {ab / a A b B}
b
. A B

(a, b) , ab (a, b) .

( AB ) p ( ab) .

p(ab) =

p(b / a) = b

p (ab) = p (a) p(b / a)

b

p(ab) = p(a) p(b)

H ( AB ) = p (ai b j ) log 2
i, j

1
p(ai b j )

H ( AB) H ( A) + H ( B )

H ( AB) = H ( A) + H ( B)

, .

A1 , A2 ,K, Ak ,
H ( A1 A2 K Ak ) H ( A1 ) + H ( A2 ) + L + H ( Ak )
1

20

4.

( A, ( A)) ( B, ( B)) ,

A = {a, b, c} , ( A) = [ 3

1 ]
5

B = {x, y} ,

( B) = [ 1

2 ]
3

3
5
3
2
3
1
3
H ( A) = log 2 + 2 log 2 5 = log 2 5 log 2 3 + log 2 5 = log 2 5 log 2 3
5
3
5
5
5
5
5
2
3 1
2
1
2
2
H ( B ) = log 2 3 + log 2 = log 2 3 + log 2 3 log 2 2 = log 2 3
3
2 3
3
3
3
3

( AB, ( AB))

6
1
2
1
2 ]
AB = {ax, ay, bx, by, cx, cy} ( AB) = [ 3
15 15 15 15 15 15

*1

,
,

p(ax) = p(a) p ( x) =

3 1 3 1
=
=
5 3 15 5

2
5
15
1
2
1

H ( AB) = log 2 5 + log 2 + 2 log 2 15 + 2 log 2

5
2
2
5
15
15

1
2
2
4
log 2 5 + (log 2 5 log 2 2) + (log 2 3 + log 2 5) + (log 2 3 + log 2 5 log 2 2)
5
5
15
15

1 2 2 4
= + + + log 2 5 +
5 5 15 15
= log 2 5 +

2 4
2 4
+ log 2 3 + log 2 2
5 15
15 15

2
2
log 2 3
3
5

H ( AB ) = H ( A) + H ( B ) .

21


2.6

H ( B / A)

. ,

H ( B / A) = H ( AB) H ( A)
, ,

H ( B / A) = H ( AB) H ( B) H ( A) + H ( B) H ( A)

H ( B / A) H ( B)
, ,
.
, , ,

H ( A / B) = H ( AB) H ( B)

H ( A / B) H ( A)

H ( A / B) = H ( A)

H ( B / A) = H ( B)

,
.

5.

( A, ( A)) ( B, ( B)) ,

( A) = [ 1

A = {a1 , a2 , a3} ,

B = {b1 , b2 , b3} .

22

1 ]
4

p (b j / ai )
:

p(b j / ai )

b1

b2

b3

a1

a2

a3

3
4
4

1
1

3
4
2

1)
2)

p(b)

3)
4)

p(ab)

5) H ( AB)
6) H ( A / B ) .
.
:

1) H ( A) =

2)

1
1
1
1 2 2 3
log 2 + log 4 + log 4 = + + = = 1.5 bits/
2
4
4
2 4 4 2

p(b1 ) = p(b1 / a1 ) p(a1 ) + p(b1 / a2 ) p(a2 ) + p (b1 / a3 ) p (a3 )

=

1 1 1 1 1 1 17
+ + =
3 2 2 4 4 4 48

, p (b2 ) =

3) H ( B ) =

4)

14
17
p (b3 ) =
48
48

48
48 17
48 14
17
1.5787 bits/
log + log + log
17
14 48
17 48
48

p(a1b1 ) = p (a1 ) p (b1 / a1 ) =

1 1 1
=
2 3 6

23

5) H ( AB ) =

p(a b ) log
i

i, j

6)

1
1
= log 6 + L = 3.0419
p(ai b j ) 6

H ( A / B) = H ( AB) H ( B) = 3.0419 1.5787 = 1.4632

H ( A) = 1.5 .

2.7

A = {a1 , a2 ,K, an } ,

A 2 = AA = {(a i a j ) / i, j = 1,2, K , n} .
, p ( ai a j ) = p ( ai ) p ( a j )

H ( A2 ) = H ( AA) = H ( A) + H ( A) = 2 H ( A)
() . ,

H ( Ak ) = kH ( A)

.

6.

( A, )

A = {0,1} = [0.1

0.9]

1
1

+ (0.9) log 2
H ( A) = (0.1) log 2
=0.469
0.1
0.9

A
,

24

00

01

10

11

(0.1)(0.1)

(0.1)(0.9)

(0.9)(0.1)

(0.9)(0.9)

=0.01

=0.09

=0.09

=0.81

( 1)

1
1
1

+ 2 (0.09) log 2
= 0.938
+ (0.81) log 2
0.01
0.09
0.81

H ( A 2 ) = (0.01) log 2

H ( A 2 ) = 2 H ( A) = 2 0.469 = 0.938

7.

( A, )

A = {0,1} = [ p

(1 p)]

A
,
000
3

001
2

p (1-p)

010

011

p (1-p)

100
2

p(1-p)

p (1-p)

101

110
2

p(1-p)

p(1-p)

( )

H ( A3 ) = 3H ( A) = 3h( p )
h( p ) Shannon.

25

111
2

(1-p)3

2.1 ()
)
)
)
) 5224.
( 14
, 1000)
;
2.2
; ,
; ( log 2 3 1.58 )
2.3 ,B,C,D

1 , 2 , 3 4
.
10
10
10
10
)

log 2 5

3
log 2 3 bits/.
10

)
;
2.4 = {a,b,c}
(A)=[0.3 0.5 x].
. ;
2.5 6

1/2

1/4

1/8

1/16

1/32

1/32

)
) ;
)
;
) )
;

26

2.6

( A, ) A = {a, b, c, d } .

) , = [
) = [

1
4

1
4

1
]
4

1 1 1 1
]
2 4 8 8

) = [0

1 1
]
2 2

) = [0

1 1023
]
1024 1024

1
4

= [0 0 0 1]

) ;
2.7 (A,(A))

A = {a1 , a2 } (A) = [0.1

0.9].

B = {b1 , b2 , b3}

p (b1 / a1 ) = 0.2

p (b2 / a1 ) = 0.3

p(b3 / a1 ) = 0.5

p (b1 / a 2 ) = 0.1

p (b2 / a 2 ) = 0.7

p(b3 / a 2 ) = 0.2

) p(b1), p(b2), p(b3)
) p(aibj)
) : H(A), H(B), H(AB), H(A/B), H(B/A).

.

2.8 (A,(A)) ={0,1} () = [

3 ].
4

3 . ( log 2 3 1.58 )

2.9 (A,(A)) ={a,b,c} () = [

1 ].
4

) 2
) () (2)
(2)=2()

27

2.1

) 1 bit,

2.2

log 2 6 = 2.58 bits

2.3

= 2 bits/

2.4

= log 2 3 bits/

2.5

) /=1,

) 31/16 bits/

) 2 bits/

2.6

) 2 bits

) 2 bits/ ,

) 23,74 bits

) 1.75 bits/ ,

) 0.001 bits/
2.7

) 24.55 bits

) 1 bit/,

) 0

) 0.11, 0.66, 0.23

)

p(ai,bj)

b1

b2

b3

0.02

0.03

0.05

0.09

0.63

0.18

()= 0.47 bits/

()= 1.23 bits/
()= 1.66 bits/
(/)= () - () = 0.43 bits/
(/)= () - () = 0.19 bits/

2.8

8 2.43 bits/

2.9

9 3 bits/

28

3.

3.1

. , ..

. ,

,
, ,
.
, ,
. , , .
0 1,
,
.

, 0 1,
.
: ,


.

3.2

,
- .

29

( )

( A, ( A))

( )

( B, ( B)) .

A = {a1 , a2 ,K, am } ,

( A) = [ p (a1 ) p(a 2 ) K p(a m )]

B = {b1 , b2 ,K, bn } ,

( B) = [ p(b1 ) p(b2 ) K p(bn )]

( A) ,

( ),
,

( B) . .
,
.

b ,

b.

pij = p(b j / ai ) = b j
ai

p(ai , b j ) = ai b j
.. a2

b3 . p(a2 , b3 ) (

),

p23 = p(b3 / a2 ) 1.

30

pij = p(b j / ai ) , ,

p11

a1

b1

p12
a2

b2

p1n

am

bn

p11
p
( B / A) = 21
M

p m1

p12

p 22

pm 2

O
K

p1n
p 2 n

p mn

,
.

mxn

( A, B )

p(ai , b j ) ai b j .

p(a1 , b1 )
p(a , b )
2
1
( A, B ) =

p(a m , b1 )

p(a1 , b2 ) L
p(a 2 , b2 ) L
O
p ( a m , b2 ) K

p(a1 , bn )
p(a 2 , bn )

p ( a m , bn )

( A)

p (a1 ) 0
D ( A) = 0
O
0
0

31

0
p (a m )
0

(
)

1.

( A) ( B / A) = ( B)
()x()= ()

2. D ( A) ( B / A) = ( A, B )
3.

( A, B ) ,

) i
) j

p(ai )

p(b j )

1.

0.9

0.9

0.1

0.1

, 0 1 90%,
10%:
0

90%

10%

10%

90%

0 1
0.4 0.6 (
)

( A) , ( B / A) , (B) , ( A, B )

) 1 ;
) 0 1;
) 1 0;

32

:
)

( A) : ( A) = [0.4 0.6]

( B / A) :

0.9 0.1
( B / A) =

0.1 0.9

(B)

0.9 0.1
=[0.42 0.58]
( B) = ( A) ( B / A) = [0.4 0.6]

0.1 0.9

( A, B )

0.4 0

( A, B) = D ( A) ( B / A) =

0 0.6
) p B (1) = 0.58 [

0.9 0.1 0.36 0.04

0.1 0.9 0.06 0.54

(B) ]

p(0,1) = 0.04 [ ( A, B ) ]

p B / A (1 / 0) = 0.1 [ ( B / A) ]

2.
A = {a, b, c, d }

1 , 1 , 1 X,
2
4
8

B = {x, y, z} ,

1
0
(/) =
0

0
1
1
4
0

0
0
3
4
1

,
. p(1)=0.58 p(1)=0.58, 1
p(1)=0.6
2
p/(1/0) p(1/0) 1
0 . .

33

) ()
d .
)
; 0;
) .
) () (,),

y .

c y.

c y.

:
)

1 1 1 1
X = 1 ( + + ) = .
2 4 8 8

I (d ) = log 8 = 3 bits,

H ( A) =

1
1
1
1
log 2 + log 4 + log 8 + log 8
2
4
8
8
1
1
1
1
7
= 1 + 2 + 3 + 3 = = 1.75 bits/
2
4
8
8
4

) 4 , .

H = 4 log 4 = log 4 = 2 bits/

4

0, ,
4 1
0.
)

a
b

y
1/4

3/4
1

34

1
(B ) =
2

( B) = ( A) ( B / A) = ( A, B) = D ( A) ( B / A)

1
4

1
2
0
( A, B ) =
0
0

1
8

0
1
4
0
0

1
8

1
0

0
0
1
8
0

0
1
1
4
0

0
0 1
3 = 2
4
1

0 1

0 0

0 0

1 0
8

0
1
1
4
0

3 1 1
+
=
32 8 2

1 1
+
4 32

0 12

0 0
3 = 0
4
1 0

9
32

7
32

0
0

1
0
4
3
1
32
32

1
0
8

3.3

y : p(y)=9/32

[ ()]

c y: p(y/c)=1/4

[ (/)]

c y: p(c,y)=1/32 [ (,)]

H ( A) = pi log
i =1

1
pi

(
) .


m

H ( A) = p(ai ) log
i =1

1
p(ai )

35

H ( B ) = p (bi ) log
i =1

1
p (bi )

H ( AB ) = p (ai , b j ) log
i, j

1
p (ai , b j )

.
,

H ( A / B) = H ( AB) H ( B)
( ).
.
.

H ( B / A) = H ( AB) H ( A)
( ).
.
.

3.4

.
.
, .

36

: ( )

I ( A, B ) = H ( A) + H ( B) H ( AB)

.

H ( A / B) H ( B / A)

I ( A, B) = H ( A) H ( A / B)

(1)

I ( A, B ) = H ( B) H ( B / A)

(2)

H ( A / B) 0.

.


.

:
:

C = max{I ( A, B)}
( A)

( A)

.

.

37


3.5

( B / A) ,

.
.
. (Deterministic Channel)

( B / A) .

A = {a1 , a2 , a3 , a4 } , B = {b1 , b2 , b3}

1
0
( B / A) =
0

0
1
1
0

0
0
0

a1
a2

a3

a4

b1

b2
b3

p(b / a) 0 1,

,
.

H ( B / A) = 0 . ,

(2)

I ( A, B) = H ( B) H ( B / A) = H ( B)

C = max{I ( A, B )} = max{H ( B)} = log 2 n bits/

( A)

( A)

( A) maximum,

.
n = 3 ,

C = log 2 3 = 1.585 bits/.

38

. (Lossless Channel)

( B / A) .
,

A = {a1 , a2 , a3} , B = {b1 , b2 , b3 , b4 , b5 }

1
3
( B / A) = 0

3
0
0

0
1
2
0

0 0

1
0
2
0 1

a1

b1

1/3

a2

2/3

b2

1/2

b3

1/2

a3

b4
1

b5

p(b / a) ( )

p(a / b) . ,

,
. , b,
. b2
a1 a 2

a3 . ,

p (a2 / b2 ) = 0 p(a3 / b2 ) = 0 .

p (a1 / b2 ) = 1

p(a / b) 1 0,

, .

H ( A / B) = 0 .

, (1)

I ( A, B) = H ( A) H ( A / B) = H ( A)

C = max{I ( A, B )} = max{H ( A)} = log 2 m

( A)

( A)

39

bits/

m = 3 ,

C = log 2 3 = 1.585 bits/.

. (Ideal Channel)
. m = n ,

C = max{H ( A)} = max{H ( B )} = log 2 n

( A)

( A)

,
.
..

1
0

1 0
.
( B / A) =

0 1
m = n = 2

C = log 2 2 = 1 bit/
. (Binary Symmetric Channel BSC)
q

1-q

q
q
1

1-q

40

( )

C = 1 q log 2

1
1
(1 q) log 2
q
1 q

C = 1 h( q )

h(q ) Shannon.

41

3.1
=={0,1}. , 0 1

.
q= 1
100

)
)

( A) , ( B / A) , (B) , ( A, B )

) 0 ;
) 0 1;
) 0 1;

3.2
A=B={a,b,c,d}.
) a,b,c,d

1/2, 1/4, 1/8 x .

x .
)
; 0;
) c d
, 10% a
b b a. (/)
.
) () b .
3.3 ={0,1}
={a,b,c}. 1 0.
0 a b 1
b c .
) ()
) (/)
) c .
) (,) 0
b.

[ log 2 3 1.59 ]

42

3.4 a, b,
c 0 1. a b
c
( a b). a 0, c
1, b 0 1 .
) ()
) (/)
) 0 1 .
) (,)
b 0.
3.5 A ={1,2} ={b1,b2},
)

( A) ( B / A) = ( B)

) D ( A) ( B / A) = ( A, B )

3.6 ={x1,x2,x3,x4,x5}
={y1,y2,y3,y4}

0
0
0
0.25
0.10 0.30
0
0

( A, B) = 0
0.05 0.10
0

0
0.05 0.10
0
0
0
0.05
0
) () ()
)

p ( y / x) (/)

)
3.7

0
0
0
1 0
0 1 / 2 1 / 2 0
0
(/) =
0 0
0 1/ 4 3 / 4

0
0
0
0 0

0
0
0

)
)

p(b4 / a3 ) , p(a3 / b4 ) , p(b5 / a2 ) , p(a2 / b5 ) .

p(a / b) ; ( a

b )
) .

43

3.8

1
1
(/) =
0

0
0
1

)
) .
3.9 (BSC) q.
=[p

1-p] ( ()=h(p)

h Shannon)
) (/), (), (,)
) ()=h(p)+h(q)
) () p
;
) (,)=()-h(q)
)
C=1-h(q)
3.10
) q=0.5
. [

) q = 1/4 ) q=3/4.

log 2 3 1.59 ]

3.11 ={0,1}, ={0,*,1}

0.9

0.1

0.1

0.9

To ,
.
*. ()=[p 1-p]
( ()=h(p) h Shannon)

44

) (/), (), (,)

) ()=h(p)+h(0.1) ()= 0.9h(p) + h(0.1)
) I(A,B)=0.9h(p)
)
C= 0.9 bits/
[: , q
C= (1-q) bits/ ]
3.12 m

m
1
( B / A) = m
M
1

1
m
1
m
M
1
m

m
1
L
m
O M
1
L

m
L

(independent channel).
C=0,
.
m=3, .

45

101

3.1

3.2

) =1.75 bits/

3.3

) () = [1/4 3/4],

300

) 3/8
3.4

300

1
100

) max= 2 bits/

) / = 11/40

() =0.81 bits/

) 1/8

) () = [1/4

1/4

1/2 ]

() = 1.5 bits/

) 3/8 5/8
) 1/8

( A) = [ p (a1 )

p (a 2 )]

p (b / a )
( B / A) = 1 1
p (b1 / a 2 )

3.5

3.6

) ()= [0.25 0.40 0.15 0.15 0.05]

p (b2 / a1 )
p (b2 / a 2 )

()=[0.35 0.35 0.20 0.10]

0
0
0
1
1 / 4 3 / 4 0
0

) ( B / A) = 0
1/ 3 2 / 3 0

0 1 / 3 2 / 3
0
0
0
1
0
)

()= 2.066 bits/

()=1.857 bits/
()=2.666 bits/
(/) = ()-() = 0.6 bits/
(/) = ()-() = 0.809 bits/

3.7

p(b4 / a3 ) = 1/4 p(b5 / a2 ) = 0 p(a3 / b4 ) =1 p(a2 / b5 ) =0

p(a / b) 0 1

) C=2 bits/
3.8

C=1 bit/

3.9

) max ()=1 p=1/2 ( q 1/2)

46

3.10

) C=0

3.11

) ()= [0.9 p 0.1 0.9(1 p)]

3.12

(B) = [

) ) C=0.19 bits/

0.9 p
0.1p
0
(,)=
0 0.1(1 p) 0.9(1 p)

1 1
1
L ]
m m
m

H ( B ) = log 2 m

( A, B ) =

p1
m
p2
m
M
pm
m

H ( AB) = log 2 m + H ( A)

47

p1
m
p2
m
M
pm
m

L
L
O
L

p1
m
p2
m
M
pm
m

48