Directed Information and Feedback

Channels
- An overview by Ketan Mehta
 Directed Information

Input

with the possible values denoted using lower case

Similarly output is

Massey, J. "Causality, feedback and directed information." In Proc. Int.

Symp. Inf. Theory Applic.(ISITA-90), pp. 303-305. 1990.
 Directed Information (contd)

Memoryless
 Directed Information (contd)
Used without feedback
 Directed Information (contd)
Used without feedback

No Feedback
 Directed Information (contd)
Used without feedback

No Feedback

Feedback
 Directed Information (contd)
Defined as,

In general,
 Directed Information (contd)
Defined as,

Theorem 1 : For a discrete channel,

with equality only if the channel is used w/o feedback

 Directed Information (contd)
Defined as,

Theorem 2 : For a DMC

with equality if are statistically independent