EVERY MESSAGE PUT OUT BY A SOURCE CONTAINS SOMEINFORMATION. SOME MESSAGES CONVEY MORE INFORMATION THANOTHERS.
How to measure the information content of a message quantitatively?
To answer this, we are required to arrive at an intuitive concept of the amount of information.Consider the following examples:A trip to Mercara (Coorg) in the winter time during evening hours,1.It is a cold day2.It is a cloudy day3.Possible snow flurriesAmount of information received is obviously different for these messages.
Message (1) Contains very little information since the weather in coorg is ‘cold’for most part of the time during winter season.
The forecast of ‘cloudy day’ contains more information, since it is not an eventthat occurs often.
In contrast, the forecast of ‘snow flurries’ conveys even more information,since the occurrence of snow in coorg is a rare event.
On an intuitive basis, then with a knowledge of the occurrence of an event, what can besaid about the amount of information conveyed?It is related to the probability of occurrence of the event.
What do you conclude from the above example with regard to quantity of information?Message associated with an event ‘least likely to occur’ contains most information.
How can the information content of a message be expressed quantitatively?The above concepts can now be formed interns of probabilities as follows:Say that, an information source emits one of ‘q’ possible messages m
as their probs. of occurrence.2