‫‪Information‬‬

‫‪Measurements‬‬
‫‪WITH DIFFERENT APPROACHES‬‬

‫إعداد الطالب ‪:‬‬

‫آدم أحمد محمد النور ‪ -‬اتصاالت‬
‫بشير حسين حمد يوسف ‪ -‬حاسوب‬
‫محمد تبن محمد ابراهيم ‪ -‬اتصاالت‬
‫عمار سليمان أحمد محمد ‪ -‬حاسوب‬
‫عمرو عبده عباس أحمد – اتصاالت‬
Pg. 01

Introduction
Information theory is the science which deals with concept of
‘information’, it is measurements and it is applications.
There are three types of information formats:
- Syntactic information, related to symbols in from which messages
are built up and to their interrelations.
- Semantic information, related to the meaning of messages, their
referential aspect.
- Pragmatic information, related to the usage and effect of
messages.
Modern information theory is founded on the ideas of Hartley and
Shannon.
In 1928, Hartley reasoned that a character symbol would convey
information, only if there existed alternative characters to be
transmitted instead. However, Hartley’s somewhat simplistic
approach gave sensible answers only in quite specific cases.
Some twenty years on. Shannon agreed with Hartley that
information exists only when there is a choice of possible
messages, but then went on to suggest that when there are a
number of possible events or messages that may occur, there will
also be a set of a priori probabilities for these messages and the
amount of information should depend upon these probabilities.
Pg. 02

Hartley’s Information Measure:-

• Messages are strings of characters from a fixed alphabet.

• The amount of information contained in a message should be a

function of the total number of possible messages.

• If you have an alphabet with s symbols, then there are s^L

messages of length, L .

• The amount of information contained in two messages should be

the sum of the information contained in the individual messages.

• The amount of information in L messages of length one should

equal the amount of information in one message of length `.

It is clear that the only function which satisfies these requirements is

the log function: Llog(s) = log(s^L). If the base of the logarithm is
two, then the unit of information is the bit.
Pg. 03

Shannon’s Information Measure:-

Let X be a discrete r.v. with n outcomes, {X1,...,Xn}. The probability
that the outcome will be xi is pX(xi). The information contained in a
message about the outcome of X is: −log pX(xi).

The avg. information or entropy of a message about the outcome of

𝑛
X is: HX = ∑𝑖=1 𝑝𝑋(𝑥𝑖)𝑙𝑜𝑔 𝑝𝑋(𝑥𝑖)

Properties of Shannon’s Measure:-

• HX is continuous in the pX(xi).
• HX is symmetric. That is, HX = HY when pY(x1) = pX(x2) and
pY(x2) = pX(x1). More generally, HX is invariant under permutation
of the distribution function, pX.
• HX is additive. That is, when X and Y are independent r.v.’s, then
HXY = HX + HY.
• HX is maximum when all of the pX(xi)’s are equal
• HX is minimum when one of the pX(xi)’s equals one.
Pg. 04

Different Approaches
Based on the string complexity:

A. A Recursive Linear-Pattern Copying algorithm:

Used for decomposing a string into a vocabulary of substrings,

where the vocabulary defines, in effect, a production history for
the string.

B. A Recursive Hierarchical-Pattern Copying algorithm:

We now introduce a somewhat more sophisticated recursive

pattern copying algorithm. As before we parse the string left-to-
right but lift the patterns pi from the string right-to-left, and further
account for consecutive repetitions of each patterns with
corresponding integer parameters, Ki . The resultant
decomposition of a string is thus given in terms of patterns pi
which tend to maximize reuse of patterns pj, j < i, i.e., minimize
the total number of steps, t, required to construct the string: t
measures the depth of the recursive pattern copy hierarchy.

Based on the source nature:

A. Natural sources, for example, random strings, DNA, protein,

and language texts.
B. Non-stationary sources: While the information rate for many
strings/sources may be assumed to be essentially constant,
other sources are observed not to be. Ex. A binary scanned
Pg. 05

document contains regions of white space interspersed by

regions of patterning.

REFERENCES:

1. C. E. Shannon, \A mathematical theory of communications",

Bell Systems Technical Journal, vol. 27, pp. 379{423,
623656, July 1948}.
2. A. Lempel and J. Ziv, \On the Complexity of Finite
Sequences", IEEE Trans. Inform. Theory, vol. 22, no. 1, pp.
75{81, January 1976}.
3. M. R. Titchener, \A Deterministic Theory of Complexity,
Information and Entropy", in IEEE Information Theory
Workshop, February 1998, San Diego.
4. M. R. Titchener, \Deterministic computation of string
complexity, information and entropy", in International
Symposium on Information Theory, August 16{21 1998, MIT,
Boston.