Information theory

24-01-2022
 Measure of Information

• Uncertainty and Information

• Self Information
• Average self information
• Mutual Information
• Average Mutual Information

24-01-2022
 Measurement of Information
Uncertainty and Information

• Sun rises in the east

• Classes go online as covid cases are increasing
• India will be corruption free from 2022
• There is heavy snow fall in Bangalore

24-01-2022
 Uncertainty and Information
Amount of information received is obviously different for these messages.
➢Message (1) Contains very little information or absolutely no information at
all since it is universal truth.
➢ The 2nd message contains some information, since it is not an event that
occurs often.
➢In contrast, the 3rd message conveys highest information as it is a rare event.

24-01-2022
 Uncertainty and Information
• On an intuitive basis, then with a knowledge of the occurrence of an event,
what can be said about the amount of information conveyed?
• We observed from the above sentences that the occurrence of a less probable
event conveys more information.
Since a lower probability implies a higher degree of uncertainty (and vice versa),
a random variable with a high degree of uncertainty contains more information.
Message associated with an event ‘least likely to occur’ contains most
information
24-01-2022
 Uncertainty and Information
• The information content of a message can be expressed quantitatively as
follows:
• The above concepts can now be formed in terms of probabilities as follows:
Say that, an information source emits one of ‘q’ possible messages S1, S2,
….. ….Sq with P1, P2,………Pq as their probs. of occurrence.

Then the amount of Information or Self Information of message 𝑆𝐾 is given

by
1
𝐼𝐾 = log
𝑃𝐾
24-01-2022
 Measurement of Information
Unit of information
Base of the logarithm will determine the unit assigned to the
information content.

1. Natural logarithm or base e : ‘nats’

2. Base - 10 : Hartleys / decits

3. Base - 2 : bits ( Which is most commonly used unit)

4. If the base in general . Is “r”, the units are called “r-ary units”
24-01-2022
 Measure of Information
Logarithmic expression is chosen for measuring information because
of the following reasons :
1. The information content or self information of any message cannot be
negative . Each message must contain certain amount of information.
2. The lowest possible self information is “zero” which occurs for a sure
event since
p(x)=1 then I(x)=0
Eg: Sun rises in the east

24-01-2022
 Measure of Information
3. More information is carried by a less likely message.
since p(x)=0 then I(x)=1
Eg: There will be snowfall on a particular day in Chennai.
4. When independent symbols are transmitted , the total self-
information must be equal to the sum of individual self –
informations.

24-01-2022
Example
As an example of independent events , suppose that you read two news
items in the news paper :
1. Earth quake rocks in Gujarat State
2. There will be snowfall on a particular day in Chennai.
It is reasonable to assume that the two events mentioned in the news
are independent and the total information received from the two
messages is same as the sum of the information contained in each of
the two news items.
24-01-2022
24-01-2022
Example: A binary symbols 0 and 1 are transmitted with probabilities ¼ and ¾
respectively. Find the corresponding self information.

Summary :

1 3
𝑃0 = and 𝑃1 = 𝑃1 > 𝑃0
4 4

𝐼0 = 2 𝑏𝑖𝑡𝑠 and 𝐼1 = 0.415 𝑏𝑖𝑡𝑠

𝐼0 > 𝐼1

It can be observed that more information is carried by a less likely message.

24-01-2022
• Example: Consider a zero memory source emitting 4 symbols S1,S2,S3 and S4with
respective probabilities 0.5, 0.25,0.125 and 0.125. Find the information present in the
message S1S4S2S3S4 and show that this is same as sum of the information carried by the
individual symbols.

24-01-2022
 Zero Memory Source
• Now, we do not stop at 1 toss. We toss the same coin several times. In fact, m
times and suppose this source is memory less that is my output of the second
toss does not belong to or does not depend on the first toss, outcome of the
first toss and subsequently any of the later tosses.
• This means that there is no connection between any two symbols that the
source has no memory. Such type of sources are called memory less or
Zero-memory sources.

24-01-2022
 Average information Content of symbols in long
independent sequence: Entropy
Consider a zero memory source producing independent sequences of symbols. Let
us consider the source alphabet S ={S1, S2,….Sq} with probabilities P = {P1, P2,
….Pq} respectively.
Consider a long independent sequence of length L symbols. This long sequence then
contains
Note:
P1L number of messages of type S1 Consider a source emitting 4 symbols S1 S2 S3 S4.
P2L number of messages of type S2 Considers a sequence of 8 symbols
S1S2S3S2S4S2S3S1
…………………………………. Now P(S3)=Total No. of S2/Total No. of symbols
Or No. of S2 in the sequence =P(S2) *8
PqL number of messages of type Sq
1
We know that self information of S1=log bits
𝑃1

24-01-2022
Therefore P1L number of messages of type S1 contain
1
P1L log bits of information and
𝑃1
1
P2L number of messages of type S2 contain P2Llog bits of information
𝑃2

…………………………………………………
1
PqL number of messages of type Sq contain PqL log bits of information
𝑃𝑞

The total information content of all these message symbols is given by

1 1 1
Itotal = P1L log + P2L log …….. PqL log bits
𝑃1 𝑃2 𝑃𝑞

𝑞 1
= Lσ𝑖=1 𝑃𝑖 log bits
24-01-2022
𝑃𝑖
 𝐼𝑡𝑜𝑡𝑎𝑙
Average self information =
𝐿
𝑞 1
= σ𝑖=1 𝑃𝑖 log bits/ message symbol
𝑃𝑖
Average self information is also called as ENTROPY of the source
denoted by H(S)
𝑞 1
Therefore H(S) = σ𝑖=1 𝑃𝑖 log bits/ message symbol
𝑃𝑖

𝑞 1
Entropy H(S) = σ𝑖=1 𝑃𝑖 log bits/ message symbol
𝑃𝑖

24-01-2022
Information Rate: Let us suppose that symbols are emitted by the
source at a fixed time rate rs symbols per second. Then the average
source information rate is defined as the product of the average
information content per symbol and the message symbol rate rs

Rs=rs H(S) bits/sec

24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
 Properties of Entropy

24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
 Extension of Zero memory source
Extension of zero memory source becomes necessary in coding
problems

24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022
24-01-2022