2.

Independence and Bernoulli Trials

Independence: Events A and B are independent if

P ( AB ) = P ( A) P ( B ). (2-1)

• It is easy to show that A, B independent implies A, B;

A, B ; A, B are all independent pairs. For example,
B = ( A ∪ A ) B = AB ∪ AB and AB ∩ AB = φ , so that
P ( B ) = P ( AB ∪ AB ) = P ( AB ) + P ( AB ) = P ( A) P ( B ) + P ( AB )
or
P ( AB ) = P ( B ) − P ( A ) P ( B ) = (1 − P ( A )) P ( B ) = P ( A ) P ( B ),
i.e., A and B are independent events.
1
PILLAI
• If P(A) = 0, then since the event AB ⊂ A always, we have
P ( AB ) ≤ P ( A) = 0 ⇒ P ( AB ) = 0,
and (2-1) is always satisfied. Thus the event of zero
probability is independent of every other event!
• Independent events obviously cannot be mutually
exclusive, since P ( A) > 0, P ( B ) > 0 and A, B independent
implies P ( AB ) > 0. Thus if A and B are independent,
the event AB cannot be the null set.
• More generally, a family of events {Ai } are said to be
independent, if for every finite sub collection
Ai , Ai , , Ai , we have
1 2 n

 n  n
P  ∩ Aik  = ∏ P( A ik ). (2-2)
 k =1  k =1 2
PILLAI
• Let
A = A1 ∪ A2 ∪ A3 ∪ ∪ An , (2-3)
a union of n independent events. Then by De-Morgan’s
law
A = A1 A 2 An (2-4)
and using their independence
n n
P ( A ) = P ( A1 A 2 An ) = ∏ P ( A ) = ∏ (1 − P ( A )). (2-5)
i =1
i
i =1
i

Thus for any A as in (2-3)

n
P ( A ) = 1 − P ( A ) = 1 − ∏ (1 − P ( Ai )) , (2-6)
i =1
a useful result.
3
PILLAI
Example 2.1: Three switches connected in parallel operate
independently. Each switch remains closed with probability
p. (a) Find the probability of receiving an input signal at the
output. (b) Find the probability that switch S1 is open given
that an input signal is received at the output.

Input Output

Fig.2.1
Solution: a. Let Ai = “Switch Si is closed”. Then P ( Ai ) = p ,
i = 1 → 3 . Since switches operate independently, we have
P ( Ai A j ) = P ( Ai ) P ( A j ); P ( A1 A2 A3 ) = P ( A1 ) P ( A2 ) P ( A3 ).
4
PILLAI
Let R = “input signal is received at the output”. For the
event R to occur either switch 1 or switch 2 or switch 3
must remain closed, i.e.,
R = A1 ∪ A2 ∪ A3 . (2-7)
Using (2-3) - (2-6),
P( R) = P( A1 ∪ A2 ∪ A3 ) = 1 − (1 − p)3 = 3 p − 3 p 2 + p 3. (2-8)
We can also derive (2-8) in a different manner. Since any
event and its compliment form a trivial partition, we can
always write
P ( R ) = P ( R | A1 ) P ( A1 ) + P ( R | A1 ) P ( A1 ). (2-9)
But P ( R | A1 ) = 1, and P ( R | A1 ) = P ( A2 ∪ A3 ) = 2 p − p 2
and using these in (2-9) we obtain
P ( R ) = p + ( 2 p − p 2 )(1 − p ) = 3 p − 3 p 2 + p 3 , (2-10)
which agrees with (2-8). 5
PILLAI
Note that the events A1, A2, A3 do not form a partition, since
they are not mutually exclusive. Obviously any two or all
three switches can be closed (or open) simultaneously.
Moreover, P ( A1 ) + P ( A2 ) + P ( A3 ) ≠ 1 .
b. We need P ( A1 | R ). From Bayes’ theorem

P ( R | A1 ) P ( A1 ) ( 2 p − p 2 )(1 − p ) 2 − 2 p + p 2 (2-11)
P ( A1 | R ) = = = .
P( R) 3p − 3p + p
2 3
3p − 3p + p
2 3

Because of the symmetry of the switches, we also have

P ( A1 | R ) = P ( A2 | R ) = P ( A3 | R ).

6
PILLAI
 Repeated Trials
Consider two independent experiments with associated
probability models (Ω1, F1, P1) and (Ω2, F2, P2). Let
ξ∈Ω1, η∈Ω2 represent elementary events. A joint
performance of the two experiments produces an
elementary events ω = (ξ, η). How to characterize an
appropriate probability to this “combined event” ?
Towards this, consider the Cartesian product space
Ω = Ω1× Ω2 generated from Ω1 and Ω2 such that if
ξ ∈ Ω1 and η ∈ Ω2 , then every ω in Ω is an ordered pair
of the form ω = (ξ, η). To arrive at a probability model
we need to define the combined trio (Ω, F, P).
7
PILLAI
Suppose A∈F1 and B ∈ F2. Then A × B is the set of all pairs
(ξ, η), where ξ ∈ A and η ∈ B. Any such subset of Ω
appears to be a legitimate event for the combined
experiment. Let F denote the field composed of all such
subsets A × B together with their unions and compliments.
In this combined experiment, the probabilities of the events
A × Ω2 and Ω1 × B are such that
P ( A × Ω 2 ) = P1 ( A), P (Ω1 × B ) = P2 ( B ). (2-12)

Moreover, the events A × Ω2 and Ω1 × B are independent for

any A ∈ F1 and B ∈ F2 . Since
( A × Ω 2 ) ∩ (Ω1 × B ) = A × B, (2-13)

we conclude using (2-12) that 8

PILLAI
 P( A × B ) = P( A × Ω2 ) ⋅ P(Ω1 × B ) = P1 ( A) P2 ( B ) (2-14)
for all A ∈ F1 and B ∈ F2 . The assignment in (2-14) extends
to a unique probability measure P ( ≡ P1 × P2 ) on the sets in F
and defines the combined trio (Ω, F, P).
Generalization: Given n experiments Ω 1 , Ω 2 , ,Ωn, and
their associated Fi and Pi , i = 1 → n , let
Ω = Ω1 × Ω2 × × Ωn (2-15)

represent their Cartesian product whose elementary events

are the ordered n-tuples ξ 1 , ξ 2 , , ξ n , where ξ i ∈ Ω i . Events
in this combined space are of the form
A1 × A2 × × An (2-16)

where Ai ∈ Fi , and their unions an intersections. 9

PILLAI
If all these n experiments are independent, and Pi ( Ai ) is the
probability of the event Ai in Fi then as before
P ( A1 × A2 × × An ) = P1 ( A1 ) P2 ( A2 ) P ( An ). (2-17)

Example 2.2: An event A has probability p of occurring in a

single trial. Find the probability that A occurs exactly k times,
k ≤ n in n trials.
Solution: Let (Ω, F, P) be the probability model for a single
trial. The outcome of n experiments is an n-tuple
ω = {ξ 1 , ξ 2 , , ξ n }∈ Ω 0 , (2-18)

where every ξ i ∈ Ω and Ω 0 = Ω × Ω × × Ω as in (2-15).

The event A occurs at trial # i , if ξ i ∈ A . Suppose A occurs
exactly k times in ω. 10
PILLAI
Then k of the ξ i belong to A, say ξ i , ξ i , , ξ i , and the
1 2 k

remaining n − k are contained in its compliment in A .

Using (2-17), the probability of occurrence of such an ω is
given by
P0 (ω) = P({ξi1 ,ξi2 , ,ξik , ,ξin }) = P({ξi1 })P({ξi2 }) P({ξik }) P({ξin })
= P( A)P( A) P( A) P( A)P( A) P( A) = pk qn−k . (2-19)
k n−k

However the k occurrences of A can occur in any particular

location inside ω. Let ω 1 , ω 2 , , ω N represent all such
events in which A occurs exactly k times. Then
" A occurs exactly k times in n trials" = ω1 ∪ ω2 ∪ ∪ ωN . (2-20)

But, all these ω i s are mutually exclusive, and equiprobable.

11
PILLAI
Thus
P(" A occurs exactly k times in n trials" )
N
= ∑ P0 (ωi ) = NP0 (ω ) = Npk qn −k , (2-21)
i =1

where we have used (2-19). Recall that, starting with n

possible choices, the first object can be chosen n different
ways, and for every such choice the second one in (n −1)
ways, … and the kth one (n −k +1) ways, and this gives the
total choices for k objects out of n to be n ( n − 1) ( n − k + 1).
But, this includes the k! choices among the k objects that
are indistinguishable for identical objects. As a result
n( n − 1) ( n − k + 1) n! ∆ n
N= = =   (2-22)
k! ( n − k )! k!  k 
12
PILLAI
represents the number of combinations, or choices of n
identical objects taken k at a time. Using (2-22) in (2-21),
we get
Pn (k ) = P(" A occurs exactly k times in n trials" )
 n  k n −k (2-23)
=   p q , k = 0,1,2, , n,
k 
a formula, due to Bernoulli.
Independent repeated experiments of this nature, where the
outcome is either a “success” ( = A) or a “failure” ( = A)
are characterized as Bernoulli trials, and the probability of
k successes in n trials is given by (2-23), where p
represents the probability of “success” in any one trial.
13
PILLAI
Example 2.3: Toss a coin n times. Obtain the probability of
getting k heads in n trials ?
Solution: We may identify “head” with “success” (A) and
let p = P (H ). In that case (2-23) gives the desired
probability.
Example 2.4: Consider rolling a fair die eight times. Find
the probability that either 3 or 4 shows up five times ?
Solution: In this case we can identify
"success" = A = { either 3 or 4 } = { f 3 } ∪ { f 4 }.
Thus
1 1 1
P ( A) = P ( f 3 ) + P ( f 4 ) = + = ,
6 6 3
and the desired probability is given by (2-23) with n = 8 , k = 5
and p = 1/ 3. Notice that this is similar to a “biased coin”14
problem. PILLAI
Bernoulli trial: consists of repeated independent and
identical experiments each of which has only two outcomes A
or A with P ( A) = p, and P ( A) = q. The probability of exactly
k occurrences of A in n such trials is given by (2-23).
Let
X k = " exactly k occurrence s in n trials" . (2-24)

Since the number of occurrences of A in n trials must be an

integer k = 0 ,1, 2 , , n , either X0 or X1 or X2 or or Xn must
occur in such an experiment. Thus
P(X 0 ∪ X1 ∪ ∪ X n ) = 1. (2-25)

But X i , X j are mutually exclusive. Thus

15
PILLAI
 n
 n  k n−k
n
P(X 0 ∪ X1 ∪ ∪ X n) = ∑ k =0
P ( X k ) = ∑   p q
k =0  k 
. (2-26)

From the relation

n
n  k n−k
(a + b) n
= ∑   a b , (2-27)
k =0 k 
(2-26) equals ( p + q ) n = 1, and it agrees with (2-25).
For a given n and p what is the most likely value of k ?
From Fig.2.2, the most probable value of k is that number
which maximizes Pn ( k ) in (2-23). To obtain this value,
consider the ratio
Pn (k )
n = 12, p = 1 / 2.

k
Fig. 2.2 16
PILLAI
 Pn ( k − 1) n! p k − 1 q n − k + 1 ( n − k )! k ! k q
= k n−k
= . (2-28)
Pn ( k ) ( n − k + 1)! ( k − 1)! n! p q n − k +1 p

Thus Pn ( k ) ≥ Pn ( k − 1 ), if k (1 − p ) ≤ ( n − k + 1 ) p or k ≤ ( n + 1) p .
Thus Pn ( k ) as a function of k increases until

k = ( n + 1) p (2-29)

if it is an integer, or the largest integer kmax less than ( n + 1) p ,

and (2-29) represents the most likely number of successes
(or heads) in n trials.
Example 2. 5: In a Bernoulli experiment with n trials, find
the probability that the number of occurrences of A is
between k 1 and k 2 .
17
PILLAI
Solution: With X i , i = 0 ,1, 2 , , n , as defined in (2-24),
clearly they are mutually exclusive events. Thus
P(" Occurrences of A is between k1 and k2 " )
k2 k2
 n  k n −k
= P( X k1 ∪ X k1 +1 ∪ ∪ X k2 ) = ∑ P( X k ) = ∑   p q . (2-30)
k = k1 k = k1  k 

Example 2. 6: Suppose 5,000 components are ordered. The

probability that a part is defective equals 0.1. What is the
probability that the total number of defective parts does not
exceed 400 ?
Solution: Let
Yk = " k parts are defective among 5,000 components ".
18
PILLAI
Using (2-30), the desired probability is given by
400
P (Y0 ∪ Y1 ∪ ∪ Y400 ) = ∑ P (Y
k =0
k )

 5000 
400
= ∑   ( 0 .1) k ( 0 .9 ) 5000 − k . (2-31)
k =0  k 
Equation (2-31) has too many terms to compute. Clearly,
we need a technique to compute the above term in a more
efficient manner.
From (2-29), kmax the most likely number of successes in n
trials, satisfy
( n + 1) p − 1 ≤ k max ≤ ( n + 1) p (2-32)
or q k max p
p− ≤ ≤ p+ , (2-33)
n n n 19
PILLAI
so that
km
lim = p. (2-34)
n→ ∞ n
From (2-34), as n → ∞, the ratio of the most probable
number of successes (A) to the total number of trials in a
Bernoulli experiment tends to p, the probability of
occurrence of A in a single trial. Notice that (2-34) connects
the results of an actual experiment (km / n) to the axiomatic
definition of p. In this context, it is possible to obtain a more
general result as follows:
Bernoulli’s theorem: Let A denote an event whose
probability of occurrence in a single trial is p. If k denotes
the number of occurrences of A in n independent trials, then
 k  pq
P   − p > ε   < . (2-35)
 n  nε 2
20
PILLAI
Equation (2-35) states that the frequency definition of
k
probability of an event n and its axiomatic definition ( p)
can be made compatible to any degree of accuracy.
Proof: To prove Bernoulli’s theorem, we need two identities.
Note that with Pn ( k ) as in (2-23), direct computation gives
n n −1 n
n! n!
∑
k =0
k Pn ( k ) = ∑
k =1
k
( n − k )! k !
p k n −k
q = ∑
k =1 ( n − k )! ( k − 1)!
p k n −k
q
n −1
n! n −1
( n − 1)!
=∑ i +1 n − i −1
p q = np ∑ p i q n −1−i
i = 0 ( n − i − 1)! i! i = 0 ( n − 1 − i )! i!

= np ( p + q ) n −1 = np . (2-36)
Proceeding in a similar manner, it can be shown that
n n n
n! n!
∑
k =0
k2
Pn ( k ) = ∑
k =1
k
( n − k )! ( k − 1)!
p k n −k
q = ∑
k = 2 ( n − k )! ( k − 2 )!
p k q n −k
n
n!
+∑ p k q n − k = n 2 p 2 + npq . (2-37)
k =1 ( n − k )! ( k − 1)!
21
PILLAI
Returning to (2-35), note that
k
− p >ε is equivalent to ( k − np ) 2 > n 2ε 2 , (2-38)
n
which in turn is equivalent to
n n

∑
k =0
( k − np ) Pn ( k ) >
2
∑k =0
n 2 ε 2 Pn ( k ) = n 2 ε 2 . (2-39)

Using (2-36)-(2-37), the left side of (2-39) can be expanded

to give
n n n

∑ ( k − np )
k =0
2
Pn ( k ) = ∑k
k =0
2
Pn ( k ) − 2 np ∑ k Pn ( k ) + n 2 p 2
k =0

= n 2 p 2 + npq − 2 np ⋅ np + n 2 p 2 = npq . (2-40)

Alternatively,
n
the left side of (2-39) can be expressed as
∑ ( k − np )
k =0
2
Pn ( k ) = ∑ ( k − np )
k − np ≤ n ε
2
Pn ( k ) + ∑ ( k − np )
k − np > n ε
2
Pn ( k )

≥ ∑ ( k − np )
k − np > n ε
2
Pn ( k ) > n 2ε 2 ∑
k − np > n ε
Pn ( k )
(2-41) 22
= n ε P { k − np > n ε }.
2 2
PILLAI
Using (2-40) in (2-41), we get the desired result
 k  pq
P   − p > ε   < . (2-42)
 n  nε 2

Note that for a given ε > 0, pq / nε 2 can be made arbitrarily

small by letting n become large. Thus for very large n, we
can make the fractional occurrence (relative frequency) kn
of the event A as close to the actual probability p of the
event A in a single trial. Thus the theorem states that the
probability of event A from the axiomatic framework can be
computed from the relative frequency definition quite
accurately, provided the number of experiments are large
enough. Since kmax is the most likely value of k in n trials,
from the above discussion, as n → ∞, the plots of Pn ( k ) tends
to concentrate more and more around kmax in (2-32).
23
PILLAI
Next we present an example that illustrates the usefulness of
“simple textbook examples” to practical problems of interest:
Example 2.7 : Day-trading strategy : A box contains n
randomly numbered balls (not 1 through n but arbitrary
numbers including numbers greater than n). Suppose
a fraction of those balls − say m = np ; p < 1 − are initially
drawn one by one with replacement while noting the numbers
on those balls. The drawing is allowed to continue until
a ball is drawn with a number larger than the first m numbers.
Determine the fraction p to be initially drawn, so as to
maximize the probability of drawing the largest among the
n numbers using this strategy.

Solution: Let “ X k = (k + 1) st drawn ball has the largest

number among all n balls, and the largest among the PILLAI 24
first k balls is in the group of first m balls, k > m.” (2.43)
Note that X k is of the form A ∩ B,
where
A = “largest among the first k balls is in the group of first
m balls drawn”
and
B = “(k+1)st ball has the largest number among all n balls”.
Notice that A and B are independent events, and hence
1 m 1 np p
P( X k ) = P ( A) P ( B) = = = . (2-44)
nk n k k
Where m = np represents the fraction of balls to be initially
drawn. This gives
P (“selected ball has the largest number among all balls”)
n −1 n −1
1 n 1
= ∑ P( X k ) = p ∑ ≈ p ∫ np = p ln k
n
np
k =m k =m k k
25
= − p ln p. (2-45)
Maximization of the desired probability in (2-45) with
respect to p gives
d
(− p ln p ) = −(1 + ln p ) = 0
dp
or
p = e −1 0.3679. (2-46)
From (2-45), the maximum value for the desired probability
of drawing the largest number equals 0.3679 also.
Interestingly the above strategy can be used to “play the
stock market”.
Suppose one gets into the market and decides to stay
up to 100 days. The stock values fluctuate day by day, and
the important question is when to get out?

According to the above strategy, one should get out 26

PILLAI
at the first opportunity after 37 days, when the stock value
exceeds the maximum among the first 37 days. In that case
the probability of hitting the top value over 100 days for the
stock is also about 37%. Of course, the above argument
assumes that the stock values over the period of interest are
randomly fluctuating without exhibiting any other trend.
Interestingly, such is the case if we consider shorter time
frames such as inter-day trading.
In summary if one must day-trade, then a possible strategy
might be to get in at 9.30 AM, and get out any time after
12 noon (9.30 AM + 0.3679 × 6.5 hrs = 11.54 AM to be
precise) at the first peak that exceeds the peak value between
9.30 AM and 12 noon. In that case chances are about 37%
that one hits the absolute top value for that day! (disclaimer :
Trade at your own risk) 27
PILLAI
We conclude this lecture with a variation of the Game of
craps discussed in Example 3-16, Text.

Example 2.8: Game of craps using biased dice:

From Example 3.16, Text, the probability of
winning the game of craps is 0.492929 for the player.
Thus the game is slightly advantageous to the house. This
conclusion of course assumes that the two dice in question
are perfect cubes. Suppose that is not the case.

Let us assume that the two dice are slightly loaded in such
a manner so that the faces 1, 2 and 3 appear with probability
1 − ε and faces 4, 5 and 6 appear with probability
6
1 + ε , ε > 0 for each dice. If T represents the combined
6
total for the two dice (following Text notation), we get 28
PILLAI
p4 = P{T = 4} = P{(1,3),(2, 2),(1,3)} = 3( 16 − ε ) 2
1 − ε 2 ) + 2( 1 − ε ) 2
p5 = P{T = 5} = P{(1, 4),(2,3),(3, 2),(4,1)} = 2( 36 6
1 − ε 2 ) + ( 1 − ε )2
p6 = P{T = 6} = P{(1,5),(2, 4),(3,3),(4, 2),(5,1)} = 4( 36 6
1 −ε2)
p7 = P{T = 7} = P{(1,6),(2,5),(3, 4),(4,3),(5, 2),(6,1)} = 6( 36
1 − ε 2 ) + ( 1 + ε )2
p8 = P{T = 8} = P{(2,6),(3,5),(4, 4),(5,3),(6, 2)} = 4( 36 6
1 − ε 2 ) + 2( 1 + ε ) 2
p9 = P{T = 9} = P{(3,6),(4,5),(5, 4),(6,3)} = 2( 36 6
p10 = P{T = 10} = P{(4,6),(5,5),(6, 4)} = 3( 16 + ε )2
p11 = P{T = 11} = P{(5,6),(6,5)} = 2( 16 + ε ) 2 .

(Note that “(1,3)” above represents the event “the first dice
shows face 1, and the second dice shows face 3” etc.)
For ε = 0.01, we get the following Table:
29
PILLAI
 T=k 4 5 6 7 8 9 10 11
pk = P{T = k} 0.0706 0.1044 0.1353 0.1661 0.1419 0.1178 0.0936 0.0624

This gives the probability of win on the first throw to be

(use (3-56), Text)
P1 = P(T = 7) + P(T = 11) = 0.2285 (2-47)
and the probability of win by throwing a carry-over to be
(use (3-58)-(3-59), Text)
10 pk2
P2 = ∑ = 0.2717 (2-48)
k =4 pk + p7
k ≠7

Thus
P{winning the game} = P1 + P2 = 0.5002 (2-49)
Although perfect dice gives rise to an unfavorable game,
30
PILLAI
a slight loading of the dice turns the fortunes around in
favor of the player! (Not an exciting conclusion as far as
the casinos are concerned).
Even if we let the two dice to have different loading
factors ε1 and ε 2 (for the situation described above), similar
conclusions do follow. For example, ε1 = 0.01 and ε 2 = 0.005
gives (show this)
P{winning the game} = 0.5015. (2-50)
Once again the game is in favor of the player!
Although the advantage is very modest in each play, from
Bernoulli’s theorem the cumulative effect can be quite
significant when a large number of game are played.
All the more reason for the casinos to keep the dice in
perfect shape. 31
PILLAI
 In summary, small chance variations in each game
of craps can lead to significant counter-intuitive changes
when a large number of games are played. What appears
to be a favorable game for the house may indeed become
an unfavorable game, and when played repeatedly can lead
to unpleasant outcomes.

32