Example 4

Let 𝑋1 , 𝑋2 ,…, 𝑋𝑛 be a random sample of size 𝑛 from a

Poisson distribution having pmf
𝑒 −λ λ 𝑥
𝑓 𝑥; λ = , 𝑥 = 0,1,2, … … , λ > 0
𝑥𝑖 !
Find the maximum likelihood estimator of λ.

Solution
𝑛

𝐿 𝑥1 , 𝑥2 , … … , 𝑥𝑛 ; λ = ෑ 𝑓 𝑥𝑖 ; λ
𝑖=1
= 𝑓 𝑥1 ; λ . 𝑓 𝑥2 ; λ … … 𝑓 𝑥𝑛 ; λ

1
 Theory of Statistics II
SA302
Lecture 5

Course Instructor: Reham Elshaer

Email: rehamelshaer@feps.edu.eg
2
 Example 4 contd.
𝑒 −λ λ𝑥1 𝑒 −λ λ 𝑥2 𝑒 −λ λ 𝑥𝑛
= . …..
𝑥1 ! 𝑥2 ! 𝑥𝑛 !
−𝑛λ σ𝑛
𝑒 λ 𝑖=1 𝑥𝑖
=
ς𝑛𝑖=1 𝑥𝑖 !
𝑛 𝑛

𝐿𝑛 𝐿 𝑥; λ = −𝑛λ + ෍ 𝑥𝑖 ln λ − ln ෑ 𝑥𝑖 !
𝑖=1 𝑖=1
𝑑 ln 𝐿 𝑥; λ σ𝑛𝑖=1 𝑥𝑖
= −𝑛 + . 1 =0
𝑑λ λ
σ𝑛𝑖=1 𝑥𝑖
𝑛=
λ෠
σ 𝑋𝑖
λ෠ = =𝑋
𝑛
You can check that λ෠ is a maximum point. 3
Properties of Estimators

4
Properties of Estimators
1. Unbiasedness:
An estimator 𝑇 is defined to be an unbiased estimator
of 𝐾(𝜃) if and only if
𝐸 𝑇 = 𝐾(𝜃)

Example 1: Show that 𝑋ത is an unbiased estimator for 𝜇.

Solution

σ 𝑋𝑖 1 1 1
𝐸 𝑋ത = 𝐸 = ෍ 𝐸 𝑋𝑖 = ෍ 𝜇 = 𝑛𝜇 = 𝜇
𝑛 𝑛 𝑛 𝑛
5
Properties of Estimators
Example 2:
Let 𝑋1 , 𝑋2 ,…, 𝑋𝑛 be a random sample from probability
distribution
𝑓(𝑥; 𝜇, 𝜎 2 ) where 𝐸 𝑋 = 𝜇 and Var 𝑋 = 𝜎 2 .
1
2
Define 𝑆 = ത 2 as an estimator for 𝜎 2 .
σ(𝑋𝑖 − 𝑋)
𝑛−1
Show that 𝑆 is an unbiased estimator for 𝜎 2 .
2

6
Properties of Estimators

Solution
1
𝑆 =2
෍(𝑋𝑖 − 𝑋) ത 2
𝑛−1
1
= ෍(𝑋𝑖 2 − 2𝑋𝑖 𝑋ത + 𝑋ത 2 )
𝑛−1
1
= (෍ 𝑋𝑖 2 − 2𝑛. 𝑋ത 𝑋ത + 𝑛𝑋ത 2 )
𝑛−1
1
= (෍ 𝑋𝑖 2 − 2𝑛. 𝑋ത 2 + 𝑛𝑋ത 2 )
𝑛−1

7
Properties of Estimators
Solution contd.
1
= (෍ 𝑋𝑖 2 − 𝑛𝑋ത 2 )
𝑛−1
1 2
𝐸 𝑆2 = (෍ 𝐸 𝑋𝑖 − 𝑛𝐸 𝑋ത 2 )
𝑛−1
2
1 𝜎
= (෍(𝜇 2 + 𝜎 2 ) − 𝑛(𝜇 2 + ))
𝑛−1 𝑛
1
= 𝑛𝜇 2 + 𝑛𝜎 2 − 𝑛𝜇 2 − 𝜎 2
𝑛−1
1
= 𝑛 − 1 𝜎2 = 𝜎2
𝑛−1
∴ 𝑆 2 is an unbiased estimator of 𝜎 2 . 8
Properties of Estimators
Example 3:
1 2
If 𝜎ො = σ 𝑋𝑖 − 𝑋ത
2
is an estimator of 𝜎 2 .
𝑛
Show that 𝜎ො 2 is a biased estimator.

Solution 1
𝜎ො = ෍ 𝑋𝑖 − 𝑋ത
2 2
𝑛
1
𝐸(𝜎ො ) = 𝐸 ෍ 𝑋𝑖 − 𝑋ത
2 2
𝑛
From Example 2, 𝐸 σ 𝑋𝑖 − 𝑋ത 2 = (𝑛 − 1)𝜎 2
1
∴ 𝐸(𝜎ො ) = (𝑛 − 1)𝜎 2 ≠ 𝜎 2
2
𝑛
∴ 𝜎ො 2 is biased estimator for 𝜎 2 9
Properties of Estimators
Definition of Bias
The quantity 𝐸 𝑇 − 𝐾(𝜃) where 𝑇 is an estimator for
𝐾(𝜃) is called the bias of the estimator 𝑇.
It can either be positive or negative.
Example: In the previous example, find the bias of 𝜎ො 2 .
Solution
Bias = 𝐸 𝜎ො 2 − 𝜎 2
𝑛−1
=( 𝜎2 − 𝜎2)
𝑛
2 𝑛−1
= 𝜎 ( − 1)
𝑛
−𝜎 2
= 10
𝑛
Properties of Estimators
Definition of Asymptotically Unbiased Estimator
We say that 𝑇 is an asymptotically unbiased estimator of
𝐾 𝜃 if lim 𝐸(𝑇) = 𝐾 𝜃 .
𝑛→∞
Example 1: Show that 𝜎ො 2 is asymptotically unbiased
estimator for 𝜎 2 .
Solution
2
1
lim 𝐸(𝜎ො ) = lim 𝐸( ෍ 𝑋𝑖 − 𝑋ത )
2
𝑛→∞ 𝑛→∞ 𝑛
(𝑛−1)
= lim 𝜎2
𝑛→∞ 𝑛
2 (𝑛−1)
=𝜎 lim = 𝜎2
𝑛→∞ 𝑛 11
Properties of Estimators

Note that:
One of the properties of maximum likelihood
estimators is that they are asymptotically unbiased.
However, moments’ estimators are not necessarily
asymptotically unbiased.

12
Properties of Estimators
Example 2:
Let 𝑋1 , 𝑋2 ,…, 𝑋𝑛 be a random sample from a uniform
distribution 𝛼, 𝛽 where 𝛼 = 0. Show that the nth
order statistic 𝑌𝑛 is a biased estimator of the parameter
𝛽. Can you modify this estimator to make it unbiased?
Solution
1
𝑓𝑋 𝑥; 𝛽 = 0≤𝑥≤𝛽
𝛽

𝑥1 𝑥
For 0 ≤ 𝑥 ≤ 𝛽, 𝐹𝑋 𝑥 = ‫׬‬0 𝛽 𝑑𝑥 =
𝛽
13
Properties of Estimators
Solution contd.
0 𝑥<0
𝑥
𝐹𝑋 𝑥 = 0≤𝑥≤𝛽
𝛽
1 𝑥>𝛽

𝑓𝑌𝑛 𝑦 = 𝑛(𝐹𝑋 𝑦 )𝑛−1 𝑓𝑋 𝑦

𝑛−1
𝑦 1
=𝑛 .
𝛽 𝛽
𝑛 𝑛−1
= 𝑛
. 𝑦 0<𝑦<𝛽 14
Properties of Estimators
Solution contd.
𝛽
𝑛
𝐸 𝑌𝑛 = න 𝑛 . 𝑦𝑛 𝑛 𝑑𝑦𝑛
0 𝛽

𝑛 𝑦𝑛 𝑛+1 𝜷
= 𝑛
𝛽 𝑛+1
𝑛 𝑛+1
𝑛𝛽
= 𝑛 𝛽 = ≠𝛽
𝛽 (𝑛 + 1) 𝑛+1

∴ 𝑌𝑛 is a biased estimator of 𝛽
15
Properties of Estimators
Solution contd.

This estimator can be modified to make it unbiased.

𝑛+1
𝑛
𝑌𝑛 is an unbiased estimator of 𝛽.

Also, note that 𝑌𝑛 is an asymptotically unbiased

estimator of 𝛽 since
𝑛𝛽
lim =𝛽
𝑛→∞ 𝑛 + 1

16