STA732
Statistical Inference
Lecture 23: Large-Sample Theory for Likelihood Ratio Tests
Yuansi Chen
Spring 2022
Duke University
https://www2.stat.duke.edu/courses/Spring22/sta732.01/
1
Recap from Lecture 22
1. Canonical linear model
𝑍0 𝜇0
⎛ ⎞ ⎛ ⎛ ⎞ ⎞
⎜ ⎜
𝑍 = ⎜𝑍1 ⎟ ∼ 𝒩 ⎜⎜𝜇1 ⎟
⎟ ⎜ ⎟ , 𝜎𝕀𝑛 ⎟
⎟
𝑍
⎝ 𝑟⎠ ⎝⎝ ⎠0 ⎠
• 𝜎2 known, 𝑑1 = 1, 𝑍-test: 𝑍𝜎1
• 𝜎2 unknown, 𝑑1 = 1, 𝑡-test: 𝑍𝜎1̂
2
‖𝑍1 ‖2
• 𝜎2 known, 𝑑1 ≥ 1, 𝜒2 -test: 𝜎2
2
‖𝑍 ‖ /𝑑
• 𝜎2 unknown, 𝑑1 ≥ 1, 𝐹 -test: 1𝜎2̂ 2 1
2. General linear model: find an orthonormal matrix 𝑄 such that
𝑄⊤ 𝑌 follows the canonical linear model
2
Goal of Lecture 22
1. Wald test
2. Score test
3. Generalized likelihood ratio test
Chap. 17.1-3 of Keener or 12.4 in Lehmann and Romano
3
Review the asymptotics of MLE
Setup
i.i.d.
𝑋1 , … , 𝑋𝑛 ∼ 𝑝𝜃 (𝑥), 𝑝𝜃 (⋅) is “regular” enough (check the
conditions in Thm 9.14 of Keener)
4
Consistency of MLE on compact Ω
Define
𝑊𝑖 (𝜃) = ℓ1 (𝜃; 𝑋𝑖 ) − ℓ1 (𝜃0 ; 𝑋𝑖 )
1 𝑛
𝑊̄ 𝑛 = ∑ 𝑊𝑖
𝑛 𝑖=1
We know that
𝔼𝑊𝑖 (𝜃) = −𝒟KL (𝜃0 ‖ 𝜃) ≤ 0
and it becomes = 0 iff 𝑃𝜃 = 𝑃𝜃0 .
Consistency result
If model is identifiable, 𝑊𝑖 continuous random function, then
𝑝
• ∥𝑊̄ 𝑛 − 𝔼𝑊̄ 𝑛 ∥∞ → 0 on compact Ω.
𝑝
• Then 𝜃𝑛̂ → 𝜃0 (convergence of argmax requires uniform convergence
5
result in Thm 9.4 Keener)
Asymptotic distribution of MLE
MLE satisfies
0 = ∇ℓ𝑛 (𝜃𝑛̂ ) = ∇ℓ𝑛 (𝜃0 ) + ∇2 ℓ𝑛 (𝜃𝑛̃ )(𝜃𝑛̂ − 𝜃0 ).
Then
−1
√ 1 1
𝑛(𝜃𝑛̂ − 𝜃0 ) = (− ∇2 ℓ𝑛 (𝜃𝑛̃ )) ( √ ∇ℓ𝑛 (𝜃0 ))
𝑛 𝑛
−1 𝑝
• (− 𝑛1 ∇2 ℓ𝑛 (𝜃𝑛̃ )) → 𝐼1 (𝜃0 )−1 (convergence of a random function
evaluated on a random point requires uniform convergence result in Thm
9.4 Keener!)
• √1 ∇ℓ (𝜃 ) ⇒ 𝒩(0, 𝐼1 (𝜃0 )) (CLT)
𝑛 𝑛 0
√
By Slutsky’s thm, 𝑛(𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝐼1 (𝜃0 )−1 )
6
√
𝑛(𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝐼1 (𝜃0 )−1 )
We can use the asymptotic distribution to compute confidence
regions!
7
Wald test
Intuition for Wald-type confidence regions (1)
Assume we have an estimator 𝐼𝑛̂ ⪰ 0 such that
1 ̂ 𝑝
𝐼 → 𝐼1 (𝜃0 )
𝑛 𝑛
Then we can use it as plug-in estimate for 𝐼1 (𝜃0 ) in asymptotic
distribution
√
Since 𝑛(𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝐼1 (𝜃0 )−1 ),
1/2 √
then (𝐼1 (𝜃0 )) 𝑛(𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝕀𝑑 ),
by Slutsky’s thm,
1/2
𝐼𝑛̂ (𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝕀𝑑 )
8
Intuition for Wald-type confidence regions (2)
Under the null hypothesis 𝐻0 ∶ 𝜃 = 𝜃0 , we have
1/2 2
∥𝐼𝑛̂ (𝜃𝑛̂ − 𝜃0 )∥ ⇒ 𝜒2𝑑
2
We can construct a test that rejects for large value of
1/2 2
∥𝐼𝑛̂ (𝜃𝑛̂ − 𝜃0 )∥ :
2
𝜙=1 1/2
2
̂ −𝜃 )∥ >𝜒2 (𝛼)
∥𝐼𝑛̂ (𝜃𝑛 0 𝑑
2
Remark
• The test might not have the correct level. It only has
asymptotic level 𝛼
• The confidence region is an ellipsoid
−1/2
𝜃𝑛̂ + 𝐼𝑛̂ 𝔹(0, 𝜒2𝑑 (𝛼))
9
Two options for 𝐼𝑛̂
1. 𝐼𝑛 (𝜃𝑛̂ ) obtained by plugging in the MLE
𝐼𝑛̂ = 𝐼𝑛 (𝜃𝑛̂ )
= Var𝜃 (∇ℓ𝑛 (𝜃; 𝑋)) ∣𝜃=𝜃 ̂
𝑛
2. Observed Fisher information
𝐼𝑛̂ = −∇2 ℓ𝑛 (𝜃𝑛̂ ; 𝑋)
Remark:
𝑝
Both should have 𝑛1 𝐼𝑛̂ → 𝐼1 (𝜃0 ) in “regular” model i.i.d. setting
10
Wald interval for 𝜃𝑗
√
Since 𝑛(𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝐼1 (𝜃0 )−1 ),
then by multiplying (1, 0, … , 0)⊤ , we obtain
√
̂ − 𝜃 ) ⇒ 𝒩(0, (𝐼 (𝜃 )−1 ) )
𝑛(𝜃𝑛,𝑗 0,𝑗 1 0 𝑗𝑗
Using 𝑛1 𝐼𝑛̂ as plug-in estimate for 𝐼1 (𝜃0 ), we obtain univariate
interval
̂ ± √(𝐼 −1
𝐶𝑗 = 𝜃𝑛,𝑗 ̂
𝑛 ) ⋅ 𝑧𝛼/2
𝑗𝑗
11
Wald interval for 𝜃𝑗
√
Since 𝑛(𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝐼1 (𝜃0 )−1 ),
then by multiplying (1, 0, … , 0)⊤ , we obtain
√
̂ − 𝜃 ) ⇒ 𝒩(0, (𝐼 (𝜃 )−1 ) )
𝑛(𝜃𝑛,𝑗 0,𝑗 1 0 𝑗𝑗
Using 𝑛1 𝐼𝑛̂ as plug-in estimate for 𝐼1 (𝜃0 ), we obtain univariate
interval
̂ ± √(𝐼 −1
𝐶𝑗 = 𝜃𝑛,𝑗 ̂
𝑛 ) ⋅ 𝑧𝛼/2
𝑗𝑗
glm function in R uses the above intervals:
with 𝐼𝑛̂ = −∇2 ℓ𝑛 (𝜃𝑛̂ )
11
Confidence ellipsoid for 𝜃0,𝑆
Want to provide confidence ellipsoid for 𝜃0,𝑆 = (𝜃0,𝑗 )𝑗∈𝑆 , |𝑆| = 𝑘
We have
√
̂ − 𝜃 ) ⇒ 𝒩(0, (𝐼 (𝜃 )−1 ) )
𝑛 (𝜃𝑛,𝑆 0,𝑆 1 0 𝑆𝑆
Then the confidence ellipsoid is
1/2
̂ + ((𝐼 −1
𝜃𝑛,𝑆 ̂
𝑛 )𝑆𝑆 ) 𝔹(0, 𝜒𝑘 (𝛼))
12
Example: generalized linear model with fixed design
Suppose 𝑥1 , … , 𝑥𝑛 ∈ ℝ𝑑 fixed
ind.
𝑌𝑖 ∼ 𝑝𝜂𝑖 (𝑦𝑖 ) = 𝑒𝜂𝑖 𝑦𝑖 −𝐴(𝜂𝑖 ) ℎ(𝑦𝑖 )
where 𝜂𝑖 = 𝛽 ⊤ 𝑥𝑖
Link function
Let 𝜇𝑖 (𝛽) = 𝔼𝛽 𝑌𝑖 . If 𝑓(𝜇𝑖 ) = 𝛽 ⊤ 𝑥𝑖 , then 𝑓 is called link function.
Common examples
⊤
ind. 𝑒𝑥𝑖 𝛽
• Logistic regression: 𝑌𝑖 ∼ Bernoulli ( ⊤ )
1+𝑒𝑥𝑖 𝛽
ind. ⊤
• Poisson log-linear model: 𝑌𝑖 ∼ Poisson(𝑒𝑥𝑖 𝛽 )
13
Confidence interval in generalized linear model
𝑛
ℓ𝑛 (𝛽; 𝑌 ) = ∑(𝑥⊤ ⊤
𝑖 𝛽)𝑦𝑖 − 𝐴(𝑥𝑖 𝛽) − log ℎ(𝑦𝑖 )
𝑖=1
𝑛
∇ℓ𝑛 (𝛽; 𝑌 ) = ∑ 𝑦𝑖 𝑥𝑖 − 𝐴′ (𝑥⊤
𝑖 𝛽)𝑥𝑖
𝑖=1
𝑛
= ∑ (𝑦𝑖 − 𝜇𝑖 (𝛽)) 𝑥𝑖
𝑖=1
𝑛
−∇2 ℓ𝑛 (𝛽; 𝑌 ) = ∑ 𝐴″ (𝑥⊤ ⊤
𝑖 𝛽)𝑥𝑖 𝑥𝑖
𝑖=1
𝑛
= ∑ Var𝛽 (𝑦𝑖 )𝑥𝑖 𝑥⊤
𝑖
𝑖=1
= Var𝛽 (∇ℓ𝑛 (𝛽; 𝑌 ))
in GLM, −∇2 ℓ𝑛 (𝛽; 𝑌 ) is not random 14
Can estimate 𝐼𝑛̂ by plug-in MLE
Apply our asymptotic directly (or do Taylor expansion from scracth)
1/2
𝐼𝑛̂ (𝜃𝑛̂ − 𝜃0 ) ⇒ 𝒩(0, 𝕀𝑑 )
15
Pros and cons of Wald test
Advantages
• Easy to invert, simple confidence regions
• Asympotically correct level
Disadvantages
• Have to compute MLE
• Depends on parameterization
• Relies on second order Taylor expansion of ℓ𝑛
• Need MLE to be consistent
• Confidence region might go outside of Ω
16
Score test
Intuition for score test
Testing 𝐻0 ∶ 𝜃 = 𝜃0 vs. 𝐻1 ∶ 𝜃 ≠ 𝜃0
We can bypass quadratic approximation by using the score as test
statistics
1
√ ∇ℓ𝑛 (𝜃0 ) ⇒ 𝒩(0, 𝐼1 (𝜃0 ))
𝑛
17
Score test
Reject 𝐻0 ∶ 𝜃 = 𝜃0 if
2
∥𝐼𝑛 (𝜃0 )−1/2 ∇ℓ𝑛 (𝜃0 )∥2 ≥ 𝜒2𝑑 (𝛼)
if 𝑑 = 1, we just use 𝑍-test instead
18
Score test
Reject 𝐻0 ∶ 𝜃 = 𝜃0 if
2
∥𝐼𝑛 (𝜃0 )−1/2 ∇ℓ𝑛 (𝜃0 )∥2 ≥ 𝜒2𝑑 (𝛼)
if 𝑑 = 1, we just use 𝑍-test instead
Advantages of score test
• No quadratic approximation
• No MLE
Disadvantage is that it might not be easy to invert the test
18
Score test is invariant to reparameterization
Assume 𝑑 = 1, 𝜃 = 𝑔(𝜉) with 𝑔′ (𝜉) > 0,
𝑞𝜉 (𝑥) = 𝑝𝑔(𝜉) (𝑥),
show that the two test statistics are the same a.s.
19
Example 1: 𝑠-parameter exponential family
i.i.d.
Suppose 𝑋1 , … , 𝑋𝑛 ∼ 𝑝𝜂 (𝑥) = exp(𝜂⊤ 𝑇 (𝑥) − 𝐴(𝜂))ℎ(𝑥). Derive
score test for 𝐻0 ∶ 𝜂 = 𝜂0
20
Example 2: Pearson 𝜒2 test
Suppose 𝑁 = (𝑁1 , … , 𝑁𝑑 ) ∼ Multinom(𝑛, (𝜋1 , … , 𝜋𝑑 )), with
density
𝑁 𝑁
𝑛!𝜋1 1 ⋯ 𝜋𝑑 𝑑
1
𝑁1 ! ⋯ 𝑁𝑑 ! ∑ 𝑁𝑖 =𝑛
𝑑
Note since ∑𝑗=1 𝜋𝑗 = 1, this is a full-rank (𝑑 − 1)-param exp family,
with the possible parameterization
⎧ 1
{ 1+∑𝑘>1 𝑒𝜂𝑘 𝑗=1
𝜋𝑗 =
⎨
{ 𝑒𝜂 𝑗
𝑗>1
⎩ 1+∑𝑘>1 𝑒𝜂𝑘
Derive score test.
21
Generalized likelihood ratio test
GLRT in simple vs composite two-sided testing
Testing 𝐻0 ∶ 𝜃 = 𝜃0 vs. 𝐻1 ∶ 𝜃 ≠ 𝜃0
Taylor expansion around 𝜃𝑛̂ gives
1
ℓ𝑛 (𝜃0 ) − ℓ𝑛 (𝜃𝑛̂ ) = ∇ℓ(𝜃𝑛̂ ) + (𝜃0 − 𝜃𝑛̂ )⊤ ∇2 ℓ𝑛 (𝜃𝑛̃ )(𝜃0 − 𝜃𝑛̂ )
2
2
1/2
1 1 √
= 0 − ∥(− ∇2 ℓ𝑛 (𝜃𝑛̃ )) ( 𝑛(𝜃0 − 𝜃𝑛̂ ))∥
2 𝑛
2
1
⇒ − 𝜒2𝑑
2
why?
Test statistic in GLRT
2 log(𝜆) = 2 (ℓ𝑛 (𝜃𝑛̂ ) − ℓ𝑛 (𝜃0 )) ⇒ 𝜒2𝑑
22
GLRT in composite vs composite
Testing 𝐻0 ∶ 𝜃 ∈ Ω0 vs. 𝐻1 ∶ 𝜃 ∈ Ω\Ω0
The generalized likelihood ratio is
supΩ 𝐿(𝜃)
𝜆= 1
supΩ 𝐿(𝜃)
0
The test statistic is
2 log(𝜆) = 2 (ℓ𝑛 (𝜃𝑛̂ ) − ℓ𝑛 (𝜃0̂ ))
where 𝜃0̂ = arg max𝜃∈Ω0 ℓ𝑛 (𝜃)
23
Asympotitic distribution of 2 log(𝜆)
Asymptotic distribution of 2 log(𝜆), see 17.2 Keener
Assume Ω = ℝ𝑑 , Ω0 𝑑0 -dim subspace. 𝜃0 in interior of Ω0 , 𝜃𝑛̂ is
consistent, 𝑝𝜃 (⋅) is “regular” (as in the asymptotic of MLE), then
2 log(𝜆) = 2 (ℓ𝑛 (𝜃𝑛̂ ) − ℓ𝑛 (𝜃0̂ )) ⇒ 𝜒2𝑑−𝑑0
where 𝜃0̂ = arg max𝜃∈Ω0 ℓ𝑛 (𝜃)
24
Intuition for the asymptotic distribution
(See rigorous derivation in 17.2 Keener)
Assume 𝜃0 = 0, 𝐼0 (0) = 𝕀𝑑 (after reparameterization), then
• 𝜃𝑛̂ ≈ 𝒩(𝜃0 , 𝑛1 𝕀𝑑 )
• locally, ∇2 ℓ𝑛 (𝜃) ≈ 𝑛𝕀𝑑 near 𝜃0
2
• ℓ𝑛 (𝜃) − ℓ𝑛 (𝜃𝑛̂ ) ≈ 𝑛
2 ∥𝜃 − 𝜃𝑛̂ ∥
2
2
• 𝜃0̂ ≈ arg min𝜃∈Ω0 ∥𝜃 − 𝜃∥̂ = ProjΩ (𝜃𝑛̂ )
2 0
•
2
2 (ℓ𝑛 (𝜃𝑛̂ ) − ℓ𝑛 (𝜃0̂ )) ≈ 𝑛 ∥𝜃𝑛̂ − ProjΩ (𝜃𝑛̂ )∥
0 2
⇒ 𝜒2𝑑−𝑑0
25
Asymptotic equivalence of the three tests
How close are the three tests asymptotically?
1/2 2
• Wald test: ∥𝐽𝑛̂ (𝜃𝑛̂ − 𝜃0 )∥
2
1/2 2
• Score test: ∥𝐽𝑛 (𝜃0 ) ∇ℓ𝑛 (𝜃0 )∥2
• GLRT: ℓ𝑛 (𝜃𝑛̂ ) − ℓ𝑛 (𝜃0 )
all are related to (for large 𝑛)
2
∥𝐼𝑛 (𝜃0 )1/2 (𝜃𝑛̂ − 𝜃0 )∥
2
26
Summary
• Wald test: test statistic based on quadratic approx
• Score test: test statistic using score
• Generalized likelihood ratio test: 2 log(𝜆)
We intuitively derived its asympotitic distribution
Read Page 362 of Keener for strengths and weaknesses
27
What is next?
• Final review
28
Thank you for attending
See you on Wednesday in Old
Chem 025
29
30
