Scribd
Upload
69 views13 pages

Exponential Family Distributions

The document discusses several key concepts in exponential families and generalized linear models (GLM): 1. Exponential families are probability distributions that can be written in a particular form involving natural parameters and sufficient statistics. Common examples include normal, binomial, Poisson, and gamma distributions. 2. GLMs extend linear regression by allowing non-normal distributions and linking the systematic component to the random component through a link function. 3. The canonical link in a GLM sets the natural parameter equal to the linear predictor. Common canonical links include identity for normal, logit for binomial, and log for Poisson distributions. 4. Logistic regression applies a logit link to model binomial outcomes as a function of predictors. It estimates

Uploaded by

Liban Ali Mohamud
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
69 views13 pages

Exponential Family Distributions

The document discusses several key concepts in exponential families and generalized linear models (GLM): 1. Exponential families are probability distributions that can be written in a particular form involving natural parameters and sufficient statistics. Common examples include normal, binomial, Poisson, and gamma distributions. 2. GLMs extend linear regression by allowing non-normal distributions and linking the systematic component to the random component through a link function. 3. The canonical link in a GLM sets the natural parameter equal to the linear predictor. Common canonical links include identity for normal, logit for binomial, and log for Poisson distributions. 4. Logistic regression applies a logit link to model binomial outcomes as a function of predictors. It estimates

Uploaded by

Liban Ali Mohamud
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Exponential Family

• Suppose Y1 , · · · , Yn are independent random


variables.
• Let f (yi ; θi , ϕ) be PMF or PDF of Yi , where
ϕ is a scale parameter.
• If we can write
[ yi θi − b(θi )
f (yi ; θi , ϕ) = exp + c(yi , ϕ)],
a(ϕ)
then we call the PMF or the PDF f (yi ; θi , ϕ)
is an exponential family.

1
Normal Distribution

Assume Yi ∼ N (µi , σ 2 ). Then, E(Yi ) = µi and σ


is a scale parameter. The PDF is
1 (y −µ )2
− i 2i
√ e 2σ
2πσ
yi µi − µ2i /2 1 yi2
= exp{ + [− log(2πσ ) − 2 ]}.
2
σ2 2 2σ
We may use θi = µi , b(θi ) = θi2 /2, ϕ = σ 2 ,
a(ϕ) = ϕ, c(yi , ϕ) = −(1/2) log(2πϕ) − yi2 /(2ϕ).

2
Binomial Distribution

Assume Yi ∼ Bin(ni , pi ). Then, E(Yi ) = ni pi .


The PMF is
( )
ni yi
pi (1 − pi )ni −yi
yi
( )
pi ni
= exp{yi log + ni log(1 − pi ) − log }.
1 − pi yi

Thus, θi = log[pi /(1 − pi )], b(θi ) = ni log(1 + eθi ),


(ni )
ϕ = 1, a(ϕ) = 1, c(y, ϕ) = − log yi .

3
Poisson Distribution

Assume Yi ∼ P oisson(λi ). Then, E(Yi ) = λi .


The PMF is
λyi i −λi
e
yi !
= exp{yi log(λi ) − λi − log(yi !)}.

Thus, θi = log(λi ), b(θi ) = eθi , ϕ = 1, a(ϕ) = 1,


c(yi , ϕ) = − log(yi !).

4
Gamma Distribution

Assume xi ∼ Γ(α, βi ), βi is unknown. Then,


E(xi ) = α/βi . Then PMF is

βiα xα−1
i
e−βi xi = exp{α log xi + α log(βi )
Γ(α)
− log(Γ(α)) − log(xi ) − βi xi }.

Assume α is known. If we choose yi = xi , then


θi = −βi (θi < 0), b(θi ) = −α log(−θi ), ϕ = 1 and
a(ϕ) = 1.
Remark: We can also choose yi = −xi and
θi = βi . Then, b(θi ) = −α log θi .

5
Negative Binomial Distribution

Assume Xi ∼ N B(k, pi ). The PDF is


( )
xi − 1 k
pi (1 − pi )xi
k−1
( )
pi xi − 1
= exp{xi log(1 − pi ) + k log − log },
1 − pi k−1
for xi = 0, 1, · · ·. We choose yi = xi . Then,
θi = log(1 − pi ), b(θi ) = −k log[(1 − eθi )/eθi ],
(yi −1)
ϕ = 1, a(ϕ) = 1 and c(yi , ϕ) = − log k−1 . Then,

k k
E(Yi ) = E(Xi ) = b′ (θi ) = =
1 − eθi pi
and

′′ keθi k(1 − pi )
V (Yi ) = V (Xi ) = b (θi ) = = .
(1 − eθi )2 p2i

6
GLM

The definition of Generalized Linear Model


(GLM) is based on exponential family. There are
three components in GLM. They are
• Random component. Assume the
distributions of the sample. Such as normal,
binomial, Poisson and etc.
• Systematic component. Describe the form of
predictor (independent) variables. Such as

ηi = β0 + β1 xi1 + · · · + βip xip .

• Link function. Connect the unknown


parameters to model. Such as

g[µi (θi )] = ηi

for some g(·), where µi (θi ) = E(yi ) is the


expected value.

7
Canonical Link

If θi = ηi (or simply write θ = η), then the


canonical link is derived.
• Normal: identity link g(µi ) = µi or simply
write g(µ) = µ (same as below).
• Binomial: logistic link g(µ) = log µ
1−µ .

• Poisson: log link g(µ) = log(µ).


• Gamma: negative inverse link g(µ) = −1/µ.
• Negative binomial: g(µ) = log[µ/k(1 + µ/k)].

8
• The most important cases are binomial and
Poisson.
• Canonical link is just one of the link
functions.
• Estimation is based on the maximum
likelihood approach.
• Except the normal case, numerical
computation is needed.

9
Link for Binomial

There are three link functions for binomial.


• Logistic link.

pi ∑p
log = β0 + xij βj
1 − pi j=1

called logistic linear model or logistic


regression.
• Inverse CDF link.
∑
p
F −1 (pi ) = β0 + xij βj .
j=1

If F = Φ, it is the probit link, called probit


model.
• Complementary loglog link.
∑
p
log[− log(1 − pi )] = β0 + xij βj .
j=1

10
Logistic Regression

Consider the simplest case. That is


pi
log( ) = β0 + β1 xi .
1 − pi

Suppose β̂ and β̂1 are the MLEs.


• Odds ratio: as x increases a units, the
estimate of odds ratio is eaβ̂1 .
• The significance of the odds ratio can be
directly read by the p-value of β1 .
• Confidence interval can also be derived
respectively.

11
An example

Table 1: Blood Pressure and Heart Disease


Blood Heart Disease
Pressure Present Absent
< 117 3 153
117 − 126 17 235
127 − 136 12 272
137 − 146 16 255
147 − 156 12 127
157 − 166 8 77
167 − 186 16 83
> 186 8 35

12
Goodness of Fit

Let n̂ij be the predicted counts of the model.


• Pearson χ2 is
∑I ∑ J
(nij − n̂ij )2
2
X = .
i=1 j=1
n̂ ij

• Loglikehood ratio χ2 is
∑
I ∑
J
G2 = 2 nij log(nij /n̂ij ).
i=1 j=1

13

You might also like