Lecture 4: Multiple Linear

Regression Models

Bangladesh University of Eng. & Tech. Slide 1 of 33 Industrial &Production Engineering

 Outline
• Introduction
• Estimating the Coefficients
• Linear Regression Model Using Matrices
• Properties of the Least Squares Estimators
• Analysis of Variance in Multiple Regression
• Inferences in Multiple Linear Regression
• Individual t-Tests for Variable Screening
• Inferences on Mean Response and Prediction
• Choice of a Fitted Model through Hypothesis Testing
• Study of Residuals and Violation of Assumptions (Model Checking)

Bangladesh University of Eng. & Tech. Slide 2 of 33 Industrial &Production Engineering

 Multiple Linear Regression Models

• The complexity of most scientific mechanisms is such

that in order to be able to predict an important response,
a multiple regression model is needed.
• When this model is linear in the coefficients, it is called a
multiple linear regression model.

Bangladesh University of Eng. & Tech. Slide 3 of 33 Industrial &Production Engineering

 Estimating the Coefficients
• We obtain the least squares estimators of the
parameters β0, β1, . . . , βk by fitting the multiple linear
regression model to the data points.

• where yi is the observed response to the values x1i, x2i, .

. . , xki of the k independent variables x1, x2, . . . , xk.
• Each observation (x1i, x2i, . . . , xki, yi) is assumed to satisfy
the following equation.

Bangladesh University of Eng. & Tech. Slide 4 of 33 Industrial &Production Engineering

 Estimating the Coefficients

• where εi and ei are the random error and residual,

respectively, associated with the response yi and fitted
value 𝑦! i.
• As in the case of simple linear regression, it is assumed
that the εi are independent and identically distributed
with mean 0 and common variance σ2.

Bangladesh University of Eng. & Tech. Slide 5 of 33 Industrial &Production Engineering

 Estimating the Coefficients

• These equations can be solved for b0, b1, b2, . . . , bk by

any appropriate method for solving systems of linear
equations.

Bangladesh University of Eng. & Tech. Slide 6 of 33 Industrial &Production Engineering

 Example 1

Bangladesh University of Eng. & Tech. Slide 7 of 33 Industrial &Production Engineering

 Example 1

Bangladesh University of Eng. & Tech. Slide 8 of 33 Industrial &Production Engineering

 Example 1

Bangladesh University of Eng. & Tech. Slide 9 of 33 Industrial &Production Engineering

 Linear Regression Model Using
Matrices
• In fitting a multiple linear regression model, particularly
when the number of variables exceeds two, a knowledge
of matrix theory can facilitate the mathematical
manipulations considerably.

Bangladesh University of Eng. & Tech. Slide 10 of 33 Industrial &Production Engineering

 Linear Regression Model Using
Matrices

Bangladesh University of Eng. & Tech. Slide 11 of 33 Industrial &Production Engineering

 Linear Regression Model Using
Matrices

• If the matrix A is nonsingular, we can write the solution

for the regression coefficients as

Bangladesh University of Eng. & Tech. Slide 12 of 33 Industrial &Production Engineering

 Example 2

Bangladesh University of Eng. & Tech. Slide 13 of 33 Industrial &Production Engineering

 Example 2

Bangladesh University of Eng. & Tech. Slide 14 of 33 Industrial &Production Engineering

 Properties of the Least Squares
Estimators
• The means and variances of the estimators b0, b1, . . . ,
bk are readily obtained under certain assumptions on the
random errors ε1, ε2, . . . , εk that are identical to those
made in the case of simple linear regression.
• When we assume these errors to be independent, each
with mean 0 and variance σ2, it can be shown that b0, b1,
. . . , bk are, respectively, unbiased estimators of the
regression coefficients β0, β1, . . . , βk.

Bangladesh University of Eng. & Tech. Slide 15 of 33 Industrial &Production Engineering

 Properties of the Least Squares
Estimators

• In addition, the variances of the b’s are obtained through

the elements of the inverse of the A matrix.
• Note that the off-diagonal elements of A = XʹX represent
sums of products of elements in the columns of X, while
the diagonal elements of A represent sums of squares of
elements in the columns of X.
Bangladesh University of Eng. & Tech. Slide 16 of 33 Industrial &Production Engineering
 Properties of the Least Squares
Estimators
• The inverse matrix, A−1, apart from the multiplier σ2,
represents the variance-covariance matrix of the
estimated regression coefficients.
• That is, the elements of the matrix A−1 σ2 display the
variances of b0, b1, . . . , bk on the main diagonal and
covariances on the off-diagonal.
• For example, in a k = 2 multiple linear regression
problem, we might write

Bangladesh University of Eng. & Tech. Slide 17 of 33 Industrial &Production Engineering

 Properties of the Least Squares
Estimators

• Of course, the estimates of the variances and hence the

standard errors of these estimators are obtained by
replacing σ2 with the appropriate estimate obtained
through experimental data.

Bangladesh University of Eng. & Tech. Slide 18 of 33 Industrial &Production Engineering

 Theorem

Bangladesh University of Eng. & Tech. Slide 19 of 33 Industrial &Production Engineering

 Properties of the Least Squares
Estimators

Bangladesh University of Eng. & Tech. Slide 20 of 33 Industrial &Production Engineering

 Analysis of Variance in Multiple
Regression
• An analysis of variance can be conducted to shed light
on the quality of the regression equation.
• A useful hypothesis that determines if a significant
amount of variation is explained by the model is

Bangladesh University of Eng. & Tech. Slide 21 of 33 Industrial &Production Engineering

 Analysis of Variance in Multiple
Regression
• Rejection of H0 implies that the regression equation
differs from a constant.
• That is, at least one regressor variable is important.

Bangladesh University of Eng. & Tech. Slide 22 of 33 Industrial &Production Engineering

 Inferences in Multiple Linear
Regression
• bj (j = 0, 1, 2, . . . , k) are normally distributed with mean
βj and variance cjjσ2. Thus, we can use the statistic

• with n − k − 1 degrees of freedom to test hypotheses and

construct confidence intervals on βj .

Bangladesh University of Eng. & Tech. Slide 23 of 33 Industrial &Production Engineering

 Example 3

Bangladesh University of Eng. & Tech. Slide 24 of 33 Industrial &Production Engineering

 Individual t-Tests for Variable
Screening
• The t-test most often used in multiple regression is the
one that tests the importance of individual coefficients
(i.e., H0: βj = 0 against the alternative H1: βj ≠ 0).
• These tests often contribute to what is termed variable
screening, where the analyst attempts to arrive at the
most useful model (i.e., the choice of which regressors to
use).
• If a coefficient is found insignificant (i.e., the hypothesis
H0: βj = 0 is not rejected), the conclusion drawn is that
the variable is insignificant.

Bangladesh University of Eng. & Tech. Slide 25 of 33 Industrial &Production Engineering

 Inferences on Mean Response &
Prediction

= standard error of prediction

Bangladesh University of Eng. & Tech. Slide 26 of 33 Industrial &Production Engineering

 Example 4

Bangladesh University of Eng. & Tech. Slide 27 of 33 Industrial &Production Engineering

 Inferences on Mean Response and
Prediction

Bangladesh University of Eng. & Tech. Slide 28 of 33 Industrial &Production Engineering

 Example 5

Bangladesh University of Eng. & Tech. Slide 29 of 33 Industrial &Production Engineering

 Coefficient of Determination
• One criterion that is commonly used to illustrate the
adequacy of a fitted regression model is the coefficient of
determination, or R2.

Bangladesh University of Eng. & Tech. Slide 30 of 33 Industrial &Production Engineering

 Adjusted Coefficient of
Determination
• Adjusted R2 is a variation on R2 that provides an
adjustment for degrees of freedom

Bangladesh University of Eng. & Tech. Slide 31 of 33 Industrial &Production Engineering

 Study of Residuals & Violation of
Assumptions

Bangladesh University of Eng. & Tech. Slide 32 of 33 Industrial &Production Engineering

 Assignment-4
• Walpole Chapter-12:
– Problems 12.1, 12.17, 12.21, 12.24, 12.29, 12.30, 12.31, 12.32.

Bangladesh University of Eng. & Tech. Slide 33 of 33 Industrial &Production Engineering