B.Sc.
(I) Statistics Paper -III Unit-III (Part-I)
The Method of Least Squares
Let we have a system of independent linear equations with ‘m’
equations and ‘n’ unknowns as follows
a 1 x 1 + b 1 x 2 + ….. + f 1 x n = u 1
a 2 x 1 + b 2 x 2 + ….. + f 2 x n = u 2
……
…….
a m x 1 + b m x 2 + ….. + f m x n = u m
where a i ’s, b i ’s….., u i ’s are constants.
Now,
(i) if m = n, we can find a unique set of values of ‘n’ unknowns
(ii) if m < n, we may have infinite no. of solutions
(iii) if m > n, then no such solution may exist.
The method of least squares given by Legendre is a standard
approach to the approximate solution of sets of equations in which
there are more equations than unknowns. "Least squares" means that
the overall solution minimizes the sum of the squares of the errors
made in solving every single equation.
By using this method we can find the values of unknowns x 1 , x 2 , …,x n
which will satisf y the given system of equations as nearly as possible.
The least squares method provides the best values (most plausible) of
unknowns when the sum, S, of squared errors (residuals)
is a minimum. A residual is defined as the difference between the
actual value of the unknown and the value predicted by the model i.e.
m
S = i 1
( a i x 1 + b i x 2 + ….. + f i x n – u i ) 2
1
Now, as the principle of least square, we have to find those values of
x 1 , x 2 , …,x n for which S is minimum. The minimum of the sum of
squares S can be found by setting the partial derivatives of S with
respect to x 1 , x 2 , …,x n to zero.
S m m
x1
0 , i.e i 1
a i .( a i x 1 + b i x 2 + ….. + f i x n ) =
i 1
aiui
S m m
x2
0 , i.e
i 1
b i .( a i x 1 + b i x 2 + ….. + f i x n ) = i 1
biui
…..
…..
S m m
xn
0 , i.e
i 1
f i .( a i x 1 + b i x 2 + ….. + f i x n ) =
i 1
fiui
W hen the above equations are solved simultaneously, we can obtain
an unique set of values of unknowns x 1 , x 2 , …,x n . These values are
called most plausible values.
2S
Further, on calculating second order partial derivatives 0,
xi2
i=1,2,..,n and substituting the values of x 1 , x 2 ,…, x n , we will find that
the second order partial derivatives are positive. Hence, S is minimum
for these values.
Curve Fitting
The device of finding a functional relationship between two
variables x and y, where x is independent and y is dependent variable
on the basis of a set of pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …, (x n ,y n )
on these variables, is called curve fitting.
Or
Curve fitting is a process of finding a mathematical relationship
between dependent and independent variables in the form of an
equation for a given set of data.
2
Fitting of a n t h degree pol ynomial by the method of Least Squares
Let we have a set of ‘n’ pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …,
(x n ,y n ) for which we want to fit a n t h degree polynomial.
Let the equation of n t h degree polynomial be
y = a 0 + a 1 x + a 2 x 2 + …. a n x n ….(1)
where a 0 , a 1 , ….a n are constants. Here the curve fitting means to
find the values of the constants a 0 , a 1 , ….a n
Now putting x=x i in equation (1), estimated value of y is
Y i = a 0 + a 1 x i + a 2 x i 2 + …. a n x i n
Hence the error of estimate for y i is
E i = y i – Y i = y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n
And the sum of squares of residuals is
S = E 1 2 + E 2 2 + ….. + E n 2
n
=
i 1
( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) 2
Now, as the principle of least square, we have to find those values of
a 0 , a 1 , ….a n for which S is minimum. The minimum of the sum of
squares S can be found by setting the partial derivatives of S with
respect to a 0 , a 1 , ….a n to zero.
S n
0 , i.e -2 ( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) = 0
a 0 i 1
S n
0 , i.e -2 x i ( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) = 0
a1 i 1
….
….
S n
0 , i.e -2 ( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) = 0
a n i 1
3
The above n+1 equations are called Normal equations and these
equations will be used to find the values of a 0 , a 1 , ….a n .
Fitting of a Straight line
Let we have a set of ‘n’ pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …,
(x n ,y n ) for which we want to fit a straight line.
Let the equation of straight line be
y = a + bx ….(1)
where ‘a’ and ‘b’ are constants. Here the curve fitting means to
find the values of the constants ‘a’ and ‘b’.
Let P(x i ,y i ) be any point in the scatter diagram.
Now putting x=x i in equation (1) estimated value of y is
Y i = a + bx i
Thus the point Q(x i ,Y i ) be on the line given in equation (1).
Hence the error of estimate for y i is
E i = y i – Y i = y i – a – bx i
And the sum of squares of residuals is
4
S = E 1 2 + E 2 2 + ….. + E n 2
= (y 1 – a – bx 1 ) 2 + (y 2 – a – bx 2 ) 2 + ….+ (y n – a – bx n ) 2
n
=
i 1
(y i – a – bx i ) 2
Now, as the principle of least square, we have to find those values of a
and b for which S is minimum. The minimum of the sum of squares S
can be found by setting the partial derivatives of S with respect to a
and b to zero.
S n
0 , i.e -2 (y i – a – bx i ) = 0
a i 1
It gives us
n n
i 1
y i = na + b x i
i 1
….(A)
Now,
S n
0 , i.e -2 x i ( y i – a – bx i ) = 0
b i 1
It gives us
n n n
i 1
xiyi = a xi + b xi2
i 1 i 1
….(B)
Equations (A) and (B) are called Normal Equations to find the least
square values of ‘a’ and ‘b’. Putting the values of ‘a’ and ‘b’ in
equation (1) we will get fitted straight line.
Change of Origin and Scale
If the values of ‘X’ are given to be equidistant at an interval say ‘h’ and
no. of pairs ‘n’ is odd then we take
X Middle Term
U =
h
But if n is even
5
X Mean of Two Middle Term
U =
h/2
Fitting of a Second degree parabola
Let we have a set of ‘n’ pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …,
(x n ,y n ) for which we want to fit a second degree parabola.
Let the equation of second degree parabola be
y = a + bx + cx 2 ….(1)
where ‘a’, ‘b’ and ‘c’ are constants. Here the curve fitting means
to find the values of the constants ‘a’, ‘b’ and ‘c’.
Now putting x=x i in equation (1) estimated value of y is
Y i = a + bx i + cx i 2
Hence the error of estimate for y i is
E i = y i – Y i = y i – a – bx i - cx i 2
And the sum of squares of residuals is
S = E 1 2 + E 2 2 + ….. + E n 2
= (y 1 –a–bx 1 –cx 1 2 ) 2 + (y 2 –a–bx 2 –cx 2 2 ) 2 + ….+ (y n –a–bx n –cx n 2 ) 2
n
=
i 1
(y i – a – bx i - cx i 2 ) 2
Now, as the principle of least square, we have to find those values of
a, b and c for which S is minimum. The minimum of the sum of squares
S can be found by setting the partial derivatives of S with respect to a,
b, and c to zero.
S n
0 , i.e -2 (y i – a – bx i - cx i 2 ) = 0
a i 1
It gives us
6
n n n
i 1
y i = na + b x i + c x i 2
i 1 i 1
….(A)
S n
0 , i.e -2 x i ( y i – a – bx i - cx i 2 ) = 0
b i 1
It gives us
n n n n
i 1
xi yi = a xi + b xi2 + c xi 3
i 1 i 1 i 1
….(B)
And
S n
0 , i.e -2 x i 2 ( y i – a – bx i - cx i 2 ) = 0
c i 1
It gives us
n n n n
i 1
xi yi = a xi + b xi + c xi4
2
i 1
2
i 1
3
i 1
….(C)
Equations (A), (B) and (C) are called Normal Equations to find the
least square values of ‘a’, ‘b’ and ‘c’. Putting the values of ‘a’, ‘b’ and
‘c’ in equation (1) we will get fitted second degree parabola.
Note: If we have to fit a non -linear curve by using Method of least
squares, then first we have to convert its equation in linear form by
making suitable transformations.
Suppose we have to fit a curve
y = ab x ….(1)
which is not in linear form. To convert it into linear form let we take log
(to the base 10) on both the sides, we get
log y = log a + x. log b
Or Y = A + Bx ….(2)
W here Y = log y, A = log a and B = log b
Now equation (2) is in linear form of Y on x, for which normal
equations will be
n n
i 1
Y i = nA + B x i
i 1
….(A)
7
n n n
i 1
xi Yi = A xi + B xi2
i 1 i 1
….(B)
Solving the above equation we have the values of A and B. And
using A and B we can find the values of ‘a’ and ’b’ as
a = antilog A and b = antilog B.
