B.Sc.

(I) Statistics Paper -III Unit-III (Part-I)

The Method of Least Squares
Let we have a system of independent linear equations with ‘m’
equations and ‘n’ unknowns as follows

a 1 x 1 + b 1 x 2 + ….. + f 1 x n = u 1

a 2 x 1 + b 2 x 2 + ….. + f 2 x n = u 2

……

…….

a m x 1 + b m x 2 + ….. + f m x n = u m

where a i ’s, b i ’s….., u i ’s are constants.

Now,

(i) if m = n, we can find a unique set of values of ‘n’ unknowns

(ii) if m < n, we may have infinite no. of solutions

(iii) if m > n, then no such solution may exist.

The method of least squares given by Legendre is a standard

approach to the approximate solution of sets of equations in which
there are more equations than unknowns. "Least squares" means that
the overall solution minimizes the sum of the squares of the errors
made in solving every single equation.

By using this method we can find the values of unknowns x 1 , x 2 , …,x n

which will satisf y the given system of equations as nearly as possible.

The least squares method provides the best values (most plausible) of

unknowns when the sum, S, of squared errors (residuals)

is a minimum. A residual is defined as the difference between the

actual value of the unknown and the value predicted by the model i.e.

m
S = i 1
( a i x 1 + b i x 2 + ….. + f i x n – u i ) 2

1
Now, as the principle of least square, we have to find those values of

x 1 , x 2 , …,x n for which S is minimum. The minimum of the sum of

squares S can be found by setting the partial derivatives of S with

respect to x 1 , x 2 , …,x n to zero.

S m m

x1
 0 , i.e i 1
a i .( a i x 1 + b i x 2 + ….. + f i x n ) = 
i 1
aiui

S m m

x2
 0 , i.e 
i 1
b i .( a i x 1 + b i x 2 + ….. + f i x n ) = i 1
biui

…..
…..
S m m

xn
 0 , i.e 
i 1
f i .( a i x 1 + b i x 2 + ….. + f i x n ) = 
i 1
fiui

W hen the above equations are solved simultaneously, we can obtain

an unique set of values of unknowns x 1 , x 2 , …,x n . These values are

called most plausible values.

2S
Further, on calculating second order partial derivatives  0,
xi2

i=1,2,..,n and substituting the values of x 1 , x 2 ,…, x n , we will find that

the second order partial derivatives are positive. Hence, S is minimum

for these values.

Curve Fitting
The device of finding a functional relationship between two

variables x and y, where x is independent and y is dependent variable

on the basis of a set of pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …, (x n ,y n )

on these variables, is called curve fitting.

Or

Curve fitting is a process of finding a mathematical relationship

between dependent and independent variables in the form of an

equation for a given set of data.

2
Fitting of a n t h degree pol ynomial by the method of Least Squares

Let we have a set of ‘n’ pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …,

(x n ,y n ) for which we want to fit a n t h degree polynomial.

Let the equation of n t h degree polynomial be

y = a 0 + a 1 x + a 2 x 2 + …. a n x n ….(1)

where a 0 , a 1 , ….a n are constants. Here the curve fitting means to

find the values of the constants a 0 , a 1 , ….a n

Now putting x=x i in equation (1), estimated value of y is

Y i = a 0 + a 1 x i + a 2 x i 2 + …. a n x i n

Hence the error of estimate for y i is

E i = y i – Y i = y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n

And the sum of squares of residuals is

S = E 1 2 + E 2 2 + ….. + E n 2

n
= 
i 1
( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) 2

Now, as the principle of least square, we have to find those values of

a 0 , a 1 , ….a n for which S is minimum. The minimum of the sum of

squares S can be found by setting the partial derivatives of S with

respect to a 0 , a 1 , ….a n to zero.

S n
 0 , i.e -2  ( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) = 0
a 0 i 1

S n
 0 , i.e -2  x i ( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) = 0
a1 i 1

….
….
S n
 0 , i.e -2  ( y i – a 0 - a 1 x i - a 2 x i 2 - …. a n x i n ) = 0
a n i 1

3
 The above n+1 equations are called Normal equations and these

equations will be used to find the values of a 0 , a 1 , ….a n .

Fitting of a Straight line

Let we have a set of ‘n’ pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …,

(x n ,y n ) for which we want to fit a straight line.

Let the equation of straight line be

y = a + bx ….(1)

where ‘a’ and ‘b’ are constants. Here the curve fitting means to

find the values of the constants ‘a’ and ‘b’.

Let P(x i ,y i ) be any point in the scatter diagram.

Now putting x=x i in equation (1) estimated value of y is

Y i = a + bx i

Thus the point Q(x i ,Y i ) be on the line given in equation (1).

Hence the error of estimate for y i is

E i = y i – Y i = y i – a – bx i

And the sum of squares of residuals is

4
S = E 1 2 + E 2 2 + ….. + E n 2

= (y 1 – a – bx 1 ) 2 + (y 2 – a – bx 2 ) 2 + ….+ (y n – a – bx n ) 2

n
= 
i 1
(y i – a – bx i ) 2

Now, as the principle of least square, we have to find those values of a

and b for which S is minimum. The minimum of the sum of squares S

can be found by setting the partial derivatives of S with respect to a

and b to zero.

S n
 0 , i.e -2  (y i – a – bx i ) = 0
a i 1

It gives us

n n


i 1
y i = na + b  x i
i 1
….(A)

Now,

S n
 0 , i.e -2  x i ( y i – a – bx i ) = 0
b i 1

It gives us

n n n


i 1
xiyi = a  xi + b  xi2
i 1 i 1
….(B)

Equations (A) and (B) are called Normal Equations to find the least
square values of ‘a’ and ‘b’. Putting the values of ‘a’ and ‘b’ in
equation (1) we will get fitted straight line.

Change of Origin and Scale

If the values of ‘X’ are given to be equidistant at an interval say ‘h’ and
no. of pairs ‘n’ is odd then we take

X  Middle Term
U =
h

But if n is even

5
 X  Mean of Two Middle Term
U =
h/2

Fitting of a Second degree parabola

Let we have a set of ‘n’ pairs of observations (x 1 ,y 1 ), (x 2 ,y 2 ), …,

(x n ,y n ) for which we want to fit a second degree parabola.

Let the equation of second degree parabola be

y = a + bx + cx 2 ….(1)

where ‘a’, ‘b’ and ‘c’ are constants. Here the curve fitting means

to find the values of the constants ‘a’, ‘b’ and ‘c’.

Now putting x=x i in equation (1) estimated value of y is

Y i = a + bx i + cx i 2

Hence the error of estimate for y i is

E i = y i – Y i = y i – a – bx i - cx i 2

And the sum of squares of residuals is

S = E 1 2 + E 2 2 + ….. + E n 2

= (y 1 –a–bx 1 –cx 1 2 ) 2 + (y 2 –a–bx 2 –cx 2 2 ) 2 + ….+ (y n –a–bx n –cx n 2 ) 2

n
= 
i 1
(y i – a – bx i - cx i 2 ) 2

Now, as the principle of least square, we have to find those values of

a, b and c for which S is minimum. The minimum of the sum of squares

S can be found by setting the partial derivatives of S with respect to a,

b, and c to zero.

S n
 0 , i.e -2  (y i – a – bx i - cx i 2 ) = 0
a i 1

It gives us

6
 n n n


i 1
y i = na + b  x i + c  x i 2
i 1 i 1
….(A)

S n
 0 , i.e -2  x i ( y i – a – bx i - cx i 2 ) = 0
b i 1

It gives us

n n n n


i 1
xi yi = a  xi + b  xi2 + c  xi 3
i 1 i 1 i 1
….(B)

And

S n
 0 , i.e -2  x i 2 ( y i – a – bx i - cx i 2 ) = 0
c i 1

It gives us

n n n n


i 1
xi yi = a  xi + b  xi + c  xi4
2

i 1
2

i 1
3

i 1
….(C)

Equations (A), (B) and (C) are called Normal Equations to find the
least square values of ‘a’, ‘b’ and ‘c’. Putting the values of ‘a’, ‘b’ and
‘c’ in equation (1) we will get fitted second degree parabola.

Note: If we have to fit a non -linear curve by using Method of least

squares, then first we have to convert its equation in linear form by
making suitable transformations.

Suppose we have to fit a curve

y = ab x ….(1)
which is not in linear form. To convert it into linear form let we take log
(to the base 10) on both the sides, we get
log y = log a + x. log b
Or Y = A + Bx ….(2)
W here Y = log y, A = log a and B = log b
Now equation (2) is in linear form of Y on x, for which normal
equations will be

n n


i 1
Y i = nA + B  x i
i 1
….(A)

7
 n n n


i 1
xi Yi = A  xi + B  xi2
i 1 i 1
….(B)

Solving the above equation we have the values of A and B. And

using A and B we can find the values of ‘a’ and ’b’ as

a = antilog A and b = antilog B.