Physics 114: Lecture 17

Least Squares Fit to

Polynomial
Dale E. Gary
NJIT Physics Department

Reminder, Linear Least

Squares
We start with a smooth
line of the form
y ( x) a bx
which is the curve we want to fit to the data. The chi-square for
this situation is
2
2

y
(
x
)
1
2 i
yi a bx

i
i

To minimize any function, you know that you should take the
derivative and set it to zero. But take the derivative with respect
2
to what? Obviously, we want to find constants a andb that
minimize , so we will form two equations:
2
1

2
1

y

bx
yi a bxi 2

i 0,
i
2
a
a i

2

yi a bxi
b
b i

xi
yi a bx i 0.
i2

Apr 12, 2010

Polynomial Least Squares

Lets now allow a curved line of polynomial form

y ( x) a bx cx 2 dx 3 ...

which is the curve we want to fit to the data.

For simplicity, lets consider a second-degree polynomial
(quadratic). The chi-square
for this situation is 2
2
y y ( x)
1
2
2 i

bx

cx
i

i
i

Following exactly the same approach as before, we end up with

three equations in three unknowns (the
2 parameters a, b and c):
1

2

yi a bxi cxi2

a
a i

2
1
2
y

bx

cx

i
i
i 0,
2

2

yi a bxi cxi2

b
b i

2

yi a bxi cxi2

c
c i

2
xi
2
y

bx

cx

i 0,
i
i
2

2
xi2
y

bx

cx

i
i
i 0.
i2

Apr 12, 2010

Second-Degree Polynomial

The solution, then, can be found from the same determinant

technique we used before, except now we have 3 x 3 determinants:

1
a

1
c

yi

i2
xi yi

i2
xi2 yi

i2

1
i2
xi

i2
xi2

i2

xi

i2
xi2

i2
xi3

i2

xi

i2
xi2

i2
xi3

i2

xi2

i2
xi3

i2

xi4

i2
yi

i2
xi yi

i2
xi2 yi

i2

1
b

1
i2
xi

i2
xi2

i2

where

1
i2
xi

i2
xi2

i2

yi

i2
xi yi

i2
xi2 yi

i2

xi

i2
xi2

i2
xi3

i2

xi2

i2
xi3

i2
xi4

i2

xi2

i2
xi3

i2
xi4

i2

You can see that extending to arbitrarily high powers is

straightforward, if tedious.
We have already seen the MatLAB command that allows polynomial
fitting. It is just p = polyfit(x,y,n), where n is the degree of the fit.
We have used n = 1 so far.
Apr 12, 2010

MatLAB Example:
2nd-Degree Polynomial Fit
First, create a set of points that follow a second degree polynomial,
with some random errors, and plot them:

Now use polyfit to fit a second-degree polynomial:

25

3.0145 -2.5130

hold on
plot(x,polyval(p,x),'r')

And the original function

p = polyfit(x,y,2)
prints p = 1.5174

Now overplot the fit

x = -3:0.1:3;
y = randn(1,61)*2 - 2 + 3*x + 1.5*x.^2;
plot(x,y,'.')

plot(x,-2 + 3*x + 1.5*x.^2,'g')

20

data1
Polyfit
y(x)

15
y = 1.5x 2 + 3x - 2

10
5
0

Notice that the points scatter about-5

the fit. Look at the residuals.
-10
-3

-2

-1

0
x

Apr 12, 2010

MatLAB Example (contd):

2nd-Degree Polynomial Fit
The residuals are the differences between the points and the fit:

The residuals appear flat and random, which is good. Check the
standard deviation of the residuals:10

resid = y polyval(p,x)
figure
plot(x,resid,'.')

std(resid)
prints ans = 1.9475

This is close to the value of 2 we

used when creating the points.

Residuals

-5

-10
-3

-2

-1

0
x

Apr 12, 2010

MatLAB Example (contd):

Chi-Square for Fit

We could take our set of points, generated from a 2nd order

polynomial, and fit a 3rd order polynomial:

The fit looks the same, but there is a subtle difference due to the
use of an additional parameter. Lets look at the standard deviation
of the new

p2 = polyfit(x,y,3)
hold off
plot(x,polyval(x,p2),'.')

resid2 = y polyval(x,p2)
std(resid2)
prints ans = 1.9312

Is this a better fit? The residuals are slightly smaller BUT check chisquare.
chisq1 = sum((resid/std(resid)).^2)
% prints 60.00

chisq2 = sum((resid2/std(resid2)).^2)
%1.0345
prints 60.00
sum((resid/std(resid)).^2)/58.
% prints
sum((resid2/std(resid2)).^2)/57.
% prints 1.0526
=> 2 nd-order
fit is preferred
They
look identical, but now consider
the reduced
chi-square.

Apr 12, 2010

Linear Fits, Polynomial Fits,

Nonlinear Fits

When we talk about a fit being linear or nonlinear, we mean linear in

the coefficients (parameters), not in the independent variable. Thus,
a polynomial fit is linear in coefficients a, b, c, etc., even though those
coefficients multiply non-linear terms in independent variable x, (i.e. cx2).
Thus, polynomial fitting is still linear least-squares fitting, even though
we are fitting a non-linear function of independent variable x. The
reason this is considered linear fitting is because for n parameters we
can obtain n linear equations in n unknowns, which can be solved
exactly (for example, by the method of determinants using Cramers
Rule as we have done).
In general, this cannot be done for functions that are nonlinear in the
parameters (i.e., fitting a Gaussian function f(x) = a exp{[(x b)/c]2}, or
sine function f(x) = a sin[bx +c]). We will discuss nonlinear fitting next
time, when we discuss Chapter 8.
However, there is an important class of functions that are nonlinear in
parameters, but can be linearized (cast in a form that becomes linear
in coefficients). We will now take a look at that.
Apr 12, 2010

Linearizing Non-Linear Fits

Consider the equation

y ( x) ae bx ,
where a and b are the unknown parameters. Rather than consider
a and b, we can take the natural logarithm of both sides and
consider instead the function
ln y ln a bx.

This is linear in the parameters ln a and b, where chi-square is

2

1
2
ln yi ln a bx .
i

Notice, though, that we must use uncertainties i, instead of the

usual i to account for the transformation of the dependent
2
variable:

(ln
y
)
1 2
1
2
i

i2

i.
i
i
i
2
y
yi
yi

Apr 12, 2010

MatLAB Example:
Linearizing An Exponential
First, create a set of points that follow the exponential, with some
0.25
random errors, and plot them:

0.2
0.15
0.1
0.05

Now convert using log(yi) MatLAB for ln(yi)

logy = log(y+dev);
plot(x,logy,.)

-1
2
-2
0

10

6 6

88

10
10

-3
-2

As predicted, the points now make a pretty good

-4
-4
-6
-5
straight line. What about the errors. You might
-8
-6
-10
think this will work:
-7
ln(y)

x = 1:10;
y = 0.5*exp(-0.75*x);
sig = 0.03*sqrt(y); % errors proportional to sqrt(y)
dev = sig.*randn(1,10);
errorbar(x,y+dev,sig)

errorbar(x, logy, log(sig))

Try it! What is wrong?

-12
-8
-14

-9

4 4

Apr 12, 2010

xx

MatLAB Example (contd):

Linearizing An Exponential

The correct errors are as noted earlier:

0.2

This now gives the correct plot. Lets go ahead

0.15
and try a linear fit. Remember, to do a weighted
0.1
linear fit we use glmfit().
0.05
y

0.25

logsig = sig./y;
errorbar(x, logy, logsig)

To plot the line over the original data:

p = glmfit(x,logy,normal,weights,logsig);
p = circshift(p,1);
% swap order of parameters
hold on
plot(x,polyval(p,x),r)
hold off
errorbar(x,y+dev,sig)
hold on
plot(x,exp(polyval(p,x)),r)

66

88

10
10

88

10
10

-2
-3
-4
ln(y)
ln(y)

Note parameters a = ln a = 0.6931, b = b = 0.75

-5
-6
-7
-8
-9

Apr 12, 2010

Summary
Use polyfit() for polynomial fitting, with third parameter giving the
degree of the polynomial. Remember that higher-degree
polynomials use up more degrees of freedom (an nth degree
polynomial takes away n + 1 DOF).
A polynomial fit is still considered linear least-squares fitting, despite
its dependence on powers of the independent variable, because it is
linear in the coefficients (parameters).
y ( x) ae bx ,
For some problems, such as exponentials,
, one can
b
linearize the problem. Another
y ( type
x) axthat
, can be linearized is a
power-law expression,

as you will do in the homework.

When linearizing, the errors2 must be handled properly, using the
(ln yi )
1 2
1
2
2

i.
usual error propagation
equation,
e.g.
i

i
i
i
2

y
y
y

i
i
Apr 12, 2010