Professional Documents
Culture Documents
ECM6Lecture11aVietnam 2014
ECM6Lecture11aVietnam 2014
Slide 1 of 7
Lecture 11a
Linear Regression
Brian G. Higgins
Department of Chemical Engineering & Materials Science
University of California, Davis
April 2014, Hanoi, Vietnam
ECM6Lecture11aVietnam_2014.nb
Background
A common task in engineering is to determine a formula that describes how a physical quantity such as
say heat capacity varies as function of a second quantity say temperature.
As a result of an experiment you have made set of measurements and have M data points that are in
the form
88x1 , y1 <, 8x2 , y2 <, 8x3 , y3 <, 8xM , yM <<
Here say xi represents a temperature measurement and yi represents the heat capacity. When we plot
the data we might get something like this
1.2
1.0
0.8
CP
0.6
0.4
0.2
0.0
0
10
15
20
Temperature
Based on your understanding of the physics you may have an idea that the data should agree with the
blue line shown in the above plot.
We call the blue line the guess function g(x).
Since the points 8xi , yi < are experimental that are not likely to lie on the curve given by the guess function. Thus
dk = g Hxk L - yk
where dk represents the vertical distance from the data point 8xk , yk < and the curve given by the guess
function g(x). Here is a graphical representation
ECM6Lecture11aVietnam_2014.nb
0.12
0.10
Hxk , yk L
CP
0.08
dk =gHxk L-yk
0.06
0.04
0.02
0.00
0.0
0.5
1.0
1.5
2.0
Temperature
Types of Errors
We can define 3 types of error
M
d1
d2
+ dM
dk
(1)
k=1
M
Square Error of g :
(2)
k=1
Maximum Error of g :
E3 HgL = max 8 d1 ,
d2 ,
d3
dM <
(3)
Note if the graph passes through all the points 8xk , yk <, then the error Ei HgL no matter how it is defined
is zero.
On the other hand as Ei HgL get larger Hsome or all of the data points do not lie on the curveL, then we say
that the guess function g(x) does not fit the data as well. In short, the smaller is Ei HgL, the better is the fit.
Squared Error
The square error E2 HgL is the most widely used estimate of the error of how well g(x) fits the data. There
are statistical reasons why this is the best way to estimate the goodness of a fit, but we will not discuss
it in these lectures.
Thus our goal will be to
M
E2 HgL = @g Hxi L - yk D2
(4)
k=1
We say then we seek a g(x) that has the least square error. Another way of saying this is the g(x) with
the minimum E2 HgL is called the least square g(x).
ECM6Lecture11aVietnam_2014.nb
Example 1
Problem Statement
Suppose we are given the following list of data points Hxk , yk L
881, 1.5<, 82, 3.9<, 84, 6.6<, 87, 11.7<, 89, 15.6<<
(i) Determine the square error as a function of k for
g HxL = k x
(ii) Find the parameter value of k that minimizes E2 HgL. What is E2 HgL for this value of k?
Next we define a function for the error E2 HgL. Note we use double strike E, as the variable E in Mathematica is the protected exponential constant equal to 2.7182...
M
Solution Step 2: Compute the Sum of Squared Error using the data
Here we have supposed that xi and yi are our data points. These are defined from the given data list
as follows:
x@i_D := data@@i, 1DD; y@i_D := data@@i, 2DD; M = Length@dataD;
i=1
i=1
(5)
ECM6Lecture11aVietnam_2014.nb
E2
2000
1500
1000
500
0
-2
k
It is clear that E2 has a minimum value for a value of k 1.8
= 0,
2 E
k2
> 0 at k = kmin
Differentiating E2 gives
D@E2 @g, MD, kD Simplify
302. H- 1.70861 + kL
ECM6Lecture11aVietnam_2014.nb
gHxL
10
0
0
10
x
Finally let us evaluate the least square error
E2 @g, MD . sol
0.448808
Note that the actual value is not small number, yet the fit looks good. We will address this point later.
ECM6Lecture11aVietnam_2014.nb
E2 HLL = Hm xk + b - yk L2
k=1
E2 HLL
= 0,
=0
Note that since the xk ' s and yk ' s are constant, we can evaluate the following sum
M
b=Mb
k=1
= 2 Hm xk + b - yk L
k=1
Hm xk + b - yk L
= 2 Hm xk + b - yk L
k=1
= 2 M b + m xk - yk
k=1
k=1
Similarly we find
E2 HLL
m
= 2 Hm xk + b - yk L xk
k=1
ECM6Lecture11aVietnam_2014.nb
= 2 m x2k + b xk - yk xk
k=1
k=1
k=1
Then setting the partial derivatives to zero gives the following linear set of equations for m and b
M
M b + m xk - yk = 0
k=1
k=1
m x2k + b xk - yk xk = 0
k=1
k=1
k=1
xk
x2k
yk
b
O=K
O
m
yk xk
ECM6Lecture11aVietnam_2014.nb
Example 2
Problem Statement
881, 5.12<, 83, 3<, 86, 2.48<, 89, 2.34<, 815, 2.18<<
and we want to find the parameters m and b that give the least square error for a guess function
LHxL = m x + b
ECM6Lecture11aVietnam_2014.nb
LHxL
10
2.5
2.0
1.5
1.0
0
10
15
20
x
It is apparent from the plots that a straight line is not particularly good fit of the data.
ECM6Lecture11aVietnam_2014.nb
11
12
ECM6Lecture11aVietnam_2014.nb
Final Comments
In these notes we have outlined the general principle for doing linear regression.
We have kept the discussion limited to fitting the data to a linear function
L HxL = m x + b
In this case there are two parameters: L and M and they appear linearly in the fitting function
The term linear regression does not mean that the function is linear in the independent variable x, but
that it is linear in the unknown parameters m and L.
For example, the same ideas and programming steps apply to fitting the data to a polynomial function of
x:
P HxL = a0 + a1 x + a2 x2 + a3 x3 + + an xn
Note that the unknown parameters a0 , a1 , , an appear linearly in the function P(x)
References
These notes and the examples are adapted from Maron (1987)
M.J. Maron, Numerical Analysis. A Practical Approach, 2nd Edition, Macmillan Publishing Company,
1987