Azərbaycan Dövlət Neft

və Sənaye Universiteti
Numerical
Methods II
Polynomial Approximation
When a complicated function in a practical problem
must be evaluated many times, it is often more
convenient to produce an efficiently computable
approximation.

In this presentation, we will be investigating

polynomials as approximations to a given function.
(The use of trigonometric and rational functions for
approximation is another option.)

Consider producing approximations to a continuous

function on the interval .
When we search for an approximation for , we need to
study the deviation

which is also continuous.

For this purpose, we introduce the notion of a for

continuous functions.

There are at least three different frequently used norms

for continuous functions.
They are analogous to the , , and norms used for
vectors.
We thus define three norms for continuous functions as
follows:

Let denote the set of polynomials of degree at most .

The polynomial for which the error is minimized is
called the from for :

In view of Eq. , the determination of is frequently

called the , and is called the to from .

The quantity is called the , or the of from in

approximation theory.
The for general functions is a difficult problem, and
we will consider it briefly later on.

The minimization of is equally difficult for general

functions because of the occurrence of the absolute
value function in the norm to be minimized.

Hence, we often consider the minimization of , which

is known as the - problem, and we will consider it
right now.
Polynomial Approximation:

Continuous Least-Squares
Letting an approximation to from be

the - problem is to find

for which the quantity

is minimized.
According to (see Multivariable Calculus), for to
attain minimum at , it is necessary that

Therefore, for each , we have

or, equivalently,

for .
As before, Eqs. describe what are called the .

In matrix form, Eq. is written as

where , , and with

The solution of Eqs. describes the least-squares

polynomial of degree that approximates
It is easy to verify that the matrix is symmetric and
non-singular. The symmetry of follows form the
definition in .

In order to show that is non-singular, we first

assume that is singular and produce a contradiction.

If is singular, then there exists a non-zero vector

for which .
Therefore, it is also true that for this vector .
If we define the polynomial

then we obtain

Thus, because

which means that as long as .

Hence , from which it follows that is non-singular.

Since

we have actually shown that is positive definite.

. Construct the least-squares straight line
approximating on the interval .

.
Let the least-squares straight line fit be given by

with .

The normal equations are given by the linear system:

.
After evaluating the integrals we obtain

which has the solution and .

Hence the least-squares straight line approximating on

is then given by .

A graph of along with the graph of on the interval is

shown in . The graph of the error in the approximation
is given in .
.

. Least-squares approximation for on the interval .

.

. Least-squares approximation for on the interval .

The magnitude of the error is .

It is apparent that the continuous least-squares
problem has properties very similar to those of the
discrete least-squares problem.

Therefore, it is reasonable to suspect that the normal

equations for the continuous problem may be just as
ill-conditioned as those for the discrete problem.

The following example shows that this is indeed the

case.
.
Consider finding the least-squares approximating
polynomial of degree for a continuous function on the
interval .
The normal equations corresponding to an
approximation of the form are given by with , , and ,
where

and .
.
For example, for the coefficient matrix is

which is the notorious Hilbert matrix of order .

As increases, the order of this matrix increases while

the structure remains the same.
It is well-known that Hilbert matrices are ill-
conditioned.
.
Thus the accuracy of the solution obtained for the
coefficients of the least-squares polynomial will be
questionable.
As we saw earlier in connection with , sometimes it
may be necessary to examine approximating
polynomials of increasing degrees before we obtain a
satisfactory approximation.

Unfortunately, as with the Vandermonde approach or

the Lagrange approach for the interpolation problem,
the construction of the least-squares polynomial of
degree does not in any way reduce the amount of
computational effort in the construction of the least-
squares polynomials of degree or higher.
Hence it is worthwhile to explore other approaches
that will take care of the condition problem, and that
can provide approximating polynomials of increasing
degrees without requiring much additional
computational effort.

We will describe the more efficient methods for the

least-squares problems stated in a more general form.

Given the function that is continuous on , we wish to

construct an approximation of the form
Here are a selected set of functions and are
parameters that are to be determined in order to
minimize the quantity

The function is called the , and it signifies varying

degrees of importance assigned to the errors in
different portions of the interval .
For example, our earlier definition of the continuous
least-squares problem in Eq. corresponds to ,
assigning the same importance to the errors at all the
points in the interval .

Another commonly used weight function is

for the interval , which assigns more importance to

errors near the ends of the interval than to those in the
middle.
Yet another weight function

for the interval , assigns more importance to errors in

the middle of the interval than at the ends.

Even though itself may be infinite, as is the case with

at and , it should possess the following properties:

𝒃
𝐤
𝐢 .∫| 𝒙| ∙ 𝝎 ( 𝒙 ) 𝕕 𝒙 <∞ 𝐟𝐨𝐫 𝐚𝐥𝐥 𝐤 ≥ 𝟎
𝒂
ii. For a continuous function ,

Some of the other commonly used weight functions are

for the interval , and for the interval .

We are particularly interested in the general least-

squares problem in which the functions are
polynomials.
For the examples we have discussed so far, we have
chosen .
Instead, we may use other types of functions for .

Once the form of the approximation has been chosen,

the minimization process can proceed as before: for
defined by Eq. to attain a minimum at , it is necessary
that
Therefore, for each , we have
Equivalently,

In matrix form, we may write this as with , , and ,

where
The linear system will be uniquely solvable, provided
the are chosen “properly”.

In order to clarify what “properly” means, we need the

following basic ideas.

. We say that the functions are on if

for all .
Otherwise, they are said to be .
. Show that the functions
for
are linearly independent.

. Suppose that the given functions are linearly

dependent. Then there exist constants , not all of which
are zero, such that
for all

However, this is impossible because the polynomial is

of degree at most and can only have at most zeros,
unless .
. Hence, the functions

are linearly independent.

. If is a polynomial of degree for , then the functions

are linearly independent on any interval .

. Let us assume that the aforementioned functions

are linearly dependent.
. Then, there must exist constants

not all of which are zero, for which .

Letting , we conclude that for all , which means that

the coefficients of all powers of in must be zero.

Since is the only polynomial of degree among the , is

the coefficient of in .
Therefore, .
. Now we have

for all .

Since is the only polynomial of degree among the , is

the coefficient of in . Therefore, .

In a similar manner, we can successively establish that

are all zero as well. Hence are linearly independent.
An immediate consequence of this is the following
corollary.

. Let be a polynomial of degree for . Then any

polynomial in can be expressed uniquely as a linear
combination of

. Let be any polynomial of degree at most . By

definition, . It is clear that may be expressed as a linear
combination of
. In order to show that such a representation is unique,
we proceed as follows.
Suppose there exist two different ways to express as a
linear combination of the .

Then there exist two different sets of constants and

satisfying
. Since the functions are linearly independent, this
means that

In other words, for , and the representation for must

be unique. ◄

. We say that the functions form a for a set of

functions if the are linearly independent and if every
function in may be uniquely expressed as a linear
combination of the .
For example, the functions

form a for .

As a matter of fact, it follows from and that any

collection of polynomials

such that is a polynomial of degree forms a for .

In the solution of the discrete least-squares problem in
its general form, we needed to solve the linear system

for .

We commented there that the linear system will be

uniquely solvable provided the functions are chosen
properly.

A similar statement was made in connection with the

continuous least-squares problem earlier.
In view of the preceding discussion, it is evident that
the choice of will be proper if they form a for the set
of approximating functions being explored.

In contrast, if the are linearly dependent, the linear

system for the expansion coefficients in both types of
least-squares problems will be singular.

Even though we have shown that the functions = are

linearly independent on any interval , their graphs
indicate that they tend to be nearly linearly dependent
on the interval .
This is the cause of the ill-conditioning that we saw in .

Therefore, it may be worthwhile to worry about other

choices for .

For example, if we choose the to be orthogonal, the

least-squares problem has an immediate solution that
is free of problems of ill-conditioning.

In the next section, we will consider the generation and

use of orthogonal polynomials in the solution of
general least-squares problem.
Polynomial Approximation:

Orthogonal Polynomials
The general statement

of the continuous least-squares problem is sometimes

called the - problem.

The corresponding discrete weighted least-squares

problem is to find an approximation to the data of the
form that will minimize
Here is the discrete analog of the weight function
introduced earlier, and form a for the approximations
sought.

In order to simplify the notation somewhat, we

introduce the idea of an .

. Let and be continuous on . Then the of and is

defined by
Note that the of two functions is a generalization of
the vector inner product that you have seen in Linear
Algebra course.

. Find the inner product of and with the weight

function on .

. According to Eq. we have

. Find the inner product of the functions and with the
weight function for the set of discrete points , where .

. According to Eq. we have

The inner product possesses the following properties.
Let and be continuous functions. Then
i. unless .
ii. .
iii. for all real numbers and .

. Find the norms and for and given in , that is for

and
with the weight function on .
.
We have

so that and .
. Two continuous functions and are said to be
orthogonal if .

We say that is a (finite or infinite) sequence of

orthogonal polynomials if is a polynomial of degree ,
and the members of the sequence are pairwise
orthogonal.
To be precise, the following two conditions hold:
i.
ii. for .
It follows from and that the sequence of orthogonal
polynomials is linearly independent and constitutes a
basis for .

It turns out that orthogonal polynomials greatly

simplify the solution of least-squares problems.
However, we have to come up with a sequence of
orthogonal polynomials before we can use them in the
solution of least-squares problems.

We may generate such a sequence by the following

procedure, known as the - .
(-)
Given the definition of the inner product, we begin by
letting .

Then we write

and require that .

From it follows that

so that
Since , and since forms a basis for polynomials, we
may write

as a generalization of .

By requiring that , we can obtain the constants for .

From it follows
.
Generate the first three members of the sequence of
orthogonal polynomials for the weight function and
the inner product defined by

. Let . Then , and

with
.
Since , we have
and .

Next we find

with

Hence . ◄
.
Generate the first four members of the sequence of
orthogonal polynomials for the weight function on the
interval and the inner product

. We have and . Then

with
.
Since

we have
and .

Next we compute
.
Hence
and .

Finally

Hence . ◄
Let be a sequence of polynomials in which is a
polynomial of degree .

Suppose we wish to find the weighted least-squares

approximation of the form

for given data , where , and for some function , or a

given function on the interval .

Let be a given weight function.

Then both the discrete and the continuous least-
squares problems in their weighted form may be
expressed as follows:

where

and is as defined in either or .

Expanding the error expression,
Further, if we construct to form an orthogonal
sequence with respect to the weight function at for
the discrete least-squares problem, or on the interval
for the continuous least-squares problem, then

Setting yields the normal equations

Hence

Note that we can now produce the coefficients without

having to solve a linear system.

Further, if we denote by the least-squares polynomial

of degree approximating , then
Hence, in order to produce least-squares polynomials
of increasing degrees, all we have to do is to obtain as
many additional coefficients as desired.

For example,
• alone will define the least-squares constant
approximation to the given function ;
• and will define the least-squares straight-line
approximation;
• , , and will define the least-squares quadratic
approximation, and so on.
.
Find for the following data , , that will minimize for :

1 2 3 4 5
1 1.2 1.5 1.8 2.1

Use an orthogonal polynomial representation to solve

this problem.
.
We have already determined in that the polynomials

are orthogonal with respect to the weight function for

the data for .

Letting

and using Eq. we obtain the following values.

.

Thus we get
.
Notice that has been obtained simply by adding
another term to .

This is much more efficient than having to form and

solve the normal equations separately for each
required. ◄
 
Several families of orthogonal polynomials occur
frequently in Numerical Analysis.

The polynomials generated in , for example, are the

first four members of the family known as the , and
they are orthogonal with respect to the weight function
on the interval .

There are many traits of Legendre polynomials; we

will mention just two.
First, if we let be the Legendre polynomial of degree ,
then an explicit expression for is given by

where is a known constant that depends only on .

This formula for Legendre polynomials is used when

developing numerical integration formulas that have
many desirable properties.
Another popular trait of Legendre polynomials is
through the three-term recurrence relation

It turns out that any set of orthogonal polynomials may

be characterized by some three-term recurrence
relation of this type.
.
Use Legendre polynomials to obtain the least-squares
approximating polynomials of degree , , and for the
function on .

. The Legendre polynomials , , , and were obtained in

as through .

In addition, we have
.
Let the least-squares approximation using Legendre
polynomials, or simply, the Legendre least-squares
polynomials for , be denoted by

If we let , then it follows from that

.
.
Hence, the desired Legendre least-squares polynomials
for on are as follows:
The continuous least-squares problem for general
intervals and may be transformed to the least-squares
problem for the interval through the change of
variables

which maps the interval to

Under this transformation, there is a one-to-one

correspondence between polynomials of degree on
and polynomials of degree on for every .
Further, a continuous function for is converted to

Also, if is any polynomial of degree then

where
Thus the minimization of on is equivalent to the
minimization of on .

: There is another orthogonal family of polynomials

known as , which is used extensively in approximation
theory.

We conclude this presentation with a strong result that

often serves as the motivation for obtaining least-
squares approximations.
. Let be the polynomial of degree that minimizes
with respect to the weight function .
When is finite, we have

. Since is the minimizing polynomial of degree , it is

true that

Let and let be arbitrary.

. Then by the Weierstrass theorem there exists a
polynomial of some degree, say , satisfying

However, since is the minimizing polynomial of

degree , it is true that
. Then, from we may conclude that for all .

This immediately yields since is arbitrary. ◄

. Note that this Theorem does not say that as .

However, if we assume additional differentiability on ,
it is possible to obtain results indicating that also in
the uniform norm.
.
Find the weighted least-squares straight-line
approximation to for with .
Use Legendre polynomials.

. The transformation maps the interval to .

Thus the least-squares straight-line approximation to

on may be obtained by finding the least-squares
approximation to on .
.
If we let , we may simply write down the
approximation desired, using results of , as

Thus the approximation to for is

shows and on .
. Approximation to by on .
Thank you for attention!
text

text

text
.
text

text

text
.
text

text

text
. text

text

text