Hello Again, Linear Algebra - A Second Look at The Subject Through A Collect

Visit http://insightfultexts.com for videos and supplementary content.
2
Chapter 1 Introduction
Linear algebra is a grand subject. Because it is fundamentally different from any high
school mathematics, and because of the wildly varying quality of instructors, not all students
enjoy learning it. If you hated it, I blame the instructor. There I said it. In either case, whether
you loved it or hated it, it takes several passes to learn linear algebra to the point that it
becomes one of your favorite tools, one of your favorite ways of thinking about practical
problems.
This little textbook invites you on your second pass at linear algebra. Of course, your
second pass may take place alongside your first pass. You may find this textbook particularly
useful if you are studying for a test. Our goal is to take a step back from the mechanics of the
subject with an eye towards gaining a larger view. A larger view, however, is achieved in
small steps. We are not hoping for a big revelation but for a few small aha! moments. It
simply takes time to put together the grand puzzle of linear algebra. You will get there, and
the point, as the cliche goes, is to enjoy the ride.
Psychology in mathematics is everything. I chose the topics according to the impact I feel
they would make on your relationship with linear algebra. The textbook's utmost goal is to
make you feel positively about the subject. You will find that some topics are surprisingly
simple, others surprisingly tough. Some topics have important applications, others have none
at all. Some were well presented in your linear algebra course, others skipped altogether.
However, I hope you will find that all topics bring you a little closer to the subject of linear
algebra.
There are several points that this textbook emphasizes over and over again. Some of these
points are made explicitly. Others are more difficult to put into words and it is my hope that
these points become clear as a result of considering the many examples analyzed here.
Therefore, I would like to make these points right here with minimal explanations.
1. The applications of linear algebra are extraordinarily broad. The subject applies to any
type of objects that can be added together and multiplied by numbers. These objects are united
by the term vector and include pointed segments, polygons or – more generally – functions,
sets of numbers organized in a column, matrices, currents, voltages, substances in chemical
reactions, strains and stresses in elastic bodies, sound signals, portfolios of financial securities.
Need I go on?
2. Linear algebra studies the commonalities among these objects. However, focusing on
the commonalities is counterproductive if one ignores the differences. It is that the fact that
these objects are all uniquely different that makes linear algebra such a remarkable tool. We
3
therefore invite the reader to accept and treat each object on its own terms. If the problem
relates to pointed segments which we call geometric vectors, by all means let us have a
geometric discussion and talk about straight lines, lengths, and angles. Do not rush to
associate a geometric vector in the plane with a pair of numbers. It is not necessary and, in
fact, detrimental to clarity, to do so. Unfortunately, just about every textbook in the land does
this and in doing so obscures much of the psychology of linear algebra. Similarly, when
dealing with , lets talk about entries, rather than lengths and angles. Sets of numbers have
entries and nothing else. Sets of numbers don't have lengths and directions, and when they do
eventually acquire these notions, it doesn't happen in the corrupted way presented in most
textbooks. Finally, functions are functions, and polynomials are polynomials. There's no need
to think of polynomials as "vectors" of coefficients. Polynomials are cool on their own terms.
Other than adding them and multiplying them by scalars, you can differentiate them and even
dilate them. Can you do that with "vectors" of coefficients?
3. Let the problem dictate the basis, or say no to Cartesian coordinates! This point is
related to the preceding point and applies to more than the subject of linear algebra. There
seems to be a lamentable tradition to refer the space of our everyday physical experience to
Cartesian coordinates, whether or not it is the best coordinate system and, even more
lamentably, whether or not coordinates are needed in the first place. If I may speak for
Descartes, he would have been appalled by this state of affairs! His ingenious invention of
coordinates was meant to unite the worlds of algebra and geometry. The knee-jerk reaction to
translate a physical problem to numbers, typically by introducing a Cartesian coordinate
systems, and often doing so tacitly, usually leads to loss of simplicity, vision, clarity, and
understanding.
In linear algebra, choosing a coordinate system is equivalent to choosing a basis and the
Cartesian syndrome manifests itself by choosing the orthonormal basis in the plane:
or the basis for the space of cubic polynomials:

4
or the basis in :
First of all, as was just discussed in point 2., don't resort to a basis in the first place,
thereby converting a "real-life" problem to numbers, unless the reasons for doing so are clear
to you. Good reasons for doing so, for instance, are described in Chapter With Or Without a
Basis. Secondly, when the time is right to convert the problem to by decomposing all
elements of the problem with respect to a basis, let the problem dictate your choice of basis.
This is much more than a matter of practicality. If a basis is chosen poorly or, worse yet,
introduced unnecessarily, the artifacts of the basis will obscure the analysis of the problem.
My first linear algebra professor Bill Thurston states in his famous essay On Proof And
Progress In Mathematics that the value understanding is greater than that of the formal
correctness of a solution. For an engineer (even if you are not an engineer, I'll think of you as
an engineer: you want to create things, don't you?) the value of understanding is even greater
because the products of your labor are required to work. So if don't wish for the bridge you
built to collapse – second guess yourself when you instinctively reach for a basis when
presented with a problem. Heal yourself of the Cartesian syndrome. Here! – I included a
portrait of Descartes, because I think that he would agree with me.
5
Chapter 2 One too many!
The figure below shows various arrangements of three vectors in the plane. Which
arrangement has three linearly independent vectors?
In our linear algebra course, we come to this question in the second week. During the first
week, we discuss linear combinations of vectors and the concept of decomposition:
representing vectors by linear combinations of other vectors. Then comes the definition of
linear independence: a set of vectors is linearly independent if none of the vectors can be
represented as a linear combination of the rest. Linear dependence is the opposite of linear
independence: a set of vectors is linearly dependent if at least one of the vectors can be
represented as a linear combination of the rest.
If you don't feel that the definition of linear dependence is obvious or, at least, natural in
some way – I'm with you! However, this definition has shown over the years to lead to a very
effective way of organizing the subject of linear algebra. This problem, which culminates in a
fundamental realization, illustrates one aspect of the effectiveness of this definition.
Let us start eliminating the linearly dependent sets. First of all, a linearly independent set
cannot contain the zero vector , since it can be expressed as a linear combination of any set
of vectors! Just set the coefficient to for every vector in the linear combination. For
example, for any two vectors and ,
This linear combination, all of whose coefficients are zero, is called the trivial linear
combination. It is a very special combination and it is perfectly valid. The definition of linear
dependence does not call for a nontrivial linear combination: just one way – any one way –
6
of expressing one of the vectors in terms of the rest.
It is also unimportant that the vector cannot be expressed in terms of and . The
definition of linear dependence only requires that there is a vector in the set that can be
expressed in terms of the rest. There is no requirement that every vector be expressible in
terms of the rest.
And so, the arrangements and are out. What about the arrangement ? The vector
certainly cannot be expressed in terms of and , since and point along the same line
and any linear combination of and necessarily points along the same straight line.
However, is a multiple of
and it is therefore a linear combination of the vectors and , and the vector is not even
needed to express . That is not particularly relevant since the definition of linear dependence
does not require that all of the remaining vectors are used in the linear combination. And even
if it did, we could simply assign a coefficient of to :
This eliminates the arrangement . Similarly, the vectors in the arrangement are linearly
dependent since
This leaves us with the arrangements and which, in a manner of speaking, are the most
linearly dependent of them all! We say this because each of the vectors can be expressed as a
linear combination of rest. (Not that the definition of linear dependence requires that every
vector be expressed as a linear combination of the rest. Just one of the vectors is enough for
linear dependence.) The vectors in the arrangement are such that
Equivalently,
and
The vectors in the arrangement are such that
7
A moment of reflection. We owe ourselves a moment of reflection. We tried, tried, and
tried again, but to no avail! No matter what configuration of three vectors in the plane we try,
we end up with a linearly dependent set. What is going on here? What we are experiencing
here is a fundamental law of life: a set of more than two vectors in the plane is necessarily
linearly dependent. The linear dependence may manifest itself in a variety of ways, as we just
discovered. But the fact of linear dependence is inescapable.
This fact is not an axiom or a postulate. This fact it is not a theorem or a corollary. This
fact is something that slowly crystallized out of mathematical human experience and gave the
geometric inspiration to the concept of linear dependence on which the subject of linear
algebra is built.
In the formal subject of linear algebra, there is a theorem that captures this experience. The
theorem states that all bases in a linear space have the same number of elements. That
common number is called the dimension of the space. (The dimension of the plane is .) It
then follows that any set that contains more vectors than the dimension of the space is linearly
dependent. With a little bit of formal thinking you will be able to explain why.
Exercise 1 Explain why augmenting a linearly dependent set with an additional

vector cannot result in a linearly independent set.
Solution If one of the vectors in the original set can be expressed as a linear
combination of the rest, for example,
then the addition of a vector , does not alter the original relationship, which can now be
written as
Exercise 2 Explore the three dimensional case and convince yourself that four or
more vectors are necessarily linearly dependent.
Exercise 3 Show that the polynomials
8
are linearly dependent. This exercise illustrates the unique power of linear algebra: you may
not be able to determine the linear combination that expresses one of these polynomials in
terms of the rest, but one can state with confidence that a relationship like that exists.
Solution Cubic polynomials form a four dimensional set. This is because the set
can serve as a basis. Therefore, being five vectors in a four
dimensional space, the polynomials , , , , and are linearly
dependent.
Exercise 4 Show that the set of vectors
is linearly dependent.
Solution Since is three dimensional (explain why), the four given vectors are
linearly dependent.
Solution These vectors are linearly dependent because
9
A set of vectors containing the zero vector is always linear dependent. In fact, the zero
vector by itself is considered linearly dependent. This will become evident from an
alternative formulation discussed in the next chapter.
Solution The vectors are linearly dependent because
Solution The argument given in the preceding exercise is no longer available, but a
more subtle argument can be given. The three given vectors all belong to the subspace
which is two dimensional (explain why). Thus, being three vectors in a two dimensional
space, the three given vectors are linearly dependent.
Exercise 8 On a clock, can the second hand always be expressed as a linear

combination of the hour hand and the minute hand? If not, during what times is it
impossible?
10
Solution It is impossible at moments when the hour hand and the minute hand are
aligned (but the second hand points in a different direction – that excludes o'clock). This
happens at times given by the equation
For example, the first time this occurs is at 1:05.45454545... o'clock. It also happens when
the hour hand and the minute hand point in the exact opposite directions.
11
Chapter 3 Or, to put it another way...
A linearly dependent set of vectors is one in which one vector can be represented by a
linear combination of the rest. There is a different, but entirely equivalent, way to say the
same thing. Namely: a set of vectors is linearly dependent if there exists a nontrivial linear
combination of the vectors that yields the zero vector. The key word is nontrivial. It means
that at least one of the coefficients in the linear combination is nonzero. The word nontrivial
is absent from the other definition. If one of the vectors in the set is the zero vector, then it is,
in fact, the trivial combination of the rest. So allowing trivial combinations is the crucial part
of the first definition.
The new definition doesn't single out one vector and treats all vectors equally. But it
requires that the linear combination is nontrivial. If the word nontrivial were missing from the
new definition, it would be completely meaningless. Because for any set of vectors there
exists a linear combination that results in the zero vector – the trivial combination, of course!
But a nontrivial combination exists only when the set is linearly dependent.
The two definitions are completely equivalent. That equivalence is easy to show, and
doesn't take long to get used to. If one vector, say , is a linear combination of the rest, say ,
, , and , in other words
(where , , , and may all be zero), then bringing all terms from the right hand side to the
left, we have
and, there you have it: we have a nontrivial linear combination that produces the zero vector.
How do we know that it's nontrivial? Because the coefficient that multiplies is , so at least
one coefficient is nonzero. This shows that if the set of vectors is linearly dependent in the
original sense, then it is linearly dependent in the new sense.
Going in the other direction is just as easy. Suppose that a set of vectors is linearly
independent according to the new definition. That means, there exists a nontrivial linear
combination that yields zero:
This linear combination is nontrivial. That means that one of the coefficients , , , , or is
12
nonzero. We don't know which one, but we know that one of them is. But whichever it is, the
vector it multiplies can be expressed in terms of the rest simply because we can divide through
by the nonzero coefficient. For example, suppose that . Then
and there you go: one vector is a linear combination of the rest. Division must always be
approached with care, lest we divide by zero, but if we know that the linear combination is
nontrivial, then we can safely pick out the one that is nonzero. So there we have it: the two
definitions are completely interchangeable!
Why do we need two equivalent definitions? We don't. But in certain situations, one
formulation is easier to appeal to than the other. For example, why are the polynomials
linearly dependent? Because their sum is zero! (Note that these four polynomials come from
the four dimensional space of cubic polynomials so the "one too many" argument does not
apply.)
Exercise 9 Explain why the following polynomials are linearly dependent:
Solution The polynomials are linearly dependent because
Here's another neat example where it is more natural to come up with a nontrivial linear
13
combination that yields zero. This example comes from geometry. Consider three unit vectors
pointing in three different directions in the plane. Of course they are linearly dependent: any
two of these vectors can express any other vector in the entire plane. But how does one
determine the linear combination geometrically? The answer comes from calculus! Or,
perhaps, you can call it physics. We all agree that a rigid body, no matter the shape, immersed
in a uniform pressure field will remain in equilibrium. However, the force of pressure acts
along the normal. Thus pressure per unit area is , where is the external unit normal.
Since the total force of pressure vanishes, we must have
where is the surface of the rigid body. This formula must, therefore, hold for the normal
field over an arbitrary surface . In fact, this formula works in two dimensions, too, if the
surface integral is replaced with a line integral. The unit normal field for a general closed
curve is shown in the figure below. The figure does not justify or prove this property of the
normal field, but it does illustrate our intuition that all the normal vectors added together may
very well cancel out to .
The same property of the normal field is valid for polygons despite the sharp corners. For
polygons, the integral can be represented by a sum with a term for each side:
where are lengths of the sides of an -sided polygon and are the corresponding
normal. For a triangle with sides , , and , this formula reads
14
and this is key for constructing a nontrivial linear combination of three vectors , , and
that yields zero. Draw a straight line perpendicular to that passes through the tip of , and
do the same for and . The straight lines will intersect to form a triangle with sides , ,
and . This is illustrated in the figure below.
For the obtained triangle, the last equation reads simply
where , , and , are the lengths of the vectors , , and . This is so because ,
, . Thus, the coefficients , , and give a nontrivial linear
combination that yields the zero vector.
Exercise 10 Devise a geometric method for constructing a nontrivial linear

combination that yields zero for four nonzero vectors pointing in four different directions in
the three dimensional space.
Solution The solution is based in constructing a tetrahedron (instead of a triangle) by

drawing planes orthogonal to each vector. The lengths of sides are replaced by areas of
faces.
The last geometric example is very appealing! But here's the real reason why this
alternative formulation of linear dependence is so valuable. It has to do with solutions of
linear systems. Or, as linear algebra people like to say instead of linear systems: .
Consider the following system with the dialpad matrix on the left hand side
15
One solution is obvious: since the right hand side matches the first column of the matrix, we
have
Are there other solutions? Of course the answer to this question can be revealed by Gaussian
elimination, but there is a more insightful approach to this problem. The key is to interpret the
linear system as a decomposition problem:
Written in this form, the problem is to decompose the vector
in terms of
OK, now what? Well, the vectors , , and are linearly dependent! The easiest way to see it,
is by the original definition. The vector is the average of the vectors and :
This observation helps us determine the fact of linear dependence, but has not yet allowed us
to capture the multitude of possible solutions to the system. That happens when the last
equation is rewritten as a nontrivial linear combination that yields zero. Bringing and to
the left hand side (and multiplying it by to make it prettier), we find
16
This is relevant because it tells that the combination
corresponds to the zero vector. We call it a fancy zero.

What happens when you add a fancy zero to the solution that we guessed:
It certainly alters the solution, but it doesn't change the right hand side because the
combination with coefficients , , corresponds to zero! More formally, you can see it this
way
So the solution is altered, but the right hand side is unchanged, so the new vector
is still a solution!
This perspective immediately leads us to a way of capturing the general solution, that is
the set of all solutions to the system. A fancy zero – since it leaves the right hand side
unchanged – can be added to a solution in any proportion and, while altering that solution (by
17
virtue of being fancy, i.e. nontrivial), will leave the right hand side unchanged. Thus, the
general solution leads
where is an arbitrary number.
Exercise 11 Find the general solution to the system
Solution The general solution can be written as
Importantly, whenever the null space is nontrivial, there are infinitely many ways to capture
a solution by an algebraic expression. For instance, another way to capture the solution of
this system is
and yet another is
18
Solution
Solution
Solution
19
Solution
Exercise 16 Among the routine matrix product exercises, we usually do one like
this:
Go ahead and calculate this product. The answer is the zero matrix. At first glance
(but only at first glance!) it's surprising that matrices with so many nonzero entries produce
the zero matrix. How do you think I come up with matrices like these?
Exercise 17 Suppose that , where is a full rank matrix and is

. What is the highest possible rank of the matrix ?
Solution Since each column of is in the null space of , the rank of is at most .
20
Chapter 4 The case is closed!
Let us add a new dimension, so to speak, to our ability to identify linear dependence
among vectors by pure insight and simple counting. This new dimension is particularly rich
and insightful. Let us gain start with the question: are the following polynomials linearly
dependent?
The short answer is yes, but it's not the short answer we want. The long answer will be much
more interesting. Right off the bat, the long answer is not based on the fact that these
polynomials are related by the linear combination
Rather, it is based on the idea of linear properties and linear subspaces.

Let me approach this by asking a simple question with a simple answer. Can the number
be represented by forming sums and differences of the numbers , , , , and ? Of
course not! Everyone can easily see the reason: one cannot get an odd number out of even
numbers by adding and subtracting. Another way to express this observation in words is to say
that the property of being even is closed under addition and subtraction. The sum of any two
even numbers is even and the difference of any two even numbers is even. In particular, the
number cannot be obtained by adding and subtracting the numbers , , , , and .
What does this nearly trivial example have to do with the problem at hand? Everything!
The four polynomials also share a property that's closed under addition. And not only under
addition, but under multiplication by numbers, as well. Therefore, these polynomials share a
property that's closed under linear combinations.
Can you see what that property is? Perhaps not yet. No problem, we'll do another example
that's about half way between the trivial example of even numbers and the actual problem at
hand: Are the following polynomials linearly dependent?
21
These four polynomials have something very striking in common: the constant term is zero. Is
that significant? Yes, because this property is closed under linear combinations. Such a
property is called linear. Indeed, the sum of any two polynomials with this property will have
a zero constant term. Multiplication of any polynomial with this property by a number will
also result in a polynomial like that. Therefore, any linear combination of polynomials with
zero constant term will be another such polynomial.
Why does this closure under linear combinations imply the linear dependence? Because
. Closure under linear combinations implies that polynomials with zero constant term
form a linear subspace. (After all, a linear subspace means than being closed under linear
combinations.) By a geometric analogy, polynomials with zero constant term form something
similar to a plane in a three dimensional space.
What is the dimension of the subspace formed by these polynomials? Being a subspace of
a four dimensional space, it has a dimension that is certainly less than . In fact, the dimension
of polynomials with zero constant coefficient is , but the important part here is that it's less
than four. And since , , and are four polynomials in a less-than-
dimensional space, they are necessarily linearly dependent.
22
Chapter 5 Find the middle ground!
Determine the column space of the matrix:
I know what you are thinking: what a messy matrix! What chance does anyone have of
determining its column space without doing much work? And I'm certainly not looking
forward to calculating the RREF of this matrix!
Well, this problem is not about number crunching but once again about closure. The
columns of this matrix belong to a subspace of that is easy to identify. Each of the column
of the matrix follows the pattern
In words, in each column, the middle entry is the average of the other two. The set of all
vectors of this form is closed under linear combinations. Therefore, it forms a linear space :
I prefer this way of capturing linear spaces. Some of you may prefer to separate and
which gives an explicit basis for .

One detail that remains to be filled in. What we have actually determined so far is that the
column space is contained , since every element of is also in . But we have not yet
shown that is all of . However, that follows from considering the dimension. The
dimension of is , which is apparent from the equation above. On the other hand, is at
least two dimensional since the first two columns of are linear independent (the only time a
23
set of two vectors is linearly dependent is when one vector is a multiple of the other).
Therefore, .
In this problem, we once again see the importance of closure under linear combinations
and the simple counting of dimensions.
Exercise 18 What is the column space of the matrix
Solution
Solution
24
Solution
Solution
Solution
25
Exercise 23 Suppose that is a matrix such that
This is not enough information to determine the column space of , but you should be
able to determine the linear space that is smaller than which contains .
Solution
26
Chapter 6 Linear or Not?
I hope that the preceding discussions have convinced you of the importance of considering
linear properties and the corresponding linear subspaces. This chapter is devoted to the linear
properties themselves, without discussing their broader implication. In hindsight, this topic
will seem to you utterly simple. You will be able to look at an expression and be able to know
right away whether it describes a linear property. The purpose of this chapter is to merely get
you there faster.
Here's the motivating exercise for the upcoming discussion: Is the following relationship
for the set of cubic polynomials linear? If so, identify a basis for the corresponding linear
subspace. The property reads:
Looking slightly ahead, the answer will be yes, it is a linear property. Here's how one goes
about arriving at this answer. A property is called linear if the set of vectors that satisfies the
property is closed under linear combinations. In fact, let me list the three synonymous ways of
saying the same thing:
1. The property is linear
2. The set of vectors that satisfy the property is closed under linear combinations
3. The set of vectors that satisfy the property is a linear space
The second statement is the most actionable, so let us concentrate on closure under linear
combinations. Breaking it down into two more elementary pieces for the example at hand, it
means that if two polynomials and satisfy this property, then the product
(for any number ) and the sum have this property as well.
Is this the case for the zero-integral property we are considering? There is a formal
approach (which I am about to show) to answering the question, but in most cases, simply
expressing the property in words works even better. Our property can be expressed by saying
that the signed area under the graph of from to is zero. Would the same be true for,
say, . Of course, because the factor of stretches the graph in the vertical direction,
thus changing the total signed area by a factor of , which keeps it at zero since .
Therefore, the zero-integral property is closed under multiplication by a scalar.
27
What about closure under addition? If two functions and satisfy the zero-
integral property, does their sum satisfy it, as well? In other words, if we
have two functions and with zero signed areas under the graph from to ,
does their sum also have zero signed area. Perhaps you can already tell that
the answer is yes by imagining how signed areas interact when two functions are added
together. However, I will use this as an opportunity to demonstrate the formal approach.
Let us begin by writing down that mathematical expressions that state that and
satisfy the zero-integral property:
Now, what would need to hold in order for to satisfy the same property.
Naturally, it is the following:
Can this identity be demonstrated on the basis of the two preceding equations. Of course, the
answer is yes
Thus we conclude that the zero-integral property is, in fact, linear.

Let me give you an example of a property that's not linear:
Let us analyze, by the same approach, whether this unit-integral property is closed under
addition. In other words, whether and the two equations
28
imply that satisfies the unit-integral property:
Let's see:
Thus, the unit-integral property is not closed under addition and is therefore not linear.
Returning to the linear zero-integral property, let us determine the basis for the linear
subspace of cubic polynomials that the property defines. Almost invariably, the question of
finding a basis for a subspace comes down question of null space for a matrix.
Consider a typical cubic polynomial
Its integral from to is
Thus, the polynomial satisfies the zero-integral property if
or
29
In other words, we have arrived at the question of null space for the matrix
which can be expressed by relating columns , , and , to column :
The three vectors from in this expression are linearly independent and form a basis for .
They correspond to the following linearly independent cubic polynomials
which form a basis for the subspace of cubic polynomials that satisfy the linear zero-integral
property.
Let us consider one more polynomial example prior to the practice exercises. Consider the
following property
In an exercise below you will determine that it is also linear. Now, let us consider the space of
polynomials that satisfy both, the zero-integral property and . In another exercise
below, you will be asked to show that a simultaneous combination of two linear properties is
itself a linear property. Let us construct a basis for the resulting linear subspace of cubic
polynomials.
For a polynomial with coefficients , , , and the property implies that
Therefore, this equation and the previous equation
must be satisfied simultaneously. This problem is equivalent to determining the null space of
the combined matrix
30
The RREF of is
which yields the null space
The two vectors that form the basis of correspond to the following polynomials that
form the basis the subspace defined by the simultaneous zero-integral equation and
:
In the following exercises, determine whether the property of functions is linear and, if so,
determine the basis for the subspace of cubic polynomials that satisfy this the property.
Exercise 24 Functions such that
Solution Yes:
Exercise 25 Functions such that (No)
Solution Yes:
31
Solution Yes. Let . Then
or
The null space of the matrix is
therefore
Exercise 29 Functions such that (Yes. Details left to the

reader.)
32
or
The null space of the matrix is
therefore
Exercise 33 Functions such that (Yes. Details left to the

reader.)
or
33
Let us row reduce the matrix above, but in a nonstandard way that takes advantage of the
small numbers at the tail end of the matrix. First, subtract row from row
Next, subtract row from row :
The resulting form is technically not the , but it is just as effective at helping us
construct the null space :
This null space corresponds to the basis
Exercise 35 Functions such that . (No)
Exercise 36 Polynomials whose coefficients add up to .
Solution Yes:
34
Exercise 37 Polynomials whose coefficients add up to . (No)
Exercise 38 Polynomials whose coefficients add up to its value at .
The rest is left to the reader.
Exercise 39 Polynomials whose coefficients add up to the value of its

derivative at . (Yes. Details left to the reader.)
Exercise 40 Polynomials whose coefficient equals .
Solution Yes:
Exercise 41 Functions such that .
Exercise 42 Functions such that . (Yes.

Details left to the reader.)
35
Exercise 43 Functions such that .
Exercise 44 Functions such that . (No.)
Exercise 45 Polynomials that include only even powers.
Solution Yes:
Exercise 46 Polynomials that include only odd powers.
Solution Yes:
(Yes. Details left to the reader.)
36
Exercise 48 Show that the set of polynomials such that is
closed under addition but not multiplication by a number.
Exercise 49 Show that the set of polynomials , such that is an

integer, is closed under addition but not multiplication by a number.
The following exercises deal with geometric vectors.
Exercise 50 Geometric vectors contained between two rays, as in the following

figure.
Solution Closed under addition, but not multiplication by a number. For example,
multiplying a nonzero vector by result in a vector not contained between the two rays.
Exercise 51 Geometric vectors contained between two straight lines, as in the

following figure.
37
Solution Closed under multiplication by a number, but not addition. Finding a
specific counterexample is left to the reader.
38
Chapter 7 RREF! RREF! All bark no bite
Calculate the row reduced echelon form (RREF) of the matrix
Like all the preceding and upcoming problems in this book, this problem only looks
complicated. But, with a little bit of insight, it's a piece of cake.
The RREF is all about the relationship among the columns. The RREF of is a matrix
that is much simpler than (i.e., has a lot of zeros) and has the same relationship among its
columns as the original matrix . For example, you will notice the following relationship
among the first three columns of :
That means that RREF of will also have this relationship among its first three columns.
Therefore, the first three columns of RREF of are
Isn't that cool? We are half way through the usually cumbersome procedure of Gaussian
elimination and Jordan back-substitution and we didn't have to do any work!
The first two columns of RREF of are
because that is what the RREF procedure does: it converts the columns that are linearly
independent from the preceding columns into this form: the -th such column is the -th
column of the identity matrix. You are guaranteed to always be able to achieve that.
39
Let us now take a look at the next two columns of . The fourth column is linearly
independent from the first three. Can you see why? Because the first three columns share the
linear property that the second entry is the average of the first and third entries. The fourth
column does have that property and is therefore linearly independent of the first three. The
fifth column is also linearly independent from the ones preceding. I leave it to you to justify
that statement. Thus, the first five columns of the RREF of read
Finally, the last column must be linearly dependent on the ones before it since, if nothing else,
we've run out of dimensions. Since
we conclude that
and we are done!
Exercise 52 Calculate the RREF of the matrix
Solution
Exercise 53 What is the RREF for the matrix consisting of two matrices
40
(from the main discussion) side by side?
Solution
Exercise 54 For a nonsingular square matrix what is the RREF of ?
Solution The identity matrix :
Solution Two identity matrices side by side
Solution
Exercise 57 Finally, for a nonsingular square matrix what is the RREF of ?

This should give you a new perspective on finding the inverse by Gaussian elimination.
41
Solution Let's ask the following question: in the matrix , what is the
relationship between the -st column (that is, the first column of ) and the preceding
columns? Of course, that relationship is given by the first column of . In order to see
this, interpret the identity
from the column-by-column point of view. That gives
Similar relationships can be written for all other columns of the identity matrix in .
Since the relationships among the columns are the same in the RREF of the matrix, we
conclude
In words, converting the matrix to RREF, results in in the right half of the
combined matrix. So that's why that method works!
Exercise 58 Explain why any matrix is represented by the matrix product
where is an matrix that consists of the pivot columns of , padded by zeros on the
right, if necessary.
42
Chapter 8 Go Back!
Suppose that the null space of a matrix is given by
What is the RREF of ?

The beauty of the RREF is that it makes the null space so easy to determine. For example,
consider the RREF of the matrix discussed in the previous example:
If you recall, we determined this matrix by identifying the relationships among the columns of
the matrix . From the educational point of view, it was insightful. But from the point of
view of real life applications, it was backwards! In the real world, the matrices are
complicated and the RREF is computed by Gaussian elimination in order to determine the
relationship among the columns!
Suppose that was the case with the matrix above: the RREF was computed by Gaussian
elimination. Given the RREF, the relationship among the columns becomes a completely
straightforward matter that requires no insight. Each nonpivot column corresponds to a basis
vector in the null space that represents its linear dependence on the pivot columns preceding
it. Thus, the null space of the matrix is
43
Going in the opposite direction, null space RREF, is a relatively straightforward matter
if the null space basis is available in this "canonical" form. For example, suppose that the null
space of a matrix with four columns is given by
then we can infer, from the locations of the 's, that the nonpivot columns are 3rd, 5th, and
6th. Meanwhile, the rest of the nonzero numbers in the null space basis vectors tell us how the
nonpivot columns are obtained as linear combinations of the pivot columns. In summary, from
this null space basis, we can restore the as follows:
Easy enough! But what if the basis for the null space is not given in the "canonical" form?
(This is the case for the vector given at the beginning of the chapter.) Then we need to find
another basis that is in the canonical form – but how do we do it?
What tools do we have at our disposal? Notice something about the RREF procedure. It
does not change the relationships among the columns (that being the whole point). But it also
leaves the row space unchanged. That is because each of the three Gaussian operations
1. multiplying a row by a nonzero number
2. switching rows
3. adding a multiple of one row to another
leaves the span of the rows unchanged.
The task we are facing here is to transition from the three vectors presented at the
beginning of the chapter to another set of three vectors (with some special properties) that
span the same space. So, in order to preserve the span, we should perform the Gaussian
operations on those vectors. Sticking to the three Gaussian operations would ensure that at
every step we have three vectors that span the null space. Now we only need to convert the
vectors to the desired form.
To determine what that desired form is, let us take another look at the "easy" example.
Combine those vectors, which already have the desired form, into a single matrix
44
Kind of looks like an RREF, doesn't it? An upside-down-and-backwards RREF! Transpose
this matrix with respect to the "other" diagonal (known as the reverse diagonal, marked in
bold in the following equation) and you will see a proper RREF matrix:
Now the strategy for converting a given basis to the canonical form is clear:
1. Combine the vectors into a matrix
2. Reverse-transpose the matrix
3. Calculate the RREF of the obtained matrix
4. Reverse-transpose the RREF
5. Interpret the columns as the new basis vectors.
For the problem at hand, this strategy yields the following results:
45
Now we have a new basis for the null space, but this one has the canonical RREF form and
the original question of the chapter (what is the RREF given the null space) is easy to answer:
Done!
Exercise 59 Confirm by forming an appropriate matrix product that the RREF

matrix in the above equation is correct.
Solution The matrix product that confirms that matches the desired null space
is
Exercise 60 Compute the RREF of a matrix whose null space is given by
Solution
46
Exercise 61 Given two sets and of vectors in , devise a strategy for
determining whether the two sets have identical spans. Start by considering only linearly
independent sets with equal number of vectors.
47
Chapter 9 It's elementary, Watson!
In each of the following two equations, determine the unknown matrix:
This discussion will be about elementary matrices. By definition – let's make it a casual
definition – an elementary matrix is one that can be obtained from identity by one – or, at
most, a handful – easy-to-see column or row operations. If you can say how to obtain a
particular matrix from the identity matrix, then that matrix is elementary. For example, the
following matrices are all elementary
Go ahead and state how each of these matrices can be obtained from the identity by row
operations and by column operations. It's important to say the operations out loud!
Now let's see what makes elementary matrices special. What makes them special is that
they are keepers of row operations and column operations. (We'll find out that, by extension,
all matrices are keepers of row and column operations. But elementary matrices store just a
handful of operations at a time making them easier to use.) As a result, elementary matrices
can represent many of the algorithms of linear algebra by multiplication. This helps reduce
algorithms to algebraic expression – a powerful idea!
Here is what we mean when we say that elementary matrices are keepers of column and
row operations. Consider the following product
48
In words, what was the effect of multiplying the matrix with letters by the elementary
matrix on the right? It was adding times the 3rd column to the 2nd. OK, good! And
how is the matrix obtained from the identity by column operations? The answer is: by
adding times the 3rd column to the 2nd. It is as if the matrix remembers how it was
built from the identity and passes those steps on to any matrix in multiplies from the right.
When the multiplication takes place from the left, rows are affected:
Let's again put in words what we just observed: the effect of multiplying by on the left is
adding times the 2nd row to the 3rd. And how is the matrix obtained form the identity
by row operations? Of course, by adding times the 2nd row to the 3rd. So we notice the
same phenomenon, but for rows instead of columns. This, by the way, is a prime example of
the matrix product being non-commutative.
Before moving on to the "inverse" problem with unknown matrices, it's a good idea to
solve a few "forward" problems involving multiplication by elementary matrices. As you
work out these exercises, note that this rule applies to all row column permutations including
the swap.
Exercise 62 Evaluate the matrix product
Solution
49
Solution
Solution
Solution
50
Solution
Solution
51
Solution
Solution
52
Solution
All right, let's now turn to the inverse problems with unknown matrices. Determine the
unknown matrix in the following identity
In order to determine the unknown matrix, we need to answer the following question: how is
the matrix on the RHS obtained from the matrix on the LHS by column operations? We are
considering column operations because the unknown matrix is on the right side of the product.
The answer to this question is: multiply the 1st column by 2, and add 3 of the 3rd column
to the 2nd. So, in order to build the elementary matrix that effects these operations, simply
apply these steps to the identity matrix. The result is
and that's the correct answer! You can double check by direct multiplication.
Now let's do a row-wise example:
In order to determine the unknown matrix, we need to state the row operations that transform
the matrix on the LHS to the matrix on the RHS. Those operations are: multiply the 1st row
by 2, and add the 3rd row to the 1st row (in that order!). Thus, construct the unknown
matrix, apply these two steps to the identity
53
Done!
If you think about, what's going on right now is quite impressive. Solving these problems
is equivalent to solving three systems, each with three equations and three unknowns (so there
are nine unknowns in total).
Let us now solve a couple of problems that are a little bit more subtle. Here is one:
In order to answer this question, you much describe the path from the matrix on the LHS to
the matrix on the RHS, in terms of column operations – since the unknown matrix multiplies
on the right. With numbers instead of letters, the necessary operations may not be as easy to
see, but I think this case is not too hard: add of the 2nd column to 1st. Thus the
unknown matrix is obtained in one step:
Here's another example
This time, it takes two steps to achieve the desired transformation: multiply 1st column by
and add 2nd column to 1st. Thus, the unknown matrix is constructed as follows
Once again, note that the order is important. Performing the exact same steps in the
opposite order leads to the wrong matrix
and the wrong result
54
Finally, a simple row-based example
Notice that the first two rows are unchanged. So the only question is how the 3rd row of the
matrix on the RHS are obtained from the rows of the matrix on the LHS. The answer is: add
1st and 2nd rows to 3rd. You may think of this as two operations. I think of it as one:
We can now revisit the original problems of this chapter:
What makes them just a little more challenging than the problems already discussed is that
they involve more than one or two steps to achieve the goal. For the first problem, the column
operations are: a). divide 3rd column by , b). add 3rd column to 1st column, c). subtract 4th
column from 1st, and d). multiply 4th column by . Thus the known matrix is obtained as
follows (showing two steps at a time)
For the second problem, the row operations are: a). divide 3rd row by , b). add of 1st row to
3rd, c). multiply 4rth row by , and d). subtract of 1st row from 4th. The result is
55
Finally, note that the steps required to achieve the desired result are not unique. For example,
the first two steps could have been chosen as a). add of 1st row to 3rd row and b). divide 3rd
row by . These steps are different but, of course, they lead to the right result and the same
unknown matrix:
Exercise 72 Determine the unknown matrix in the product
Solution The necessary column operations are: i). subtract 2nd column from 1st, ii).
subtract 3rd column from 2nd, iii). subtract 4th column from 3rd. These operations are
captured by the following matrix:
56
Solution The necessary row operations are: i). subtract 3rd row from 4th, ii). subtract
2nd row from 3rd, iii). subtract 1st row from 2nd. These operations are captured by the same
matrix as in the preceding exercise:
Exercise 74 Notice that the answers in the two preceding exercises were identical.
This is unlike all of the previous examples where the answers were different depending on
whether the unknown matrix appeared on the right or on the left. Explain, why this case is
different.
Solution
57
Solution
Exercise 77 Show that in matrix product , the result is unchangedif the columns
of and the rows of undergo identical swaps. This can be explained without elementary
matrices. However, with the help of elementary matrices, the argument can be reduced to an
algebraic identity.
Exercise 78 Suppose
What is ? Answer the question without calculating first.
Solution
58
Chapter 10 Use the force, LU!
The LU is a matrix factorization that corresponds to Gaussian elimination. It is very much
tied to the use of elementary matrices. The point of this chapter, however, is to take a step
back and to investigate the overall structure of the LU decomposition.
Find the LU decomposition of the matrix
The key to answering this question is realizing the in the product is what's left of
after we are through with Gaussian elimination, where it is assumed that the pivots are not
normalized to unity but rather left as they are. Therefore, in this case, can be determined
without going through row operations:
In order to determine consider the identity
Interpreting as an elementary matrix, it becomes apparent that
Exercise 79 Find the LU decomposition of the matrix
59
Solution , .
The following few exercises are related to elementary matrices.
Exercise 80 Find the LU of the matrix
Solution
Solution
60
Solution
Solution
61
Solution
62
Chapter 11 You can run, but you can't hide!
Determine all four eigenvalues of the following matrix
Let us use this problem as an opportunity to highlight several special features of matrices in
which some of the eigenvalues and the corresponding eigenvectors can be easily determined.
Feature 1: A single nonzero entry in a column occurs on the diagonal. Consider the
following matrix
where denotes the entries of the matrix that are inconsequential to this discussion. Since for
we have
Therefore, we conclude that is an eigenvector and that is the corresponding eigenvalue.

Feature 2: All rows add up to the same value. Consider the matrix
63
whose rows all add up to . Since for
we have
we conclude that is an eigenvector and that is the corresponding eigenvalue.

Feature 3. The matrices and have identical eigenvalues (but different sets of
eigenvectors). This is true, because the matrix and its transpose have identical determinants.
Thus the matrices and leads to identical polynomials in
whose identical zeros are the eigenvalues of the matrices and . Of course, at the same
time, the matrices and cannot have identical eigenvectors (corresponding to the same
eigenvalues) since two different matrices cannot have identical eigenvalues and eigenvectors.
Thus the matrix
has an eigenvalue of and a yet-to-be-determined corresponding eigenvector. Similarly, for

the matrix
one of the eigenvalues is and the corresponding eigenvector is yet to be determined.

Feature 4. The eigenvalues of a matrix add up to the trace, that is the sum of the
diagonal elements. The product of the eigenvalues equals the determinant of the matrix.
The (shortest) reason for these features is slightly technical. For some of the eigenvalues, it's
easy to show that the zeros of a polynomial
add up to , and that for the characteristic polynomial of is precisely .
For the product of the eigenvalues, it is easy to show that the product of the zeros of the
64
eigenvalues is . Now, is , isn't it? Meanwhile , so there
you go.
The determinant property of the eigenvalues is not particularly useful since calculating the
determinant is a labor intensive task. However, the trace is easy to calculate and can help
determine an eigenvalue when all but one of the eigenvalues have already been discovered.
For example, in the matrix
two of the eigenvalues are and (why?). Therefore, the remaining eigenvalue is
.
Feature 5. For a singular matrix, there are as many zero eigenvalues as the dimension of
the null space. The elements of any basis of the null space can be considered the
corresponding eigenvectors. This is so, because for a vector in the null space, by definition
For example, for the matrix
the vector
is an eigenvector corresponding to .
Feature 5. The eigenvalues of a diagonal matrix appear on the diagonal. That's clear,
because the determinant of a diagonal matrix is the product of the diagonal entries, therefore,
the characteristic polynomial of
is .
65
Armed with these insights, we can now determine the eigenvalues of the originally posed
matrix
a). Since is alone in its row, we conclude , but won't know the corresponding
eigenvector without further work. b). Since each column adds up to , we have , but
again we don't know the eigenvector without further work. c). Since the 3rd column is the
average of the 2nd and 4th, we have and the corresponding eigenvector is
Finally, d). since we know three out of the four eigenvectors, the trace tells us the fourth:
.
Exercise 85 Find the spectrum of the matrix
Solution
Exercise 86 Find the spectrum of the Laplace matrix
Solution
66
Solution
Exercise 88 Find the spectrum of the Laplace matrix with "wraparound"
Solution
Exercise 89 Find one eigenvalue and the corresponding eigenvector of the

"magic square"
67
Solution
Solution
Exercise 91 Is it possible each row of a matrix to add up to and each column to

add up to , such that ?
Solution No, because if the matrix is , then and both represent the
sum of all the entries in the matrix.
Exercise 92 Find one eigenvalue and the corresponding eigenvector of the matrix
68
Solution
Exercise 93 Find one eigenvalue of the matrix
Solution
Solution
69
Solution
Exercise 96 Find as many eigenvalues and eigenvectors as you can for the
following matrix.
Solution
Exercise 97 Find as many eigenvalues and eigenvectors as you can for the matrix
Solution
70
Exercise 98 There are three easy to find eigenvalues and eigenvectors here:
Solution
Exercise 99 Also three here:
Solution
71
Solution
Solution
Solution
72
Solution
Solution
73
Solution
Exercise 106 A Pascal matrix
Solution This matrix is defective. The eigenvalue has algebraic multiplicity

with a single corresponding eigenvector
Exercise 107 A different kind of a Pascal matrix
74
Solution This matrix is also defective. The eigenvalue corresponds to
however the eigenvalues and each have algebraic multiplicity with only a
single corresponding eigenvector
Exercise 108 Determine all eigenvalues and two of the three corresponding
eigenvectors for the matrix
Solution
75
Solution
Solution
76
Solution
Solution
Solution
77
Solution
Solution
78
Solution
Solution
Solution
79
Solution
80
So you want me to calculate
Chapter 12
eigenvalues. Where's the matrix!?

First, a motivating exercise.
Exercise 121 What are the eigenvalues of the reflection transformation in the
plane with respect to the straight line ? This transformation is illustrated in the figure below.
In order to find the image of a vector , draw a straight line orthogonal to through the
tip of the vector . The point where that line intersects is the tip of .
The key point of the exercise is to illustrate that the concept of eigenvalues and
eigenvectors applies not just to matrices, but to general linear transformations, too. In fact,
that's the primary interpretation of eigenvalues and eigenvectors. The eigenvalues and
eigenvectors of a matrix are a special case of this more general view.
For a general transformation , the eigenvalues and the corresponding eigenvectors
are determined by an equation that is nearly identical to the matrix eigenvalue equation:
In words, an eigenvector is a special vector whose image is a scalar multiple of .

When dealing with the special case of geometric vectors, it is often said that is an
eigenvector when and are parallel, although that statement doesn't quite hold up
when . But it is nice to use a geometric word like parallel when talking about geometric
vectors. Finally note, that parenthesis in are usually dropped. Then the eigenvalue
equation reads
81
and now it's all but identical to the matrix eigenvalue equations .
What's the strategy of determining the eigenvalues and eigenvectors of a general linear
transformation ? The robust approach is to represent the linear transformation by a matrix
multiplication with respect to a particular basis and subsequently analyze the resulting matrix
by the available methods. This approach is highlighted in Chapter With Or Without a Basis.
Here, we insist on a basis-free approach. It is less practical, but far more insightful.
Actually, the approach I'm really advocating here is trial and error. So it is far less practical,
but when spectrum can be determined simply by thinking about the transformation, great
insights are gained. The figure below illustrates the trial and error approach to analyzing the
reflection transformation. It shows eight pairs of vectors and their reflections. One of the
vectors that lines along , is its own reflection so it appears as a single vector.
Among the pairs of vectors in the figure, can you identify those that satisfy the eigenvalue
relationship ? In other words, those pairs in which one vector is a scalar multiple of
the other? Yes you can: it's the vectors that lie along (and appear on top of each other as a
single vector) and those perpendicular to ! Call those vectors and , and and .
These pairs of vectors satisfy
Thus, we have found the eigenvectors and and the corresponding eigenvalues
and . And we did it by pure geometric reasoning without writing down a single
number, let alone a matrix. The main point, however, is not so much the mechanics of what
we did, but the very idea is that the concept of eigenvalues and eigenvectors is geometric!
82
While we are at it, let me show you another beautiful basis-free way of determining the
eigenvalues of a reflection. What happens when the reflection is applied to the result of a
reflection? In other words, what is ? The answer is the original vector :
This is true for any vector . But if is an eigenvector of , then this identity can be
analyzed further, since
and therefore
In combination with , the above equation applies
Thus,
This equation, satisfied by the eigenvalues of , has two zeros:
I just managed to make a short argument appear long, so I owe it to you to make it appear
as short as it's ought to be. You see, any lover of linear algebra would write as .
It should make intuitive sense to you because, if nothing else, it is reminiscent of the second
derivative symbol in calculus, which is used in place of . With this
new notation, equation is rewritten as
and holds for all . Now, any lover of linear algebra would shorten this identity even further.
Note that this identity is saying that the applied to leaves unchanged. In other words,
, as a linear transformation, does nothing! And what's a good letter for denoting a
transformation that does nothing? The letter , of course. Thus, we may write
One nice thing about is that it focuses exclusively on the operator itself and not its
argument. This is possible when certain properties of an operator are true for all arguments.
83
Another nice thing about is that it reminds us of the algebraic equation ,
which is the same as with a different letter for the unknown. Thus, equation
which took several simple steps to establish, can be "derived" in a single step from equation
.
Exercise 122 Determine the eigenvalues and eigenvectors of the projection

transformation illustrated in the figure below.
Solution The eigenvalue corresponds to an eigenvector that lies on the line .

The eigenvalue corresponds to an eigenvector perpendicular to the line .
Exercise 123 Note that the projection transformation is characterized by the

relationships
or
which corresponds to the algebraic equation , with zeros . Is this consistent

with your answers for the preceding exercises?
Exercise 124 Find the eigenvalues and eigenvectors of the reflection operator
with respect to a plane in the three dimensional space. Note that one of the eigenvalues is
repeated, so you must find two linearly independent eigenvectors corresponding to it.
84
Solution The eigenvalues correspond to eigenvectors in the plane. The
eigenvalue corresponds to an eigenvector perpendicular to the plane.
Exercise 125 Find the eigenvalues and eigenvectors of the reflection operator
with respect to a straight line in the three dimensional space. Note that one of the
eigenvalues is repeated, so you must find two linearly independent eigenvectors
corresponding to it.
Solution The eigenvalue corresponds to an eigenvector that lies on the

straight line.. The eigenvalue correspond to eigenvectors that lie in the
plane perpendicular to the line.
Exercise 126 Find the eigenvalues and eigenvectors of the projection operator
onto a plane in the three dimensional space. Note that one of the eigenvalues is repeated, so
you must find two linearly independent eigenvectors corresponding to it.
Solution The eigenvalue correspond to eigenvectors in the plane. The

eigenvalue corresponds to an eigenvector perpendicular to the plane.
Exercise 127 Find the eigenvalues and eigenvectors of the projection operator
with respect to a straight line in the three dimensional space. Note that one of the
eigenvalues is repeated, so you must find two linearly independent eigenvectors
corresponding to it.

straight line.. The eigenvalue correspond to eigenvectors that lie in the plane
perpendicular to the line.
Exercise 128 Consider two straight lines and that pass through the origin.
Suppose that and form the angle. Let the transformation be: projection onto
followed by projection onto . In other words
85
What are the eigenvalues and the eigenvectors of ?

line .. The eigenvalue corresponds to eigenvectors that lie in the plane
perpendicular to the line .
86
Chapter 13 Functions Are Vectors, Too!
This chapter expands on the central idea of the preceding chapter, where the idea of linear
transformations and eigenvalues was expended to geometric vectors. But, of course, those
concepts are even more general than that! They work in all linear spaces – in other words, for
all objects that can be added together and multiplied by a number. One of the most important
types of objects that can be analyzed from the linear algebra perspective is functions.
Functional analysis is a prominent branch of mathematics that utilizes this perspective.
As an example, consider the following linear transformation , called dilation, on the
space of cubic polynomials
In words, dilation is the operation of substituting for . For example, the polynomial
becomes
The operation of dilation has important applications in signal processing that you can read
about in Gilbert Strang's textbook Wavelets and Filter Banks.
Exercise 129 Show the following relationships
Let us ask the same question regarding as we did regarding the reflection
transformation in the preceding chapter: What are the eigenvalues and eigenfunctions of ?
(The term eigenfunction means the same thing as eigenvector but in the context where vectors
are functions.)
87
Before we analyze this question, let us take half a step back and answer a few simpler
eigenvalue questions. Let's begin by considering the grandest linear operator of them all: the
derivative operator
What are the eigenvalues and eigenfunctions of ? Of course, the answer that immediately
comes to mind is the celebrated exponential function
We can enrich the answer by putting an arbitrary constant into the exponent:
Since
we note that
Translating this familiar calculus identity into the language of linear algebra, we state that
is the eigenfunction of the derivative operator and is the corresponding eigenvalue.
What about the second derivative operator ? Now the eigenfunction family of
eigenfunctions is richer. It still includes the exponentials, since
Thus any positive number is an eigenvalue with the corresponding eigenfunction.

Note that the function has the same property
So, the linearly independent functions and correspond to the same eigenvalue .
Further, now sines and cosines join the eigenparty. Since
we conclude that any negative number is an eigenvalue with and cos the
88
corresponding eigenfunctions.
In conclusion, every real number is an eigenfunction of the second derivative operator.
The , the multiplicity of the eigenvalue is . For , the multiplicity is since the
only eigenfunction is the constant function.
Exercise 130 Show that, for a fixed , the set of functions
forms a linear subspace. What is the dimension of this subspace?
A very interesting phenomenon takes place when we restrict our attention to functions
such that
First important thing to realize is that this property does define a linear subspace. (Doesn't it?
We devoted a considerable amount of time to this question earlier.) Within this subspace, we
can once again ask the question of spectrum (that is, of eigenvalues and eigenfunctions). It
turns out that the eigenfunctions are given by
with corresponding eigenvalues
For unrestricted functions, any number is an eigenvalue. With the space of function
restricted by the linear property , the eigenvalues become discretized.
Thus it is said that boundaries quantize the spectrum. This phenomenon is central to quantum
mechanics.
Exercise 131 What is the spectrum of for functions that satisfy the linear
property
Solution Each nonzero eigenvalue has multiplicity with

corresponding eigenvalues
89
By now you must be comfortable with approaching functions from the linear algebra point
view. So let us return to the original question of the dilation spectrum on the space of cubic
polynomials. How does one approach determining the spectrum of an unfamiliar
transformation? There is a robust approach. It involves selecting a basis and converting the
problem from polynomials, or whatever the space may be, to . This approach is discussed
in the next chapter. In this and the preceding chapters, the point is not so much the techniques
for finding the spectrum as it is discussing various objects from the linear algebra point view.
We shall therefore, as we did in the preceding chapter, identify the spectrum by trial and error.
One of the eigenfunctions of the dilation operator is relatively easy to see: the constant
polynomial. Since,
we conclude that is an eigenfunction and is the corresponding

eigenvalue.
From the exercises above, one particular example may jump out at you as the
eigenfunction: . Since
we conclude that is an eigenfunction and is the corresponding

eigenvalue.
Once this eigenfunction is identified, the remaining eigenfunctions are perhaps easy to
guess. The next eigenfunction is and since
we determine that the corresponding eigenvalue is . Finally, is the

remaining eigenfunction and the corresponding eigenvalue is .
Exercise 132 Determine (by guessing) the spectrum of the operator among
quadratic polynomials:
90
Solution
Exercise 133 Determine (by guessing) the spectrum of the operator among
quadratic polynomials
Solution Note that, luckily, the eigenfunctions of operator

are the same as those of the dilation operator. Therefore
91
Chapter 14 With Or Without a Basis
The main emphasis of this textbook is on the breadth of linear algebra: its ability to
analyze objects of which so little is required: only being able to be added together and
multiplied by numbers. This idea was explored in most of the chapters of this textbook as we
discussed directed segments, polynomials and general functions as vectors. It is now time to
give the respect that it deserves – for it is the linear space of the robust algorithms of
linear algebra.
This is the case of several powerful reasons. First, there is no denying that provides a
convenient common medium for analyzing linear spaces. Once one chooses a basis and
represents all elements of a linear space with respect to that basis, resulting in -tuples of
numbers, each finite dimensional linear space becomes indistinguishable from . In the
formal mathematical language, this fact is expressed by saying that all final dimensional linear
spaces are isomorphic to , where , of course, is the dimension of the linear space.
The second reason is that , which consists of sets of numbers, can be analyzed by the
powerful methods of algebra that have been developed to extraordinary heights since algebra
and geometry united and lent each other their mutual forces.
The final reason and one that, perhaps, trumps the other two, is computers. In most applied
engineering problems the number is very large, often reaching billions. When problems
become large, solving them by a computer is really the only way to go. As a result, an analysis
of real life problems – such as the stability of a bridge – usually follows the following three
step procedure:
1. Select a basis and represent all elements of the problem with respect to this basis. This
converts the real life problem to .
2. Solve the resulting matrix problem. The resulting matrix problem is typically huge
(billions of entries), but computers can handle it. This step takes place entirely in .
3. Interpret the solution in the real life setting. This step is synthesis, that is the inverse of
decomposition. Synthesis is simply reconstructing vectors from their coefficients. This final
step brings us back from to real life.
Let us use the dilation transformation for an illustration. In the preceding chapter, we
analyzed the entire problem on its own terms without ever selecting a basis. That approach
may be called basis free or coordinate free. It skipped the three steps altogether. It's elegant
and insightful, but in most situations it's simply not possible to carry out. And it gives us no
92
chance to exploit the power of computers. So this time, let us analyze the problem by the
robust approach outlined above. Of course, we had better arrive at the same answer we did
before.
Our ultimate task is to determine the eigenvalues and eigenvectors of the dilation operator
on the set of cubic polynomials. The process starts with selecting a basis. Let's select the
following basis
The next task is to construct the matrix that represent the dilation operator with respect to
the basis . It is constructed according to the following rule: the -th column is the
decomposition of the result of applying the operator to . That is, the column of will
present the vectors
More precisely, the column of will contain the coefficients of these polynomials with
respect to :
We leave it as an exercise to show that the entries of are indeed the coefficients of the
vectors , , and with respect to the basis .
What can the matrix do? Lots of things. Eventually, it'll help us to determine the
eigenvalues and the eigenfunctions of the operator it represents. But first, let us show that the
matrix can be used to apply the dilation in component space. Consider the polynomial
Under dilation, it becomes
Now, let's perform the same operation in the three steps outlined above.
93
1. Convert to component form
2. Apply the transformation in component space, where the operator amounts to

multiplication by the matrix ! Thus
3. Finally, translate the answer in component space back to the polynomials:
and we have obtained the same answer as before! Was it worth the effort? Probably not in this
case, but it illustrates the template for solving industrial size linear algebra problems.
Now, back to the eigenvalue and eigenfunctions of the dilation operator. With regard to
the three step algorithm, we are at Step 2. Our task now is to determine the eigenvalues and
eigenvectors of the matrix . Since the matrix is diagonal, its eigenvalues are easily seen (see
Chapter You Can Run But You Can't Hide) to be , , and in the following pairs
94
This completes step : the eigenvalue problem is solved in the component space.
The final step is to translate the answers to the polynomials. After all, the vectors , ,
and are not the answers themselves, but rather the components of the answers with respect
to the basis . The eigenfunctions , and are synthesized as follows
In other words,
just as before!
This time, was it worth the effort? I think this time the answer is yes. Recall, that during
the original discussion of dilation it took quite a bit of ingenuity to determine the eigenvalues
and eigenvectors. Had the operator been any more complicated, we may have failed to
determine the eigenvalues altogether. (For example, try to determine the eigenvalues of the
operator . On the other hand, the component approach
did not require any ingenuity along the way. Indeed, we could have taught a computer to do
all the steps for us.
As a final demonstration, let us repeat the analysis by using a different basis. It is
important to observe what changes and remains the same. Let the new basis be
95
Then
Thus,
Therefore, is
and this completes Step 1. I'll leave it as an exercise to confirm that the eigenvalues (which
are, of course, , , and ) and eigenvectors of the matrix are
which competes Step 2. The final step is rebuilding the eigenfunction from the eigenvectors of
:
In other words,
96
which is the same answer as before, save for the inconsequential factor of for !
Exercise 134 Repeat the analysis of the dilation operator with respect to the basis
Solution Apply the dilation for each of the basis elements
Therefore, with respect to , the dilation is represented by the matrix
whose spectrum is
It is left as an exercise for the reader to confirm that these eigenvectors correspond to the
previously determined eigenfunctions.
Exercise 135 Describe what changes occur in the analysis when the order of the
elements in the basis is changed, for example
97
Chapter 15 What's Next?
What's next is new everyday encounters with linear algebra.
Interpolation? A linear algebra problem!
Least Squares Fit? A linear algebra problem!
Immunization of a bond portfolio? A linear algebra problem!
Vibrations of a drum? A linear algebra problem!
Resonance? A linear algebra problem!
Stability of a bridge? A linear algebra problem!
Guessing answers in physics by dimensional analysis? A linear algebra problem!
Correlation statistic? A linear algebra concept!
This list is truly infinite. And it is an extraordinarily effective way of seeing the world: to
view it through the prism of linear algebra. It's not the only mathematical framework for
organizing ideas, but it is probably your first one and it's certainly one of the most elegant and
powerful.
So when you are presented with a new problem, look for the linear algebra structure
within it. Once you find it, most if not all of the problem will have been solved.
98
Table of Contents
Chapter 1 Introduction 2
Chapter 2 5
Chapter 3 11
Chapter 4 20
Chapter 5 22
Chapter 6 26
Chapter 7 38
Chapter 8 Go Back! 42
Chapter 9 47
Chapter 10 58
Chapter 11 62
Chapter 12 80
Chapter 13 Functions Are Vectors, Too! 86
Chapter 14 91
Chapter 15 What's Next? 97
99

Hello Again, Linear Algebra - A Second Look at The Subject Through A Collect

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Hello Again, Linear Algebra - A Second Look at The Subject Through A Collect

Uploaded by

Copyright:

Available Formats

Visit http://insightfultexts.com for videos and supplementary content.

or the basis for the space of cubic polynomials:

The vectors in the arrangement are such that

Exercise 1 Explain why augmenting a linearly dependent set with an additional

Exercise 3 Show that the polynomials

Exercise 4 Show that the set of vectors

Exercise 5 Show that the set of vectors

Solution These vectors are linearly dependent because

Exercise 6 Show that the set of vectors

Solution The vectors are linearly dependent because

Exercise 7 Show that the set of vectors

Exercise 8 On a clock, can the second hand always be expressed as a linear

Exercise 9 Explain why the following polynomials are linearly dependent:

Solution The polynomials are linearly dependent because

For the obtained triangle, the last equation reads simply

Exercise 10 Devise a geometric method for constructing a nontrivial linear

Solution The solution is based in constructing a tetrahedron (instead of a triangle) by

Written in this form, the problem is to decompose the vector

corresponds to the zero vector. We call it a fancy zero.

where is an arbitrary number.

Exercise 11 Find the general solution to the system

Solution The general solution can be written as

and yet another is

Exercise 12 Find the general solution to the system

Exercise 13 Find the general solution to the system

Exercise 14 Find the general solution to the system

Exercise 17 Suppose that , where is a full rank matrix and is

Rather, it is based on the idea of linear properties and linear subspaces.

which gives an explicit basis for .

Exercise 18 What is the column space of the matrix

Exercise 19 What is the column space of the matrix

Exercise 20 What is the column space of the matrix

Exercise 21 What is the column space of the matrix

Exercise 22 What is the column space of the matrix

Thus we conclude that the zero-integral property is, in fact, linear.

Its integral from to is

Thus, the polynomial satisfies the zero-integral property if

which can be expressed by relating columns , , and , to column :

Therefore, this equation and the previous equation

which yields the null space

Exercise 24 Functions such that

Exercise 25 Functions such that (No)

Exercise 26 Functions such that

Exercise 28 Functions such that

Solution Yes. Let . Then

The null space of the matrix is

Exercise 29 Functions such that (Yes. Details left to the

Exercise 30 Functions such that (No)

Exercise 31 Functions such that (No)

Solution Yes. Let . Then

The null space of the matrix is

Exercise 33 Functions such that (Yes. Details left to the

Exercise 34 Functions such that

Solution Yes. Let . Then

Next, subtract row from row :

This null space corresponds to the basis

Exercise 35 Functions such that . (No)

Exercise 36 Polynomials whose coefficients add up to .

Exercise 38 Polynomials whose coefficients add up to its value at .

Solution Yes. Let . Then

The rest is left to the reader.

Exercise 39 Polynomials whose coefficients add up to the value of its