Numerical Methods

ECE 3340
Electrical and Computer Engineering
University of Houston
Dr. David Mayerich
Goals and Expectations
• Understand fundamentals of numerical methods
• Experience in implementation (C++ and Matlab)
• I want to give you a set of important numerical tools and
practical experience implementing them

• This class is mostly about:

• translating mathematics into algorithms
• implementing those algorithms with constraints imposed by digital
computers
• Programming will be an important part of the class!
• I expect a challenge but not an insurmountable one.
Everyone can and should get an A.
Prerequisites
• ECE 3331 – Programming Applications in ECE
• MATH 3321 – Engineering Mathematics

• Linear algebra, differential equations

• C/C++
• Know how to format and compile your own code
• Basics of Visual Studio
• Matlab
• Using the massive number of available Matlab functions
• Vectorizing your code
Recommended Textbooks
• R.W.
  Hamming, Numerical Methods for Scientists and
Engineers
• Mathematical basis for numerical methods
• Short and precise explanations for different aspects of numerical
methods
• $14.00 on Amazon.com

• W.H. Press, et al., Numerical Recipes

• Detailed discussions and code for more in-depth understanding of a
given topic
• Here you’ll learn how to build an LCG
• low-dimensional integration, sorting, modeling, etc.
• NR will describe the Mersenne Twister algorithm
• scientific applications, high-dimensional problems, encryption
Grading
• Assignments
• 5% Written homework – work together, turn in your own
• 30% Programming assignments – individual work
• 40% Mid-Terms 1 and 2 (20% each)
• 25% Final Exam
• Standard 10-point scale (90+ = A, 80-89 = B, etc.)

• Programming Assignments
• Finding Zeros (C/C++)
• Scaled Partial Pivoting (C/C++)
• Random Numbers & Monte-Carl Methods (C/C++)
• The Kessel Run – Solving ODEs (Matlab)
Course Material
• Homework assignments
• Taken from class lecture (your notes) and similar math problems
• You’re encouraged to work together, but don’t hand in copied work
• Grading will not be harsh, but it will exist
• Programming Projects
• You are expected to work independently
• Document your code – average one line of comments / line of code
• Explain “why” in the comments, not “what”
• Exams – Material will be taken from:
• Homework assignments (often outside material, reading)
• Lectures
• Programming assignments (focus on algorithms, not code)
• A 5” x 8” crib sheet is allowed for the exam
About Me
• B.S. Computer Science – SWOSU (Oklahoma)
• Ph.D. Computer Science – Texas A&M University
• Beckman Fellowship – University of Illinois

• Background
• Computer Graphics, Visualization
• Biomedical Imaging, Microscopy, Spectroscopy
• High-Performance Computing, GPU Computing

• Personal
• video games, fencing, climbing
Research Interests

• Biomedical imaging – brain and cancer imaging

• Microscopy – large data sets

0.6

100µm
0.5mm
a.u.

0.0

IR image tissue map

0.5mm
Research Interests

• GPU Computing – visualization and image processing

nVidia Fermi

• Physically-based modeling – optical systems, fluid dynamics

• Computer graphics – interactive visualization and games

𝒌 
𝑑  𝜃 
𝒑 
Why numerical methods?
• Implementation depends on need and constraints
• high-level applications can use Matlab, external libraries
• low-level applications are under pressure – memory, speed,
architecture
• Most low-level implementations are simple
• algorithms are available
• mostly have to worry about implementation
• architecture, memory, precision, speed, discretization errors

• First major hurdle – dealing with numerical errors

Reasons for Errors

• Representation
  is limited – n bits allow values
• 1-bit: 0, 1
• 2-bits: 00, 01, 10, 11
• What do we mean by 8-, 32-, and 64-bit systems?

• Different representations map binary to decimal

• Most common data types we’ll be working with:
• Integers (for representing numbers in ℤ)
• Floating-point values (for representing numbers in ℝ)

• Quantization errors – mapping analytical processes to countable

sets results in errors that include:
(1) round-off, (2) truncation, (3) feedback, (4) aliasing
Round-off Error
• Correct
  value is “clamped” to the nearest representable
value
• .3977587 becomes .397759 (or .397758)

• Problem when performing iterative computations

• Intel Core i7-4770K (Haswell) GFLOPS

• Subtracting similar numbers causes “catastrophic

cancellation” (which we’ll study in detail):
.397758 - .397756 = .000002
 Round-off Error (Example)
•  Bessel Functions of the First Kind:

0.7

0.5

0.3 n=0
𝐽  𝑛 (𝑧)

n=1
0.1 n=2
n=3
n=4
-0.1

-0.3

-0.5
 𝑧
“It is well known that a recurrence relation is numerically unstable in the direction in which the
function is decreasing. Successive values are computed as small differences between nearly equal
terms and all accuracy is soon lost.”
-A.R. Barnett, The Calculation of Spherical Bessel Functions and Coulomb Functions
Truncation Error
• Take a common Taylor series approximation:
∞ (2 𝑛+1)
  𝑥
sin 𝑥=∑ (− 1)𝑛 where 𝑥 ∈ ℝ
𝑛 =0 ( 2𝑛+1 ) !
• error introduced by not completing the evaluation of a
series
• This error can usually be bounded:

Magnitude of Taylor Series Terms

x=0.5
30
x=1
25 x=1.5
x=2
20 x=2.5
Axis Title

x=3
15
x=3.5
10 x=4
x=4.5
5 x=5

0
1 2 3 4 5 6 7 8
Feedback Error
• error introduced when an iterative algorithm takes a result
from one cycle and uses it in another

• Example: Euler method for

Δ𝑡
 
solving ordinary differential
equations

• Stability – do errors increase or

decrease over several
iterations?
 Aliasing

0.9

0.4

-0.1

-0.6

-1.1

7 Hz

10 Hz samples
 Aliasing

0.9

0.4

-0.1

-0.6

-1.1

13 Hz

10 Hz samples
 Aliasing

0.9

0.4

-0.1

-0.6

-1.1

3 Hz

10 Hz samples
 Aliasing

0.9

0.4

-0.1

-0.6

-1.1

7 Hz

10 Hz samples
 Aliasing
• How do we select the correct signal?

0.9

0.4

-0.1

-0.6

-1.1

7 Hz 3 Hz 13 Hz

10 Hz samples
 Aliasing
• How do we select the correct signal?
• Applying a band limit eliminates high frequencies

0.9

0.4

-0.1

-0.6

-1.1

7 Hz 3 Hz

10 Hz samples
 Aliasing
•  How do we select the correct signal?
• Applying a band limit eliminates high frequencies
• Sample above the Nyquist limit: sample rate

0.9

0.4

-0.1

-0.6

-1.1

7 Hz 7 Hz 3 Hz

20 Hz samples
 Aliasing
•  How do we select the correct signal?
• Applying a band limit eliminates high frequencies
• Sample above the Nyquist limit: sample rate

0.9

0.4

-0.1

-0.6

-1.1

7 Hz

10 Hz samples
Image Aliasing
• Aliasing in images is very easy for humans to spot
• Problem in video games and visualization
• Apply the same technique: band limit (blur) the image
Image Aliasing
• Anti-aliasing techniques are often used in real-time graphics
• Human eye is highly sensitive to aliasing, so we trade
aliasing for noise (MSAA) or blurring (SSAA)

World of Warcraft AA demo, eyesofthebeast.com

Temporal Aliasing
• Cameras take discrete samples, resulting in aliasing effects
in video:

See Brasspup:
https://www.youtube.com/watch?v=uENITui5_jU

• Temporal blurring isn’t practical, so we’ve just gotten used

to seeing this stuff. Our eyes behave the same way.
Homework
• Describe ‘aliasing’ in two sentences.
• Choose an argument x for cos(x).
• What is the minimum (Nyquist) sample rate necessary to
accurately reconstruct this function?
• Choose a sample rate below the Nyquist limit and plot:
• cos(x)
• the sample points for your selected sampling criterion
• the resulting cosine function reconstructed from your selected
samples – this will be an aliased representation of cos(x)