Robust Regression

Robust regression procedures dampen the influence of outlying cases, as

compared to ordinary least squares estimation, in an effort to provide a
better fit for the majority of cases. Numerous robust regression
procedures have been developed. Here we will discuss a few.
♠ LAR or LAD and LMS Regressions

• LAR (Least Absolute Residuals) or LAD (Least Absolute Deviation)

is minimum L1 -norm regression.
The criterion to be minimized, denoted by L1 , is
n
X
L1 = |Yi − (β0 + β1 Xi1 + β2 Xi2 + · · · + βk Xik )|.
i=1

11-2
• The LMS (Least Median of Squares) is to minimize the median
squared deviation:

the median{(Yi −(β0 +β1 Xi1 +β2 Xi2 +· · ·+βk Xik ))2 , i = 1, 2, . . . , n}

with respect to the regression coefficients.

• Comments
– Both LAR (or LAD) and LMS estimates of β can be obtained only
via computer.
– Steven P. Ellis (1998, Statistical Science, Vol. 13, 337-350) pointed
out that LS is better than LAD and LMS in terms of the instability
at data sets, in the sense that small changes to the data sets can
substantially change the fit.

11-3
♠ IRLS Robust Regression

• Iteratively reweighted least squares (IRLS) robust regression uses

the weighted least squares procedures to dampen the influence of
outlying observations.

• Instead of weights based on the error variances, IRLS robust

regression uses weights based on how far outlying a case is, as
measured by residual for that case.

• Steps
(1) Choose a weight function for weighting the cases.
(2) Obtain starting weights for all cases.
(3) Use the starting weights in weighted least squares and obtain the
residuals from the fitted regression function.
(4) Use the residuals in step 3 to obtain revised weights.
(5) Continue the iterations until convergence is obtained.

11-4
• Weight Function
Two widely used weight functions are the Huber and bisquare
weight functions:

1 |u| ≤ 1.345
Huber: w =
 1.345 |u| > 1.345
|u|
h i2
 1− u

2
4.685
|u| ≤ 4.685
Bisquare: w = ,
0 |u| > 4.685

where w denotes the weight and u denotes the scaled residual. 1.345
and 4.685 are called tuning constants. They were chosen to make the
IRLS robust procedure 95 percent efficient for data generated by the
normal error MLR model.

11-5
• Starting Values
– When the Huber weight function is employed, the initial residuals
may be those obtained from an ordinary least squares fit.
– The bisquare function calculations are more sensitive to the starting
values. To obtain good starting values for the bisquare weight
function, the Huber weight function is often used to obtain an initial
robust regression fit, and the residuals for this fit are then employed
as starting values fir several integrations with the bisquare weight
function.
– Alternatively, LAR regression is used to obtain starting residuals for
the bisquare weight function.

11-6
• Scaled Residuals
The scaled residual ui is defined by
ei
ui = ,
M AD
where
1
MAD = median{|ei − median{ei }|},
0.6745
and the constant 0.6745 provides an unbiased estimate of σ for
independent observations from a normal distribution.

11-7
 Bootstrapping

Bootstrap methods are the computer-intensive resampling methods,

which can be used for obtaining the sampling distributions of some
complicated estimators or test statistics.
For standard fitted regression models, Bootstrap methods can be used
for obtaining standard errors and confidence intervals for regression
coefficients without making any assumptions on error distribution.
♠ Bootstrap Algorithm

Step 0. Obtain residuals {ei , i = 1, 2, . . . } and the fitted values

Ŷ = (Ŷ1 , Ŷ2 , . . . , Ŷn )′ from an ordinary least squares fit.

Step 1. Obtain the empirical cdf F̂ of the residuals e = (e1 , e2 . . . , en )′ ,

which assigns mass n1 at ei , i = 1, 2, . . . , n.

11-8
Step 2. Draw a bootstrap sample from F̂ , denoted by
e∗ = (e∗1 , e∗2 , . . . , e∗n )′ . This consists of drawing each e∗i
independently with replacement and with equal probability
1/n from the observed data {e1 , e2 , . . . , en }.

Step 3. Compute Yi∗ = Ŷi + e∗i .

Step 4. Obtain b (the LS estimate of β) from an ordinary least squares

fit using the data {(Yi∗ , Xi1 , Xi2 , . . . , Xik ), i = 1, 2, . . . , n}.

Step 5. Independently repeat Steps 2 — 4 a large number of times, say

B times, and obtain bootstrap replications b∗1 , b∗2 , . . . , b∗B .

Using the bootstrap sample {b∗1 , b∗2 , . . . , b∗B }, we can easily compute the
standard errors and CI’s for β.

11-9
 Loess Method

♠ Introduction

• The loess method is a more refined nonparametric method than band

regression.

• It obtains a smooth curve by fitting successive linear regression

functions in local neighborhoods.

• The name loess stands for locally weighted regression scatter plot
smoothing.

• LOESS, originally proposed by Cleveland (1979) and further

developed by Cleveland and Devlin (1988), specifically denotes a
method that is (somewhat) more descriptively known as locally
weighted polynomial regression.

11-10
• At each point in the data set a low-degree polynomial is fit to a
subset of the data, with explanatory variable values near the point
whose response is being estimated.

• The polynomial is fit using weighted least squares, giving more

weight to points near the point whose response is being estimated
and less weight to points further away. The value of the regression
function for the point is then obtained by evaluating the local
polynomial using the explanatory variable values for that data point.

• The LOESS fit is complete after regression function values have

been computed for each of the n data points. Many of the details of
this method, such as the degree of the polynomial model and the
weights, are flexible.

11-11
♦ Localized Subsets of Data

• The subsets of data used for each weighted least squares fit in
LOESS are determined by a nearest neighbors algorithm. A
user-specified input to the procedure called the ”bandwidth” or
”smoothing parameter” determines how much of the data is used to
fit each local polynomial.

• The smoothing parameter, q, is a number between (d + 1)/n and 1,

with d denoting the degree of the local polynomial.

• The value of q is the proportion of data used in each fit.

• The subset of data used in each weighted least squares fit is

comprised of the nq (rounded to the next largest integer) points
whose explanatory variables values are closest to the point at which
the response is being estimated.

11-12
• q is called the smoothing parameter because it controls the flexibility
of the LOESS regression function. Large values of q produce the
smoothest functions that wiggle the least in response to fluctuations
in the data. The smaller q is, the closer the regression function will
conform to the data. Using too small a value of the smoothing
parameter is not desirable, however, since the regression function
will eventually start to capture the random error in the data.

• The textbook recommends 0.4 ≤ q ≤ 0.6.

11-13
♦ Degree of Local Polynomials
• The local polynomials fit to each subset of the data are almost
always of first or second degree; that is, either locally linear (in the
straight line sense) or locally quadratic.
• Using a zero degree polynomial turns LOESS into a weighted
moving average. Such a simple local model might work well for some
situations, but may not always approximate the underlying function
well enough.
• Higher-degree polynomials would work in theory, but yield models
that are not really in the spirit of LOESS.
• LOESS is based on the ideas that any function can be well
approximated in a small neighborhood by a low-order polynomial
and that simple models can be fit to data easily.
• High-degree polynomials would tend to overfit the data in each
subset and are numerically unstable, making accurate computations
difficult.

11-14
♦ Distance Measure

• A Euclidean distance measure is employed.

• For case of two predictor, suppose that we wish to obtain the fitted
value at (Xh1 , Xh2 ), and then this measure is denoted by di and is
defined as
di = [(Xi1 − Xh1 )2 + (Xi2 − Xh2 )2 ]1/2 .

• When the predictor variables are measured on different scales, each

should be scaled by dividing it by its standard deviation.

11-15
♦ Weight Function
• The weight function gives the most weight to the data points nearest
the point of estimation and the least weight to the data points that
are furthest away.
• The use of the weights is based on the idea that points near each
other in the explanatory variable space are more likely to be related
to each other in a simple way than points that are further apart.
Following this logic, points that are likely to follow the local model
best influence the local model parameter estimates the most. Points
that are less likely to actually conform to the local model have less
influence on the local model parameter estimates.
• The traditional weight function used for LOESS is the tri-cube
weight function,
(
[1 − (di /dq )3 ]3 if di < dq
wi =
0 if di ≥ dq ,

where dq denotes the Euclidean distance of the furthest case in the

neighborhood of (Xh1 , Xh2 ).

11-16
♦ Local Fitting
Given the weights wi for the n cases, weighted least squares is used to fit
either the first-order polynomial model or the second-order polynomial
model. After the regression model is fitted by weighted least squares, the
fitted value Ŷh at (Xh1 , Xh2 ) then serves as the nonparametric estimate
of the mean response at these X levels.
♠ Advantages of LOESS
• The biggest advantage LOESS has over many other methods is the
fact that it does not require the specification of a function to fit a
model to all of the data in the sample. Instead the analyst only has
to provide a smoothing parameter value and the degree of the local
polynomial.
• LOESS is very flexible, making it ideal for modeling complex
processes for which no theoretical models exist.
• These two advantages, combined with the simplicity of the method,
make LOESS one of the most attractive of the modern regression

11-17
 methods for applications that fit the general framework of least
squares regression but which have a complex deterministic structure.

♠ Disadvantages of LOESS

• LOESS is not efficient in using data. LOESS requires fairly large,

densely sampled data sets in order to produce good models.

• LOESS does not produce a regression function that is easily

represented by a mathematical formula. This can make it difficult to
transfer the results of an analysis to other people.

• In order to transfer the regression function to another person, they

would need the data set and software for LOESS calculations.

• In nonlinear regression, on the other hand, it is only necessary to

write down a functional form in order to provide estimates of the
unknown parameters and the estimated uncertainty. Depending on
the application, this could be either a major or a minor drawback to
using LOESS.

11-18
♠ References

• Cleveland, W.S. (1979), ”Robust Locally Weighted Regression and

Smoothing Scatterplots,” Journal of the American Statistical
Association, 74, 829-836.

• Cleveland, W.S. and Devlin, S.J. (1988), ”Locally Weighted

Regression: An Approach to Regression Analysis by Local Fitting,”
Journal of the American Statistical Association, 83, 596-610.

11-19