Simple Linear Regression

- There is only one independent and

dependent variable

Independent variable is the variable

that can possibly predict the
dependent variable

Correlation vs Regression

Scatter plot – shows relationship

between two variable (IV DV)

Correlation – used to measure the

strength of the association (linear
relationship) between two variables.

Regression analysis used to: Positive - upward

Predict the value od a DV based on Negative - downward
the value of at least one IV Equation of Simple Linear Regression:

Explain the impact of changes in an IV

on DV

*Income and spending, wage

DV: the variable we wish to predict or

explain
Beta sub 0 is constant
IV: variable used to predict or explain
the DV

Example:

Only one IV, X

Relationship between X and Y is

described by a linear function The representation of the linearity of
Changes in Y are assumed to be relationship if they weak are scattered
related to changes in X
Measures of Variation
When using the least-squares method
to determine the regression
coefficients for a set of data, you need
to compute three measures of
variation. The first measure, the
total sum of squares (SST), is a
measure of variation of the Yi values
around their mean, Y. The
total variation, or total sum of
squares, is subdivided into explained
variation and unexplained variation.
The explained variation, or
regression sum of squares (SSR),
represents variation that is explained
by the relationship between X and Y,
and the unexplained variation, or
error sum of squares (SSE),
represents variation due to factors
other than the relationship between X
and Y. Figure 13.6 shows these
different measures of variation.
Assumptions
When hypothesis testing and the
analysis of variance were discussed in
Chapters 9 through 12, the importance
of the assumptions to the validity of
any conclusions reached was
emphasized.
The assumptions necessary for
regression are similar to those of the
analysis of variance because both are
part of the general category of linear
models (reference 4).
The four assumptions of regression
(known by the acronym LINE) are as
follows:
• Linearity
• Independence of errors
• Normality of error
• Equal variance
The first assumption, linearity, states
that the relationship between variables
is linear. Relationships between
variables that are not linear are
discussed in Chapter 15.
The second assumption,
independence of errors, requires
that the errors Ei are independent of
one another. This assumption is
particularly important when data are
collected over a period of time. In such
situations, the errors in a specific time
period are sometimes correlated with
those of the previous time period.
The third assumption, normality,
requires that the errors are normally
distributed at each value of X. Like the
t test and the ANOVA F test,
regression analysis is fairly robust
against departures from the normality
assumption. As long as the distribution
of the errors
at each level of X is not extremely
different from a normal distribution,
inferences about 130 and 131 are not
seriously affected.
The fourth assumption, equal
variance, or homoscedasticity,
requires that the variance of the errors
Ei be constant for all values of X. In
other words, the variability of Y values
is the same when X is a low value as
when X is a high value. The equal-
variance assumption is important
when making inferences about [30
and [31. If there are serious
departures from this assumption, you
can use either data transformations or
weighted least-squares methods (see
reference 4).
Residual Analysis
Residual analysis visually evaluates
these assumptions and helps you to
determine whether the regression
model that has been selected is
appropriate.
The residual, or estimated error value,
q, is the difference between the
observed (Yi) and predicted (Yi)
values of the dependent variable for a
given value of Xi. A residual appears
on a scatter plot as the vertical
distance between an observed value
of Y and the prediction line.
Equation (13.14) defines the residual.
RESIDUAL
The residual is equal to the difference
between the observed value of Y and
the predicted value of Y.
Evaluating the Assumptions
Recall from Section 13.4 that the four
assumptions of regression (known by
the acronym LINE) are linearity,
independence, normality, and equal
variance.
Linearity. To evaluate linearity, you
plot the residuals on the vertical axis
against the corresponding Xi values of
the independent variable on the
horizontal axis. If the linear model is
appropriate for the data, you will not
see any apparent pattern in the plot.
However, if the linear model is not
appropriate, in the residual plot, there
will be a relationship between the Xi
values and the residuals, q.
You can see such a pattern in Figure
13.9. Panel A shows a situation in
which, although there is an increasing
trend in Y as X increases, the
relationship seems curvilinear
because the upward trend decreases
for increasing values of X. This
quadratic effect is highlighted in Panel
B, where there is a clear relationship
between Xi and q. By plotting the
residuals, the linear trend of X with Y
has been removed, thereby exposing
the lack of fit in the simple linear
model. Thus, a quadratic model is a
better fit and should be used in place
of the simple linear model. (See
Section 15.1 for further discussion of
fitting curvilinear models.)

To assess linearity, the residuals are
plotted against the independent
variable (store size, in thousands of
square feet) in Figure 13.11. Although
there is widespread scatter in the
residual plot, there is no clear pattern
or relationship between the residuals
and X. The residuals appear to be
evenly spread above and below 0 for
different values of X. You can
conclude that the linear model is
appropriate for the Sunflowers Apparel
data.
Independence. You can evaluate the
To determine whether the simple
linear regression model is appropriate,
return to the evaluation of the
Sunflowers Apparel data. Figure 13.10
displays the predicted annual sales
values and residuals.
assumption of independence of the
errors by plotting the residuals in the
order or sequence in which the data
were collected. If the values of Y are
part of a time series (see Section 2.6),
one residual may sometimes be to conclude that you should not be
related to the previous residual. overly concerned about departures
from this normality assumption in the
If this relationship exists between Sunflowers Apparel data.
consecutive residuals (which violates
the assumption of independence), the
plot of the residuals versus the time in
which the data were collected will
often show a cyclical pattern.
Because the Sunflowers Apparel data
were collected during the same time
period, you do not need to evaluate
the independence assumption for
these data.
Normality. You can evaluate the
assumption of normality in the errors
by organizing the residuals into a
frequency distribution as shown in
Table 13.3. You cannot construct a
meaningful histogram because the
sample size is too small. And with
such a small sample size (n = 14), it
can be difficult to evaluate the
normality assumption by using a stem-
and-leaf display (see Section 2.5), a
boxplot (see Section 3.3), or a normal Equal Variance. You can evaluate the
probability plot (see Section 6.3). assumption of equal variance from a
plot of the residuals with Xi. For the
Sunflowers Apparel data of Figure
13.11 on page 540, there do not
appear to be major differences in the
variability of the residuals for different
Xi values.
Thus, you can conclude that there is
no apparent violation in the
From the normal probability plot of the assumption of equal variance at each
residuals in Figure 13.12, the data do level of X.
not appear to depart substantially from
a normal distribution. The robustness To examine a case in which the equal-
of regression analysis with modest variance assumption is violated,
departures from normality enables you observe Figure 13.13, which is a plot
of the residuals with Xi for a
hypothetical set of data. This plot is
fan shaped
because the variability of the residuals
increases dramatically as X increases.
Because this plot shows unequal
variances of the residuals at different
levels of X, the equal-variance
assumption is invalid.
Multiple regression models use two
or more independent variables to
predict the value of a dependent
variable.
Compare the multiple regression
model to the simple linear regression
model [Equation (13.1) on page 522]:
In the simple linear regression model,
the slope, ß1, represents the change in
the mean of Y per unit change in X
and does not take into account any
other variables. In the multiple
regression model with two
independent variables [Equation
(14.2)], the slope, ß1, represents the
change in the mean of Y per unit
change in X1, taking into account the
effect of X2.
As in the case of simple linear
regression, you use the least-squares
method to compute sample regression
coefficients (ß0, ß1, and ß2) as
estimates of the population
parameters
(ß0, ß1, and ß2). Equation (14.3)
defines the regression equation for a
multiple regression model with two
independent variables.
MULTIPLE REGRESSION
EQUATION WITH TWO
INDEPENDENT VARIABLES
r2, Adjusted r2, and the Overall F Test
This section discusses three methods
you can use to evaluate the overall
multiple regression model: the
coefficient of multiple determination, r,
the adjusted r2, and the overall F test.
Coefficient of Multiple Determination
Recall from Section 13.3 that the
coefficient of determination, r,
measures the proportion of the
variation in Y that is explained by the
independent variable X in the simple
linear regression model. In multiple
regression, the coefficient of
multiple determination represents
the proportion of the variation in Y that
is explained by the set of independent
variables.
Equation (14.4) defines the coefficient
of multiple determination for a multiple
regression model with two or more
independent variables.
In the OmniPower example, from
Figure 14.2 on page 580, SSR =
39,472,730.77 and SST = adjusted
52,093,677.44. Thus,
The coefficient of multiple
determination (r2 = 0.7577) indicates
that 75.77% of the variation in sales is
explained by the variation in the price
and in the promotional expenditures.
The coefficient of multiple
determination also appears in the
Figure 14.2 results on page 580, and
is labeled R Square in the Excel
results and R-Sq in the Minitab
results.
Adjusted r2
When considering multiple regression
models, some statisticians suggest
that you should use the adjusted r2 to
take into account both the number of
independent variables in the model
and the sample size. Reporting the
adjusted r is extremely important when
you are comparing two or more
regression models that predict the
same dependent variable but have a
different number of independent
variables. Equation (14.5) defines the
r2.
Therefore, 74.21% of the variation in
sales is explained by the multiple
regression model adjusted for the
number of independent variables and
sample size. The adjusted r also
appears in the Figure 14.2 results on
page 580, and is labeled Adjusted R
Square in the Excel results and R-
Sq(adj) in the Minitab results.
Test for the Significance of the
Overall Multiple Regression Model
You use the overall F test to
determine whether there is a
significant relationship between the
dependent variable and the entire set
of independent variables (the overall
multiple regression model). Because
there is more than one independent
variable, you use the following null
and alternative hypotheses:
Using a 0.05 level of significance, the
critical value of the F distribution with
2 and 31 degrees of freedom found
from Table E.5 is approximately 3.32
(see Figure 14.4 below). From Figure
14.2 on page 580, the FSTATtest
statistic given in the ANOVA summary
table is 48.4771.
> 3.32, or because the p-value =
0.000 < 0.05, you reject Ho and con-
Because 48.4771 include that at least
one of the independent variables
(price and/or promotional
expenditures) is related to sales.
--------------------------------------------------
Factor Analysis
Exploratory factor analysis (EFA)
- only be used for if you can’t find or
identify a variable in the RRL.-
processed by forming the variables
into a structure called factors.
Pseudo-factor
- variables come from item-indicators Bartlett’s test, it should be significant
or item-questions that describe the (0.05 or lesser) to make the factor
factor. analysis reliable.
- basis: higher factor loading - the Total Variance Explained:
higher number in the group will be the
name of the factor loading. The cumulative variance explained
should be 60% or higher to make it
- respondents: min. 100 observations. valid and reliable.
- if you are going to split the samples Communalities of Variables
for validation purposes the
Below 0.50 is a candidate for deletion
respondents must be 200 or higher.
or failing to explain the other variables.
General rule:
If it falls into on or above 0.40, but
Minimum is to have at least five times below 0.50; it depends to researcher’s
as many observations as the number discretion if the he will remove it or it
of variables to be analyze. More has signifacnt factor
acceptable is 10:1 (10 respondents
per one variable). Scree Plot
Interpretation: - 1 the eigen value should be greater
than 1 to make it significant.
Measures of Sampliing Adequacy
(MSA) through KMO is needed to - if below 1, the factors are not
proceed factor analysis and Barlett’s considered anymore in the analysis.
Test of Spherecity.
KMO (Kaiser-Meyer Olkin Test) it
should be greater than 0.5 for a
satisfactory factor analysis to proceed.
0.90 or above, marvelous
0.80 or above, meritorious
0.70 or above, middling
0.60 or above, mediocre
0.50 or above, miserable
Below 0.50, unacceptable