Table of Content

1. INTRODUCTION................................................................................................................1

1.1 Background.......................................................................................................................1

1.2 Statement of Problem.......................................................................................................2

1.3 Objectives and Methodology............................................................................................2

1.4Significances......................................................................................................................2

1.5Limitations.........................................................................................................................2

2. DISCUSSION AND ANALYSIS........................................................................................3

2.1 Ordinary least squares (OLS)...........................................................................................3

2.2 OLS Assumptions.............................................................................................................4

2.3 Deriving OLS Estimates for a Simple Regression Model................................................8

2.4 The Gauss-Markov Theorem..........................................................................................14

2.5 Properties of OLS Estimator..........................................................................................14

2.6 Applications of OLS estimators.....................................................................................17

3. CONCLUSION...............................................................................................................18

References...............................................................................................................................19

1
2
1.INTRODUCTION

1.1 Background
The ordinary least squares (OLS) method is a linear regression technique that is used to estimate the
unknown parameters in a model. The method relies on minimizing the sum of squared residuals
between the actual and predicted values. The OLS method can be used to find the best-fit line for
data by minimizing the sum of squared errors or residuals between the actual and predicted values.
And, the calculus method for minimizing the sum of squares residuals is take the partial derivative of
the cost function with respect to the coefficients of determination, set the partial derivatives equal to
zero and solve for each of the coefficients. The OLS method is also known as least squares method
for regression or linear regression.
The simple linear regression model is a statistical model, based on the idea that the relationship
between two variables can be explained by the following formula:

Where εi is the error term, and α, β are the true (but unobserved) parameters of the regression. The
parameter β represents the variation of the dependent variable when the independent variable has a
unitary variation.
Now, the idea of Simple Linear Regression is finding those parameters α and β for which the error
term is minimized. To be more precise, the model will minimize the squared errors: indeed, we do not
want our positive errors to be compensated by the negative ones, since they are equally penalizing for
our model.

This procedure is called Ordinary Least Squared error — OLS.

Linear models can model curvature by including nonlinear variables such as polynomials and
transforming exponential functions.

1
1.2 Statement of Problem
The OLSmethod enables to minimize the prediction error, between the predicted and real values.One
may ask themselves why we choose to minimize the sum of squared errors instead of the sum of
errors directly. The pitfalls of applying least squares are not sufficiently well understood by many of
the people who attempt to apply it.There are some serious difficulties while using the OLSmethod
due to its strict assumptions. Several basic assumptions are important to allow the use in practice of
the conventional formulae for the OLS method. The OLS formulae are potentially misleading and
more appropriate estimators have to be used, in particular when the statistical hypothesis is violated:
zero mean of the residuals, serial interdependence of the errors, homoscedasticity.Presence of
outliers and high laverage point, the actual nonlinear relation between response and the predictor
makes the conclusion drawnfrom OLS method null and void. Also, the accuracy of the method may
drop significantly.

1.3 Objectives and Methodology

The general objective of this study is to understand about the OLS method and OLS Estimator. The
specific objectives of the study include:
 To know about the important assumptions of OLS method
 To study about the method to derive OLS Estimates for a Simple Regression Model
 To get acquainted with the Gauss-Markov Theoremand the BLUE properties of OLS Estimator
 To know about the applicability of OLS method and OLS estimator
The methodology of this study is based on the review of the secondary data. Secondary information
were collected from various literatures including journal articles, proceedings, books, lecture notes,
reference materials provided in the class etc. The relevant information were arranged systematically.
The key findings after the review of these sources were summarized and presented in the article.
1.4 Significances
 This paper helps us to understand about the the important assumptions ofOLS method.
 It discusses about method to derive OLS Estimates for a Simple Regression Model.
 It helps us to familiarize with the Gauss-Markov Theoremand properties of OLS Estimator.

1.5 Limitations
The study is completely based on review of the sources including journal articles, books, lecture
notes, class lectures and other online media. There is no any empirical finding or deduction.

2
2. DISCUSSION AND ANALYSIS

2.1 Ordinary least squares (OLS)

It is a type of statistical method for the estimation of unknown parameters in a linear regression
model (with fixed level-one effects of a linear function of a set of explanatory variables). It helps to
estimate the relationship between one or more independent variables and dependent variable. It
estimates the parameters by the principle of least squares; minimizing the residual sum of squares
(differences between the observed values of dependent variable and values predicted by the linear
function of the independent variable).
OLS method estimates the regression parameters by making the squares of the error term the least or
minimum.
Geometrically, this is seen as the sum of the squared distances, parallel to the axis of the dependent
variable, between each data point in the set and the corresponding point on the regression surface—
the smaller the differences, the better the model fits the data. The resulting estimator can be
expressed by a simple formula, especially in the case of a simple linear regression, in which there is
a single regressor on the right side of the regression equation.
The OLS estimator is consistent for the level-one fixed effects when the regressors
are exogenous and forms perfect colinearity (rank condition), consistent for the variance estimate of
the residuals when regressors have finite fourth moments [1] and—by the Gauss–Markov theorem—
optimal in the class of linear unbiased estimators when the errors are homoscedastic and serially
uncorrelated. Under these conditions, the method of OLS provides minimum-variance mean-
unbiased estimation when the errors have finite variances. Under the additional assumption that the
errors are normally distributed with zero mean, OLS is the maximum likelihood estimator that
outperforms any non-linear unbiased estimator.
Ordinary Least Squares (OLS) is the most common estimation method for linear models—and that’s
true for a good reason. As long as the model satisfies the OLS assumptions for linear regression, the
best possible estimates can be found.
Regression is a powerful analysis that can analyze multiple variables simultaneously to answer
complex research questions. However, the OLS assumptions is not satisfied, the results cannot be
trusted.

3
2.2 OLS Assumptions
Like many statistical analyses, ordinary least squares (OLS) regression has underlying assumptions.
When these classical assumptions for linear regression are true, ordinary least squares produces the
best estimates. However, if some of these assumptions are not true, remedial measures or use other
estimation methods are needed to be employed to improve the results.
Many of these assumptions describe properties of the error term. Unfortunately, the error term is a
population value that will be never known. Instead, the residuals can be used. Residuals are the
sample estimate of the error for each observation.
Residuals = Observed value – the fitted value
OLS Assumption 1: The regression model is linear in the coefficients and the error term
This assumption addresses the functional form of the model. In statistics, a regression model is linear
when all terms in the model are either the constant or a parameter multiplied by an independent
variable. You build the model equation only by adding the terms together. These rules constrain the
model to one type:

In the equation, the betas (βs) are the parameters that OLS estimates. Epsilon (ε) is the random error.
In fact, the defining characteristic of linear regression is this functional form of the parameters rather
than the ability to model curvature. Linear models can model curvature by including
nonlinear variables such as polynomials and transforming exponential functions.
To satisfy this assumption, the correctly specified model must fit the linear pattern.
OLS Assumption 2: The error term has a population mean of zero
The error term accounts for the variation in the dependent variable that the independent variables do
not explain. Random chance should determine the values of the error term. For the model to be
unbiased, the average value of the error term must equal zero.
Suppose the average error is +7. This non-zero average error indicates that the model systematically
under-predicts the observed values. Statisticians refer to systematic error like this as bias, and it
signifies that this model is inadequate because it is not correct on average.
Stated another way, we want the expected value of the error to equal zero. If the expected value is +7
rather than zero, part of the error term is predictable, and we should add that information to the
regression model itself. We want only random error left for the error term.

4
It is not needed to be worried about this assumption when the constant is included in regression
model because it forces the mean of the residuals to equal zero.
OLS Assumption 3: All independent variables are uncorrelated with the error term
If an independent variable is correlated with the error term, the independent variable can be used to
predict the error term, which violates the notion that the error term represents unpredictable random
error. We need to find a way to incorporate that information into the regression model itself.
This assumption is also referred to as exogeneity. When this type of correlation exists, there is
endogeneity. Violations of this assumption can occur because there is simultaneity between the
independent and dependent variables, omitted variable bias, or measurement error in the independent
variables.
Violating this assumption biases the coefficient estimate. To understand why this bias occurs, keep
in mind that the error term always explains some of the variability in the dependent variable.
However, when an independent variable correlates with the error term, OLS incorrectly attributes
some of the variance that the error term actually explains to the independent variable instead. For
more information about violating this assumption, read my post about confounding variables and
omitted variable bias.
OLS Assumption 4: Observations of the error term are uncorrelated with each other
One observation of the error term should not predict the next observation. For instance, if the error
for one observation is positive and that systematically increases the probability that the following
error is positive, that is a positive correlation. If the subsequent error is more likely to have the
opposite sign, that is a negative correlation. This problem is known both as serial correlation and
autocorrelation. Serial correlation is most likely to occur in time series models.
For example, if sales are unexpectedly high on one day, then they are likely to be higher than
average on the next day. This type of correlation isn’t an unreasonable expectation for some subject
areas, such as inflation rates, GDP, unemployment, and so on.
In the graph for a sales model, there is a cyclical pattern with a positive correlation.

5
To predict the error term for an observation, that information must be incorporated into the model
itself. To resolve this issue, an independent variable might be needed to add to the model that
captures this information. Analysts commonly use distributed lag models, which use both current
values of the dependent variable and past values of independent variables.
For the sales model above, variables that explains the cyclical pattern is needed to be added.
Serial correlation reduces the precision of OLS estimates. Analysts can also use time series analysis
for time dependent effects.
An alternative method for identifying autocorrelation in the residuals is to assess the autocorrelation
function, which is a standard tool in time series analysis.
OLS Assumption 5: The error term has a constant variance (no heteroscedasticity)
The variance of the errors should be consistent for all observations. In other words, the variance does
not change for each observation or for a range of observations. This preferred condition is known as
homoscedasticity (same scatter). If the variance changes, we refer to that as heteroscedasticity
(different scatter).
The easiest way to check this assumption is to create a residuals versus fitted value plot. On this type
of graph, heteroscedasticity appears as a cone shape where the spread of the residuals increases in
one direction. In the graph below, the spread of the residuals increases as the fitted value increases.

6
Heteroscedasticity reduces the precision of the estimates in OLS linear regression.
OLS Assumption 6: No independent variable is a perfect linear function of other explanatory
variables
Perfect correlation occurs when two variables have a Pearson’s correlation coefficient of +1 or -1.
When one of the variables changes, the other variable also changes by a completely fixed proportion.
The two variables move in unison.
Perfect correlation suggests that two variables are different forms of the same variable. For example,
games won and games lost have a perfect negative correlation (-1). The temperature in Fahrenheit
and Celsius have a perfect positive correlation (+1).
Ordinary least squares cannot distinguish one variable from the other when they are perfectly
correlated. If you specify a model that contains independent variables with perfect correlation, your
statistical software can’t fit the model, and it will display an error message. You must remove one of
the variables from the model to proceed.
Perfect correlation is a show stopper. However, your statistical software can fit OLS regression
models with imperfect but strong relationships between the independent variables. If these
correlations are high enough, they can cause problems. Statisticians refer to this condition as
multicollinearity, and it reduces the precision of the estimates in OLS linear regression.

7
OLS Assumption 7: The error term is normally distributed (optional)
OLS does not require that the error term follows a normal distribution to produce unbiased estimates
with the minimum variance. However, satisfying this assumption allows you to perform statistical
hypothesis testing and generate reliable confidence intervals and prediction intervals.
The easiest way to determine whether the residuals follow a normal distribution is to assess a normal
probability plot. If the residuals follow the straight line on this type of graph, they are normally
distributed. They look good on the plot below!

These assumptions are extremely important because violation of any of these assumptions would
make OLS estimates unreliable and incorrect. Specifically, a violation would result in incorrect signs
of OLS estimates, or the variance of OLS estimates would be unreliable, leading to confidence
intervals that are too wide or too narrow.

2.3 Deriving OLS Estimates for a Simple Regression Model

The Simple Regression:
In Econometrics, a simple regression is a tool used to establish a relationship between 2 variables.
One of the variables (Y) is called the dependent variable or the regressand, while the other variable
(X) is called the independent variable or the regressor. Mathematically, a simple regression model is
expressed as:

8
Here α and β are the regression coefficients i.e. the parameters that need to be calculated to
understand the relation between Y and X. i has been subscripted along with X and Y to indicate that
we are referring to a particular observation, a particular value associated with X and Y. εᵢ is the error
term associated with each observation i.
Estimating α and β:
So, how do we estimate α and β? One of the most common approach used by statisticians is the OLS
approach. OLS stands for Ordinary Least Squares. Under this method, we try to find a linear function
that minimizes the sum of the squares of the difference between the true value of Y and the predicted
value of Y. Let the true value of Y associated with each observation be Yi and the value predicted by
our model be α +βXᵢ. So, we are essentially trying to minimise:

where i, goes from 1 to n, indicates that we have n observations in our dataset for values of X and Y.
Graphs can be more intuitive:

9
Derivation
For estimating α and β:
Step 1: Defining the OLS function
OLS, as described earlier is a function of α and β. So our function can be expressed as:

Step 2: Minimizing our function by taking partial derivatives and equating them to zero.
First, we take the partial derivative of f(α, β) with respect to α, and equate the derivative to zero to
minimize the function over α.

Equation 1
Note: We have replaced α and β with α-hat and β-hat to indicate that we are finding an estimate for
the regression coefficients.
Similarly, we take the partial derivative of f(α, β) with respect to β, and equate the derivative to zero
to minimize the function over β.

10
 Equation 2

Step 3: We solve equation 1 In order to obtain a relation between α-hat and β-hat
From Equation 1,

Dividing by n (number of observations) on both sides of the equation:

On splitting up the expression and solving further:

Equation 3

11
Now, we define X̅ and Y̅ as the means of the observations under X and Y from our dataset.

Mathematically:

Using these in Equation 3:

Equation 4

Step 4: We solve for equation 2 using results from equation 1 and 4 to get an estimate for β-
hat.
First, we multiply equation 1 by X̅:

Subtracting this from equation 2:

12
Using equation 4,

Substituting the value of α-hat in the previous equation:

This is the required expression for estimating β-hat.

To obtain the expression for calculating α-hat, we substitute the expression for β-hat in equation 4:

13
2.4 The Gauss-Markov Theorem
The Gauss-Markov Theorem is named after Carl Friedrich Gauss and Andrey Markov.
Let the regression model be: Y =βo+βiXi+ε
Let βo and βi be the OLS estimators of  βo and βo
According to the Gauss-Markov Theorem, under the assumptions A 1 to A5 of the linear regression
model, the OLS estimators βo and  }βi are the Best Linear Unbiased Estimators (BLUE) of βo and 
βi.
In other words, the OLS estimators βo and βi have the minimum variance of all linear and unbiased
estimators of βo and βi. BLUE summarizes the properties of OLS regression. These properties of
OLS in econometrics are extremely important, thus making OLS estimators one of the strongest and
most widely used estimators for unknown parameters. This theorem tells that one should use OLS
estimators not only because it is unbiased but also because it has minimum variance among the class
of all linear and unbiased estimators.
2.5 Properties of OLS Estimator
Property 1: Linear
This property is more concerned with the estimator rather than the original equation that is being
estimated. In assumption A1, the focus was that the linear regression should be “linear in
parameters.” However, the linear property of OLS estimator means that OLS belongs to that class of
estimators, which are linear in Y, the dependent variable. Note that OLS estimators are linear only
with respect to the dependent variable and not necessarily with respect to the independent variables.
The linear property of OLS estimators doesn’t depend only on assumption A1 but on all assumptions
A1 to A5.
Property 2: Unbiasedness
If you look at the regression equation, you will find an error term associated with the regression
equation that is estimated. This makes the dependent variable also random. If an estimator uses the
dependent variable, then that estimator would also be a random number. Therefore, before
describing what unbiasedness is, it is important to mention that unbiasedness property is a property
of the estimator and not of any sample.
Unbiasedness is one of the most desirable properties of any estimator. The estimator should ideally
be an unbiased estimator of true parameter/population values.
Consider a simple example: Suppose there is a population of size 1000, and you are taking out
samples of 50 from this population to estimate the population parameters. Every time you take a

14
sample, it will have the different set of 50 observations and, hence, you would estimate different
values of βo and βi. The unbiasedness property of OLS method says that when you take out samples
of 50 repeatedly, then after some repeated attempts, you would find that the average of all
the βo and βi from the samples will equal to the actual (or the population) values of βo and βi.
Mathematically,
E(bo) = βo
E(bi) = βi
Here, ‘E’ is the expectation operator.
In layman’s term, if you take out several samples, keep recording the values of the estimates, and
then take an average, you will get very close to the correct population value. If your estimator is
biased, then the average will not equal the true parameter value in the population.
The unbiasedness property of OLS in Econometrics is the basic minimum requirement to be satisfied
by any estimator. However, it is not sufficient for the reason that most times in real-life applications,
you will not have the luxury of taking out repeated samples. In fact, only one sample will be
available in most cases.
Property 3: Best: Minimum Variance
The efficient property of any estimator says that the estimator is the minimum variance
unbiased estimator. Therefore, while taking unbiased estimators of the unknown population
parameter, the estimator will have the least variance. The estimator that has less variance will have
individual data points closer to the mean. As a result, they will be more likely to give better and
accurate results than other estimators having higher variance. In short:
1. If the estimator is unbiased but doesn’t have the least variance – it’s not the best!
2. If the estimator has the least variance but is biased – it’s again not the best!
3. If the estimator is both unbiased and has the least variance – it’s the best estimator.
Now, talking about OLS, OLS estimators have the least variance among the class of all linear
unbiased estimators. Efficiency property says least variance among all unbiased estimators, and OLS
estimators have the least variance among all linear and unbiased estimators.
Just denoting mathematically,
Let bobe the OLS estimator, which is linear and unbiased. Let bo∗ be any other estimator of βo
which is also linear and unbiased. Then,
, 

Var(bo)<Var(bo∗)

15
Let bi be the OLS estimator, which is linear and unbiased. Let bi∗ be any other estimator of βi, which
is also linear and unbiased. Then,
Var(bi)<Var(bi∗)
The above three properties of OLS model makes OLS estimators BLUE as mentioned in the Gauss-
Markov theorem.
It is worth spending time on some other estimators’ properties of OLS in econometrics. The
properties of OLS described below are asymptotic properties of OLS estimators. So far, finite
sample properties of OLS regression were discussed. These properties tried to study the behavior of
the OLS estimator under the assumption that you can have several samples and, hence, several
estimators of the same unknown population parameter. In short, the properties were that the average
of these estimators in different samples should be equal to the true population parameter
(unbiasedness), or the average distance to the true parameter value should be the least (efficient).
However, in real life, you will often have just one sample. Hence, asymptotic properties of OLS
model are discussed, which studies how OLS estimators behave as sample size increases. Keep in
mind that sample size should be large.
This property is what makes the OLS method of estimating  and  the best of all other methods. When
there are more than one unbiased method of estimation to choose from, that estimator which has the
lowest variance is best. (Variance is a measure of how far the different  and  are from their mean; the
variance is the average distance of an element from the average.)
An estimator (a function that we use to get estimates) that has a lower variance is one whose
individual data points are those that are closer to the mean. This estimator is statistically more likely
than others to provide accurate answers. The OLS estimator is one that has a minimum variance.
This property is simply a way to determine which estimator to use.
 An estimator that is unbiased but does not have the minimum variance is not good.
 An estimator that has the minimum variance but is biased is not good
 An estimator that is unbiased and has the minimum variance of all other estimators is the best
(efficient).
The OLS estimator is an efficient estimator.
Property 4: Asymptotic Unbiasedness
This property of OLS says that as the sample size increases, the biasedness of OLS estimators
disappears.

16
Property 5: Consistency
A consistent estimator is one which approaches the real value of the parameter in the population as
the size of the sample, n, increases.
An estimator is said to be consistent if its value approaches the actual, true parameter (population)
value as the sample size increases. An estimator is consistent if it satisfies two conditions:
a. It is asymptotically unbiased
b. Its variance converges to 0 as the sample size increases.
Both these hold true for OLS estimators and, hence, they are consistent estimators. For an estimator
to be useful, consistency is the minimum basic requirement. Since there may be several such
estimators, asymptotic efficiency also is considered. Asymptotic efficiency is the sufficient condition
that makes OLS estimators the best estimators.

2.6 Applications of OLS estimators

OLS estimators, because of such desirable properties discussed above, are widely used and find
several applications in real life.
Example: Consider a bank that wants to predict the exposure of a customer at default. The bank can
take the exposure at default to be the dependent variable and several independent variables like
customer level characteristics, credit history, type of loan, mortgage, etc. The bank can simply run
OLS regression and obtain the estimates to see which factors are important in determining the
exposure at default of a customer. OLS estimators are easy to use and understand. They are also
available in various statistical software packages and can be used extensively.
OLS regressions form the building blocks of econometrics. Any econometrics class will start with
the assumption of OLS regressions. It is one of the favorite interview questions for jobs and
university admissions. Based on the building blocks of OLS, and relaxing the assumptions, several
different models have come up like GLM (generalized linear models), general linear models,
heteroscedastic models, multi-level regression models, etc.
Research in Economics and Finance are highly driven by Econometrics. OLS is the building block of
Econometrics. However, in real life, there are issues, like reverse causality, which render OLS
irrelevant or not appropriate. However, OLS can still be used to investigate the issues that exist in
cross-sectional data. Even if OLS method cannot be used for regression, OLS is used to find out the
problems, the issues, and the potential fixes.

17
3. CONCLUSION

To conclude, linear regression is important and widely used, and OLS estimation technique is the
most prevalent. In this article, the assumption of OLS and properties of OLS estimators were
discussed because it is the most widely used estimation technique. OLS estimators are BLUE (i.e.
they are linear, unbiased and have the least variance among the class of all linear and unbiased
estimators). Amidst all this, one should not forget the Gauss-Markov Theorem (i.e. the estimators of
OLS model are BLUE) holds only if the assumptions of OLS are satisfied. Each assumption that is
made while studying OLS adds restrictions to the model, but at the same time, also allows making
stronger statements regarding OLS. So, whenever, a linear regression model using OLS is planned to
use, it is always necessary to check for the OLS assumptions. If the OLS assumptions are satisfied,
OLS can be directly usedfor the best results.

18
 References

Gujarati, D. (2000). Two-variable regression model: the problem of estimationchapter (3). Basic
Econometrics 4(Th) Edition,i, 697.
Kloppers, P.H., and Shatalov, M.Y., (2012). A new method for least squares identification of
parameters of the transcendental equations, International Journal of the Physical Sciences, (7),
5218–5223.

19