Application of Outlier Detection using Re-Weighted
Least Squares and R-squared for IoT Extracted Data
Moath Awawdeh Tarig Faisal
Department of Electrical Engineering Department of Electrical Engineering
Higher College of Technology Higher College of Technology
Abu Dhabi, UAE Abu Dhabi, UAE
*mawawdeh@hct.ac.ae tfaisal@hct.ac.ae
Amjad Sheikh
Department of Chemical Engineering
Higher College of Technology
Abu Dhabi, UAE
asheikh@hct.ac.ae
Abstract—The problem of outlier detection for measurements
extracted from IoT platform has been addressed. An approach to
validate the detected values via the coefficient of determination
analysis is presented by applying a combination procedure of
weighted least square, bisquare algorithm and robust fit. We
fit the model firstly by weighted least square then we used the
method of bisquare weight where the weight of each measure is
assigned based on the distance of that value to the generated best
fit. The map of weights for each measure in the dataset determine
if the value is a possible outlier by searching for those measures
with zero weight. In the second part of the paper we apply
some of the most common method of outlier processing with
logarithmic and square root transformation where the reweighted
least square has been used to detect the outliers. We Analyze
three cases of dealing with outliers including transformation and
outlier removal then we measure the coefficient of determination
for each case together with other statistic. The results are shown
by mean of simulation with estimation result and weight labeling.
Keywords—Weighted least squares, outlier detection, coeffi-
cient of determination, outlier transformation.
I. I NTRODUCTION
Data mining has been widely investigated in the literature
in term of data preprocessing and transformation where a
useful patterns are generated using analysis tools and al-
gorithms [1]. For most of the applications in data analysis
a huge amount of data are recorded including the outlying
variables that would considerably affect the robustness of the
model regression. Those observations are usually considered
as noise which may lead to generate a bad estimator and
incorrect models although they may in many situations hide
useful information. Therefore, it is crucial task to identify
outliers before applying data analysis. [2]–[4]. Hawkins [5]
defined the concept of an outlier based on the deviations
among those measures considering the mechanism in which
those observations are generated. The detection methods of
outliers have been studied and suggested for many applications
This work is supported by Higher College of Technology Research Pro-
gram, 2018
Authorized licensed use limited to: Higher College of Technology. Downloaded on September 26,2021 at 14:29:13 UTC from IEEE Xplore. Restrictions apply.
Anees Bashir
Department of Electrical Engineering
Higher College of Technology
Abu Dhabi, UAE
abashir1@hct.ac.ae
such as, intrusion detection systems [6], [7], wireless sensors
networks [6], [8], and more applications to outlier detection
in satellite image analysis, motion segmentation, and severe
weather prediction, can be found in [2]–[4], [6], [9]–[11]
where surveys and analytic comparison of different methods
and applications are presented. The effect of those outliers on
the regression analysis can be reduced by applying one of the
two options (delete or transform) of dealing with outliers. If
the measures contain a limited number of outliers so deleting
those values is a common option to do in such cases, still
the observations may carry useful knowledge. This option
is widely used with Tukey’s box-and-whisker plot. Deleting
the variables instead, would be the easiest choice in case of
many outliers presence in the variables. The other option is
transformation, whether to transform the values by changing
them to the neighbor value or transforming the variables
instead of changing each outlying observation individually
[12]–[14]. In any IoT platfrom, where sensors are sending
measures from the field remotely, the data can be affected
by noise or any other factor like sensor malfunction where
outlying events occurs. Analyzing those reading with no pre-
processing generate a bad estimator, and hence miss leading
decision. An IoT platform equipped with Matlab analysis and
visualization tools can be used to accomplish dataset prepa-
ration and processing. In this paper we introduce a method
for validating outlier detection processing via coefficient of
determination check analysis. The data are considered to be
extracted from an IoT platform, where the dataset can be
created using those measures stored in the cloud and then
processed and analyzed. Taking a decision using data in an IoT
platfrom need a primary step of data preparation, together with
robust interpretation of the information, in this stage our paper
is contributing. This paper is organized as follow in Section
II, a brief of IoT data processing is addressed, Section III, we
introduce the polynomial regression least squares, in Section
VI, re-weighted least square modeling is presented, and finally
Simulation example is drawn in Section V.
 II. I OT- BASED DATA E XTRACTION AND M ODELING
Using the concept of Internet-of-Thing (IoT), sensors mea-
sure data of different physical quantity and communicate those
data in various forms. An IoT platform provides the sensors
with the ability to send data to the cloud where it can be
analyzed and stored. Such a platform is equipped with Matlab
visualization and analyzing tool, where stored data can be
easily explored and interpreted. This feature gives the user an
access to Matlab to explore more features and extract different
knowledge using the built-in tools. The data coming from
those sensors are real-time data and have not been processed
or transformed. Thus, interpreting those data and extracting
useful patterns without applying primary steps of data pre-
processing and transformation may lead to misinterpreted
pattern that effect greatly the decision making. A crucial step
in the chain of data processing is outlier detection, where data
must be clean and correct. Such a step could be integrated in
an IoT platform using the access of Matlab to improve and
validate the analysis result.
III. P OLYNOMIAL R EGRESSION , L EAST S QUARES (LS)
Polynomial regression present the fitting line of the nonlin-
ear model between x and y with nth degree polynomial. It can
be considered as a form of estimation statistical processes.
The basic regression model of a dependent variable Y
on a set of k independent variables X1 , X2 , · · · , Xk can be
expressed as [15]:
yi = β0 + β1 xi1 + β2 xi2 + · · · + βk xik + ei
i = 1, 2, · · · , n. Where yi is the value of Y for the ith case,
xij is the value of the j th independent variable Xj for the ith
case, and β0 is the Y -intercept of the regression surface,see
[15], [16] for model assumptions and components.
For the polynomial curve fitting, we consider the general
form for a polynomial of order j:
j
X squares are minimized. The distance between a point and the
f (x) = β0 + βk x k
k=1
and the general least squares error (residuals):
!#2
j
n
"
X X
r2 = yi − β0 + βk x k
i=1 k=1
In regression analysis the polynomial regression is a primary
step which has the crucial impact on the design. for our scope,
a Matlab-based tool [17] has been used which provide the
decision support system users with an interactive plot of the
result in a graphical interface. The result is interactive plotting
in graphical interface where user can change the parameters
of the fit and exporting fit results to the Matlab’s workspace
for more analysis.
Authorized licensed use limited to: Higher College of Technology. Downloaded on September 26,2021 at 14:29:13 UTC from IEEE Xplore. Restrictions apply.
IV. R E -W EIGHTED L EAST S QUARES (R E -WLS)
M ODELING
For estimating the known parameter in regression analysis
the ordinary least squares is popular technique to fit the
model by the concept of minimizing the sum of squares
between observed and predicted values. The validity of least
squares estimation depend on the validity of the assumptions
underlying the regression model. Least squares (LS), generate
a bad estimation result in case of outlier, or generally in case
when some measures deviate so much from other measures
in the dataset, this is because of the concept of LS, where
the non-normal points (those differ from the rest points) will
greatly affect the analysis of the data (effect on the slope of the
regression line. Regression parameter can be estimated using
weighted least squares, this estimator is also can be affected by
outliers like ordinary least squares. Thus, we will be proposing
the re-weighted least squares algorithms using robust fitting,
see [18] for detection of outliers in weighted least squares
regression and modeling description.
A. Weighted Least Square (WLS) Regression
The weighted least square model is given by
Y = Xβ +  (4)
where Y is an n×1 vector of dependent observations, X is an
n×p0 full column rank matrix of known explanatory variables,
β is a p0 × 1 vector of unknown parameters to be estimated,
and error term  is an n × 1 vector of independent random
2
errors with zero mean and unknown variance
ˆ P σ . The TWLS
(1) estimator βw is obtained by minimizing wi (yi − xi β)2 ,
ˆ T −1 T
that is βw = (X Dw X) X Dw Y . The WLS algorithm
providing the variance of the estimator, the vector of weighted
fitted value, the vector of residuals, and the unbiased estimator.
B. Outlier Detection using Re-WLS
As discussed earlier, the least-squares fitting is sensitive to
the outlying observations. To minimize this effect, the data are
fit using Bisquare weights method, which weighted sum of
(2) estimation line determines the weight that a point will carry.
Nearest points will get the highest weight while those point
that are very far from the fitted line will get zero weight and
considered as outliers. Matlab provides a combination pro-
cedure of WLS and bisquare weight algorithms for detecting
outliers and it can be described as follow
(3)
1) WLS for model fitting
2) Computing the adjusted residuals
ri
radj = √ (5)
1 − hi
ri are the usual least-squares residuals and hi are lever-
ages that adjust the residuals
3) Standardizing the adjusted residuals
radj
u= (6)
Ks
 where K is 4.685, and s is the robust variance given by
M AD
s= (7)
0.6745
where M AD is the median absolute deviation of the
residuals.
4) Bisquare weight computation
(1 − u2i )2

|ui | < 1
wi = (8)
0 |ui | ≥ 1
5) Finding the zeros in the weight matrix and label each
zero’s weight index with the value’s position in the data.
This labeled data is to be considered as outlier. If the fit
converges, and the outliers are detected then the process
is done. Otherwise, to repeat form step 1.
We consider en example of 20 measures as shown in Fig. 1. In
Fig. 2 the using WLS for outlier detection is applied. We used
a data set of 20 measures with single synthetic outlier at time
Fig. 1. Least square fitting with a single outlier instance t = 16. The model has been fitted using WLS. We
have followed the steps that mentioned earlier. Fig. 3 shows the
weights labeling where each measures has a bisquare weight
according to Equation (8).
C. Coefficient of determination R2
R-squared (R2 ) is a measure of the goodness of the re-
gression line to the measures, it is known as coefficient of
determination. This tool can be a very useful in assessing the
quality of the estimator in linear regression. As mentioned be-
fore about LS, unfortunately it can be said also that coefficient
of determination is also sensitive to outlier and the goodness
of the model can be greatly affected. It is assigned with values
zero to one, where a poor fitting line represented by a value
of closer to 0, while values near 1, represents the best fitting.
We have used this tool as a measure of fitting accuracy in the
presence of outlying observation and how do outliers affect the
regression analysis. The R2 can be presented in the following
ratio [19]:
ESS RSS
Fig. 2. Outlier detection via WLS R2 = =1− (9)
T SS T SS
P n 2
(yi − yˆi )
= 1 − Pi=1 n 2
(10)
i=1 (yi − ȳ)
where ESS, T SS and RSS are the explained, total and resid-
ual sum of squares respectively. When there is an intercept
term in the linear model, this coefficient of determination
is actually equal to the square of the correlation coefficient
between yi and yˆi
 2
Pn ¯
(yi − ȳ)(yˆi − ŷ)
R2 =  qP i=1  (11)
n 2
Pn ¯ 2
(y
i=1 i − ȳ) (
i=1 iy
ˆ − ŷ)
with ŷ¯ the mean predicted responses. Following the line of
[19], [20]. The adjusted R2 is defined as
 
2 2 n−1
Radj = 1 − (1 − R ) (12)
n−p−1
 
p
= R2 − (1 − R2 ) (13)
Fig. 3. Weights labeling for the measurements in Fig. 2 n−p−1

Authorized licensed use limited to: Higher College of Technology. Downloaded on September 26,2021 at 14:29:13 UTC from IEEE Xplore. Restrictions apply.

Fig. 4. Original data with three synthetic outliers and data transformation
where p is the total number of regressions in the linear model,
and n is the sample size. In the following pages, we show the
effect of outliers on R2 by the mean of simulation with sta-
tistical regression analysis. We use the same observations for
all comparisons with synthetic outliers. Data transformation is
used to reduce the effect of outlier in case of not removing
them. Hence, we show also the effect of transformation on R2
and we compare that with the case of removing outliers. The
measurements which have been used and their transformation
are shown in Fig. 4. Outliers at time instance t = 4, 10, 20 have
been considered in the original measures, we have counted
for three outliers instead of one as in Fig. 2 to increase the
model misfitting and to count for more than single outlier
case. Data transformation has been used and we show the
effect of outliers on R2 with in next pages. Fig. 5 and Fig. 6
show the detection of outliers using WLS and weight labeling
respectively, where three measures have been assigned with
weight zero which can be considered as outliers. No any outlier
processing has been applied and the resulted R2 is around 51
%, more regression and statistical analysis are extracted in
Table I and II.
V. S IMULATION E XAMPLE
A set of 20 measures as shown in Fig. 4, has been
considered for our analysis. Three different values have been
assigned as outliers, the values as 18, 26 and 32. The same
approach of last section with different outlier processing has
been implemented considering three cases of dealing with
outlier as following, outlier removal (Fig. 7-8 and its statistic in
Table III and IV), square root and logarithmic transformation
for Fig. 9-10, and Fig. 11-12 respectively. For the same
measure, the data can be reversely analyzed, with an indicator
of the best dealing option according to its fitted model. The
approach can be briefed as step 1, detect the outlier using
Re-WLS, step 2, process the outlier as per model assumption,
step 3, find the R2 for each model, last step is processing
the outlier based on the model that generate the maximum
R-squared except for the removal case.
C ONCLUSION
A method for visualizing and dealing with detected outlier
has been proposed. The use of R2 to measure the optimal
Authorized licensed use limited to: Higher College of Technology. Downloaded on September 26,2021 at 14:29:13 UTC from IEEE Xplore. Restrictions apply.
action of dealing with outlier is presented and the simulation
result shows the different statistic when remove and transform
those values. Considering the transformation,one can say that
logarithmic transformation for the given data-set shows better
fitting line than other transformation, still of course removing
all values give us the maximum R2 . The reverse measure tool
can be used in a way that finding the value of the coefficient of
determination for the all advised outlier processing methods,
then approaching the maximum value as the indicator to
the best fit, then to apply that chosen method for the same
confidence.
R EFERENCES
[1] U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth. ”From Data Mining to
Knowledge Discovery in Databases” AI Magazine American Association
for Artificial Intelligence, pp.37–54, Fall 1996.
[2] I. Ben-Gal ”Outlier Detection” In: Maimon O. and Rockach L. (Eds.)
Data Mining and Knowledge Discovery Handbook: A Complete Guide
for Practitioners and Researchers, Kluwer Academic Publishers, 2005.
[3] G. J. Williams ,R. A. Baxter ,H. X. He ,S. Hawkins, and L. Gu ”A
Comparative Study of RNN for Outlier Detection in Data Mining” IEEE
International Conference on Data-mining (ICDM’02), Maebashi City,
Japan, CSIRO Technical Report CMIS-02/102, 2002.
[4] H. Liu, S. Shah, and W. Jiang. ”On-line Outlier Detection and Data
Cleaning” Journal of Computers and Chemical Engineering, vol. 28,
pp. 1635–7647, 2004.
[5] D. Hawkins ”Identification of Outliers”. Chapman and Hall, 1980.
[6] C. C. Aggarwal ”Outlier Analysis”. Kluwer Academic Publishers.
[7] N. Devarakonda, S. Pamidi, V. V. Kumari, A. Govardhan ”Outliers
Detection as Network Intrusion Detection System Using Multi Layered
Framework” Advances in Computer Science and Information Technology
Communications in Computer and Information Science. Springer Vol.
131, pp. 101–111, 2011.
[8] O. Ghorbel, M.W. Jmal, W. Ayedi, H. Snoussi, and M. Abid. ”An
overview of outlier detection technique developed for wireless sensor
networks” IEEE 10th International Multi-Conference on Systems,
Signals Devices (SSD), pp. 1–6, 2013.
[9] S. Cateni, V. Colla, and M. Vannucci ”Outlier Detection Methods for
Industrial Applications” Advances in Robotics, Automation and Control,
Book edited by: Jess Armburo and Antonio Ramrez Trevio, I-Tech,
Vienna, Austria, 2008.
[10] V. J. Hodge, and J. Austin ”A Survey of Outlier Detection Methodolo-
gies.”. Kluwer Academic Publishers, 2004.
[11] V. Chandola, A. Banerjee, and V. Kumar ”Outlier Detection : A Survey”.
University of Minnesota.
[12] J. Osborne ”Notes on the use of data transformations”. Practical
Assessment, Research Evaluation, Vol. 8(6), 2002.
[13] J. W. Osborne, and A. Overbay ”The power of outliers and why
researchers should always check for them”. Practical Assessment,
Research Evaluation, Vol. 9(6), 2004.
[14] J. W. Osborne ”Improving your data transformations: Applying the Box-
Cox transformation”. Practical Assessment, Research Evaluation, Vol.
15(12), 2010.
[15] E. Ostertagova ”Modelling using polynomial regression ”. ”Procedia
Engineering, Elsevier”, Vol 48, pp. 500–506, 2012.
[16] A. D. Aczel ”Complete Business Statistics”. Irwin, ISBN 0-256-
05710-8, 1989.
[17] Matlab and Statistics Toolbox Release 2012b The MathWorks
Inc., Natick, Massachusetts, United States. Available online on
http://www.mathwork.com
[18] Sohn, B.Y. Kim, G.B. ”Detection of outliers in weighted least squares
regression”. ”Korean J. Comp. Appl. Math”, Vol. 4(2), pp. 441–452,
1997.
[19] O. Renaud, and M.-P. Victoria-Feser ”A robust coefficient of determi-
nation for regression”. Journal of Statistical Planning and Inference,
Vol. 140(7), pp. 1852–1862, 2010.
[20] W. Greene. ”Econometric Analysis (third ed.)”. Prentice Hall, 1997.
Fig. 5. Outliers detection using Re-WLS and no outliers processing Fig. 7. Outlier removal with Re-WLS detection

Fig. 6. Weight labeling with outliers label 0.00 Fig. 8. Weight labeling with 3 measures have been removed

TABLE I TABLE III

R EGRESSION S TATISTICS R EGRESSION S TATISTICS

Statistic Value Statistic Value

Multiple R 0.71939 Multiple R 0.94209
R2 51.752 % R2 88.753 %
Adjusted R2 0.49072 Adjusted R2 0.88003
Standard Error 5.88154 Standard Error 2.04944
Outliers 3 Outliers Removed 3
Outlier process No Action Outlier process Removed

TABLE II TABLE IV
S TATISTICAL A NALYSIS S TATISTICAL A NALYSIS

No action to outlier Outliers have been removed

Coefficient St. Error t stat Coefficient St. Error t stat
Intercept 1.432 2.732 0.524 Intercept -0.341 1.039 -0.328
Variable 1.002 0.228 4.394 Variable 1.103 0.101 10.88

Authorized licensed use limited to: Higher College of Technology. Downloaded on September 26,2021 at 14:29:13 UTC from IEEE Xplore. Restrictions apply.
 Fig. 9. Outlier detection and square root transformation Fig. 11. Outlier detection and logarithmic transformation

Fig. 10. Transformed data weight labeling Fig. 12. Transformed data weight labeling

TABLE V TABLE VII

R EGRESSION S TATISTICS R EGRESSION S TATISTICS

Statistic Value Statistic Value

Multiple R 0.77557 Multiple R 0.80056
R2 60.151 % R2 64.090 %
Adjusted R2 0.57937 Adjusted R2 0.62095
Standard Error 0.80536 Standard Error 0.23256
Outliers 2 Outliers 2
Transformation Square root Transformation Logarithmic

TABLE VI TABLE VIII

S TATISTICAL A NALYSIS S TATISTICAL A NALYSIS

Square root transformation Logarithmic transformation

Coefficient St. Error t stat Coefficient St. Error t stat
Intercept 1.529 0.374 4.088 Intercept 0.413 0.108 3.822
Variable 0.162 0.031 5.212 Variable 0.051 0.009 5.667

Authorized licensed use limited to: Higher College of Technology. Downloaded on September 26,2021 at 14:29:13 UTC from IEEE Xplore. Restrictions apply.