On the Use of Predictive Analytics Techniques for
Network Elements Failure Prediction in Telecom
Operators
Ahmed F. Fahmy Ahmed H. Yousef
Computers & Systems Computers & Systems
Department, Faculty of Department, Faculty of
Engineering, Ain Shams Engineering, Ain Shams
University University
Cairo, Egypt. Cairo, Egypt.
Ahmed.fahmy@eng.asu.edu.eg ahassan@eng.asu.edu.eg
Abstract— Reliability of the telecom operator optical network
infrastructure is one of the most important competitive advantages
of the operator to provide the required quality of service to its
customer. Massive amounts of voice and data traffic can be lost
due to network failure, especially in the case of failures of nodes
(optical switches). In this paper, we use predictive analytics
techniques in one of the biggest telecom operators in the Middle
East to predict the node failures in advance to take the
precautionary measures before it fails.
Keywords— data mining; predictive analytics; telecom;
network element; failure
I. INTRODUCTION
Significant amounts of data and voice traffic can be lost
due to network failure. There may be cases in which only
parts of the node fail, and some parts remain functional. For
example, only some of the links on an optical switch may
cease to function because of some hardware failure in the
switch. In such cases, the protection scheme may be able to
treat them as individual link failures and take appropriate
measures to recover them. However, determining the nature
and the extent of the failure may be difficult. There is another
problem with allowing different failure modes for nodes:
Most protection schemes do not support simultaneous
recovery from several failures. Therefore, we assume that
none of the channels on the failed node remain active.
In this paper, we propose a methodology for building
Node Failure Prediction Model that will help to predict the
node failures in advance to take the precautionary measures
before it fails. Predictive analytics [1] will be applied to
achieve this objective. The term "predictive analytics"
describes any data mining approach with four attributes:
focus on prediction more than description, classification or
clustering, short time to insight (measured in minutes or
hours), focus on the resulting insights and its relevance to the
business, and focus on usability by making the tools
accessible to business users. Predictive analytics can bring
clarity and consistency to any situation where the likely future
behavior or condition is uncertain. The paper organization is
as follows: First, Section 2 describes the background and
978-1-5386-4266-5/17/$31.00 ©2017 IEEE
Hoda K. Mohamed
Computers & Systems
Department, Faculty of
Engineering, Ain Shams
University
Cairo, Egypt.
hoda.korashy@eng.asu.edu.eg
related work, then Sections 3 and 4 demonstrate the
methodology and the results from deploying it in a real
environment. Finally, Section 5 displays the conclusion.
II. BACKGROUND AND RELATED WORK
To the extent of our knowledge, the predictive analytics
algorithms have not been used in optical network equipment
failure prediction using our proposed methodology. The work
in this field focuses on the protection and restoration of failures
in fiber-optic networks. For example, [2], [3] presented the
efforts towards network protection algorithms. But, these
algorithms detect the failure only after it occurs. The same in
[4], the authors tried to propose four different protection
schemes for converged access network architecture, where a
hybrid passive optical network is deployed to provide high
bandwidth to the end users. However, they did not consider how
to forecast the failure. In Summary, some of the above
researchers were able to handle the protection and restoration
of failures in fiber-optic networks. However, no method was
designed to build a generic framework to predict failure of
nodes (optical switches) in telecom Operators.
III. METHODOLOGY
To explain our methodology, we need first to define the
research problem, research challenges, and the solution
A. Research Problem
To maintain the high quality of fiber optic nodes, detection of
failed or malfunctioning node is essential. The failure of the
node is either because of communication device failure or
battery, environment, and sensor device-related problems. To
check the failed node manually in such environment is
troublesome. As per the current process, nodes are
maintained either through periodical maintenance schedule
or as and when there is failure reported. This process is
expensive as there is no information available on which nodes
should be prioritized over others for maintenance and also
since the failure is fixed only after the node has failed, it leads
to bad customer experience. Therefore, there is need to be
250
 able to predict which nodes are more likely to fail and address
them on priority. So, the research question is “How to use
Measurement/customer/call/Field technician information to
predict the co-axial node failure much ahead in time so that
network operations team can fix it before it fails thereby
reducing the cost of maintenance, improving customer
experience and improving the node lifetime?”
B. Research Challenges
There are several complications involved in predicting a node
failure. Below are some major complications which need to
be addressed to predict a node failure.
▪ Multiple Reasons for failure – Failure of a co-axial cable
Node can happen because of several reasons. Failure can also
occur because of non-controllable factors such as
thunderstorms, cyclones, extreme weather conditions,
accidents, etc. Other reasons for Node failure can be
controllable such as technical problems, poor specifications
of the equipment, poor infrastructure, stolen bandwidth, etc.
It is a challenge to cleanly capture all these factors into a data
model which can be used for predictive analytics
▪ Actionability of prediction – It is of utmost importance to
give a fair prediction of failing nodes much ahead of time for
operations/maintenance team to be able to effectively use the
prediction ability. This further adds to the complexity of the
predictive model.
▪ Seasonality – Relationship between failure and affecting
reasons can change with time and may lead to decay of
accuracy of predictive model.
▪ Data captured from different sources – It is a challenge to
be able to accurately and effectively map together with the
data captured from different sources at one place. A Fair
amount of efforts and business understanding are required to
roll up the data which is captured at different levels and
different sources and prepare a single version of the truth.
C. Solution Overview
The Proposed solution to existing node maintenance process
is to create a predictive model. This predictive model will be
able to predict the nodes which have higher propensity to fail
as compared to other nodes and help maintenance team in
prioritizing node maintenance and pro-actively maintain
nodes that are likely to fail. This proposed process will help
in decreasing the overall node failure rate, increase the
customer experience and reduction in costs of visits made to
node site for maintenance.
The proposed Node Failure predictive model to solve the
research problem is shown in Fig. 1.
IV. EXPERIMENT
The data mining activities are carried out using Rapid Miner
v7.5 [5]. The experiment was done on an HP desktop computer
model: Envy. The processor is Intel(R) Core (TM) i7-6560 U
CPU @ 2.20 GHz; RAM is 16GB; and 64-bit operating system.
Rapid Miner license used is an educational licensee to avoid the
limitation of the trial version of dealing only with ten thousand
rows as a maximum.
A. Data Understanding and Preparation
Data pre-processing is often a critical and essential part of
the prediction process [6]. The primary purpose of data pre-
processing is to have a final and correct data set that is only
containing those columns that will contribute well to the data
mining algorithms and model. Data pre-processing helps to
focus on [7] the most critical issues within the famous
knowledge discovery from data process [8], [9].
Fig. 1. The proposed Node Failure predictive model
1) Defining Node Failure
There are multiple ways to define a Node Failure. Network
Maintenance or Engineering team can define and record when
a Node failed and what was the reason for failure. In our case,
we have used outage flag to determine Node Failure.
2) Identify and Define Predictor Variables
A set of predictor variables which are believed to have an
impact on Node performance are created using data sets which
are 24 hours before the Node failure from different sources
such as Measurement Data, Customer Data, Call interaction
data and Field technician data. Table I shows the attributes of
these datasets. For developing any analytics model, the first
step is to build one dataset with all the necessary data that will
go as an input in the model. To build a predictive Node Failure
Model, we need to create various calculated fields/metrics,
which are derived from various data sources. All the different
data information which is available is rolled up at Node Day
level so that all these metrics can be directly used for
exploratory and model training exercise.
TABLE I. ATTRIBUTES OF DATA SETS USED IN THE PREDICTION.
Data Set Definition
Measurement /Technical Data These variables capture the technical
parameters such as Signal to Noise
Ratios (SNR), Correctible and
Uncorrectable FECC (Forward Error
Corrections), Signal Alarm Counts,
Count of Flux Devices, etc.
251
 Customer Data These variables capture customer
information such as Lifestyle
segments of customers, product
profile of customers, billing amounts
of customers associated with the
Node, Video on Demand orders by the
customers, etc.
Call Interaction Data These variables capture the volume
and increase in Repair/Technical calls
made by customers for Cable
Boxes, Phone, and Modems, etc.
Field Technician Data These variables capture the count of
Field technician opened, completed,
etc. The higher the number of Field
technician orders opened the higher
the chances of an issue at the Node.
Below Table II is a sample of the variables created from the
consolidated datasets.
TABLE II. SAMPLE OF THE VARIABLES CREATED FROM THE CONSOLIDATED
DATASETS
Variable Description
Var 1 Installations done from the previous day.
Total No of IVR calls about set-top
Var 2
boxes/modem/phone
Var 3 (MAX) Signal Alarm Cable counts. The
number of set top boxes that show signal
alarms in Unified.
Var 4 USNR - (MAX) Average upstream signal
to noise ratio for the node as polled
from CMTS Port. This variable equal
1 if it is between 28 and 35. Else 0.
Var 5
Open Field Technician Orders (72Hours). The
number of Field operations orders that were
opened on the in the last 72 hours. This will
represent the pending trouble calls.
Var 6 (Environmental Temperature (ET). The node
health tool will update at 15-minute intervals.
Var 7 A number of connect CPEs (Customer
premises equipment)
3) Variable Creation and Variable Selection
As discussed above, we have outage flag to define Node
failure. One additional cleaning that we have to is to remove
consecutive outage days. In real scenarios, an outage may
remain consecutively for days. However, for modeling
purpose, we would try to predict only the first day of the
outage. Therefore, in the modeling data, we would only keep
the first day of every outage sequence and delete consecutive
outage days on each Node. In this step, new variables are
created which are to be used in the model. Since in the model
we will only be using 24 hours prior information, all the
variables are created using only 24 hours prior information to
the actual day of the event. Data is created at Node-Day level.
Below are different types of variables which are created;
1. Min/Max/Avg. value of different technical variables rolled
up at Node-Day Level.
2. Percent/Proportion variables created based on total
accounts attached to a Node. For example, percentage of set
boxes with Signal Alarm can be created using a ratio of Count
of set boxes with Signal Alarm and total accounts.
3. Based on exploratory analysis of each variable, we define
flag variables for those technical variables which shows step
relationship with outage. For example – USNR (upstream
single to noise ratio) – outage is high if this variable is either
below 28 or above 35. Therefore, for this variable we create a
flag variable which takes value 1 if USNR is between 28 and
35; takes value 0 otherwise.
4. Variables such as # of calls in last X days (where X =
2,3,5,7,15 etc.), # of calls/customer in last X days etc.
Data pre-processing is often a critical and essential part of the
prediction process. It is well known that data pre-processing or
data preparation stage can take a considerable amount of
processing or analysis time and effort [10] and as we have a
massive amount of data which will affect the mining process.
So, it was mandatory to execute some techniques of data
reduction. Once we have created different types of variables
that can be used in the model, next step is to select best variables
which will be used for modeling. One of the essential
techniques that were used for that purpose is feature selection
FS [11], [12]. ‘‘A feature can be considered irrelevant if it is
independent of the class labels’’ [13]. As per Hall in [14]:
“Feature selection FS is the process of identifying and removing
as much irrelevant and redundant information as possible.”. The
benefits of FS is detailed in the literature [15], [16]. Feature
selection can be classified into two main categories, filter
methods [17] and Wrapper methods [16]. In the experiment, the
Rapid miner operator “Weight by Chi Squared Statistic” was
used. This operator estimates the of the attributes’ relevance by
calculating the value of the chi-squared statistic for each
attribute of the input data set concerning the class attribute. The
higher the weight of an attribute, the more relevant it is
considered. Then, the features were ranked in descending order
according to their score and the ranking weights were
normalized between 0 and 1.
The chi-square statistic [18] is a nonparametric test used in
statistics to determine if the observed frequencies distribution
differs from the theoretically expected frequencies.
χ 2=   
Where χ 2 is the chi-square statistic, O is the observed
frequency, and E is the expected frequency.
The features were ranked in descending order according to their
score and the ranking weights were normalized between 0 and
1, based on that top 7 variables were chosen for modeling.
B. Modeling
Predictive modeling is a set of machine learning techniques
which search for patterns in data sets and use those patterns to
create predictions for new situations. Then start training the
models, changing some parameters in the experiments while
keeping the remaining classifiers’ parameters set to the default
values of Rapid Miner Studio 7. In this step, we will try three
different machine learning algorithms to train the model to
predict Node failure. Different learning algorithms make
different assumptions about the data and
252
 Fig. 3. Decision Tree diagram

Fig. 2. Stage of generating the prediction model using Rapid Miner and the 2- Train with Ensemble Model [20]. This approach uses a
Prediction Stage vote classification technique based on the majority
using the predictions of the inner learners. The rapid
have different rates of convergence. The one which works best, miner vote operator is an operator that has a sub-
i.e., minimizes some cost function of interest (failure rate in our process. The sub-process must have two learners at least
case) will be the one that makes assumptions that are consistent which are called base learners using KNN and Deep
with the data and has sufficiently converged to its error rate. learning ANN as base learners). This methodology has
Table III shows a quick comparison of three techniques. a benefit of not being based on “a priori” assumptions
and of allowing detection of links between factors the
TABLE III. A QUICK COMPARISON OF THREE USED MACHINE LEARNING conventional techniques such as logistic regression may
TECHNIQUES.
not be able to detect. This algorithm is capable of
modeling much more complex relationship between
Technique Advantages Disadvantages target and predictors
3- Train with Logistic Regression [21]. Logistic
Decision Trees -Simplicity of Results -No model scores. It is
-No implicit assumption of difficult to fix cut- off regression technique is used for prediction of a binary
linear relationship scores variable. We have defined Node Failure as a binary
- If tree becomes very event – Node has failed/not failed. Therefore, we can
large, it becomes apply logistic regression technique to predict whether
Logistic difficult toapresent
-Gives probability scores as an -Assumes log-linear the Node will fail or not in next 24 hours based on a set
Regression output relationship
-Specifically designed for - Co-linear variables of predictor variables that we have created. Logistic
predicting binary outcomes cannot be used regression will measure the relationship between the
outage and predictor variables by using probability
Ensemble -Ability to model extremely -Complexity of the scores as the predicted values of the outage. That is, this
Model complex non-linear results is difficult to model will provide a probability that a node will fail in
relationships which could go interpret from a business next 24 hours. These probability scores computed by
un-noticed in other technique point of view
logistic regression model can be used to prioritize the
node maintenance schedule. Fig. 2 represents the
second stage which is responsible for applying the
1- Train with Decision Tree [19]. A decision tree is a generated model to the validation. Along with the new
decision support tool that displays a tree-like graph of entries, the model generated is also provided as stage
possibilities and their possible consequences, including input, then applied through the “apply model” operator
chance event outcomes, resource costs. We have defined in Rapid Miner.
Node failure as a chance outcome, and we will use a
decision tree to identify variables which would help us C. Validation
identify nodes which are more likely to fail as compared To ensure that predictive model that we have built is as
to other nodes. Decision tree in Fig. 3 can be used to accurate as possible and it is actually fitting the “true”
identify certain combinations of the key variables which relationship between the variables and not just fitting the
lead to higher node failure propensity. Tree nodes in the characteristics of the data sample, it must be validated
above tree which are color-coded in red have the higher through out-of-sample testing. Therefore, to validate the
failure rate as compared to overall population, and hence, accuracy and robustness of the model we keep a hold-out
the nodes which qualify the criteria as per these red sample (validation sample) of 20% records, which are not
colored tree nodes should be prioritized for maintenance. used for model training. Once the model is trained on training
Below are the lift calculations if the maintenance is sample, its performance is checked on validation sample to
prioritized on decision tree ensure that model is performing as good on the validation
sample as it is in the training sample. Therefore, the whole
data is divided into two parts – Training (80%) and Validation

253
(20%). Model training is done on the 80% sample, and then
the model is validated on 20% sample. This data partitioning
is done using simple random sampling procedure. To validate
the model, we compare the model performance on training
and validation. If the model performance does not deteriorate
on the validation sample as compared to training sample, then
we can say that the model is stable. Performance of the model
can be measured in terms of Area under the ROC curve.
1- Lift Calculation for Decision Tree
% population of red colored tree nodes: 16.6%
% of total node failures covered: 31% Lift: 88 % [= (31%-
16.6%)/16.6%]
2- Lift Calculations for Ensemble models
The whole population can be divided into 10 bins (10 pctl
each) as shown in Fig. 4 based on the model scores and then
populations under the top few bins can be prioritized for
maintenance as these nodes will be more likely to fail as
compared to other nodes.
% of total population selected = 20%
% of total outages can be captured without using Model =
20%
% of total outages can be captured based on rank ordering by
Model = 38% Lift = (38%-20%)/20% = 90%
A lift of 90% means that the site visits' cost to determine and
service this 38 % of the failing nodes can be reduced to almost
half of what it would cost if done without the help of the
model.
Fig. 4. The Lift chart for Ensemble Model
3- Lift Calculations for Logistic Regression
The whole population can be divided into 20 bins (5 pctl
each) as shown in Fig. 5 based on the model scores and then
populations under the top few bins can be prioritized for
maintenance as these nodes will be more likely to fail as
compared to other nodes.
% of total population selected = 20%
% of total outages can be captured without using Model =
20%
% of total outages can be captured based on rank ordering by
Model = 38% Lift = (38%-20%)/20% = 90%
A lift of 90% means that the site visits' cost to determine and
service this 38 % of the failing nodes can be reduced to almost
half of what it would cost if done without the help of the
model.
Fig. 5. The Lift chart for Logistic Regression
Table IV shows, the comparison of the performance of the three
machine learning algorithms used to train the model to predict
Node failure
TABLE IV. THE COMPARISON OF THE PERFORMANCE OF THE THREE MACHINE
LEARNING ALGORITHMS USED TO TRAIN THE MODEL TO PREDICT NODE FAILURE
Algorithms’ Decision Tree Ensemble Model Logistic
Lift % Regression
88% 90% 90%
Selection of the best model depends primarily on factors such
as stability & robustness of the model, the simplicity of the
model output and lift (business benefit) provided by the model.
Based on these metrics logistic model is proving better than
other two techniques for the data sample that we had used to
develop these models. Logistic is showing higher lift as
compared to Decision Tree and also the flexibility of choice as
much population as it is possible from an operational standpoint
where Decision Tree breaks the population into fix segments
and makes it difficult to fix the size of the population that
operations team may want to target. When compared to
Ensemble Model. Both logistic and Ensemble Models are
giving same lift, however, given the complexity of the output
from Ensemble Models – Logistic model is preferred.
D. Controlling
In this phase, we start to define and implement controls to
sustain improvement and set up for continuous mining of the
data, and finally plan for monitoring and maintenance of the
model. As discussed in the Solution Overview section,
proposed process with the predictive model will help
maintenance team in prioritizing the technician’s visits to nodes
that are more likely to fail in near future than others. Hence,
reducing the costs of visits to the node as well as reducing the
time lag between failing and fixing the node leading to
improvement in customer experience. Below figures illustrate
the existing reactive maintenance process as shown in Fig. 6 VS
proposed predictive maintenance process as shown in Fig. 7
which is based on the predictive model described in this paper.
The model which we have developed based on the available
data sample can capture (35%-40%) of the failing nodes by
monitoring just 20% of the total nodes, giving a lift of 90%-
300%. This means that one will be able to visit (20%- 45%) of
the failing nodes in almost half the number visits which would
254
be required as per the traditional process, leading the reduction
in costs of visits by almost (50%-75%).
Fig. 6. Existing reactive maintenance process
Fig. 7. Proposed predictive maintenance process
V. CONCLUSION
In this paper, we proposed a methodology for building Node
Failure Prediction Model that will help to predict the node
failures in advance to take the precautionary measures before it
fails in the optical based network. The model was able to
capture (35%-40%) of the failing nodes by monitoring just 20%
of the total nodes, giving a lift of 90%-300%. This means that
one will be able to visit (20%- 45%) of the failing nodes in
almost half the number visits which would be required as per
the traditional process, leading the reduction in costs of visits
by almost (50%-75%), which will lead to an improvement in
customer experience and more operational efficiency by
avoiding data loss of optical network.
[11] F.Sahin, G.Chandrashekar,”A survey on feature selection methods”,
Journal of Computers and Electrical Engineering Vol. 40 ,pages 16–28,
2014.
[12] S. Garcõa, J. Luengo, F. Herrera, “Tutorial on practical tips of the most
influential data preprocessing algorithms in data mining”, Elsevier -
Knowledge-based Systems, 2015.
[13] Law MH, F. Mario AT, J. AK, “Simultaneous feature selection and
clustering using mixture models”. IEEE Trans Pattern Anal Mach Intel,
2004.
[14] M.A. Hall, “Correlation-based feature selection for machine learning”,
Ph.D. thesis, Waikato University, Department of Computer Science,
1999.
[15] Guyon, A. Elisseeff, “An introduction to variable and feature selection”,
Journal of Machine Learning Research Vol .3, pages 1157–1182,2003.
[16] Y. Dhote , S. Agrawal ,A. Jayant Deen , “A Survey on feature selection
techniques for internet traffic classification”, IEEE international
conference on Computational Intelligence and Communication Networks
(CICN) ,2015.
[17] L.C., Taminau J, Meganck S, Steenhoff D, Coletta A, Molter C, et al. “A
survey on filter techniques for feature selection in gene expression
microarray analysis,” IEEE/ACM Trans Computer Bioinform , 2012.
[18] J. Ignacio Serrano1, J.P. Romero, M. Castillo1, E. Rocon ,E D. Louis,J.
Benito-León, “A data mining approach using cortical thickness for
diagnosis and characterization of essential tremor“,
www.nature.com/scientificreports ,2017.
[19] M. Zhao, “The Decision Tree evaluation analysis for the relationship of
the marketing and enterprise strategy “, International Conference on
Robots & Intelligent System (ICRIS) ,2017.
[20] M. Fang, “A novel multiple classifiers integration algorithm with pruning
function,”, Fifth International Conference on Fuzzy Systems and
Knowledge Discovery, 2008.
[21] T.Masamha,E.Mnkandla,A.Jaison,“Logistic regression analysis of
information communication technology projects' critical success factors:
A focus on computer networking projects “, IEEE AFRICON ,2017.

ACKNOWLEDGMENT
The author is feeling grateful for the support provided by
the management of the studied telecom operator.

REFERENCES
[1] M. Gupta, “Improving software maintenance using process mining and
predictive analytics,” IEEE International Conference on Software
Maintenance and Evolution (ICSME),2017.
[2] S. Ramamurthy, L. Sahasrabuddhe, and B. Mukherjee, “Survivable WDM
mesh networks,” J. Lightwave Technol. 21(4), 2003.
[3] X. Shao, Y. Bai, X. Cheng, Y.-K. Yeo, L. Zhou, and L. H. Ngoh, “Best
effort SRLG failure protection for optical WDM networks,” J. Opt.
Commun. Netw. 3(9), 739–749 ,2011.
[4] A. Shahid, C. Machuca, L. Wosinska, J. Chen,“ Comparative analysis of
protection schemes for fixed mobile converged access networks based on
hybrid PON “ , Conference of Telecommunication, Media and Internet
Techno-Economics (CTTE)”,2015.
[5] https://rapidminer.com [Aug. 19, 2017].
[6] Lian Yan, Richard H. Wolniewicz, Robert Dodier ,“Predicting customer
behavior in telecommunications”, IEEE Computer Society 1094-7167/04,
,2004.
[7] S. Garc´ıa, J. Luengo, F. Herrera, Data preprocessing in data mining.
Springer Publishing Company, Incorporated, 2015.
[8] J. Han, M. Kamber, J. Pei, Data mining: concepts and techniques.
Morgan Kaufmann Publishers Inc., 3rd edition, 2012.
[9] M.J. Zaki, W. Meira, Data mining and analysis: Fundamental concepts
and algorithms .Cambridge University Press, New York, NY, USA, 2014.
[10] V.N. Vapnik, The nature of statistical learning theory. Springer-Verlag
New York, Inc., New York, NY, USA, 1995.

255