2018 IEEE International Conference on Big Data (Big Data)
Using Big Data Analytics to Create a Predictive
Model for Joint Strike Fighter
Ryan Norman, Jason Bolin,
and Dr. Edward T. Powell
Big Data and Knowledge Management
Test Resource Management Center (TRMC)
Alexandria, VA
ryan.t.norman.civ@mail.mil; jason.bolin@tena-sda.org;
epowell@tena-sda.org
The amount of information needed to acquire knowledge on
today’s acquisition systems is growing exponentially due to
more complex, higher resolution, software-intensive
acquisition systems that need to operate in System-of-Systems
(SoS), Family-of-Systems (FoS), Joint, and Coalition
environments. Unfortunately, the tools and methods necessary
to rapidly collect, aggregate, and analyze this information have
not evolved as a whole in conjunction with this increased
system complexity and, therefore, has made analysis and
evaluation increasingly deficient and ineffective. The Test
Resource Management Center’s (TRMC’s) vision is to build a
DoD test and evaluation (T&E) knowledge management (KM)
and analysis capability that leverages commercial big data
analysis and cloud computing technologies to improve
evaluation quality and reduce decision-making time. An
evaluation revolution, starting with the Joint Strike Fighter
(JSF) program, is underway to ensure the T&E community
can support the demands of next-generation weapon systems.
The true product of T&E is knowledge ascertained through the
collection of information about a system or item under test.
However, the T&E community’s ability to provide this
knowledge is hampered by more complex systems, more
complex environments, and the need to be more agile in
support of strategic initiatives, such as agile acquisition and the
3rd Offset Strategy. This increased complexity and need for
speed cause delayed analysis and problems that go undetected
during T&E. The primary reason for these shortfalls is
antiquated tools and processes that make data hard to locate,
aggregate, and convert into knowledge. In short, DoD has not
evolved its evaluation infrastructure as its weapon systems
have evolved.
Conversely, commercial entities, such as medical observation
and diagnosis, electric power distribution, retail, and industrial
manufacturing, have embraced agility in their methodologies
while modernizing analytics capabilities to keep up with the
massive influx of data. Raw physical sensors could provide
data, higher-quality image or video cameras, radio frequency
identification (RFID) devices, faster data collectors, more
detailed point-of-sale information or digitized records, and
ultimately is providing more data to analysts in size and
complexity than ever before. As more data has become
available, an interrelated phenomenon is the desire of analysts
to ask more detailed questions about their consumers and their
978-1-5386-5035-6/18/$31.00 ©2018 IEEE
Sanket Amin and John Nacker
Joint Strike Fighter Program Office (JPO)
Naval Air Systems Command (NAVAIR)
Arlington, VA
sanket.amin@jsf.mil; john.nacker@jsf.mil
business infrastructure. To drive the process of implementing
big data analytics, businesses have begun establishing analytics
centers which either take pre-defined business cases and apply
methods to address them or implement existing knowledge
within the data architecture to create a higher level of
awareness to business groups or the company at-large. To
meet these demands, data storage and computation
architectures have become more sophisticated, dozens of
technologies were developed for large-scale processing (such as
Apache Hadoop or GreenPlum), and streaming architectures
which allow data to be processed and actioned on in real-time
as it is collected have become commonplace. The net result of
these commercial best practices is a solid foundation for the
DoD to transform how it uses data to achieve faster, better,
and smarter decisions throughout the acquisition lifecycle.
Big Data, Data Analytics, Knowledge Management, Data
Management, Virtualization, Cloud Computing, Predictive
Maintainance, Department of Defense, Test and Evaluation
I. KNOWLEDGE MANAGEMENT
Embracing big data analytics in the DoD starts with
improved knowledge management. Today, acquisition
system data is generated and stored in locations across the
globe over many years of prototyping, development, testing,
upgrading, and fielding. The result is T&E data that is
currently isolated and compartmentalized with little
discovery or reuse outside of each respective acquisition
program. Consequently, many programs across the DoD are
continually reinventing the wheel or rerunning tests that
another program has already performed. This redundancy is
often because program managers and analysts simply do not
know that a similar test was performed within another
program. They do not know what is already known, or
where to go to access the knowledge. A more systematic
reuse of knowledge across the DoD will result in a
substantial cost benefit as T&E data collected by one
program is valuable for reuse by another.
There are four major functional areas of KM: 1)
gathering data into the KM system; 2) warehousing the data
for long periods of time while making it available to users in
a timely fashion when they need it; 3) providing analysis
tools and capabilities, such as big data analytics (BDA), that
3590
can perform computations on large amounts of data in
parallel, yielding products that would normally not be
derivable on such large data sets; and 4) visualization and
reporting, whereby both the vast amounts of gathered data as
well as the analysis products can be both visualized
intelligibly and reported automatically to the appropriate
analyst. Any KM system must also meet system-wide
requirements such as manageability, security, reliability,
redundancy, scalability, usability, etc., while providing easy-
to-use setup and configuration tools such that the KM system
itself can be managed efficiently and without excessive cost.
Type Style and Fonts
II. BIG DATA ANALYTICS
As both the requirements and the systems that meet them
increase in complexity, the amount of data collected in test
events has mushroomed beyond the capacity of normal
analytic tools to reduce and analyze more than just a small
fraction of the data collected. This state of affairs means that
there is a tremendous amount of information relevant to the
performance and capabilities of the systems under test that is
being ignored, making testing costlier and less responsive to
warfighter needs. Big data analytic techniques are designed
to get at this hidden information and provide it to the
analysts and program managers.
Evaluating all of the data, evaluating one program’s data
against another program’s data, and looking for unknown
unknowns are the primary motivations for bringing the T&E
community into the BDA world. Today, much of the time
spent conducting analysis consists of answering specific
questions about small portions of the collected data, rather
than taking a broad look at all the collected data to discover
new insights. Specific questions, after all, can be answered
in a reasonable amount of time by querying the data for very
small subsets, then performing a specific analysis on just that
subset. So, while the T&E community is successful today at
answering these important specific questions, it is often
simply technically impossible to even ask larger, more
expansive questions. Most analysts are properly focused on
answering the questions raised in the Test and Evaluation
Master Plan (TEMP); however, it is important to ask
additional questions so that problems or issues not
anticipated at the time, known as “unknown unknowns,” can
be detected and acted upon in a timely and cost-effective
manner.
Commercial big data capabilities serve as a good starting
point to modernize T&E capabilities. However, most
existing commercial BDA techniques are suited towards
unstructured or semi-structured data, such as Google
searches or Twitter feeds, which are not relevant to the
mostly highly-structured time series data collected in T&E.
To perform their mission more effectively, analysts require
tools and algorithms that apply big data techniques to the
following areas: anomaly detection, causality detection,
trend analysis / failure prediction, regression analysis /
dataset comparison, and pattern recognition / fault code
analysis. Each of these areas can be analyzed separately or
together to give a uniquely comprehensive view of the test
article across a large number of tests and versions.
3591
III. TRMC EFFORTS
The DoD’s Test Resource Management Center is
charged with overseeing the nation’s T&E ranges and
facilities and making investments in technology and
capabilities that will benefit the entire T&E community. The
TRMC envisions a DoD T&E environment where
knowledge is readily available and discoverable across an
acquisition program’s entire lifecycle, thus enabling data
analytics to occur. Ideally, analysts, program managers,
logisticians, and other decision-makers would have access to
all data associated with an acquisition system, available at
their fingertips from a logically-centralized, permissions-
based, enterprise KM system. The KM system would
include data from the initial conceptual research and
development (R&D) work, through all phases of T&E, to
data gathered while fielded in-theater and operations and
maintenance (O&M) records for all acquisition systems.
To ensure the department’s T&E analytic needs are met,
TRMC has created a clear vision for transforming the current
T&E data infrastructure. The TRMC’s vision is to build a
DoD T&E knowledge management and analysis capability
that leverages commercial big data analysis and cloud
computing technologies to improve evaluation quality and
reduce decision-making time. To accomplish this, analysts
and evaluators need an improved T&E infrastructure flexible
enough to meet current and future T&E requirements. This
infrastructure must rely on a scalable, adaptable universal
architecture framework to ensure that all future data streams
can be managed and analyzed accurately and efficiently
within a secure and protected multi-level security
environment. In addition, in times of limited or uncertain
DoD budgets, transforming the existing T&E infrastructure
must be cost effective with clear benefits on associated
investments. To realize this vision, the TRMC identified the
need for a Big Data Analytics initiative.
The BDA Initiative consists of a number of sub-projects,
studies, and other efforts designed to improve the T&E
community’s knowledge management and data analysis
capabilities. Lessons learned and capabilities will be
captured in the culmination of a DoD-wide architecture
framework and T&E capability for improved knowledge
management and BDA. The project addresses a variety of
capabilities including (1) high-speed data extraction from the
test articles; (2) use of secure networks and cloud storage to
enhance data sharing capabilities; and (3) use of state-of-the-
art KM, data analytics, and big data tools to gain greater
insights from the data collected.
As part of the Big Data Initiative, the TRMC established
two proofs of concept spanning the acquisition lifecycle to
inform the big data architecture framework. The first proof
of concept, Collected Operational Data Analytics for
Continuous T&E (CODAC-TE), leverages historical data
collected in-theater to support continuous system
improvement while providing valuable knowledge to a next-
generation system's engineering research and development
(ERD) phase. CODAC-TE is exploring two distinct uses of
big data analytics. First, it is fusing Mine-Resistant
Ambush-Protected (MRAP) platform data collected during
both in-theater and continental U.S. (CONUS) testing. The
use of BDA tools to simultaneously analyze these large
databases is expected to answer customer questions about
availability and sustainability. Second, the project is
assessing whether large volumes of network data analyzed
with BDA tools can supplement or replace current methods
of evaluating tactical communications effectiveness for
programs such as Warfighter Information Network-Tactical
(WIN-T).
The second proof of concept, the Joint Strike Fighter
Knowledge Management (JSF-KM) project is partnering
with a current acquisition program conducting complex
developmental tests (DTs) and operational tests (OTs) at
multiple locations. The TRMC is utilizing the JSF-KM
project to ascertain how recent advancements in data mining,
big data analysis, and cloud computing technologies could
apply to T&E datasets and data centers. This effort is
investigating tools, techniques, resources, policies, and
procedures required to implement private cloud technologies
and big data analytics to use data more efficiently and
effectively in support of the JSF acquisition program. This
private cloud utilizes the latest in virtualization technologies,
big data analytics, continuous trend analysis, automated
regression testing, and tiered-storage with automated
movement between fast / commodity storage based on usage.
Through the usage of technologies such as Platform as a
Service (PaaS), Software as a Service (SaaS), and virtual
desktop infrastructure (VDI), computations and analytics
will be executed within a safe and controlled environment.
This will allow a properly permissioned user to remotely
perform analysis, utilizing virtualized software that is co-
located with the data, removing the latency historically
associated with remote analysis. Techniques, technologies,
and lessons learned from this effort are discussed in-depth
below and will form the foundation for an enterprise T&E
Knowledge Management capability.
IV. JSF BIG DATA ANALYTICS EARLY DETAILS
The early analysis performed in the current KM system
focuses on aircraft data for test flights in which a certain
non-critical component failure has put stress on inventory.
To demonstrate the value of big data techniques, exploratory
techniques were performed to characterize the flights, and a
targeted method was implemented for predictive
maintenance of the aircraft. The exploratory techniques are
intended to support ad-hoc study for analysts, giving a quick
glance at what each aircraft did during each test, as well as
pointing out potential anomalous flights which can be
investigated further in the raw data. The predictive
maintenance framework is intended to support field teams to
give them an indication of when to perform physical
inspections, thus saving time, as well as helping to diagnose
the problem.
The data set used for this effort consists of approximately
42 billion data points collected during 1,400 flights
consisting of three parts. First, over 150 columns of numeric
sub-second measured parameters are collected. Second, over
40 metrics are measured at the end of each flight. Third, a
varying set of around 900 events are triggered by the
3592
onboard software and captured, along with associated shorter
time-series data. To limit bias and perform a real-world test
of the value of big data techniques, the data analyst team was
not informed which flights within the dataset experienced
component failures. The questions posed and techniques
used by the data analyst team to find the failures and
discover other anomalies are described in the sections that
follow.
V. OUTLIERS
Datasets must always have an understanding of
observations which have very different characteristics than
the rest of the dataset. In this early study, only statistics of
the time series data per flight were studied. The maximum,
minimum, average, and standard deviation were computed
for each time series, as well as the event number per flight.
The distributions of these statistics are now characteristic of
the sample. These distributions were found to be quite
varied: some were bell-shaped, some skewed with long tails,
and some bimodal distributions, often with a sharp peak at
zero. Even when using only two points, discrepancies were
found and eliminated from further analytical transformations.
In further work, outliers will be studied for combinations of
variables, using the sparsity metric to compare the amount of
data expected in a cell of the subspace of that variable set.
VI. FLIGHT CLASSIFICATION
The simplest question asked by an analyst when provided
data from flights is what maneuvers the aircraft performed.
Some flights don’t leave the ground, while others may land
briefly and take off again. Others may perform maneuvers
that other flights do not. A simple classification of the data
can provide general answers to these questions in a quick
fashion by focusing on the parameters that indicate
movement.
Here, different parameter sets were studied as input into a
hierarchical clustering algorithm which supports an
interpretation of the aircraft state regarding movement in a k-
dimensional space of those parameters. This algorithm
allows analysts to tune the number of flight segment types by
the dissimilarity between data points. A further requirement
on top of the algorithm is that the time points clustered
together must be in a contiguous time range, and last longer
than 1 second. The obtained classifications are regions in
which the aircraft moves, passing through temporary areas.
Figure 1 plots the parameter set as a function of time for
one flight, normalized by the mean and standard deviation of
each parameter separately. The shaded regions highlight
each flight segment. The largest are the yellow ‘in-flight’
and red ‘on-ground’ segments. There are smaller segments
for take-off maneuvers. Two of the smaller segments are
expanded in Figure 2.
There is some variability in the approach: either changing
the dissimilarity cutoff for assigning clusters or including
different parameters or parameter transforms from the
current flight. Afterward, one can easily profile the rest of
the flights in the database and gain a condensed view of what
each aircraft was doing. Furthermore, one can characterize
the parameter behavior within each cluster, for example,
 Figure 1. Normalized plot for flight classification metrics
Figure 2. Closer look at two smaller clusters in Figure 1
through correlations. One can also investigate the non-
clustered time portions, as they could be viewed as outliers.
VII. ASSOCIATIONS
Another common question is whether well-known
relations can be derived from parameters in the data. Once
these are known, could unknown relationships be
uncovered? A flexible approach can be created using
association rules by defining a sufficient set of events of
interest. Here, event data is studied to understand if the
software-defined events have investigative value.
The a priori algorithm for determining the rules
computes estimated probabilities for each event occurring in
the dataset, and then binary combinations, triple
combinations, etc., until a given threshold on the
probabilities prevents more rules. Confidence is calculated
for each rule, that is, the conditional probability of an event
occurring. In Figure 3, rules with high confidence (in this
3593
case, over 99%) are shown graphically, each node
representing an event, and a line connecting them
representing a rule. There is a cluster of closely related rules
around events 484, 485, 486, 495, as there are common rules
for each of these with each other, as well as with
combinations of themselves. Events 430 and 431 are
commonly found together, as well as events 480 and 481.
Going further, flights are profiled in terms of how many
times each rule misclassified an event, meaning a case where
one of the pairs of events occurred, but the other event of the
pair did not occur. For high-confidence rules, this indicates
possible anomalies that could be investigated in more detail.
In Figure 4, a heat map reveals the misclassifications.
Note that for some flights, neither of the events in the rule
occurred, as indicated by dark blue. As indicated in dark red,
one flight with many misclassifications for rules 11 and 13
certainly warrants inspection. Further, rules 11 and 13 have
other flights with many misclassifications. Note the flight
with the second-highest set of misclassifications suffered a
component failure. Ideally, the algorithm can be tuned to
reveal the failure more explicitly. Another point of note is
that there are two flights with misclassifications across
several of the rules, 84 and 90. Future work would be to
tune the algorithm to reveal other rules or include more
events.
VIII. PREDICTIVE MAINTENANCE
For these aircraft, a maintenance problem has arisen for a
specific component whose underlying root cause was not
revealed to the data analyst team. The question at hand is
whether it is possible to know ahead of time which aircraft
may experience the next failure, enabling field teams to
know when to make the physical inspection. Addressing the
concern, a modeling strategy using logistic regression which
can be tuned by a data scientist and inspected by engineering
teams is implemented. This modeling strategy has the
benefit of providing predicted probabilities of failure, either
for the flight just performed or for future flights, giving
engineers a way of triggering a manual inspection only when
needed. This implementation has the potential of
considerably reducing the amount of time spent for
inspection and reducing the number of failures in-flight.
Additionally, it can provide a trend for specific parameters
with the probability, giving possible indicators to the
engineering team for whether increasing or decreasing values
are associated with the failure. This purely data-driven
addition to the engineering team’s knowledge could lead to
the underlying cause.
As flights are regularly conducted, there are two choices
for prediction: a model can predict whether the flight just
performed suffered the failure, or the model can predict
whether the next flight or next few flights will suffer the

Figure 3. Association rule graph for the event data. Nodes correspond to events, and a link indicates a rule. Some nodes are joint events, indicated by
the pipe delimiter.

Figure 4. Heat map of misclassifications for each of the rules. The x-axis is the index for each flight. The y-axis contains the 54 rules found in Figure
3. Dark blue indicates the rule never occurred. Off-color white indicates few misclassifications, while dark red indicates a large number of
misclassifications.

3594
failure. This prediction gives two different views with which TABLE I. CAPPED MEAN OF PREDICTED PROBABILITIES FOR THE
MODEL
the engineers can use to diagnose the failure. A time-to-
failure model was not chosen initially due to the fact that the Aircraft Capped Mean
cause is expected to be sudden and not a symptom of long- Probability
term use. This can, of course, be included in the future as a 02 0.6%
03 1.0%
parallel model.
05 2.9%
The framework implemented included a flexible data 04 4.2%
aggregation for each flight with a variable selection process 10 4.5%
that keeps statistical considerations and the benefits above in 11 5.4%
mind. Thousands of models were generated, and we note 09 8.7%
that those models that did perform the best-contained 08 9.8%
variables which do make sense from an engineering 07 11.4%
perspective. This means the strategy is pointing toward 06 22.0%
01 23.3%
relevant data parameters.
To examine the predictive power of the probability
estimates, the distributions for each flight over aircraft are The final step is to score these models over the unknown
illustrated in Figure 5. They do appear to be distinctive for flights in the hold-out set to determine if the unknown
some aircraft and not others. The aircraft that did not fail do failures can be determined. This determination was left as
indeed have a sharp distribution of small probabilities, future work for the project.
though there are high outliers for aircraft 4 and 5. This IX. SUMMARY OF FINDINGS
suggests that removing those outliers could provide a better
health indicator. Aircraft 1, 6, 7, 8 and 9 do have wide So, what does all of this mean? Data Analysts with no
distributions. Aircraft 10 and 11 peak as sharply as the knowledge of any failures or any expertise with aircraft were
healthy ones; only there is more data for those two. A able to identify problems using commercially-matured big
capped average of the distributions provides complete data techniques. They were also able to build a predictive
differentiation, as in Table 1. model to determine when future problems may occur. The
model has been well-trained on 1,400 flights to predict the
presence of component failures. Out of billions of data

Figure 5. Distributions of the predicted probabilities over the eleven aircraft used in the train / test methodology. Aircraft 2, 3, 4 and 5 were the
only aircraft that did not fail.

3595
points and hundreds of predictors, the model has picked out a
handful of data points, which also happen to make sense
from an engineering perspective. The model as-is identified
roughly 70% of the failures within the dataset. Future work
on the model will be to validate it against an additional
dataset of 3,600 flights to see if the model correctly identifies
additional “unknown” failed components. Once validated,
the model can be trusted in production to identify component
failures and hopefully prevent component failure in-flight.
X. CONCLUSION
The Joint Strike Fighter program has embraced the need
for BDA to ensure the best system is delivered to the
warfighter. By embracing an enterprise approach to KM that
leverages commercially-proven big data analytic
technologies, the DoD can efficiently and continuously
improve its knowledge base throughout the lifecycle of
today’s complex systems. Using more collective and
systematic processes will enable programs to learn from each
other's mistakes and take advantage of each other's
achievements. Federating distributed data repositories will
enable execution and automated search scenarios that cannot
occur today, enabling analysts and program managers to find
and reuse data collected within other programs. Historical
data will be readily available for next-gen systems, ensuring
lessons are learned, and previous deficiencies are anticipated
and mitigated. Problems will be identified earlier in the
acquisition lifecycle when they are cheaper to fix and a lower
risk to both program schedule and our warfighters.
Anomalies will be easier to identify, diagnose, and resolve
through a lifetime of readily available data - even if the
anomaly or problem was not noted at the time of the original
test. Big data analytics can identify anomalies, trends, and
patterns that a human analyst might miss, enabling the
discovery of “unknown unknowns” which are the greatest
expense and risk to the development of a complex
acquisition system.
Moving the T&E community into the Big Data Analytics
arena will take time, effort, commitment, and resources.
However, as tests become more productive and as more
relevant information can be gleaned from the limited tests
performed, the quality of our acquisition decisions—and thus
the support to the warfighter those systems provide—will
increase dramatically. Big data analytics has the potential to
reinvent the “Evaluation” part of T&E, bringing more
insight, better decisions, and better warfighting capability to
the nation. The BDA Initiative is the start of this process and
when complete will provide faster, more efficient methods
for ingesting and analyzing large quantities of test data.
Going forward, weapon systems will only grow more
complex; we cannot afford to wait any longer to change how
the T&E community does business. The revolution begins
now.
BIOGRAPHIES
RYAN NORMAN serves as both Program Manager for
the Test and Training Enabling Architecture (TENA)
Software Development Activity (SDA) and as the Test
Resource Management Center’s (TRMC) Initiative Lead for
3596
Big Data and Knowledge Management. Since joining the
TRMC in 2008, Mr. Norman has held several positions
including Project Manager for the Cyber Operations
Research and Network Analysis (CORONA) project;
TRMC’s Senior Engineer for Cyberspace Test Capabilities;
the Joint Mission Environment Test Capability (JMETC)
Deputy Program Manager for Systems Engineering; and
TRMC Lead for Army Range Oversight. Mr. Norman holds
a bachelor of science degree in computer science from the
Georgia Institute of Technology and is a DoD-certified
acquisition professional in the T&E, Information
Technology, and Program Management career fields.
JASON BOLIN is a Senior Systems Engineer with
KBRWyle supporting the TRMC Initiative for Big Data and
Knowledge Management. Before joining KBRWyle in
2013, Mr. Bolin held several positions in the DoD including
Chief of Emerging Technologies at Defense Logistics
Agency, Chief Technology Officer at Redstone Test Center,
and Enterprise Software Development Lead at Redstone Test
Center. Mr. Bolin holds a Master’s of Science degree in
Software Engineering from the Naval Postgraduate School, a
bachelor of science degree in computer science from the
Tennessee Technological University and was a DoD-
certified acquisition professional in the T&E, Information
Technology, and Program Management career fields.
Dr. EDWARD T. POWELL is a lead architect for the
Test and Training Enabling Architecture. After receiving his
Ph.D. in Astrophysics from Princeton University, he worked
for the Lawrence Livermore National Laboratory performing
simulation-based analysis. He moved to SAIC (now Leidos)
in 1994 and participated as lead architect in some of the most
complex distributed simulation programs in DoD, including
the Joint Precision Strike Demonstration (JPSD), the
Synthetic Theater of War (STOW), and the Joint Simulation
System (JSIMS). He then worked in the intelligence
community on architectures for integrating large-scale,
diverse ISR systems. He has been the lead architect for
TENA for fifteen years and is currently working on
expanding the applicability of TENA and integrating TENA
with broader DoD-wide Data Management and Big Data
analysis systems.
SANKET AMIN and JOHN NACKER are the subject
matter experts for NAVAIR and Joint Strike Fighter
Program Office (JPO) in the area of propulsion system
diagnostics. Combined, Sanket and John have 20+ years in
turbomachinery diagnostics and prognostics with primary
expertise in vibration-based diagnostics. To further advance
DoD state-of-the-art in this area, Sanket and John are
exploring the application of Big Data and Data Science
techniques to improve earlier detection of turbomachinery
faults to impact JSF readiness, safety, and ownership costs
positively. Sanket has earned BS and MS degrees in
Mechanical Engineering from Penn State University. John
Nacker has earned BS and MS degrees in Engineering
Mechanics from the University of Wisconsin and the
University of Texas, respectively.