See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/303640832

New Measures of Effectiveness for the Air and Missile Defense Simulation
Community

Article · December 2015

CITATIONS READS

5 626

2 authors, including:

Brian Wade
US Army
9 PUBLICATIONS 24 CITATIONS

SEE PROFILE

All content following this page was uploaded by Brian Wade on 29 May 2016.

The user has requested enhancement of the downloaded file.


New Measures of Effectiveness
for the
Air and Missile Defense
Simulation Community
MAJ Brian Wade, Center for Army Analysis, brian.m.wade.mil@mail.mil; and Mr. Paul Chang, Center for
M
Army Analysis, paul.m.chang.civ@mail.mil
odern Air and Missile
Defense (AMD) systems
use radar paired with a
gun or missile weapon
system to detect and
destroy an array of threat
types that includes theater ballistic
missiles (TBMs), cruise missiles
(CMs), and unmanned aerial systems
(UASs). The typical Measures of
Effectiveness (MOE) used to evaluate
these AMD systems and their method
of employment, or Concept of the
Operations (CONOPS), is a single
quantity that represents the total
number of incoming threats that were
not shot down. These threats that were
not shot down are commonly referred
to as “leakers.” This single number
does not provide enough fidelity for
current and future analyses where the
incoming threat raid will be large and
made up of a mix of the three threat
types. To evaluate AMD CONOPS
against these large and complex raids
properly, analysts need new MOEs that
better describe the overall outcome
of the raid. These new MOEs should
be the total weight of ordnance that
impacts critical areas of the defended
asset and the number and percentage
of leakers by system type.
Current AMD Analysis
The analysis of AMD systems has
traditionally centered on a single MOE:
PHALANX – D E C E M B E R 2 0 1 5
PROFESSIONAL FEATURE
the total number of incoming threats
not shot down or leakers. This MOE is
a summation of all the different threat
types that were not destroyed by all
the different AMD systems defending
an asset. This MOE can manifest itself
in one of several ways: the number of
threats destroyed, the number of threats
not destroyed, or the percentage of
threats destroyed or not destroyed.
The percentage of threats destroyed
is called the achieved protection level.
This MOE is often used because it
can be compared to the commander’s
desired protection level or the
percentage of threats the commander
has specified which must be destroyed
to meet his or her operational
49
PROFESSIONAL FEATURE
objectives. However, regardless of its
form, the MOE is essentially expressing
the same information: the total number
of threats of all different types destroyed
and not destroyed by the combination
of all the different AMD systems
defending an asset.
Traditionally, this aggregate measure
has provided enough information for
analysts and commanders to make
decisions about AMD systems and
tactics primarily because the threat
raids were small and were composed
of a single threat type. An analysis of
the first and second Gulf Wars shows
that the missile raids against US and
coalition forces were composed of one
to four missiles and were homogenous,
made up of only TBMs or CMs in a
single raid (Gormley, 2008). With such
small and uniform raids, the number of
leakers MOE in any form was sufficient.
Future conflicts may not prove so easy.
New MOEs to Quantify Effects
in AMD Analysis
The proliferation of missile technology
has changed since the early days of
the Patriot Missile System and the
wars in the Persian Gulf. Today’s
enemies have vast stockpiles of TBMs,
CMs, and UASs (US Air Force, 2013;
Feicket, 2005; Cali, 2013). Future raids
will most likely be large and composed
of a combination of these threat types
(Freedberg, 2015). This is problematic
because these different threats
approach their targets at very different
speeds, trajectories, and altitudes
and each has a vastly different radar
cross section (RCS, http://www.
globalsecurity.org/military/world/
stealth-aircraft-rcs.htm). A typical TBM
will approach at an extreme angle
(60–80 degrees from the horizon) at
speeds approaching mach 6.0 (Lin,
1990; Zhang, 2013). CMs will approach
flying anywhere from “nap of the earth”
to several thousand feet in the air at
speeds typically ranging from mach 0.6
50 PHALANX – D E C E M B E R 2 0 1 5
to mach 2.0 (Werrell, 1985). UASs will
approach at altitudes ranging from tens
to hundreds of feet at speeds from 40
to 120 knots (US Army, 2010). TBMs
and CMs will have average RCS values
on the order of one to several tens of
square feet. UASs can have average
RCS values on the order of square
inches to several square feet.
The diverse characteristics of each
threat type make it difficult for AMD
systems to effectively detect and
engage all targets simultaneously. If an
AMD system is set to TBM-only mode
and has its radar turned up at a high
angle, it will see TBMs but it may miss
CMs or UASs. A radar with its filters
set to avoid the clutter from birds,
which are slow and have small RCS
values, may miss a UAS or CM (Kopp,
2014). If filters are widened, the targets
could be lost in the resulting clutter.
In this type of battle, aggregating the
leaker information together simplifies
the analysis, but conceals critical
information from commanders and
analysts. The threats destroyed or
leaked, by type, should be counted
separately, because they reveal critical
vulnerabilities to specific threat types.
Not only is it difficult to detect
and engage all three threat types
simultaneously, the threats posed by
each system are not equal. Typically,
TBMs will carry a more explosive
payload than CMs, and CMs will carry
a more explosive payload than UASs
(Marshall Institute, 2012; Gormley,
2002). This means that TBMs are the
most dangerous and UASs are the
least dangerous threats in terms of
pure explosive power and potential
damage. However, the payloads must
also be accurately delivered. Typically,
CMs and UASs have more accurate
guidance systems and are more
maneuverable than TBMs (Cali, 2013),
making them more dangerous than
TBMs in terms of accuracy. Recent
advances in guidance technology has
narrowed this accuracy gap; however,
on a whole TBMs are generally less
accurate than CMs or UAVs.
The diversity of the threat
characteristics also affects the time
the AMD system has to shoot down
each threat, called the engagement
window. A longer engagement window
means the AMD system has more
opportunities to engage the threats.
The engagement window depends on
the threat’s speed, range, and RCS
(Richards, 2010). Typically, TBMs
and CMs have similar average RCS
values and a longer range than UASs.
However, TBMs travel much faster than
CMs, so their engagement windows are
relatively short. Slower moving CMs,
which typically travel longer distances,
have longer engagement windows.
UASs have relatively small RCS values
but travel relatively slowly. This means
that their engagement windows could
be equal to, longer than, or shorter
than CMs, depending on the balance
of RCS and speed. However, more
advanced radars can detect objects
with very small RCS values at longer
distances, therefore engagement
windows of UASs are typically longer
than CMs. These relationships are
summarized in Table 1.
Table 1 shows that TBMs have the
shortest engagement windows and
carry the most payload, but have
relatively low accuracy. CMs have
the longest engagement windows,
but are most accurate. UASs fall
somewhere the in the middle of these
two threat types; they carry small
payloads, but have medium accuracy
and engagement windows. If all three
threats were launched simultaneously
from their maximum range, an AMD
system could begin engaging the
CMs before the TBMs; however, once
the TBMs entered their engagement
window, they would only be in that

Table 1. Threat characteristics and level by threat type.
Characteristic
Explosive payload
Accuracy
Engagement Range
window Speed
Radar cross section (RCS)
Resulting engagement window
window for a short period of time.
Therefore, an AMD commander may
want to stop engaging the CMs and
focus on the TBMs. He or she could
then reengage the CMs once the
TBM engagement was complete.
Another commander facing a group of
inaccurate TBMs along with a group
of CMs may argue that the CMs pose
a greater threat due to their accuracy
and may want to continue to engage
the CMs for as long as possible. The
job of an AMD systems analyst is
to evaluate these CONOPS in order
to find the one that minimizes the
damage to the defended assets.
For most AMD analyses, the damage
can be quantified by the amount of
explosive that impacts critical areas
of the defended asset. The total
mass of explosives that detonates in
critical areas of the defended asset
is a better MOE than the aggregate
number of leakers because it quantifies
the effects of an enemy attack and
measures all threats with a common
metric. Good MOEs should focus
on the output of a system (Williams,
2013). In this case, the enemy wants
to damage or destroy critical assets or
facilities and the amount of damage is
based on the mass of ordnance that
detonates on or near those targets. By
quantifying the amount of ordinance
that impacts the target, and not
simply the number of leakers, analysts
are able to articulate the outcomes
in terms of operational effects on
friendly forces. In more sophisticated
analyses, analysts can also include
TBM
Large
Low
Far
Fast
Medium
Short
the dud rate to help quantify the
threats that may leak through the
defenses and hit a critical area, but fail
to detonate. In either case, the total
mass of explosives hitting the critical
areas of the target communicates to
commanders the expected effect on
friendly forces he or she can expect.
This is the best MOE to compare
CONOPS based on effects, but
additional MOEs are needed to show
vulnerabilities to certain threat types.
To show specific vulnerabilities,
analysts should include additional
MOEs that report, by threat type,
the number of threats that impact
the defended asset (number of
leakers by type) and the percentage
of threats that impact the defended
asset (percentage of leakers by
type) compared to the total amount
that were launched. Examining
both the number and percentage of
threats that leaked through the AMD
defenses illuminates weaknesses
in CONOPS to certain quantities of
threat systems and explains why one
resulted in more explosives impacting
the defended asset compared to
another. The percentage of leakers
highlights vulnerabilities while the
actual number of leakers shows the
magnitude of that vulnerability. If a
CONOPS allows a large percentage of
a certain threat type to leak through
the AMD defenses, that CONOPS has
a weakness against that threat type.
However, if the raid consisted of only
a few systems of that type, the actual
number of leakers would be very
PHALANX – D E C E M B E R 2 0 1 5
PROFESSIONAL FEATURE
CM UAS
Medium Small
High Medium
Medium Close
Medium Slow
Large Small
Long Medium
small and might be acceptable
to the commander.
Comparing CONOPS based on the
overall operational effect, the mass of
ordinance MOE, and their strengths
and weaknesses to certain system
types and quantities, the percentage
and number of leakers by type MOE,
allows analysts and commanders to
make decisions based on operational
requirements and current intelligence
reports. Together, these new MOEs
are shown in Table 2.
New MOEs Example
This section gives an example of
how to use these new MOEs for an
attack on an airfield. For each of the
recommended CONOPS, the analysts
should express the probability of a
given amount of ordnance impacting
the target; e.g., with a given CONOPS
there is a 95 percent chance that less
than 1,000 kgs of ordnance will impact
the runway. The ordnance that impacts
the target can be determined by first
finding the probability distribution of
the number of threats, by type, that
are expected to “leak” through the
defenses and land within their lethal
radius in the critical areas of the
defended asset. Typically, analysts
can find the probability distribution by
running a Monte Carlo AMD simulation,
such as the Extended Air Defense
Simulation (EADSIM). This distribution
can then be converted into a
cumulative distribution function (CDF).
The CDF will show the maximum
number of expected leakers, for each
51
PROFESSIONAL FEATURE
Table 2. New MOEs to evaluate CONOPS in AMD simulations.
New MOEs
Threat leakers by % TBM leakers
type
% CM leakers
% UAS leakers
TBM leakers
CM leakers
UAS leakers
Total amount of explosive that impacts the
defended asset
threat type, for a given confidence
level; e.g., there is a 95 percent chance
that 1.4 or fewer CM leakers and 2.7
or fewer TBM leakers will damage the
critical areas in a given CONOPS.
To calculate the expected damage,
the number of leakers for each type is
multiplied by the expected payload for
each system. If the dud rate is known,
this can be incorporated into this
calculation. The result is a confidence
level for a given amount of explosives
that will impact a critical area of the
target. Analysts can then quantify the
expected damage from this amount of
explosives. This additional information
allows commanders to make informed
decisions on the benefits and risks of
different CONOPS as it relates to the
expected damage resulting from a
raid and not simply the total number
of leakers.
As an example, an analyst conducts the
steps outlined above for two CONOPs
(CONOP A and B) against a raid of three
CMs carrying 250 kgs of explosives
each and 10 TBMs carrying 500 kgs of
explosives each. In the first case, at a
Table 3. AMD MOE example.
CONOP Total Mass of Explosives
CONOP A 875 kg
CONOP B 1,375 kg
52 PHALANX – D E C E M B E R 2 0 1 5
Description
Percentage of TBMs that impact critical areas compared to total launched that
raid.
Percentage of CMs that impact critical areas compared to total launched that
raid.
Percentage of UASs that impact critical areas compared to total launched that
raid.
Number of TBMs that impact critical areas in a raid.
Number of CMs that impact critical areas in a raid.
Number of UASs that impact critical areas in a raid.
Total mass of explosives, carried by all leakers, which impact critical areas in a
raid.
95 percent confidence level, CONOPS leakers into a common metric, the
A resulted in an average of 0.5 TBM mass of explosives, allows analysts
leakers and 2.5 CMs leaking through to correctly aggregate their effects
the defenses and hitting the critical and describe the overall outcome on
area. The average combined mass of friendly forces.
explosives hitting the target was 875 kg
(0.5*500kg + 2.5*250kg). In the second Looking at the vulnerabilities of each
case, again at a 95 percent confidence case, CONOPS A has a significant
level, CONOPS B allowed an average of disadvantage against CMs allowing 83
2.5 TBMs and 0.5 CMs to leak and hit percent CM leakers vs. 16.7 percent
the critical area. The average combined CM leakers in CONOPS B. However,
mass of explosives hitting the target CONOPS B was more balanced
was 1,375 kg (2.5*500kg + 0.5*250 kg). between the two threats, allowing
The MOEs for these two CONOPS are 25 percent TBM and 16.7 percent
summarized in Table 3. CM leakers. The difference in the
percentage of CM leakers between the
Table 3 shows that both CONOPS two CONOPS appears extreme, but
resulted in an average of three it represents only 2.0 leakers or 500
total leakers (2.5 of one system kg of explosives. Commanders and
and 0.5 of the other); however, analysts must consider these results
CONOPS A resulted in an average along with intelligence on enemy
of 500 kg less explosives impacting capabilities. CONOPS A will most likely
the critical areas. Using only the become overwhelmed by CM more
total number of leakers, analysts easily that CONOPS B. A parametric
and commanders might conclude analysis of different threat raids can
there was no difference between show at what number of threats the
the two CONOPS even though CONOPS becomes overwhelmed. This
CONOP B resulted in 57 percent can then be compared to expected
more explosives impacting the enemy capabilities.
defended asset. Converting the
TBM CM
% Leakers # Leakers % Leakers # Leakers
5.0 % 0.5 83.3 % 2.5
25.0 % 2.5 16.7 % 0.5

Conclusion
The current MOEs used in AMD
analysis based on the total number
of leakers does not provide enough
fidelity for commanders and analysts.
The proliferation of missile technology
means that future AMD fights will
consist of large raids made up of a
mix of TBMs, CMs, and UASs. In
order to ensure the AMD systems
can defend against these large and
complex raids, new MOEs are needed.
The overall MOE should be the total
amount of explosive that impacts on
target because this better quantifies
the effect of CONOP and measures all
threat systems with the same metric.
Additionally, counting the number of
leakers by system type illuminates why
more ordnance may impact the target
in different CONOPS. These new
MOEs allow analysts to recommend
different CONOPS based on the effect
of that CONOP and its susceptibility to
different system types.
Author Statement
The views expressed in this paper
are those of the authors and do not
reflect the official policy or position of
the Center for Army Analysis, the US
Army, the Department of Defense, or
the US Government.
References
Cali, C. M. 2013. UAV Prolif-
eration and the Challenge of
Change, Georgetown Journal
of International Affairs (October
9); http://journal.georgetown.
edu/2013/10/09/uav-proliferation-
and-the-challenge-of-change-by-
c-michael-cali/.
Davenport, K. 2012. Worldwide
Ballistic Missile Inventories. Last
accessed March 7, 2014, http://
www.armscontrol.org/factsheets/
missiles.
Feicket, A. 2005. Cruise Missile
Proliferation, Congressional
Research Service, Editor. The
View publication stats
Library of Congress:
Washington DC.
Freedberg Jr., S. 2015. Army
Explores New Missile
Defense Options, Breaking
Defence (February 18); http://
breakingdefense.com/2015/02/
army-explores-new-missile-
defense-options.
George C. Marshall Institute. 2012.
Missile Threat. Last accessed
August 15, 2014, http://
missilethreat.com/missiles-of-
the-world.
Gormley, D. M. and McMahon, K.
S. 1995. Controlling the Spread
of Land-Attack Cruise Missiles.
American Institute for Strategic
Cooperation.
Gormley, D. M. 2008. Missile
Contagion: Cruise Missile
Proliferation and the Threat to
International Security, Naval
Institute Press.
Gormley, D. M. 2002. The
Neglected Dimension: Controlling
Cruise Missile Proliferation, The
Nonproliferation Review, Vol 9,
No 2.
Kopp, C. 2014. Bypassing the
National Missile Defence System:
The Cruise Missile Proliferation
Problem. Air Power Australia.
Lin, H. 1990. Rationalized Speed/
Altitude Thresholds for ABM
Testing, Science & Global
Security, Vol 2, 87–101.
N.A.a.S.I. Center, Editor. 2013.
Ballistic and Cruise Missile Threat.
NASIC Public Affairs Office:
Wright-Patterson AFB, OH.
Richards, M. A., Scheer, J.A., and
Holm, W.A. 2010. Principles of
Modern Radar, Volume I: Basic
Principles. SciTech Publishing.
US Army UAS CoR. 2010. US Army
UAS Road Map 2010-2035. US
Army UAS Center for Excellence:
Fort Rucker, AL.
Werrell, K. P. 1985. The Evolution of
the Cruise Missile, A. University,
PHALANX – D E C E M B E R 2 0 1 5
PROFESSIONAL FEATURE
Editor. Air University Press,
Maxwell Air Force Base, AL.
Zhang, W.-d., Wang, Y.-b., and
Liu, Y. 2013. Aerodynamic
Study of Theater Ballistic Missile
Target, Aerospace Science
and Technology, Vol 24, No 1,
221–225.
About the Authors
Major Brian M. Wade is an Army
aviation officer in the functional
area 49 (Operations Research
and Systems Analysis - ORSA).
He holds a bachelor’s of science
degree in mechanical engineering
from the US Military Academy and
a master’s of science degree in
aerospace engineering from the
Georgia Institute of Technology. MAJ
Wade is currently serving as the
team leader of the Air and Missile
Defense (AMD) Analysis Team at
the Center for Army Analysis (CAA).
His previous assignments include
teaching Aerospace Engineering
at the US Military Academy, ORSA
analyst in Afghanistan, and company
commander in Korea.
Mr. Paul M. Chang is an operations
research analyst at the Center for
Army Analysis (CAA). He has been
with the CAA since 2000. He holds
a bachelor’s of science degree
in mechanical engineering from
University of Maryland at College
Park. Mr. Chang is currently serving
as an Air and Missile Defense (AMD)
analyst at the Center for Army Analysis
(CAA). He has worked as a combat
system engineer with the Naval Sea
Systems Command from 1987 to
2000 designing surface combatants
including LPD-17 and DD-21. In
addition, he served as a program
manager for test and evaluation of
combat systems for surface ship
and a board member for the Navy’s
Weapon System Explosive Safety
Review Board.
53