Operations Research and the Role of

Probability and Statistics in Military Analysis

21 February 2014

Dr. Rafael E. Matos

WBB, Inc.
(703) 448-6081 x108
Terminology: OR and OA

• Operations Research (OR)

– The academic discipline comprising a range of tools for analyzing
operations for purposes of improving or optimizing various business or
functional processes

• Operations Analysis (OA)

– The discipline of applying OR tools to specific subsets of functional
processes, in your case military operations

2
 How OA Can Help You

• Provide analytic tools in helping to assess budget priorities

• Oversee Directed Studies
– Keeping tabs on outsourced studies

• Manage your own study

– How to run your own study

• Review studies from other sources

– Knowing strengths, weaknesses, good studies and bad ones

• Synthesize an analytic position based on other studies

– Putting the pieces together

3
Operations Analysis

Operations Analysis “classical” definition: “A scientific method of providing

executive departments with a quantitative basis for decisions regarding the
operations under their control”
• Methods of Operations Research, Morse & Kimball, 1951

Decision
Objective Inputs Subjective Inputs
Array of facts: Quantitative Largely provided by the decision-
comparison of the meaningful maker: judgment and experience
elements of the problem factors
“The OA Domain”
Accurately defining the Problem is critical

4
The Military focus has changed
PRE – 911 9-11-2001 POST – 911

Major Regional Conflict/

Major Theater War

2 MRC – 2 MTW

CP HA/
PE DR
PK

C-Piracy
Humanitarian
CT Assistance/
Disaster Relief
Peace–keeping/
Enforcement
C-Terrorism

Strategy and capabilities required for Spectrum of Strategy and capabilities required for post-9/11
Conflict were subsets of 2MRC/MTW force structure environment are not subsets
of MCO force structure

5
 When you change the way you look at things,
the things you look at change…

6
 An Example of Data
Anscombe's Quartet comprises 4 data sets of 11 points each:
I II III IV
x y x y x y x y
10 8.04 10 9.14 10 7.46 8 6.58
8 6.95 8 8.14 8 6.77 8 5.76
13 7.58 13 8.74 13 12.74 8 7.71
9 8.81 9 8.77 9 7.11 8 8.84
11 8.33 11 9.26 11 7.81 8 8.47
14 9.96 14 8.10 14 8.84 8 7.04
6 7.24 6 6.13 6 6.08 8 5.25
4 4.26 4 3.10 4 5.39 19 12.50
For all four: 12 10.84 12 9.13 12 8.15 8 5.56
• Mean of the x values = 9.0 7 4.82 7 7.26 7 6.42 8 7.91
• Mean of the y values = 7.5 5 5.68 5 4.74 5 5.73 8 6.89

• Equation of the least-squared regression line is: y = 0.5x + 3

• Sums of squared errors (about the mean) = 110.0
• Regression sums of squared errors What does this mean ?
(variance accounted for by x) = 27.5 • Analysis of the data show that it
• Residual sums of squared errors (about the regression line) is similar
= 3.75 • But is it ?
• Correlation coefficient = 0.82 • Let’s look at the data when it
• Coefficient of determination = 0.67 is plotted
(F.J. Anscombe, "Graphs in Statistical Analysis," American Statistician, 27 [February 1973], 17-21)

7
 Completely Different Data Sets
Using line plots reveals the differences among the data sets

I III y = 0.50x + 3.00

R² = 0.67
16 y = 0.50x + 3.00 16
R² = 0.67
12 12

8 8

4 4

0 0
0 5 10 15 20 0 5 10 15 20

II IV y = 0.50x + 3.00
R² = 0.67
16 y = 0.50x + 3.00 16
R² = 0.67
12 12

8 8

4 4

0 0
0 5 10 15 20 0 5 10 15 20

This quartet is used as an example of the importance of looking at your data before analyzing it in
Edward Tufte's book, The Visual Display of Quantitative Information.

8
 History of OA
• First wide use of OA methods was during the
Battle of the Atlantic in World War II
• Anti-Submarine Warfare Operations Research
Group (ASWORG) established as part of US
Atlantic Fleet Headquarters to evaluate: “How to
protect a large number of cargo ships against
enemy submarines with limited escorts”
• Study of the ASW problem provided crucial
Tanker torpedoed off US coast - 1942 insights regarding:
– Convoy size - number of ships in each
– Search techniques - how to best employ sensors
to maximize the probability of detecting
submarines
– Screening - where to position escorts to put them
in the best position to detect and counter
submarine attacks

Atlantic convoy at sea

Convoy Protection Problem

• Assumption: each ship in a convoy • A 49-ship convoy has a vulnerable

takes up a certain area that has to be perimeter of 7X4=28 miles and
defended by the escorts (1 mi2) requires 11 escorts (11.2)
• 96% increase in convoy size, but only
40% increase in vulnerable
5 perimeter: requires 33% fewer
escorts than same number of ships
divided into 2 convoys
• For a 25-ship convoy, vulnerable
perimeter is 5X4=20 miles
• If an escort can protect 2.5 miles, this
convoy requires 8 escorts 7
• 2 similar convoys would require 16
escorts

Application of relatively simple mathematics to a complex warfare problem

10
OA Today
• The use of OA has expanded greatly beyond ASW and
Naval Warfare to other services and the commercial
sector
– Department of Defense - uses OA to address the problem
of which programs to fund in the face of a changing
world with a limited amount of resources
– Transportation Industry - OR tools and techniques are
readily applicable to the problem of optimizing
transportation networks and assets in different areas
with varying conditions
– Many recent business improvement schools of thought
(Total Quality Management, Business Process Re-
Engineering, Six Sigma) are all based on Operations
Research theory
• Military and civilian businesses leadership increasingly
turning to analytic process to inform decisions

11
 Example Naval Warfare OA Problem

• Issue: How many ships should the Navy have in the future?
– What types of ships should they be?
“The Global Concept of Operations requires a fleet of approximately 375 ships that will
increase our striking power from today's 12 carrier battle groups, to 12 Carrier Strike
Groups, 12 Expeditionary Strike Groups, and multiple missile-defense Surface Action
Groups and guided-missile submarines. These groups will operate independently
around the world to counter transnational threats and they will join together to form
Expeditionary Strike Forces—the "gold standard" of naval power—when engaged in
regional conflict.” Sea Power 21

The Navy's report provides few details about how many ships the service would have
to buy each year to implement either the 260- or 325-ship plan--and thus how big a
budget it would need for ship construction.

Some combinations may not be affordable, feasible or mission effective

12
Spectrum of Analysis

Basic Analysis: • What or Who is being studied?

Investigative Journalism – Objects or Methods
– Materiel or Doctrine
Who, What, When, Where, Why, and How

Who
• When and Where is it being studied?
What – Micro or Macro level
– Phenomenological or Campaign
When

Where
• How and Why is it being studied?
– Simple evaluation of one
Why – A choice between two or more
– An optimal mix of several
How
Who and What
Who or What is going to be analyzed
Military Application:
• What is the mission that needs to be accomplished?
• Can the mission be accomplish with the current
constraints?
• Is there a better way to use available resources to
accomplish the mission?
• If additional resources are applied how will it improve the
ability to accomplish to this mission?

Example Problem Statement:

“How many ships should the Navy have in 2020-2030?”

14
When and Where

Which missions are

important to the
Does it matter?
overall campaign ?
(SO WHAT?)
CAMPAIGN
What functions are
critical in those Can you use it?
missions? MISSION
What parameters How does it work?
are critical to ENGAGEMENT
those functions?
What is it that does it?
ENGINEERING

Doctrinal studies are usually Technical studies are usually

conducted at Campaign or conducted at Engineering levels
Mission levels, and sometimes at or below, and sometimes at the
the Engagement level Engagement level
15
Why and How

• Straight Evaluation of one subject against a set of benchmarks

– Grades
– Stock price or profit margin

• Choice between two or more subjects

– Fly-off
– Competition

• Optimization
– Determine the optimal mix of resources to apply to a specific (fixed) situation
• Force Structure
– Trade Studies measure the effect of different levels of capability in one area or
attribute versus another
• Cost / performance trade-offs determine the balance between design alternatives
based on both performance and cost

16
 Total Spectrum of Analysis

Who / What • Regardless of where a

Optimization Optimization study falls in the who /
Choice
Choice
Evaluation Evaluation
what / when / where / how
/ why spectrum, each has

When / Where
Campaign

Material Doctrine
Campaign Campaign
Evaluation Evaluation the same characteristics
When / Where

Material Doctrine – Foundation is based on

Mission

Mission Mission
Evaluation Evaluation scientific methods
– Features logical, common
Engineering Engagement

Material Doctrine
Engagement Engagement
Evaluation Evaluation study components and
Material Doctrine stages
Engineering Engineering
Evaluation Evaluation
Material Doctrine
Who / What

17
 Foundation of OA

• Regardless of subject, level or method, all Operations Analysis is based on

scientific methods

• A more thorough definition:

– “...the application of scientific knowledge toward the solution of problems which occur
in operational activities (in their real environment). Its special technique is to invent a
strategy of control by comparing, measuring, and predicting possible behavior through
a scientific model of [the] activity”
Naval Operations Analysis, USNI Press, 1984

• “Solution of problems” - the study Objective

• “Operational activities” - the study Subject Breaking a problem
• “Real environment” - the study Context down into these
• “Comparing” - the study Basis of Comparison factors is the first
• “Measuring” - the study Metrics step in solving it

18
 A Typical OA Study Plan
2. Define This type of plan
Studies 3. Execute
is scalable to
Studies
small and large
issues
1. Establish Plan
Foundation
Prioritize Study Areas
Defense Establish Study Spectrum
Planning
Develop Basis of Comparison Generate Data
Guidance
Select Metrics
4. Synthesize
May 2001
5a. Refine / Results
Focus Plan
Define Objective
Establish Study Subject
Develop Alternatives
Define Context
Is the Study still
supporting the Objective? Process Data
5b. Check Generate Information
Review with Decision-Makers
Impact

5. Facilitate
What Happened?
the Decision
Did it support the Objective?

19
Descriptive Statistical Measures
• Graphical displays provide a better sense of the data by transferring
numerical information into a picture - another way is to use Descriptive
Measures
• Measures of Central Tendency (or location)
– Descriptions on where the middle lies
– Generally attempt to estimate the underlying population mean
– Examples include the mean, median & mode
• Measures of Dispersion (or spread)
– Descriptions on the spread of the data
– Examples include the range, variation, standard deviation & percentile

Central Tendency Dispersion looks

looks for the middle for the spread
X X
X X X X
X X X X
X X
XX X
XX XX XX X X XX XX XX XX X X

20
 Interpreting Statistics 20

Student
AVG = 77
• Sampling distribution is obtained by computing 15 Group A

statistics for a large number of samples drawn 10

from the same population 5

– Measures of Central Tendency / Location 0

A A- B+ B- C+ C- D+ D- F
• Mean
• Median
20

Student
AVG = 77
• Mode 15
Group B
– Measures of Dispersion 10

• Range 5

• Variance 0

• Standard Deviation A A- B+ B- C+ C- D+ D- F

• Percentile 20

Student
• Once observations have been made and 15
Group C AVG = 77
statistics derived, what can we infer about the 10

population from which the sample was 5

collected?
0
A A- B+ B- C+ C- D+ D- F

21
 Trend Analysis
104%

102%

100%

98%

96%

94%

92%

90%

88%

86%

84%
C1/C2 Readiness % of BA filled E5-E9
C1/2 ERB
82% and Top 5 Manning Trend
Stock Price over time 80%
Trend

95

95

96

96

97

97

98

98

99

99

00

00
96

97

98

99

00

01
6

1
-9

-9

-9

-9

-0

-0
p-

c-

p-

c-

p-

c-

p-

c-

p-

c-

p-

c-
n-

n-

n-

n-

n-

n-
ar

ar

ar

ar

ar

ar
Se

De

Se

De

Se

De

Se

De

Se

De

Se

De
Ju

Ju

Ju

Ju

Ju

Ju
M

M
Source: Pers-452 (LOOMIS) C1/C2 vs. E5-E9 BA Filled

• An element of Time Series Analysis

• An investigation of data (i.e. system performance) over a
spectrum of one variable – usually time
• Trend Analysis can estimate or predict…
– within the data set (interpolation)
– outside the data set (extrapolation)
 Analysis with Regression
• Regression Analysis evaluates the relationship between a
variable of interest (a dependent or response variable) and one
or more independent or predictor variables
– Distance from target and kill probability
– Years of training and promotion rates
– Time and target movement
• Often used to predict the response variable from the knowledge
of the independent variables
– Can evaluate either a linear relationship or more complex one
• Regression methodologies are complicated, but the EXCEL
spreadsheet package can perform both simple and complex
regression analysis quickly
A quick indicator is the coefficient of determination (R2) in the results.
You want this number to be close to one.
23
 Regression: Interpolation Example
Missile X
Target Aspect vs. Miss Distance • Test performance data set for Missile X
Test Target Aspect Miss • In addition to the miss distance data, the
Number at Launch Distance
1 5.0 3.0
target aspect at launch time was also
2 10.0 5.0 recorded
3 6.0 3.5 • The chart shows the relationship between
4 9.0 4.0
5 7.0 3.5
the target aspect at launch and the
6 15.0 10.0 missile miss distance at intercept
7 20.0 15.0 – For example for Test # 7, target aspect at
8 40.0 20.0
launch was 20 degrees and miss distance
9 35.0 21.0
10 30.0 18.0
was 15 feet
Target Aspect (Degrees)
Miss Distance (Feet)
• Is there a mathematical relationship
between these two variables such that for
Missile X Performance any given target aspect one can predict
25 the miss distance?
Miss Distance (Ft)

20
15 • What would you say the miss distance
10
5
would be for a target aspect of 25
0 degrees?
0 10 20 30 40 50
Target Aspect at Launch (Degrees)
 Regression: Extrapolation
• Extrapolation estimates or predicts outside the data set
– When time is the independent variable, predicts system performance in the future based on
how it has functioned in the past
• Validity depends on…
– …the consistency of the original data set
– …the assumption that current conditions will continue outside the data set
• Regression works well for interpolation, but not necessarily for extrapolation
– R2 value (a measure of how well the line fits the data) only applies within the data set
• When extrapolating, keep your window short and use some common sense
Missile X Performance How far out would
25 you extrapolate
y = 0.5626x + 0.3426 this data set?
Miss Distance (Feet)

20
R² = 0.9477
15
10 y = -0.0126x 2 + 1.1116x - 3.5103
5 R² = 0.9801
0
0 10 20 30 40 50

Target Aspect at Launch (Degrees)

 Extrapolation Example
• The day before the final launch of the Challenger Space Shuttle,
Morton Thiokol faxed 13 charts to NASA opposing the launch
due to possible O-ring failure.
– In the charts, the following information was scattered throughout and
summarized in the conclusions:
• O-Ring damage at 53F
• O-Ring damage at 75F
• Temperature on 1-28-86 launch initially estimated to be 29F - 38F
• RECOMMENDATION: O-Ring temperature must be > 53F at launch
– NASA officials recommended reconsideration because:
• Analytical graphics failed to communicate the magnitude of
risk that was in fact present
– Reassessing the situation, Morton Thiokol stated
– The evidence presented by their engineers was inconclusive,
– Cool temperatures were not linked to O-ring problems, and
– They now favored a launch the next day
Example from Visual Explanations, Edward R. Tufte, Graphics Press, 1997

26
Extrapolation Example (Cont.)
Temperature Damage Index
53 11 Analysis illustrates risk with launch day
57 4
58 4 temperature forecast of 26-29 degrees F
63 2
66 0 O-Ring Damage
67 0
67 0
67 0 12
Linear R2 = 0.4116
68 0
10
69 0 Polynomial R2 = 0.6286

Damage Index
70 4 8
70 0 6
70 4
4
70 0
72 0 2
73 0 0
75 0
-2 0 20 40 60 80 100
75 4
76 0 Temperature
76 0
n

(y
78 0
 yˆ ) 2
79 0 R2is the square of the SSE i
81 0
residual errors from the line R 2
 1   1  i n1
 ( yi  y ) 2
SST
How well the line approximates the points
i 1

27
 Why is basic probability theory important?
– Probability is the basis for statistics, data analysis, hypothesis
testing, sampling, surveys, experimentation, prediction … other
useful tools in the experimentation trade.

– Decision making under uncertainty is largely based on application

of statistical data analysis for probabilistic risk assessment of
decisions …probability provides the theoretical framework to be
certain of how uncertain you are.

Probability theory provides necessary foundations

for the ability to generate statistics

28
 General Concepts
• Basic Probability Theory
– What is it?
• That branch of mathematics that is concerned with calculating the
likelihood of outcomes of experiments -- modeling the phenomenon
of chance or randomness
• That way of thinking in which we make inferences from a sample to a
population, and then measure the accuracy of those inferences
Probability of a strike by a tropical storm

Katrina

29
 Operational Issues
What are the chances
How many enemy aircraft are
that
detected in a four hour
I can destroy an inbound
period?
missile raid with
Number of Enemy Aircraft my overhead CAP?
Inbound Missile Raid
Detected in a four hour destruction
period Binomial Distribution
Poisson Distribution
What is the Pk for my
How reliable is missiles?
my Single Shot missile
Air Search kill
Radar? Bernoulli Trial
Probability that radar will fail
Exponential Distribution How well does my
Flight deck crew
Are there any anti-ship perform?
mines
located in this area? Probability that flight deck
crew
Mine Location in Chokepoint will maintain advancement
Uniform Distribution rates
Normal Distribution

30
 General Concepts
• What are statistics?
– Classical definition: a science of inferring generalities from specific
observations ,i.e., a way of working with numbers to answer questions
1

about various phenomena

• E.g. Analyzing missile test firings to evaluate overall missile performance
• Goals of statistics are to:
– Descriptive: Describe a set of numbers, and
– Inferential: Make accurate inferences about groups based upon
incomplete information
• E.g., Does the type of missile meet key performance parameters
• Making accurate inferences requires groundwork. In order to
reach an informed conclusion, one must:
– Gather data (numerical information); then,
– Organize it (sometimes in a graphic); then,
– Analyze it
Statistics use “observed data” to draw conclusions
about an “unobserved population”
1-Dixon and Massey, Introduction to Statistical Analysis
31
 Missile AoA Dispersion Calculations
Let’s look at the spread miss distance dispersion data for our 3
candidate missile systems. Missile 1 Missile 2 Missile 3
Test 1 5.0 10.0 3.0
Test 2 5.0 10.0 3.0
Accuracy of 5m with the spread data? Test 3 5.0 10.0 3.0
Test 4 5.0 10.0 3.0
Accuracy of 10m with spread data?
Test 5 5.0 10.0 3.0
Least amount of dispersed shots? Test 6 5.0 10.0 10.0
Test 7 5.0 10.0 15.0
Greatest amount of dispersed shots? Test 8 5.0 10.0 20.0
Test 9 5.0 10.0 20.0
Test 10 100.0 10.0 20.0
Variance 902.5 0.0 63.3
Std Deviation 30.0 0.0 8.0
Range 95.0 0.0 17.0
Mean 14.5 10.0 10.0
Median 5.0 10.0 6.5

Miss distances in meters (m)

32
Value Statement

You can build a fairly comprehensive warfighting model

using Probability Distributions and Statistical Analysis

33
Operations Research and the Role of
Probability and Statistics in Military Analysis

21 February 2014

Dr. Rafael E. Matos

WBB, Inc.
(703) 448-6081 x108