Reliability Engineering and System Safety 152 (2016) 197–204

Contents lists available at ScienceDirect

Reliability Engineering and System Safety

journal homepage: www.elsevier.com/locate/ress

Using data envelopment analysis to evaluate the performance

of post-hurricane electric power restoration activities
Allison C. Reilly a,n, Rachel A. Davidson b, Linda K. Nozick a, Thomas Chen c,
Seth D. Guikema d
a
Cornell University, Department of Civil and Environmental Engineering, 220 Hollister Hall, Ithaca, NY 14853, USA
b
University of Delaware, Department of Civil and Environmental Engineering, 301 Du Pont Hall, Newark, DE 19716, USA
c
Johns Hopkins University, Department of Geography and Environmental Engineering, 313 Ames Hall, Baltimore, MD 21210, USA
d
University of Michigan, Industrial and Operations Research, 1205 Beal Ave., Ann Arbor, MI 48109, USA

art ic l e i nf o a b s t r a c t

Article history: Post-hurricane restoration of electric power is attracting increasing scrutiny as customers’ tolerance for
Received 2 December 2015 even short power interruptions decreases. At the peak, 8.5 million customers were without power after
Received in revised form Hurricane Sandy and over 1 million customers were without power more than a week after the storm
7 March 2016
made landfall. Currently, restoration processes are typically evaluated on a case-by-case basis by a
Accepted 10 March 2016
Available online 26 March 2016
regional public service commission or similar body and lack systematic comparisons to other restoration
experiences. This paper introduces a framework using data envelopment analysis to help evaluate post-
Keywords: hurricane restorations through comparison with the experiences of other companies in similar storms.
Hurricane The method accounts for the variable severity of the hurricanes themselves, so that companies are not
Electric-power system restoration penalized for outages that are long only because the hurricane that caused them was particularly severe.
Data envelopment analysis
The analysis is illustrated through an application comparing 27 recent post-hurricane restoration
experiences across 13 different electric power companies in the United States. The results of the study
show some consistency in performance among individual utilities after the hurricanes they experience.
The method could be applied to other types of infrastructure systems and other extreme events as well.
& 2016 Elsevier Ltd. All rights reserved.

1. Introduction enforcement strategy to protect public safety (e.g., traffic safety).

Hurricanes can cause widespread outages that last for days. In
Hurricane Sandy in 2012, for example, 439,000 Public Service
Electric and Gas (PSE&G) customers in New Jersey were still
without power a week after the storm made landfall [14]. While
these are extreme, relatively infrequent events, their recent trends
are disconcerting; the number of outage events that affect more
than 50,000 customers has steadily increased since 2003 along
with the incidents of extreme weather [13]. Power outages force
many businesses to close, disrupt other infrastructure sectors that
rely on power, such as cellular communication, and may alter law
Abbreviations: BBC, Banker, Charnes and Cooper Model. Also referred to as the
Technical Efficiency Model; CCR, Charnes, Cooper, and Rhodes Model. Also referred
to as the Scale and Technical Efficiency Model; DEA, Data Envelopment Analysis;
DMU(s), Decision Making Unit(s)
n
Corresponding author. Present adress: University of Michigan, Industrial and
Operations Research, 1205 Beal Ave., Ann Arbor, MI 48109, USA.
E-mail addresses: acreilly@umich.edu (A.C. Reilly),
rdavidso@udel.edu (R.A. Davidson), lkn3@cornell.edu (L.K. Nozick),
tchen45@jhu.edu (T. Chen), sguikema@umich.edu (S.D. Guikema).
As society's dependence on and expectation of uninterrupted
electric power increases, regulators have increased their focus on
power companies’ natural disaster responses and post-disaster
investigations by public utility commissions have become
increasingly common (e.g., [11]).
In practice, public utility commissions review power system
performance during and after a hurricane, earthquake, and other
extreme events on a case-by-case basis following the event, and
the public makes its own assessment. It would be helpful to have a
more consistent and transparent method for evaluating and
comparing performance after an event and with this evaluation, to
be able to identify where improvements in the restoration process
are warranted. It is difficult, however, to set standards regarding
what is considered an acceptably fast restoration (such as X% of
customers should be restored in Y days). Storm intensity and size
vary along with characteristics of the service area itself. A standard
should depend both on the societal impacts associated with dif-
ferent outage durations, and on what outage durations are possible
to achieve in practice given that it takes some time for crews to
move around and undertake the repairs and other activities
necessary to restore power. Both of these dimensions are difficult

http://dx.doi.org/10.1016/j.ress.2016.03.007
0951-8320/& 2016 Elsevier Ltd. All rights reserved.
198 A.C. Reilly et al. / Reliability Engineering and System Safety 152 (2016) 197–204
to estimate. An alternative approach is to base restoration per-
formance evaluations on a comparison to other companies’
experiences in similar events. Data envelopment analysis (DEA) is
well-suited to this problem.
Developed by Charnes et al. [3], DEA is a nonparametric
method to measure the relative efficiency of an organization
(called decision making unit, DMU), given the performance of
other organizations that perform similar functions. In a DEA, the
efficiency of a DMU can be measured by considering any number
of inputs and outputs. The inputs and outputs can be non-
commensurate (i.e., they do not need to be in the same units), and
there is no need to specify the relative importance of each input
and output. DEA produces a single scalar measure of efficiency for
each DMU so the results of the analysis are easy to understand and
communicate. With this efficiency measure, efficient levels of
input or outputs can be computed, so DMUs know by how much
they must reduce or augment particular inputs or outputs to
become efficient. Finally, DEA provides for the inclusion of non-
discretionary inputs. These are factors that are not under the
control of the DMU but that influence its ability to create output.
Since the evaluation of the performance of each DMU is max-
imized given the performance of other DMUs, the focus of the
analysis is on each DMU rather than on the estimation of the
parameters of a single model. This means the DEA does not require
the specification of the functional form of the relationship
between the independent and dependent variables. Rather, the
DEA analysis produces an estimate of the functional form of the
efficient frontier [3]. Unlike statistical regression methods that
measure performance based on deviations from average or “best
fit” behavior, DEA uses the best-observed performance as the
frame of reference.
The goal of this paper is to develop a wholly new, transparent,
and consistent evaluation process for the restoration of electric
power after a hurricane. This paper provides an illustrative case
study using real data to compare 27 recent post-hurricane
restoration experiences of U.S. electric power companies. While
the data in the case study are real and collected from utility
websites and news articles, they are ultimately masked; the pur-
pose of this paper is not to say how specific utilities responded to
recent storms but rather to develop a framework for consistent,
unbiased evaluation of these responses. The evaluation metrics we
choose may not suffice for a true evaluation. Those that are chosen
reflect a combination of the authors’ perception of what are
valuable metrics for evaluating response and limited amounts of
publically available data. Ultimately, it is for utilities, public service
commissions, and other stakeholders to decide the metrics that
are appropriate for comparison.
Under this paper's framework, each DMU is a post-hurricane
electric power restoration performed by a specific utility, such as
12 CCR Technical and Scale
Efficiency Frontier
10
8 C
Output
6
R Frontier
4 E
E*
2 B
A
0
0 1 2 3 4 5
Input
Efficient Restoration Process
Fig. 1. Example DEA model.
Dominion Virginia Power's restoration after Hurricane Isabel. The
most efficient restorations are those for which there is no other
restoration or linear combination of restorations that was faster
given the money spent or the storm severity. The most efficient
DMUs serve as an efficient frontier to which all other DMUs are
compared. Hurricane severity is considered a non-discretionary
input, so that the analysis acknowledges that a utility does not
have control over the hurricane severity, and therefore should not
be penalized for restorations that take longer because they were
associated with very severe hurricanes.
Many other studies have modeled the post-disaster restoration
processes of various infrastructure systems in an effort to estimate
expected restoration times, and several have tried to optimize
post-disaster restoration strategies. Previous work in these areas is
summarized in Liu et al. [8,20], and Nateghi et al. [10]. No pub-
lished work could be found, however, related to the evaluation of
restoration processes, or using DEA to evaluate system perfor-
mance in extreme events more generally.
DEA has been used in the electric power industry more broadly
to evaluate the relative efficiency of, for example, electricity dis-
tribution utilities in the U.S. Pahwa et al. [12], power plants in
Israel [6], service centers in Taiwan [4], and the impact of the clean
air act on coal-fired generators [18]. Chien et al. [4] and Yang and
Lu [21] offer reviews of this literature. DEA has not been used to
develop infrastructure system, including electric power systems,
performance comparisons for extreme events.
The paper is organized as follows. In Section 2, DEA models are
reviewed, and the type of formulation used in this study is
developed. The post-hurricane electric power restoration case
study is described in Section 3, and the paper concludes with a
summary of the strengths and limitations of this approach in
Section 4.
2. DEA models and example
The original DEA model was developed by Charnes et al. [3],
and is often referred to as the CCR model for Charnes, Cooper and
Rhodes. It assumes decision-making units (DMUs) operating with
constant returns-to-scale are efficient. Banker et al. [2] developed
a modification of the CCR model (often referred to as the BCC
model for Banker, Charnes, and Cooper) that relaxes the constant
returns-to-scale requirement in favor of variable returns-to-scale.
In this paper, we use the BCC model because it is less restrictive
and allows for a richer exploration of the data. Before developing
the DEA models to be used in evaluating the performance of post-
hurricane electric power restoration activities, the simple case of
one discretionary input and one output (Fig. 1) is used to illustrate
four key concepts: (1) the character of CCR and BCC efficiency
D
BCC Technical Efficiency
6 7 8 9 10
Inefficient Restoration Process
 A.C. Reilly et al. / Reliability Engineering and System Safety 152 (2016) 197–204
12
Non-Discretionary Input, x2
10
x2
8
L**
6 L*
0
0 1
Efficient Restoration Process
Fig. 2. Schematic illustration of the impacts of non-discretionary inputs.
frontiers; (2) the concept of an input-oriented DEA model; (3) how
to identify the reference set of DMUs that are preventing an
inefficient DMU from being efficient; and (4) the use of non-
discretionary inputs.
In Fig. 1, Points A, B, C, D, and E represent an input and output
of five DMUs which could be, for example, utility companies, fast
food franchises, or different schools within the same district. Given
the revealed performance based on these DMUs it is evident that a
DMU must use at least 2 units of input to obtain output. The CCR
model assumes DMUs operating with constant returns-to-scale
are efficient, which corresponds to a linear efficient frontier
through the origin. In many situations, however, it is reasonable to
assume that when the size of an operation is very small or very
large, output will not be linearly proportional to input, meaning
the optimal ratio of output to input is not constant as input
increases. In that case, the assumption of constant returns-to-scale
is relaxed and the Pareto efficient frontier is the envelope (or
convex hull) defined by the efficient DMUs (Points B, C, and D in
this case). All the ratios of output to input on this frontier are
deemed efficient; hence this efficiency frontier shows the best
performers in the given set of DMUs.
In an input-oriented model, like the one assumed here, the
efficiency score ranges from zero to one, with one being efficient
(e.g., Points B, C, and D). For an inefficient DMU to become efficient
in an input-oriented model (e.g., Point E), a reduction in input is
necessary, while, generally speaking, the output remains unchan-
ged. The exception to this is points along the efficient frontier like
DMU A. The input for DMU A is the same as for DMU B, two units.
However, the output for DMU B (two units) is greater than that of
DMU A (one unit). Without changing input, DMU A should be able
to eliminate this slack and increase the output by one unit. Even
though DMU A has the optimal level of input, slack exists, so it is
not technically efficient. The same is true with any point on the
dashed line in Fig. 1.
As mentioned, the DMUs on the BCC efficient frontier have
efficiency scores of 1. Each inefficient DMU's input is projected
onto the efficient frontier to find the level of input reduction
necessary to become efficient. In the case of Point E, its input is
projected onto the efficient frontier to find its efficient level of
input, at Point E*. The technical efficiency score of DMU E is the
ratio of the technically efficient amount of input, at E*, to the
actual amount of input at E.
RE
2:80
¼ ¼ 0:7
RE 4 is discretionary.
199
L***
L Section of BCC Production
Possiblities Frontier
2 3 4,x1 5 6 7 8 9 10
Discretionary Input, x1
Inefficient Restoration Process
Therefore, given DMU E's production scale, if it were to
decrease its input by 30%, it would be efficient when compared to
other DMU's.
Given that DMU E is technically inefficient, one can identify
which other DMUs are preventing it from being efficient. The set of
DMUs that prevent a given DMU from being technically efficient
are referred to as the reference set. The efficient point on the BCC
efficiency frontier for DMU E is E*, (2.8, 4). E* divides the line BC,
showing that DMU's B and C are DMU E's reference set. Line BE* is
40% the length of BC, corresponding to 40% of the change in input
between B and C and 40% of the change in output between B and
C. DMU B contributes 60% of it inputs and output and DMU C
contributes 40% of its inputs and output to create DMU E*.
Often the DMUs have no control over the amount of some types
of input to the process, such as, in this analysis, the severity of the
hurricane that precipitates restoration activities. It is, therefore
unfair to penalize DMUs that have large amount of non-
discretionary input and it is meaningless to find a target reduc-
tion of input for a non-discretionary input. The non-discretionary
inputs collectively create the “environment” the DMU operates in,
but over which it has no control. Banker and Morey [1] reformu-
lated the BCC model to include the presence of non-discretionary
inputs. Similar to the previous models, it finds the technical effi-
ciency and reference set of each DMU relative to the entire set of
DMUs. However, in the modified formulation, DMUs are not
penalized for having excessive non-discretionary input.
To illustrate the impact of non-discretionary inputs, consider a
case with one discretionary input, one non-discretionary input,
and one output (Fig. 2). By how much can the inputs of DMU L be
reduced while still achieving the same output? The production
possibilities frontier shows the optimal combinations of inputs
that produce the same output as DMU L. If both inputs, x1 and x2,
are discretionary inputs, the optimal reduction of inputs is to the
point L*. However, if input x2 is non-discretionary, then DMU L is
not able to reduce it. The most efficient location is then L**, a
location on the efficient frontier. The points L**, L***, and S are BCC
efficient. The difference in non-discretionary levels between point
S and L** simply represents slack in the model. Since a reduction in
x2 from L*** to L** or similarly S to L** is both impossible and
meaningless to the DMUs, the efficiency score is the same for both
points. Point L**, lacking slack, is used as its BCC efficient point.
Notice that DMU S has a more severe environment than that of
DMU L yet requires less discretionary input. The reduction in
discretionary input x1 when non-discretionary input x2 is present
that is necessary for DMU L to be efficient is greater than or equal
to the reduction in discretionary input x1 when the other input, x2,
ð1Þ
200 A.C. Reilly et al. / Reliability Engineering and System Safety 152 (2016) 197–204
DEA can easily be extended from this simple example of one
discretionary input, one non-discretionary input and one output to
multiple inputs (both discretionary and non-discretionary) and
outputs through the use of linear programming. The optimization
is performed N times, once for each DMU. For each of the N DMUs
in the primal formulation, the ratio of a weighted sum of outputs
minus a weighted sum of non-discretionary inputs minus a free
variable to a weighted sum of discretionary inputs is maximized,
while enforcing a constraint that no ratios can be greater than one.
The linear program solves for these weights. The dual is solved in
this paper, which minimizes the efficiency score subject to a
convexity constraint and constraints that prohibit the weighted
contribution of inputs and outputs of the DMUs in the reference
set from exceeding the realized values of non-discretionary input,
optimal discretionary input, and output of the DMU.
In standard notation, the Banker and Morey [1] BCC model with
non-discretionary input for DMU o is as follows:
( !)
X X
min θ0  ε þ
si0 þ 
sr0 ð2Þ
iAD r AS
subject to:
X more nondiscretionary inputs.
λj xij þ si0þ ¼ θ0 xi0 i A D
jAN
X inputs. Stage 1 uses only discretionary inputs and outputs to
λj zkj þ sk0
þ
¼ zk0 k A F
jAN
X that all DMUs have the same environment. In Stage 2, the entire
λj yrj  sr0

¼ yr0 r A S
jAN
X harshness of the environment, zj. The idea is that when non-
λj ¼ 1
jAN
λj ; si0þ ; sr0

Z0
θ0 unrestricted
Here xij is the amount of discretionary input i for DMU j, zkj is the
amount of non-discretionary input k for DMU j, and yrj is the
amount of output r for DMU j. There are N DMUs, ε is assumed to
be a small number (10  8 in this analysis), and iA D, k AF, and r A S
are the sets of discretionary inputs, non-discretionary inputs, and
outputs, respectively. The variables are: θo is the technical effi-
ciency of DMU o, λj is the level of contribution DMU j has on the
technical efficiency score of DMU o, sioþ and sko þ
are model slack and
the amount by which the inputs i and k can be reduced for DMU o

without changing its technical efficiency and sro is the augmen-
tation in the output for DMU o which can be achieved when the
inputs are also reduced consistent with Eqs. (3) and (4). Note that
the model reports the potential for reductions in non-
discretionary inputs but those reductions do not change the effi-
ciency score, because those reductions are often not possible.
In the objective function, Expression (2), the ε parameter is
used so that the slack variables are maximized as a second stage,
after θo is minimized. Eqs. (3) and (4) require that the optimal
levels of inputs are at least the production possibilities evident in
the set of DMUs. Eq. (5) requires that the output of DMU o be at
most the realized production possibilities of the set of DMUs.
Taken together, Eqs. (3)–(5) ensure that the recommended
reductions in discretionary inputs and opportunities for output
augmentation are consistent with the production possibilities set
of the DMUs. Eq. (6) ensures that the production possibilities
frontier is convex. Notice that the target reduction in each dis-
cretionary input i necessary for DMU o to become efficient is
xio ð1  θo Þ þ sioþ . Additionally, output r for DMU o needs to be

augmented by sro to be efficient.
Ruggiero [15] shows that the Banker and Morey BCC model
with non-discretionary data may create a production possibility
frontier that is infeasible for a DMU because its technical efficiency
score is limited by DMUs that have values for the non-
discretionary inputs that overall indicate a more favorable envir-
onment. Since these inputs are non-discretionary, the DMU cannot
implement changes that would allow its technical efficiency score
to improve. To overcome this difficulty, Ruggiero [15] eliminates
Eq. (4) and excludes peers with better environments by adding the
following constraint:
λj ¼ 0 8 zkj 4 zi0 ; k A F; j A N ð9Þ
Eq. (9) excludes all DMUs with conditions that are better in at
least one non-discretionary category. Hence it eliminates DMU j
from being DMU o's peer if it has a better environment according
to at least one non-discretionary input, even if overall the envir-
onment is the same or worse for DMU j. Ruggiero [16], however,
argues that this rule can lead to biased estimates of technical
efficiency because some of the DMUs that have been eliminated
have an overall environment that is as or less favorable than that
of DMU o even though they have more favorable values for one or
ð3Þ To remedy this, Ruggiero [16] develops a 3-Stage modeling
process to estimate technical efficiency with non-discretionary
ð4Þ obtain a First Stage efficiency score FSj for j¼1,…, N by using Eqs.
(2)–(8) without (4). FSj shows technical efficiency of DMU j given
ð5Þ set of nondiscretionary inputs, F, are linearly regressed on the
efficiency scores FSj obtained in Stage 1 to create a measure of the
ð6Þ discretionary inputs are disregarded, the resulting efficiency
should tend to be inversely related to the non-discretionary inputs
if the non-discretionary inputs are truly an impediment to
ð7Þ
efficiency.
In Stage 3, a DEA model with the objective given in Eq. (2) and
ð8Þ
the constraints in Eqs. (3), (5)–(8) and (10) is solved.
λj ¼ 0 8 zj 4 z0 ; j A N ð10Þ
where zj is the measure of the harshness of the environment for
DMU j obtained from the regression equation developed in Stage
2. Muñiz et al. [9] confirm the robustness of the Ruggiero 3-Stage
method when using multiple non-discretionary inputs. We use
this method in the restoration analysis in this paper.
3. Evaluation of post-hurricane electric power restoration
3.1. Inputs and outputs
In exploring the performance of electric power companies in
restoring service after hurricanes, we consider 27 restorations
(utility-storm combinations, or DMUs to be consistent with the
DEA literature). This includes 13 utility companies and 12 hurri-
canes that occurred between 1996 and 2012 in the Mid-Atlantic,
Northeastern, and Southeastern regions in the United States
(Table 1). The output of interest is customer-days without power
(y), which measure the amount of time it takes to restore power to
customers. There is one discretionary input, restoration costs in
2005 dollars (x), and five non-discretionary inputs: service area (in
square miles) with peak gust wind speed above 75 miles per hour
(z1) and 100 miles per hour (z2), percentage of service area with
peak gust wind speed above 75 miles per hour (z3) and 100 miles
per hour (z4), and peak number of customers without power (z5).
Restoration costs (x) typically include wire, pole, and transformer
replacement costs; the cost of clean-up and restoration crews, who
Table 1
Inputs, outputs, technical efficiencies, reference sets, and target reductions for DEA analysis.

Restorationa Storm Utility Storm year Number of Customer- Restoration cost Service area with peak wind Percentage of service area Peak number θ0 Reference set BCC target reduc-
customers at days without (in millions USD, gust (sq. mi.) with peak wind gust of customers tion of restora-

A.C. Reilly et al. / Reliability Engineering and System Safety 152 (2016) 197–204
time of storm power (y) 2005 dollars) (x) without power tion cost (millions
Z 75 mph Z 100 mph Z 75 mph Z 100 mph (z5) USD, 2005 dollars)
(z1) (z2) (z3) (z4)

2 Frances A 2004 4,300,000 7,040,000 $326.7 6044 2944 24% 12% 2,800,000 1.00 2 $326.7
3 Jeanne A 2004 4,300,000 3,160,000 $332.9 4079 1445 16% 6% 1,740,000 1.00 3 $332.9
6 Charley A 2004 4,300,000 2,650,000 $260.5 2562 761 10% 3% 870,000 0.50 2, 4 $128.9
7 Isabel B 2003 2,000,000 7,320,000 $135.9 14,007 345 47% 1% 1,690,000 1.00 7 $135.9
13 Floyd B 1999 2,000,000 430,000 $21.0 2351 0 5% 0% 520,000 0.55 7, 19 $11.6
9 Frances C 2004 1,500,000 2,710,000 $134.4 2981 0 14% 0% 830,000 0.39 7, 19 $52.7
10 Jeanne C 2004 1,500,000 1,660,000 $86.8 3731 0 18% 0% 720,000 0.39 7, 19 $33.8
5 Charley C 2004 1,500,000 2,260,000 $157.1 2020 835 10% 4% 500,000 0.71 2, 4 $111.4
4 Ivan D 2004 1,400,000 1,990,000 $99.2 8774 1115 20% 2% 830,000 1.00 4 $99.2
11 Katrina D 2005 1,400,000 1,410,000 $75.0 3420 0 8% 0% 630,000 0.39 7, 19 $29.3
8 Dennis D 2005 1,400,000 260,000 $35.0 3236 161 7% 0% 240,000 1.00 8 $35.0
12 Isabel E 2003 1,150,000 2,220,000 $84.9 0 0 0% 0% 650,000 0.52 7, 19 $43.9
23 Irene E 2011 1,250,000 1,150,000 $70.3 0 0 0% 0% 480,000 0.35 7, 19 $24.6
25 Sandy E 2012 1,250,000 260,000 $32.3 0 0 0% 0% 210,000 0.27 7, 19 $8.5
14 Isabel F 2003 750,000 1,560,000 $74.3 2 0 0% 0% 370,000 0.43 7, 19 $32.0
15 Isabel G 2003 1,300,000 330,000 $14.3 3477 0 10% 0% 320,000 0.43 18 $6.2
20 Charley G 2004 1,300,000 100,000 $13.5 0 0 0% 0% 110,000 1.00 20 $13.5
16 Jeanne H 2004 645,000 810,000 $35.3 2490 0 62% 0% 290,000 0.52 7, 19 $18.5
17 Frances H 2004 645,000 460,000 $26.1 1672 0 42% 0% 270,000 0.45 7, 19 $12.2
18 Fran I 1996 2,300,000 330,000 $21.7 1123 0 5% 0% 260,000 0.45 7, 19 $9.8
19 Isabel I 2003 2,300,000 130,000 $6.2 0 0 0% 0% 130,000 1.00 19 $6.2
1 Ike J 2008 382,202 2,610,000 $523.4 11,413 4043 60% 21% 382,202 1.00 1 $523.4
21 Ike K 2008 2,244,000 13,120,000 $680.3 1945 0 39% 0% 2,150,000 1.00 21 $680.3
22 Irene L 2011 325,000 440,000 $21.7 0 0 0% 0% 160,000 0.54 7, 19 $11.8
26 Sandy L 2012 325,000 620,000 $34.0 0 0 0% 0% 187,000 0.44 7, 19 $15.0
24 Irene M 2011 1,100,000 970,000 $67.7 0 0 0% 0% 410,000 0.32 7, 19 $21.4
27 Sandy M 2012 1,100,000 3,740,000 $535.9 371 0 10% 0% 1,000,000 0.13 7, 19 $71.3

a
The restoration number corresponds to the severity of the storm as determined by Stage 2. Restoration 1 is the most severe storm while Restoration 27 is the least severe.

201
202 A.C. Reilly et al. / Reliability Engineering and System Safety 152 (2016) 197–204

8
7
2

Customer-days without Power, y, (in millions)

7

3 3
1
5' 9 6
5
2 4

1
19
8
-
$0 $100 $200 $300 $400 $500 $600
Restoration Cost, x, (in millions USD, 2005 dollars)
Efficient Restoration Process
Inefficient Restoration Process
Efficient Frontier for Restorations 1 through 3
Efficient Frontier for Restorations 1 through 6
Efficient Frontier for Restorations 1 through 8
Fig. 3. Analysis of Restorations 1 through 9 and 19.

are often recruited from neighboring utilities unaffected by the was run using HAZUS in the deterministic mode, and the company
storm; and the cost of additional support personnel. The larger the service boundaries were overlain on the resulting estimated wind
area that experiences strong winds (z1 and z2), the more damage speed maps to extract the required data. For Hurricanes Irene and
that is likely to have occurred, thus increasing the restoration Sandy, a gradient wind field model based on that developed by
demands. The percentage of service area inputs (z3 and z4) are Huang et al. [7] was used. This model inputs storm track and other
included to measure the idea that if a large portion of a utility's storm characteristics to get localized estimates of the sustained
service area experiences strong winds, it is more likely to strain its peak 3-second wind gusts.
restoration resources and capabilities. The actual area experien- Since DEA requires that more output be considered desirable
cing strong winds and the relative area experiencing strong winds and more input be associated with easier conditions for creating
are related yet extract different information. Consider a utility output, the output and all inputs except for restoration costs have
with a small service area that experiences a harsh storm and been rescaled in the manner proposed in Seiford and Zhu [17].
another utility with a large service area that experiences a milder Customer-days without power (y) and service area with peak wind
storm. Both may have identically sized regions with wind speeds gusts above 75 and 100 miles per hour (z3 and z4) were each
in excess of some speed, yet presumably, the second utility has rescaled by taking the largest value across the 27 utility-storm
access to more resources given its size. The data show that the combinations and subtracting the original value from it (ymax  yj).
non-discretionary inputs, z1 through z5, are each positively corre- Percentage of service area with peak wind gusts greater than 75
lated with each other to varying degrees. To improve model fit and and 100 miles per hour (z1 and z2) were rescaled by taking one
reduce multicollinearity in Step 2, customer-days without power, minus the value.
y, is regressed against all combinations of non-discretionary Steps 1 through 3 of the model were solved using MATLAB,
inputs, z, and the best model is selected. version R2014b.
One could argue that the peak number of customers without
power (z5) should be a discretionary input because the more
robust a utility makes its network, the fewer customers will lose 3.2. Results
power. However, the focus here is on how long it takes to restore
service, given the number of outages that occurred, so it is con- In Stage 1 of the three-stage method, efficiencies were deter-
sidered non-discretionary. mined using only output (y) and discretionary input (x). In Stage 2,
The outage duration data were collected from the DOE Elec- the non-discretionary inputs were regressed on the efficiencies
tricity Delivery and Energy Reliability website [19], media reports, determined in Stage 1. To improve the linearity of the regression,
and utility press releases. Restoration costs were obtained from the natural log of the Stage 1 efficiency and all discretionary
media reports, utility press releases, and filings with state public variables (z1–z5) is taken. The best model contained only the log of
service commissions for storm reparations and rate adjustments the service area with peak wind gusts greater than 100 mph, log
requests. Peak gust wind speeds were found using HAZUS, the (z2), which has a p-value ¼7.0(10  3). The R2adj ¼0.24. Note that
standardized national multi-hazard loss estimation software since only one non-discretionary input was significant and
developed by the Federal Emergency Management Agency (FEMA) therefore used, this model turns out to be equivalent to the Rug-
for all storms except Hurricanes Irene and Sandy [5]. Those storms’ giero [15] formulation, (Eq. (1)–3), and (5)–(9). However, the sta-
properties were not available via HAZUS during the time frame tistical significance of the non-discretionary variables would not
under which this research was conducted. Each historical storm be known without having used the Ruggiero 3-Stage method.
 A.C. Reilly et al. / Reliability Engineering and System Safety 152 (2016) 197–204
The fact that the best model in Step 2 only contains the service
area with peak wind gusts greater than 100 mph could be for one
of two reasons: the size and severity of a storm is not truly an
impediment to an efficient restoration except in cases with
excessive winds (and conversely that which impedes efficient
restorations is due to some process(es) over which the utilities
have control), or that the nondiscretionary data that are selected
by the authors do not fully describe each utility's experience.
Likely both occur here. Since this is a proof of concept using
publically available data and reasonable approximations for storm
severity, the case study continues using this model. Should
decision-makers use the method however, they should find con-
sensus for fully describing storm severity and utility experience.
The results from Stage 3 are presented in Table 1. The
restoration number is the storm severity rank according to the
regression in Stage 2. Restoration 1 experienced the worst storm in
terms of the non-discretionary input, whereas Restoration 27
experienced the mildest storm. Restoration 1 is only compared
itself; comparing it to more favorable experiences would be
unreasonable. Conversely, Restoration 27 is compared to all other;
restoration processes; the experiences of the other storm-utility
pairings are the same or worse than Restoration 27. Although no
company experienced more than three hurricanes, there is some
consistency with regards to performance across storms. For
example, Utilities A and D both experienced two storms for which
their responses were efficient or very close to efficient (i.e., θo ¼ 1)
– Hurricanes Jeanne and Frances for Utility A, and Hurricanes
Dennis and Ivan for Utility D. Note however, that Utility A
experienced among the most severe storms, limiting the number
of other experiences to which they can be compared. Utilities A
and D also experienced one storm in which their response was
inefficient according to the model. Similarly, many utilities that
responded inefficiently during one storm responded similarly
inefficiently for other storms. For example, see Utilities C, E, H, L,
and M. Utilities F, J and K only experienced one storm.
Fig. 3 illustrates graphically how the efficiency and reference
set are determined. It shows, for Restorations 1 to 9 and 19, the
relationship between restoration cost (discretionary input, x),
customer-days without power of a given restoration, y, and whe-
ther or not they are deemed technically efficient. Restorations
9 through 27 experienced the same storm severity based on
Step 2.
The order of the discussion that follows is from most severe
storm (i.e., Restoration 1) to least severe. Restoration 1 is techni-
cally efficient not because it spent less or had fewer customer-days
without power but rather because it experienced the worst storm
and can be compared to no other experiences. Compared to
Restoration 1, Restoration 2 is efficient; Restoration 2 had more
customers-days without power yet it spent less money on repairs
than Restoration 1, which displays the tradeoff existing between
the two. For the similar reason, Restoration 3 is efficient. It spent
(slightly) more than Restoration 2 yet had fewer customer-days
without power.
When comparing only Restorations 1–3, the convex hull is the
dashed line in Fig. 3. The cost per customer-day without power is
$46, $105, and $201 for Restorations 2, 3 and 1, respectively.
Restoration 2, the least costly of the three, has the most customer-
days without power and Restoration 1, the most costly, has the
fewest. Since marginal returns-to-scale is assumed, this demon-
strates that it is increasingly expensive to reduce the number of
customer-days without power. Should the next worst storm
experience lay northeast of the dashed line, it would be deemed
inefficient.
Restoration 4 changes the shape of the efficiency frontier (i.e.,
the solid line in Fig. 3). While it experienced a milder storm, it
spent less money and had fewer customer-days without power
203
than Restorations 1–3. Restoration 4 spent only $50 per customer-
day without power. Restorations 5 and 6 are deemed inefficient
because of the presence of Restorations 2 and 4. Restorations 2 and
4 experienced worse storms yet were more efficient with the
money they did spend. For Restoration 5 to be efficient, it would
need to move to Point 5’ in Fig. 3. This point has the same number
of customer-days without power as Restoration 5 but also has an
expenditure reduction of 29% (i.e., $111 million). Had Restoration
3 experienced a storm milder than Restoration 4, it too would be
deemed inefficient.
The next two most intense storms, Restorations 7 and 8, form a
new convex hull, thus showing improved efficiency (likely)
stemming from milder storms. Restorations 9 through 27 (not
shown in Fig. 3, except for Restorations 9 and 19) experienced
similar intensity storms according to the z statistic from Step 2.
Thus, Restorations 9 through 27 may be compared to each other
and to the experiences of Restorations 1 through 8, since
Restorations 9 through 27 had milder experiences. When all
restoration processes are considered, Restorations 7 and 19 creates
the final convex hull and prevent other restoration process,
including Restoration 9, from being efficient.
Comparisons of technically inefficient restorations to their
reference set (e.g., Restoration 5 to Restorations 2 and 4) provide
an opportunity to identify characteristics making certain restora-
tions technically efficient and the others inefficient. These com-
parisons may not only highlight differences in restoration opera-
tions, such as number and location of tree crews and the order in
which areas restore power, based on location characteristics, but
also show inconsistencies in how utilities report items such as
restoration cost and customers without power. Identifying opera-
tional aspects that technically inefficient utilities can adopt creates
a potential for more efficient future restorations through lessons
learned.
4. Conclusions
This paper develops a framework using DEA to systematically
evaluate the performance of electric power companies in restoring
service following hurricanes, and demonstrated its promise using
an illustrative application. It provides a transparent, reproducible,
quantitative method for comparing restoration activities, including
a mechanism to effectively consider the severity of the storms
experienced. The illustrative results suggest that in this sample,
the performance of specific utility companies is relatively con-
sistent across hurricanes. The approach could be adapted to eval-
uate the restoration activities for other infrastructure systems in
other extreme events as well.
The data used in the analysis is the best available and suffi-
ciently accurate for demonstrating the promise of the method. Yet,
they are a reflection of the authors’ views on what are valuable
metrics for evaluating performance and of limited amounts of
publically available data. To truly evaluate performance and base
decisions on the results of this method, one would need to ensure
data quality and that the metrics chosen reflect the values of
stakeholders. Large national organizations such as the Institute for
Electrical and Electronic Engineers (IEEE) – a large professional
organization which covers the industry – or the Federal Energy
Regulatory Commission (FERC) might be able to facilitate con-
sensus gathering to understand the performance measures valued
by consumers, etc. They could also oversee consistent data col-
lection across companies and ensure large samples by collecting
data for every hurricane. As part of such an effort, standardizing
definitions of what should be included in the restoration cost and
the frequency with which customer outage estimates are made
204 A.C. Reilly et al. / Reliability Engineering and System Safety 152 (2016) 197–204
(which affects calculation of customer-days without power) would
be helpful.
If data became available to allow it, application of the method
would be further enhanced by using more specific inputs and
outputs. For example, rather than comparing overall restoration
costs, comparing number (or cost) of tree crews, line crews, and
support staff may provide insight into performance in those spe-
cific aspects of restoration. However, the number of additional
input and outputs is limited by the number of restorations that are
included. As the number of discretionary inputs and outputs are
increased, so is the number of technically efficient restorations.
DEA forms a good first step for doing comparative analyses
among storm restoration processes. DEA produces a reference set
for each restoration, which consists of technically efficient
restorations preventing the restoration under investigation from
being technically efficient. This lends itself naturally to move to
qualitative comparisons among inefficient processes and those
processes that are preventing them from being efficient. By
understanding the differences in the practices of technically inef-
ficient restorations compared to their technically efficient
restorations, additional best practices can be identified.
Finally, because DEA is a strictly comparative evaluation, the
more restorations (DMUs) considered, the more insight can be
gained. In the current data sample, for example, since the
experiences of Restorations 1, 2, and 3 were so much worse than
the others in the sample, it is difficult to draw strong conclusions
about their performance. Further, considering a restoration to be
technically efficient based on the DEA analysis does not necessarily
mean that it cannot be improved. If it was desirable to set a
standard for restoration performance, hypothetical standard
restorations could be added to the selection of restorations for
comparison, as in [6].
Acknowledgments
This work is funded by the National Science Foundation (NSF)
Graduate Research Fellowship Program and by NSF Grants
1149460 and 1331399. Any opinions, findings, conclusions or
recommendations presented in this paper are those of the authors
and do not necessarily reflect the view of the National Science
Foundation.
References
[1] Banker RD, Morey RC. Efficiency analysis for exogenously fixed inputs and
outputs. Oper Res 1986;34(4):513–21.
[2] Banker RD, Charnes A, Cooper WW. Some models for estimating technical and
scale inefficiencies in data envelopment analysis. Manag Sci 1984;30(9):1078–92.
[3] Charnes A, Cooper WW, Rhodes E. Measuring the efficiency of decision
making units. Eur J Oper Res 1978;2(6):429–44.
[4] Chien C-F, Lo F-Y, Lin JTA. DEA study to evaluate the relative efficiency of the
service center and improve operation efficiency through reorganization. IEEE
Trans Power Syst 2003;18(1):366–73.
[5] Federal Emergency Management Agency. HAZUS-Multi Hazard Tool. Available
at: 〈www.fema.gov/hazus/hazus-mh-overview〉; , 2013 [accessed 26.11.15].
[6] Golany B, Roll Y, Rybak D. Measuring efficiency of power plants in Israel by
data envelopment. Anal. IEEE Trans Eng Manag 1994;41(3):291–301.
[7] Huang Z, Rosowsky DV, Sparks PR. Hurricane simulation techniques for the
evaluation of wind-speeds and expected insurance losses. J Wind Eng Ind
Aerodyn 2001;89(7):605–17.
[8] Liu H, Davidson R, Apanasovich T. Statistical forecasting of electric power
restoration times in hurricanes and ice storms. IEEE Trans Power Syst 2007;22
(4):1–10.
[9] Muñiz M, Paradi J, Ruggiero J, Yang Z. Evaluating alternative DEA models used to
control for non-discretionary inputs. Comput Oper Res 2006;33(5):1173–83.
[10] Nateghi R, Guikema ST, Quiring SM. Forecasting hurricane-induced power
outage durations. Nat Hazards 2014;74(3):1795–811.
[11] North Carolina Public Utility Commission (NCPUC). Raleigh (NC): Report of the
North Carolina Utilities Commission and the Public Staff to the North Carolina
Natural Disaster Preparedness Task Force, Response of the Electric Utilities to
the December 2002 Ice Storm. [Online]. Available from: 〈http://www.ncuc.
commerce.state.nc.us/reports/part1ice.pdf〉; 2003 September, 76 p.
[12] Pahwa A, Feng X, Lubkeman D. Performance evaluation of electric distribution
utilities based on data envelopment analysis. IEEE Trans Power Syst 2002;17
(3):400–5.
[13] President's Council of Economic Advisers. US Department of Energy, Office of
Electricity Delivery and Energy Reliability, and White House Office of Science
and Technology. Washington (DC): Economic Benefits of Increasing Electric
Resilience to Weather Outages [Online]. Available from: 〈http://energy.gov/
sites/prod/files/2013/08/f2/Grid%20Resiliency%20Report_FINAL.pdf〉; August
28, 2013.
[14] Public Service Electric and Gas (PSE&G). PSE&G Service Restoration Update –
Sunday, November 4, 2012 at 9:00 p.m. Newark (NJ). [Online]. Available from:
〈https://www.pseg.com/info/media/newsreleases/2012/2012-11-04b.jsp〉;
November 4, 2012.
[15] Ruggiero J. On the measurement of technical efficiency in the public sector.
Eur J Oper Res 1996;90(3):553–65.
[16] Ruggiero J. Non-discretionary inputs in data envelopment analysis. Eur J Oper
Res 1998;111(3):461–9.
[17] Seiford LM, Zhu J. Modeling undesirable factors in efficiency evaluation. Eur J
Oper Res 2002;142(1):16–20.
[18] Sueyoshi T, Goto M. Returns to scale vs. damages to scale in data envelopment
analysis: an impact of US clean air act on coal-fired power plants. Omega
2013;41(2):164–75.
[19] US Department of Energy. Office of Electricity Delivery and Energy Reliability
Emergency Situation Reports. [Internet]. Washington, DC. Available from:
〈http://www.oe.netl.doe.gov/emergency_sit_rpt.aspx〉 [cited 2015 April 21].
[20] Xu N, Guikema SD, Davidson RA, Nozick LK, ÇağnanZ, Vaziri K. Optimizing
scheduling of post-earthquake electric power restoration tasks. Earthquake
Eng Struct 2006;36(2):265–84.
[21] Yang C, Lu W-M. Assessing the performance and finding the benchmarks of
the electricity distribution districts of Taiwan Power Company. IEEE Trans
Power Syst 2006;21(2):853–61.