Hidden Failures in Protection Systems and their Impact on Wide-area Disturbances

David C. Elizondo* J. de la Ree* Arun G. Phadke* Stan Horowitz**

(Student Member) (Senior Member) (Fellow) (Fellow)

*Center for Power Engineering **Consultant

The Bradley Department of Electrical Engineering Columbus, Ohio
Virginia Tech, Blacksburg, VA 24061

Abstract: This paper explores Hidden Failures in protection
systems, which have been identified as key contributors in the
cascading of Power System wide-area disturbances. The original
definition of Hidden Failure, which is a failure that remains
undetected and is uncovered by another system eveng is included
as are the developments of Hidden Failure sequence of events and a
methodology for Hidden Failure identiilcation. This method is
based on Protection Element Functionality Defects (PEFD). Hidden
Failures Modes are developed based on the relaying
implementation for Primary protection, Back-up protection and
Special Protection Systems. Wide-area disturbances based on North
American Reliability Council (NERC) reports are analyzed and
Hidden Failures are identified employing the developed
methodology. The mechanisms in the wide-area disturbances are
summarized. Regions of Vulnerability and Areas of Consequence
are important concepts that identi& the critical elements from the
point of view of Hidden Failures.
Keywords: Hidden Failures, Blackouts, Wide-area
disturbances, Primary Protection, Back-up protection, Special
Protection Systems.
In this reference, five Special Protection Systems (SPS) are
described and analyzed for Hidden Failures as well.
Examples of such failures are included in the paper.
A number of wide-area disturbances were analyzed for
Hidden Failures based on the NERC disturbance reports [3].
A methodology for the identification of Hidden Failures on
wide-area disturbances and the mechanisms of the wide-area
disturbances were developed and are presented in this paper.
Regions of Vulnerability are identified as Power System
physical regions in which the occurrence of an event will
uncover a Hidden Failure of a nearby protection system [1].
Areas of Consequence are regions of the power system
which become vulnerable to Hidden Failures in succession
as a wide-area disturbance progresses to its final state.
Analysis of regions of Vulnerability and Areas of
Consequence represents the first step towards a possible
approach to problem solution.

I. INTRODUCTION H. HIDDEN FAILURE DEFINITION,

PHILOSOPHIES AND SEQUENCE OF EVENTS
The role of protection systems during wide-area disturbances
in power networks is critically important. Several The work done in Reference [1] defined Hidden Failure as:
disturbances have started with a single contingency caused “a permanent defect that will cause a relay or a relay system
by natural events and have ended with substantial electric to incorrectly and inappropriately remove a circuit
service interruptions and system separation. Hidden Failures element(s) as a direct consequence of another switching
on the protection systems have been found as key players in event.” Hidden Failure analyses developed in [1] are based
many of the disturbance sequence of events, and a number of on permanent defects in the form of hardware failures of
these protection system mis-operations have been cataloged protective relays. This analysis is based on relay
as main contributors to the wide-area disturbance. functionality problems from the operational point of view as
it affects their expected behavior.
The purpose of this paper is to identifi the modes in which
the protection systems may fail to operate correctly and the A Hidden Failure is a defect from which any of the
consequences of these failure modes. The identification of protection system elements may suffer and it is applicable to
the failure modes in the real wide-area disturbances is potential transformers (PT), current transformers (CT),
included. This paper emphasizes Hidden Failures on cables, lugs and connectors, all kinds of relays,
Protection Systems. These failures remain undetected and communication channels, etc. The fundamental difference is
are uncovered by the occurrence of another event in the that these defects, by themselves, will not produce ~
Power System [1]. immediate action on the part of the relays but will remain
undetected until some other system event occurs.
The modes in which the protection systems may fail- the
Hidden Failure Modes- have been included for Primary and The most peculiar and dangerous characteristic of Hidden
Back-up protection systems for generators, transformers, Failures is the fact that their effects appear when the Power
buses and transmission lines in Reference [2]. System is under stressed conditions, such as during or
immediately after faults occur, under-voltages, overloads, or

0-7803-6674-3/00/$10.00
0-7803-6672-7/01/$10.00
(C)(C)
2000
2001
IEEE
IEEE 710
 as a consequence of another switching event. The local
effect of Hidden Failures is also an important factor since
the circuits tripped by Hidden Failures are a second
contingency coupled to the first contingency.
The first step in identifying Hidden Failures is to classify
Hidden Failure Modes for each protection scheme: relay or
relay system used to protect transmission lines, generators,
buses, transformers, an Special Protection System, etc.
HIDDEN FAILURE SEQUENCE OF EVENTS
The first event in the Hidden Failure mechanism is a
Protection Element Functionality Defect (PEFD). However,
having a PEFD does not automatically result in a Hidden
Failure. In general, a PEFD occurs as a random event and as
a result protection systems are unable to perform their
designed and expected actions. Usually the defects which
concern us lead to increased insecurity. These defects can be
present on any of the protection system elements, and may
take the form of hardware failures. outdated settings and
human negligence or errors.
Some examples of PEFD are a relay’s contacts that are
always open or closed, a timer that operates instantaneously
regardless of its pre-assigned time delay, an outdated setting
in a relay, a human error in relay coordination, etc. PEFD
related to hardware failures are referred to as PEFD-A, and
PEFD related to relay settings, human errors or negligence
are defined as PEFD-B.
The logic involved around the PEFD will determine if this
first event will result in an operation due to a Hidden
Failure. It is important to note that the determining factor for
an undetected PEFD is the logic of the protection scheme. In
the following, Hidden Failure Modes are defined depending
on the logic associated with the specific protection scheme
being analyzed.
111. HIDDEN FAILURE MODES, A THEORETICAL
APPROACH
An example of these ideas is described in Figure 1, which
shows the control circuit of a three zone step distance relay.
A PEFD-A, which affects T3 contacts, will result in a
Hidden Failure Mode. It should be noted that a PEFD-A on
Z 1 will not result in a Hidden Failure Mode.
Nf’rF
\ $,,,,:-‘3
Figure 1: Hidden Failure Modes, a beginning.
0-7803-6674-3/00/$10.00
0-7803-6672-7/01/$10.00
(C)(C)
2000
2001
IEEE
IEEE
A PEFD-A, which causes the Z 1 contact to close
permanently, will not cause a Hidden Failure Mode since the
logic involved with Z 1 PEFD-A does not allow the failure to
remain undetected. At the instant when Z 1 fails closed, Zone
1 distance relay will trip the line circuit breaker. This indeed
is a relay mis-operation, but not a Hidden Failure.
If the PEFD-A takes place in T3 closing the contact
permanently, the logic will allow this PEFD-A to remain
undetected. At the instant when T3 closes its contacts
nothing will happen in terms of line trips, because both Z3
and T3 are required to close before tripping the line. Thus
PEFD-A will remain Hidden until a fault occurs within
Zones 1,2 or 3 of the transmission line. Zone 2 of the
adjacent line will also close its contact but is prevented from
tripping by virtue of the timer T2. For a zone 1 fault, the
only erroneous effect of the Hidden Failure will be to drop a
zone 3 target as well, while for zones 2 or 3 faults there will
bean instantaneous trip of the line.
In accordance with the Hidden Failure definition [1], a
“failure to operate” will not be considered to be a Hidden
Failure. In other words, if there is a PEFD on any of the
protective system elements, with the consequence of not
clearing a fault, this protection system failure will not be
classified as a Hidden Failure. This is a “failure to operate”
event, and is not considered a Hidden Failure due to the fact
that some other protection systems will react and clear the
fault. Power Systems are biased towards dependability, and
one or the other of the duplicated protection systems should
clear all faults.
The coordination of the different protective schemes on the
Power System is one of the most critical tasks. Part of the art
of protective relaying is based on the engineering judgment
of the relaying community. Relay settings are the result of
multiple Power System simulations and reflect the expected
harmony of the Power System reactions for all possible
contingencies. Human errors may occur during this process,
resulting in protective relay mis-coordination. This type of
failure is considered a Hidden Failure since its behavior falls
under the Hidden Failure definition and sequence of events.
A PEFD, such as mis-coordination, remains undetected until
another event triggers it. A typical example would be a
generator excitation limiter protection, which was,
mistakenly, set to trip the generator unit for a value under its
overall capacity. This failure will be hidden until an
increment on the Power System reactive power requirement
produces an unwanted trip of the generator. The effects of
this Hidden Failure may be serious, and will depend on the
system strength and the prevailing system conditions.
Human error also will fit under this category, where the
relay settings as made initially are appropriate in terms of
coordination, but due to changes in system conditions or
topologies they become obsolete, inadequate or even wrong.
711
 A human error related Hidden Failure was one of the initial Goshen Idaho ~ Wyoming
events registered in the famous New York 1965 blackout
%
[4].
Khrport

IV. HIDDEN FAILURES IN A RECENT WIDE- 1

AREA DISTURBANCE Borah

This section applies the concepts developed earlier to .......................................................

2, HF1
identifi Hidden Failures on a wide-area disturbance, which
IQ G
occurred on the Western Systems Coordinating Council
(WSCC) system on July 2, 1996, and which has been Jim Bridger
..........................................................
documented extensively in the literature [5]. A single phase
to ground fault at the Jim Bridger-Kinport 345-kV line
Figure 2: WSCC-07/02/96, HFl localization,
ultimately resulted in system separation and electric service
interruption to more than 2 million customers. The complete
account of this wide–area disturbance is beyond the scope of The relay involved with the Bridger-Goshen 345-kV line
this paper, but since the Hidden Failures did occur at the unwanted trip was a segregated phase comparison, solid
very beginning, a detailed explanation of the initial events is state relay. The relay had a PEFD-A, a faulty ground
included. element - local delay timer - had failed in the “closed”
position. Technical staff confirmed that Jim Bridger-Goshen
Table T- 1 shows that the first event was a phase to ground relay mis-operation is a Hidden Failure sequence of events.
fault. Twenty milliseconds afler the Bridger-Kinport 345-kV The logic of the phase comparison relay is shown in Figure
line trip (correct operation), the Bridger side of the Bridger- 3.
Goshen 345-kV line was also tripped due to a Hidden
Failure. This event is identified as Hidden Failure 1, (HFI).
0 Anrsngtnpit~
Cumrlttwuor

“TD
R?nnre
Table T- 1: WSCC-07/02/96, Sequence of initial Events. square
waves
tkxn
ColnNnimtion —
CFand semltQ Trip
Event Time Comment
Jim Bridger-
Kkrport 345-Kv,
1424:37.180
MAST
Line Sag to close to a
Tree. 4)
r-l- —
% OR

Phase to mound P
fault. - -
Jim Bridger- 1424:37.200 HF 1, Faulty ground 0 OR
Goshen 345-kV MAST element at Bridger, LlxaJSqaar?
trip, I PEFD-A. - Whves AND
RAS started, JB 1424:37.200 RAS was correct, 1040 I N
loose 2 lines. \ MAST \ MW Off.. —
Ail Generators \ Freq. Went to 59.9 Hz. I
respond to I 1- 1 Figure 3: Phase comparison relay internal logic schematic.
Generation lack...
Round Up- 1424:39.200 HF2, bad connectors in
LaGrande 345-kV MAST a distance relay, PEFD- The single phase comparison protective scheme receives as
trip. A. inputs the local and remote wave forms, which are compared
Mi[lCreek- 1425:01.052 HF3, unwanted
in order to determine if the fault is external or internal and
Antelope Trip MAST operation of Back-up
relay, PEFD-B decide if a trip signal will be sent to the circuit breaker, see
numbers 1 and 2 in Figure 3. The biggest rectangle shown in
the sketch represents an AND gate, which receives as inputs
Figure 2 shows the fwst fault on the Bridger-Kinport 345-kV three signals. One of these signals is the element with a
line (see number 1), and the circuit breaker unwanted trip PEFD-A (failed in the closed position) shown in bold, see
caused by HF 1 (see number 2). number 3. The remaining two signals are a channel security
check and an Arming Input Current Detectov see numbers 4
and 5 respectively.

A PEFD-A, the timer which failed in the closed position,

remained hidden until the fault at Bridger-Kirtport line
forced a current high enough to satisfi the condition for the
Arming Input Current Detector to operate, providing its
“positive signal” to the AND gate. Since the channel
security checks were verified, all three inputs to the AND

0-7803-6674-3/00/$10.00
0-7803-6672-7/01/$10.00
(C)(C)
2000
2001
IEEE
IEEE 712
 gate were satisfied, the Arming Input Current Detector, the conforms to the PEFD-A definition and in this case it is
sanity checks and the PEFD-A (which was already with related to the relay connectors and lugs.
“positive signal”). Consequently the relay system sent a trip
signal to the Jim Bridgercircuit breaker,and second An excerpt from [7] states “Jim Bridger Remedial Action
contingency was caused by the Hidden Failure in the Scheme should have ensured stability and prevented further
protection scheme. outages. Several near simultaneous switching events,
however, had some detrimental effects: A 230 kV line
The Jim Bridger SPS was immediately activated, due to the relayed in Eastern Oregon”. This 230 kV line is the Round
fact that Jim-Bridger generation plant had lost 2 Up – LaGrande, which was tripped due to HF2.
transmission lines. This SPS operation was appropriate and
did work as designed, disconnecting 2 units from the Jim- HF3 will be described next, which is related to the
Bridger plant. Generators from the entire WSCC MillCreek-Antelope 230 kV transmission line, where the
intercomection responded to the ffequency deviation, 59.9 MillCreek station breaker had an unwanted trip. This is a
Hz. Almost 2 seconds after the first event, another relay- Hidden Failure occurring over a Back-up protection system;
unwanted trip disconnected the Round Up-LaGrande 230 kV in fact it can be cataloged as an unwanted operation of a
transmission line. This event is cataloged as HF2. Back-up relay. An excerpt from [7] states: “Relays installed
to detect short circuits must not operate for mild overload
The defective relay was identified as an electro-mechanical and mild voltage depression”. The relay did not do anything
distance relay. Figure 4 describes HF2 sequence of events. wrong, it tripped because the Power System conditions
The relay operation is based upon a balance between changed and the apparent impedance encroached under the
operating and restraining forces created by the current and zone 3 of the distance relay. The relay reacted to the low
voltage inputs [6]. For distance relays, the restraint force apparent impedance resulting from the Power System
overcomes the operating force during out-of-zone faults. In conditions.
the present instance, corrosion under the voltage restraint
crimp-on lug produced a poor connection, reducing the The PEFD associated with HF3 is a PEFD-B. This Hidden
restraint force. In time the corrosion was complete and the Failure is related to human error in relay settings, in the
restraint force was practically eliminated and the distance sense that these Power System conditions presented on the
relay (mis)operated, closing its contacts. July 2, 1996 disturbance were not previously considered in
the contingency analysis studies. This line trip and the 300
v MW interruption caused power swings leading to rapid
overload, voltage collapse and angular instability [7].

v. MECHANISMS INTO THE WIDE-AREA

& DISTURBANCES

The mechanisms encountered in the wide-area disturbance

are summarized in Figure 5, which represent a generic

@F
protection scheme. Hidden Failures occur when this
protection scheme is “armed” by the PEFD-A on one of the
relays, closing its contacts permanently

...............
Figure 4: HF2, Sequence of Events.
Toother
‘1
logic,.,

--r “
The time when the corrosion caused the distance relay to

T
R2, with a PEFD
operate is unknown. This relay condition remained
undetected due to the fact that a fault detector supervises this
relay, i.e., some others conditions are required before
sending a tripping signal to the circuit breaker. From the
time when the distance relay was defective due to corrosion
until the time the Round Up-LaGrande 230 kV line was
4
L
T
Trip Coil

52a

tripped, the system conditions were “normal”. As mentioned

before Hidden Failures are triggered or uncovered by some
Figure 5: Mechanisms in the Wide-area Disturbances
another “event” which could be a fault, overload, under-
voltage, etc. On July 2, 1996 the system did not have normal
conditions since two lines were tripped, initiating a SPS, The application of this scheme may take the form of Primary
dropping 1040 MW of generation. HF2 was triggered by protection, Back-up protection or Special Protection
these abnormal conditions. This Hidden Failure event Systems, as can be seen in Table T- 2.

0-7803-6674-3/00/$10.00
0-7803-6672-7/01/$10.00
(C)(C)
2000
2001
IEEE
IEEE 713
 Table T-2: Mechanisms into the Wide-area Disturbances, Application, VIII. ACKNOWLEDGMENTS
Scheme R1 R2 Application This work is partially supported by a grant from EPRUDOD.
Primarv I Fault I Directional I Transmission line

Ix. REFERENCES

1. Surachet Tamrong[ak, “Atralysis of Power System Disturbances due

to Relay Hidden Failures”, Ph.D. Dissertation. Virginia Polytechnic
relay and State Universi~, Blacksburg, Virginia. March 1994,
Special I Fault I Power tlow I Generation 2. David C. Elizondo, “Hidden Failures in Protection Systems and its
I Protection I detection I relays I Reiection \ impact on wide-area disturbances”. MSEE Thesis, Virginia
System relays Polytechnic and State University, Blacksburg, Virginia. April 2000.
Special Under- Timer Under-Frequency 3. “NERC Disturbance Reports” North American Electric Reliability
I fiotection I frequency I \ load shedding I Council, New Jersey, 1986-1995.
System relay 4, “Prevention of Power Faihrres”j a report to the president by the U.S.
Specird Under- Timer Under-Voltage load Federal Power Commission, July 1967, Volumes 1-3.
Protection voltage shedding 5. WSCC, “Disturbance Report for the Power System Outages that
System relay occurred on the Western Interconnection on July 2 19961424 MAST
and July 3 1996 1403 MAST”. Approved by the W SCC Operations
Committee on September 19, 1996.
6. Stanley H. Horowitz and Arun G. Phadke “Power System Relaying”,
w. REGIONS OF VULNERABILITY AND AREAS second edition. Research Studies Press Ltd., England, and John Wiley
OF CONSEQUENCE. and Sons Inc. New York, 1995.
7. Carson W. Taylor, Dennis C, Erickson, “Recording and Analyzing the
July 2 Cascading Outage,” IEEE Computer Applications in Power.
Regions of Vulnerability and Areas of Consequence January 1997, pp. 26-30,
definitions represent the initial efforts towards the problem
solution. Regions of Vulnerability are identified as Power x. BIBLIOGRAPHIES
System physical areas in which the occurrence of an event
will uncover a Hidden Failure of a nearby protection system David C. Elizondo (Student Member) was born in Monterrey, Nuevo Leon
[1]. Areas of Consequence represent the regions of the Mexico, in 1972. He received the BSEE degree from the Instituto
power system which become vulnerable to other Hidden Tecnologico y de Estudios Superiors de Monterrey in 1994. His
professional experience took place from 1994 to 1998. His job was related
Failures following the redistribution of network flows after
to the electrical business in some Mexican cities, being involved in the
the initial contingency. Regions of Vulnerability and Areas Industry as well as Utilities. Mr. Elizondo completed his MSEE in April
of Consequence were identified for a number of NERC 2000 and he is currently enrolled in the Ph.D. program at Virginia Tech.
wide-area disturbances during the research work [2].
,Jaime De La Ree (Senior Member) was born in Hermosillo Sonora,
Mexico, in 1957. He received the BSEE degree with distinction from the
VII. CONCLUSIONS [nstituto Tecnologico y de Estudios Superiors de Monterrey in 1980. and
the MS and The Ph.D. degrees from the university of Pittsburgh, Pittsburgh,
Pennsylvania in 198 i and 1984 respectively. In 1984, he joined the faculty
This paper explored Hidden Failures in protection systems, of Virginia Polytechnic Institute and State University, Blacksburg, Virgini<
which have been identified as some of the key contributors Where is an associate professor. His research interest in the areas of power
in the degradation of Power System wide-area disturbances. systems and rotating machinery. He is a member of the IEEE, PES, IAS, as
well as Tau Beta PI and Eta Kappa Nu HonorarySocieties.
Hidden Failure definition as well as developments on
Hidden Failure sequence of events and a methodology for A.G. Phadke (Fellow) is a University Distinguished Professor at Virginia
Hidden Failure identification based on Protection Element Tech. His primary research area is the microcomputer based monitoring,
Functionality Defects (PEFD) were presented. protection, and control of power systems. He is a co-author of two books on
relay ing: Computer Reloying for Power Systems, and Power System
Relaying. He was awarded the IEEE Third Millennium Medal in 2000,
Hidden Failure Modes have been classified as PEFD-A named the Outstanding Power Engineering Educator by the IEEE in 1991,
related to hardware failures and PEFD-B related to and received the Power Engineering Educator Award of the EEI in 1986.
erroneous settings. A practical approach to such an analysis He is Chairman of the Technical Committee of USNC CIGRE’, and was
elected to the National Academy of Engineering in 1993.
was presented with the help of the widely reported event of
Juiy 2, 1996.
Stanley H. Horowik (Fellow) is a consultant, author and lecturer. Mr.
Horowitz co-authored a textbook entided, “Power System Protection”,
We have presented a technique for cataloging and analyzing edited the IEEE Pressbook, “Protective Relaying for Power Systems”,
possible Hidden Failures in the protection systems of a Volumes I and 11. He is a Life Fellow of the IEEE. He has been awarded the
PSRC Distinguished Service award and a prize paper award. He was elected
power network, and introduced concepts of regions of
to the National Academy of Engineering in 1995 and he is a recipient of the
vulnerability and regions of consequence. These analytical IEEE Third Millennium Medal. He has been the editor-in-chief of the
tools help formulate a realistic view of contingency selection Power Engineering Society magazine “Computer Applications in Power”
in an Energy Management System, and offer some pointers since 1996.

for implementing supervisory systems which would reduce

the likelihood of protection system Hidden Failures
contributing to wide-area disturbances. Additional research
in this area is continuing at this time.

0-7803-6674-3/00/$10.00
0-7803-6672-7/01/$10.00
(C)(C)
2000
2001
IEEE
IEEE 714