T

THE RETAIL INDUSTRY IN THE UNITED STATES KNOWS THAT

every year between Thanksgiving and Christmas the hustle and bustle is on
to sell one more video game or one more sweater. For the North American
electric utility industry, every year between the beginning of May and the
beginning of June the hustle and bustle is on to get one more transformer
humming or one more substation online. It should come as no surprise then
that the livening of a small transmission substation on a blue-sky Saturday
late in May of 2006 in a small town in Michigan could go unnoticed. This
substation features the use of an Ethernet local area network (LAN), reliance
on IEC 61850, the capture of nonoperational data, and a station human-
machine interface (HMI). This is pretty obviously an example of “substation
automation” and yet that term may not tell the whole story.
To help appreciate how much “substation automation” has become a part of the
lexicon, a recent Google of the phrase got the same approximate number of hits as
the phrase “protective relay” (145,000). Though the definition of “protective relay”
can be debated, the term “substation automation” can be actively deceptive. It
implies that this concept is within the walls of the substation and seems to refer to
the automatic operation of things like voltage control, load transfer, and tap
changes. The Michigan substation is meant to realize the organic implementation
of automation technologies. It reflects how a system would evolve around these
technologies as opposed to having substation automation added to a system. This
substation was an attempt at realizing the vision of automation.

Strategy and Methodology

In 2004, Michigan Electric Transmission Co. embarked on a program to devel-
op a sweeping business and technical strategy to replace the aging protection

32 IEEE power & energy magazine 1540-7977/07/$25.00©2007 IEEE may/june 2007

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 © EYEWIRE

and control equipment. This aggressive investment and replacement strategy

can be the most cost-effective solution for system-wide upgrading.
The implementation methodology, addressing impact on capital and oper-
ating costs, is described in more detail and is based on the following tasks:
✔ integrating all relaying, control, monitoring, automation, and enterprise
functions through Ethernet LANs in the substations and EPRI’s com-
mon information model (CIM)
✔ introducing IEC 61850 LAN integration system and protocol as rapidly
as is practical, to replace control wiring, and to simplify integration and
data flow
✔ organizing protective functions using the newest generations of relays to
improve dependability and security, while drastically reducing the num-
ber of units required and complying with or exceeding all agency
design requirements
✔ looking at recent operating issues, relaying problems, industry trends,
and recent wide-area system events; and designing a solution that aims
squarely at improving performance on these specifics.
Key steps to implement the standardized substation protection and control
design are being carried out. The first step was to develop a practical and inno-
vative technical strategy for system-wide wholesale upgrading, including
studying the existing system design and operating issues, recent industry
events (e.g., the August, 2003 Northeast U.S. blackout), and the design
requirements of North American Electric Reliability Council (NERC). The
strategy describes how the functions in the latest microprocessor relays can be
arrayed for the most cost-effective and fully redundant protection, while
drastically reducing the amount of equipment from what was needed with

may/june 2007 IEEE power & energy magazine 33

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 preceding designs. The strategy shows how the IEC 61850
LAN protocol is to be introduced into the system design.
Monitoring functions that feed directly into the asset manage-
ment program are included in the design.
Levels of Implementation
These key steps in the strategy can be viewed at three levels
of implementation at the substation. The first level of imple-
mentation is that the substation must be designed to operate
successfully. Recent technological advances were assessed
for the ability to be immediately and successfully used to
operate the system. The second level of implementation is to
insure reliability of the substation; how well it can continue
to operate after the loss of any component. The third level of
implementation is the visibility of the substation system to
being able to track the other two levels. For example, when
substations’ supervisory controls were limited and the pro-
tection and control system was electromechanical, the sys-
tem was adequately operable, often reliable enough, but
almost completely opaque. Presence on-site might increase
the visibility of the substation’s operations, but postevent
analysis often involved deductive reasoning based on clues
gathered, such as the overcurrent relay didn’t have a target,
the negative sequence relay had a target, and the operator
said he smelled something in the yard.
Operability
The core function of a transmission substation is to facilitate
the flow of power through a bulk electrical system. Therefore,
whatever design is used, be it fully automated or not, the
breakers have to close when power is needed to flow. Trans-
formers need to be energized when the load requires it. The
newest technologies may promise much, but if they can’t be
implemented to reliably operate a system then they shouldn’t
be called on to do so. The design philosophy for the program
was summed up by the phrase “this design needs to be lead-
ing edge, not bleeding edge.” Figure 1 summarizes the result-
ing design reflecting the strategy.
The substation operating system features an Ethernet
LAN that allows data gathering along with protection and
control commands to be exchanged. This LAN connects all
the relays with most of the other intelligent electronic devices
(IEDs) in the substation. The operating commands sent to the
yard equipment are sent over hard-wire connections. The
design for the off-site operation of yard equipment uses a cor-
porate wide-area network (WAN) via data communication
services. Operation by personnel on-site can be performed at
the HMI by mousing over the elements featured on a one-line
representation of the yard (Figure 2). Personnel can also
operate the 138-kV circuit breakers from buttons on the front
of relays (Figure 3).
Multifunction relays were used so even redundant protec-
tion could be afforded without using large amounts of panel
space. The relays, being IEDs, not only performed the crucial
function of protection but also specific features were required
to execute the design of operation. These IEDs are enabled
for IEC 61850 and featured programmable front-panel but-
tons (Figure 3) that could functionally take the place of test
switches. In addition, the design of the IEDs was flexible
enough that the core functions could be changed without
removing the relay from the panel. In other words, a trans-
former relay can become a line relay by reconfiguring the
relay and addressing the external connections; replacing the
hardware platform is not required.

Other Control
METC Enterprise Service Providers SCADA/EMS
Centers
Corporate WAN
via
Primary and Hot Standby Data
Line A Relay 1 PMU 1 Communications Services
IEC 61850 and COMTRADE/IEEE
DNP 3.0 1344

Managed Optical Ethernet Switches - LAN 1

Xfmr Relay 1 Bus Relay 1

IEC 61850 IEC 61850
and DNP 3.0 and DNP 3.0

Substation
Physical and Electrical Isolation of Redundant Protection Systems dDFR Host Routers
Automation Host

Line A Relay 2 PMU 2 Monitoing IEDs

IEC 61850 COMTRADE/IEEE Serial Comms Protocol
and DNP 3.0 1344 Local Historian
GPS Clock
Local HMI
Managed Optical Ethernet Switches - LAN 2

Xfmr Relay 2 Bus Relay 2

Connections for 1 ms
IEC 61850 IEC 61850
Time Stamp Synch
and DNP 3.0 and DNP 3.0

figure 1. New substation protection and control LAN architecture.

34 IEEE power & energy magazine may/june 2007

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 The end result of all these features was great freedom in
the physical design of the relay panels since the necessary
functions could be either hard wired, programmed into the
relay, or executed over the LAN. With this great flexibility in
physical design, standards could be created that would be
applicable to more scenarios and wouldn’t change with every
new relay feature. Add to this a standardized entrance of
cables to the control house and how the cables are run to each
panel and now standardization can go beyond the panel. The
layout of the building could be standardized based on the
future build of the site. Any changes to the bulk electrical sys-
tem that may happen in the yard (the addition of a line or
transformer or a reassigned bus position) can be accommo-
dated with minimal physical construction.
No matter how far technology can take us, it must be
assumed that people will have to be able to operate the sys-
tem on-site. To help insure that the implementation of the
strategy would be able to be operated, a human factors engi-
neering evaluation was performed. The
objective was to inform the final design
and build out of the control houses and
their control panels of any problems
with the human factors and ergonomics
of the workspace, as well as the user
interface with the control-house controls
and displays. Recommendations of the
study were incorporated into the design.
system and control-house components. The second is the
external threat to the substation instigated by forces from the
outside. The NERC identifies two types of electrical sector
threats: cyber and physical (see http://www.nerc.com/
cip.html). Cyber security addresses the attacks on the corpo-
rate WAN that pose external threats to the communication
between the bulk operating system and the substation. The
project strategy was to emphasize cyber security and design a
comprehensive on-site security system to address the external
threats to the substation’s physical plant.
Internal Threats
The goal was to maintain the reliable operation of the substa-
tion by eliminating the effects of any single credible contin-
gency on the control house. The decision was that at the
345-kV level the control systems were to be fully redundant.
At the 138-kV level the systems weren’t required to have full
redundancy, but they must have backups in place for each

Reliability
It is not enough that a substation operates
properly; it must also operate reliably
under credible contingency situations. In
the U.S.-Canada Power System Outage
Task Force report from April of 2004
titled “Final Report on the August 14,
2003 Blackout in the United States and
Canada: Causes and Recommendations,”
it was identified that one of the common
causes of the significant outages of the
last 30 years on the bulk electrical system
level was a lack of “safety nets” where figure 2. An HMI displaying the substation one-line diagram.
“A safety net is a protective scheme that
activates automatically if a pre-specified,
significant contingency occurs.” This is
an important concept at the substation
level as well and any good design needs
to address “safety nets.” There are two
ways to fulfill the requirement for ade-
quate safety nets: either install those nets
or eliminate the credibility of a signifi-
cant contingency. It must be further iden-
tified that there are two types of threats to
the substation that require safety nets.
One is the internal threat to the system
due to normal failure over time of power figure 3. Front view of an 11-1/RH30 IED.

may/june 2007 IEEE power & energy magazine 35

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 credible contingency. The reason for the difference is that a One of the early concepts behind the design was the identi-
redundant system reflects the criticality of the electrical sys- fied value of redundant battery systems. A common practice in
tem components at any one substation at this voltage. There- transmission system protection is to protect for all credible sin-
fore, a redundant system is one where even extreme gle contingencies including battery failure. Since it is common
contingencies have no effect on the operability of the substa- for a substation to have one battery, its failure would leave the
tion, whereas a backup system may have reduced operability substation unable to take any action to clear a fault condition.
under equally extreme circumstances. These redundant sys- Therefore, all remote sites have to act in place of the site with
tems can be referred to as System 1 and System 2. the failed battery. With this requirement in place, distance
relays at the remote sites (sometimes referred to as Zone 3) had
Redundancy to be set to see the other remote sites, which can be a very large
What was found during design was that the technologies setting. The operation of the Zone 3 distance function during
adopted allowed the benefits of a standardized solution for periods of high load was identified as a contributing cause for
both levels of operation to outweigh the costs of most of the more than one of the major outages covered in the U.S.–Cana-
redundancy required at 345 kV. In the world of multifunction da Power System Outage Task Force report. The redundant
protective IEDs, the marginal cost between a redundant device battery minimizes the likelihood of a single credible event dis-
and a backup device, one that may have fewer functions, is not abling all operations at a substation, removing one of the needs
significant; therefore, redundant devices were installed. Fur- for the Zone 3 distance relay to be set high.
thermore, it had been decided that the benefits of diversity of In the final design, the largest difference between the 138-
manufacturer was not significant enough to preclude evaluat- kV and the 345-kV systems is this requirement of two sepa-
ing benefits gained from using the same manufacturer on all rate batteries (Figure 4) at the higher voltage substations and
relays. So the design was free to use an identical device as the the physical separation of the redundant systems. Beyond the
redundant, driving standardization farther. need for two batteries and physical separation, the differences
between the systems using redundancy
or backup were subtle.
Since the control and protection sys-
1 in Conduit
(Weather Station)
tem relies heavily on the Ethernet LAN,
Exhaust
the failure of one of the switches or of a
Fan fiber connection must have no effect on
either the ability or speed of communica-
Eye 125 VDC Battery Bank (Half) Fan tion. To accomplish this, a design strate-
Wash 1 - 7 ft-0 in × 1 ft-81/2 in Rack Control gy similar to the redundant dc system
Weather was adopted. The implementation of
Station
redundancy can be seen in the two views
System 1 of the system architecture presented in
Battery Room Figure 1 and Figure 5, specifically in the
application of redundant LANs.
Redundancy isn’t enough, though.
Louver 125 VDC Battery Bank (Half) r You can have two eggs, but if they’re in
e
1 - 9 ft-0 in × 1 ft-81/2 in Rack at the same basket then the second egg may
He
not be worth much. An effort was made
He to have physical separation between the
125 VDC Battery Bank (Half) at
Louver er redundant elements. In this design,
1 - 9 ft-0 in × 1 ft-81/2 in Rack
redundant relays have a 6-ft aisle
between them. The two batteries are not
only in two different rooms but there are
System 2
Battery Room two battery chargers and two tray sys-
tems for getting cables from the batteries
to the relay panels. The redundant Ether-
Fan net switches are also separated by an
Eye 125 VDC Battery Bank (Half) Control
Wash 1 - 7 ft-0 in × 1 ft-81/2 in Rack aisle. However, whereas pains were taken
to eliminate the close proximity of wires
from System 1 to wires from System 2, it
Exhaust
Fan was recognized that the communication
infrastructure had to be different. LAN 1
figure 4. Detail of the building layout showing two batteries. needs to know the status of LAN 2 and

36 IEEE power & energy magazine may/june 2007

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
vice versa. Therefore, the design required a physical connec-
tion between the two LANs.
This application of physical separation was extended to the
termination cabinet design. The termination cabinet is the
point where all the cables come in from the yard that are
assigned to a system. This is a large wall-mounted box filled
with columns of terminal blocks. It was observed that the
effects of the failure of any wire termination followed by a fire
could wipe out an entire system. To address this, the physical
design of the box was revised to include metal plates between
the columns of terminal blocks. The blocks themselves were
mounted on plates that raised them from the back of the box.
This allowed the physical access to the blocks that the separat-
ing plates took away. The results are that the separating plates
will limit the effect that the heat and smoke of a fire at the ter-
minal blocks has on adjacent columns of blocks. Figure 6 is a
close shot of terminal block in the termination cabinet. As
you’ll notice, there is space on either side of the block closer
to the back wall of the cabinet. The separating plate is visible
as is a portion of a second block on the other side of the plate.
The pursuit of reliability has led us to the following: redun-
dant dc systems: redundant relays and redundant communica-
tions. Combine these with the prevalence of redundant trip
coils on transmission system circuit breakers and the need for
physical separation between redundant elements and a virtual-
ly complete System 1/System 2 approach is a natural out-
growth. This redundancy is complete to the point that the
entire System 1 could theoretically be taken out of service and
the bulk power system could still be operated through System
2 at no loss of efficacy or speed. Redundancy not only brings
the system to a high level of reliability but also further enables
the modularity of design and increases the benefits of stan-
dardization. A line panel on System 1 is identical to a line
panel on System 2. The panel line up for System 1 reflects the
panel line up for System 2. The termination cabinet where the
System 1 cables come into the control house can be nearly
identical to the termination cabinet for System 2. This concept
simplifies the design efforts significantly.
External Threats: Cyber Security Plans for the Project
The many aspects and dimensions of cyber security for a
project like this are like the multiple ugly heads of the
mythical monster called the Hydra. Worst of all, when you
think you’ve dispatched one, another one grows to take its

Existing Control Building Master Site

Control Control Secure Server
Center 1 Center 2 Server Runtime

Modem Modem

Modem Modem Wireless MPLS

Splitter Splitter
Backup Network MQTT

Local Server Substation

Surveillance Runtime
Existing RTU Monitor
Inverter
New Gateway
Substation RTU
No MISC No
Switch (Security
System
HMI No Alarms)

Switch Switch Switch

Server
L7 Relay L4 Relay L1 Relay
T1 Relay
DVMre Receiver PLC PLC

T1 Relay L7 Relay L4 Relay L1 Relay

PLC PLC
Transformer
Monitors
Red Bus B kr Relay L8 Relay L5 Relay L2 Relay
GPS Clock Relay PLC PLC

Pots B kr Relay L5 Relay L2 Relay

T2 Relay L8 Relay
PLC PLC

Phone
Legend Switch B kr Relay L3 Relay
T2 Relay
B kr Relay B kr Relay PLC

Copper Telephone Blue Bus B kr Relay L3 Relay

100 Base FX B kr Relay B kr Relay
10/100 Base TX Relay
10/100 Base FL

345 kV 138 kV

figure 5. System architecture with greater connection detail.

may/june 2007 IEEE power & energy magazine 37

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 place. Some of the ugly heads include security management layer, closing down all ports is a real pain operationally. It
practices, access control systems, network and telecommu- virtually guarantees that any new application won’t run until
nications security, security architecture, encryption, applica- that port/service is enabled. However, cyber security was
tion security, and physical security to name a few. never about making life easy but rather minimizing risk, so
This section will focus on the network/telecommunica- “no pain no gain.” Recognize that once all the applications
tions security and the overall security architecture. Figure are implemented and the firewall is properly configured,
5 gives you an overall view of the architecture of the proj- there are numerous “holes” punched through the firewall,
ect. As with most cyber security architectures, much of and hence the notion of a wall of Swiss cheese. Typically,
the “defense in depth” comes from the multiple security the firewalls are used to implement several additional layers
levels or zones. The most secure area of the architecture of security. First, the “real time” systems are fire walled off
is the local substation network that connects directly to from the corporate network and the corporate network is fire
the relays. As designed, each relay has its own IP switch walled off from the Internet. One of the key techniques to
connection. Not only does this eliminate any potential support this isolation is the use of dedicated/fixed IP
collisions but these switches have media access control addresses for all servers. This allows firewall rules to be
(MAC) address filtering capability, thereby adding anoth- written that expressly allow traffic to and from a specific
er level of security in the overall architecture. The relays server on a specific port. Corporate applications traffic such
have multiple levels of passwords and audit logging to as e-mail, file sharing, terminal server, etc., are prohibited
insure access only by authorized personnel. The next level from entering the “real-time/substation network.” Most of
up includes the computer that is responsible for scanning the real-time data are stored in relational or specialized time-
the relays and then reporting the results “to the world.” series databases, thereby further isolating the substation
This security layer isolates the relays from communicat- LAN from direct contact with the “outside” world.
ing with multiple clients, leaving them free to do their job Finally, we get to the “dreaded” Internet with all its virus-
of protecting the grid. es and cyber threats. At a minimum, only those servers that
So how does the information go from “the IEDs to the absolutely need to serve Web traffic have a connection to the
Boardroom”? Well, enter the multiprotocol label switching outside world. Also, where possible, we’ve established a
(MPLS) network. Since a corporate SONET network was not “DMZ” (demilitarized zone, a semiprotected LAN segment)
available throughout Michigan, the use of a public/private where the Web servers operate, communicating through one
network is the next logical alternative. Enter MPLS or net- of the firewall ports back to the corporate servers.
works that have now become a fundamental building block This design provides layer upon layer of security. One
used by many of the large Internet service providers as their more step in keeping the ugly Hydra/cyber security heads
backbone. The key to the security of these networks is that dispatched: AUDIT, AUDIT, AUDIT! No matter how many
the entire IP address space is available to each client that sub- sleepless nights you have spent designing the most bullet-
scribes to the network service and that the core network rout- proof architecture, you don’t really know until you bring in
ing protocols are completely invisible the “white hats” (friendly hackers)
to the client and visa versa. MPLS to attack the network and look for
networks have quietly been providing vulnerabilities on how good your
IP connectivity for several years to design really is. A key part of the
business-critical applications. The plan is to perform these audits using
next layer in the security architecture an independent firm and one that is
are the boundary routers that are pro- familiar with recent work going on
grammed to route only specific IP at the national labs. In particular, the
addresses from one place to another. Department of Energy jointly estab-
This network connects all the substa- lished the National Supervisory
tions in a many-to-many communica- Control and Data Acquisition
tions network that includes the (SCADA) Test Bed program at
corporate data center. Idaho National Laboratory and San-
We now enter the corporate data dia National Laboratory.
center (Figure 7) and the notorious
corporate firewall or, more appropri- Physical Security
ately, the corporate wall of Swiss Among the requirements NERC sets
cheese. The classic firewall model is in CIP-006, which addresses physical
to close all ports until it is demon- security, are physical access controls,
strated that there is a need to have the monitoring physical access, and log-
port open. While this is a good strate- ging physical access. The substation
gy and represents our next security figure 6. Detail of a termination cabinet. design leveraged technology to effec-

38 IEEE power & energy magazine may/june 2007

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 ICCP Applications Database FEP
RTU
Server Server Server Server Modem
Developer Operator
Workstation Workstation
Modem
Radio RTU

Switch
RTU
SCADA LAN
Historian
ICCP WEB Database Remote
Server Server Server DMZ Workstation
Firewall

Internet
Switch Firewall
DMZ LAN

DMZ
Corporate Firewall
Workstation
Internet

Internet
Switch Firewall
Corporate LAN

figure 7. Pathways between the IEDs and the boardroom.

tively address these requirements. Critical control houses are The cameras installed for security purposes also pro-
built with a card reader on the doors with the intent that the vide literal visibility of the operation of the substation.
control of the physical access can both monitor and log that The implementation of the cameras includes presets for
access. Also at critical substations, perimeter cameras with camera position on substation events. For example, if a
motion sensing are deployed along the fence line. The motion transformer relay operates on a fault, nearby cameras will
sensors catch suspected intruders and the cameras swing to swing to the preset position pointed at the transformer.
monitor and log the access. Logging is done by recording the Now when remote personnel access the video images, the
images to an on-site digital video recorder. Long-term storage camera is already in place to survey the equipment in dis-
of the images recorded is done as needed. tress in real time or retrieve the most recently recorded
video. The cameras can also be manipulated remotely to
Visibility
Of the three levels of implementation discussed here, techno-
logical advances in the digital realm have most benefited visi-
bility. The same device that protects equipment can monitor
that equipment at no extra cost. It is now much easier to com-
municate waves of instantaneous data along with video
images, oscillography files, and event logs.
The first requirement for visibility is that remote operat-
ing personnel have the real-time data at hand on which to
base informed decisions. Another benefit of redundancy and
the reliance on IEDs for system data is that if the relay that
is the source of current and voltage information fails, then
the redundant device automatically takes on the duty of pro-
viding the data. Likewise, if the device used to operate a
breaker fails, its redundant partner takes over. This provides
a nearly redundant SCADA system. figure 8. Records and events collected from external sources.

may/june 2007 IEEE power & energy magazine 39

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 The design philosophy for the program
was summed up by the phrase “this design
needs to be leading edge, not bleeding edge.”
check the surrounding area for indications of why the trip
took place.
One of the first devices that greatly improved visibility
in substations was the digital fault recorder (DFR). DFRs
are stand-alone devices connected to representative inputs
that provide crucial information regarding system condi-
tions during events. The project design strategy has an
alternate approach. Since the relays installed already
record oscillographic data, a computer was installed dedi-
cated to the retrieval of these files from all the relays.
These files were then stored by time. As more substations
come online, the implementation plan for these computers
was to upload the oscillography to a central server organ-
ized by substation. Figure 8 is a screen shot showing
events gathered from numerous sources arranged chrono-
logically. Office analysis of system-wide events could now
happen within minutes of the event.
Digital data acquisition affords a level of visibility never
considered possible in the electromechanical era of substa-
tions. Exhaustive routine maintenance of every connection
between relays and auxiliary relays was a requirement
because this was the only way the failure of wires or their
terminations could be discovered. With the adoption of IEC
61850 for the delivery of protection related commands,
every data connection between relays is under constant
scrutiny. If, for whatever contingency, that protection func-
tion is unavailable, not only is there a redundant function in
place but also this failed state is alarmed immediately. This
is the equivalent of having every wire tested every few
minutes in an older substation.
BKR FAIL 1 TRIP OP 30H9BFR ON (VO39)
CONTROL PUSHBUTTON 2 ON
116
AND
LATCH 1 ON
BFR RST On (VI21)
117
AND
LATCH 1 OFF
figure 9. Relay software-generated logic diagram.
40 IEEE power & energy magazine
Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
With the dependence on IEDs for protection, control,
and communication, it is crucial that the IEDs themselves
are visible. As described, the design of the project reduces
physical installation. Auxiliary relays have been eliminated,
test switches are nearly extinct, and instead of hundreds of
wires strung between relays, there are now two pairs of
fibers from each relay to Ethernet switches. However, with
the elimination of the physical comes the proliferation of
the digital. What was once communicated with detailed dc
schematics and recorded relay settings now must be accom-
plished with settings files and logic diagrams. And control
of the configuration of the relays is critical to the reliable
operation of the substation. These newer functions are being
supported by IED manufacturers’ efforts to make the work-
ings of multifunction microprocessor relays increasingly
visible. The IEDs used in these substations have software
available that automatically converts relay configuration
into logic diagrams and easily understandable settings
reports. The software will also document inter-relay rela-
tionships, reducing the time spent on documentation. Figure
9 shows a small portion of the logic diagram representation
of the configuration for one of the relays.
Even though all this visibility exists, it is at a resolution
that is not easily understood. It would be an input overload
if someone were trying to assess the data in a real-time
manner. So once you access the information, it must be
stored for later analysis. It is at this point that the invest-
ment in automation really comes through and the vision of
automation is realized. The next step is to convert the data
into business intelligence.
118
OR 30H9BFR OFF (VO40)
may/june 2007
 It is not enough that a substation operates properly;
it must also operate reliably under credible
contingency situations.

Data Warehousing
and Information Access
The newest relays and communications systems selected for
the project present the enterprise with a massive stream of
substation data that must be automatically stored, managed,
analyzed, and presented in useful forms that improve busi-
ness and technical operations. The data warehousing fea-
tures of modern information architectures are essential to
provide end users with easy access to the wealth of data and
information substation devices provide. Three types of data
are being created and stored for later retrieval as needed:
✔ sequence of events records
✔ high-speed time-series data records such as COM-
TRADE oscillography and phasor measurement files,
including binary status such as relay trip and close or
unit line protection communications signals
✔ analog power system measurements and reports from
equipment monitors such as transformer analysis IEDs.
All the data are collected, organized, and archived at the
data-hosting center using the modeling standard CIM and
providing easy access by the staff. Today, CIM is embodied
within IEC standards 61968 and 61970 and the project is
benefiting from these standards through its use of readily
available adapters that can be used to rapidly integrate data
from various applications. Various diagnostic tools, including
automatic preprocessing and dashboard reporting, are being
developed to aid the end user in analyzing this wealth of
information. In conjunction with the upgrade project is an

Utility Databases Analytics Stack

Financial Optimization
DB's
Computations
Event
DB Dashboard
Sources:
Performance Tools
Staff
Contractors DB OLAP and
Suppliers GIS Hypercubes
ETL

Customers DB CIM Custom

CMMS Data Real-Time
Field Force Analytics Notification
DB Warehouse Event
Regulators Server
EMS Analyzers Data Miner
DB

Data
Historian IEC 61968/61970 Compliant Middleware

Monitor Monitor
Points Events
Remote Asset Expert Portal Server Application
Calc Calc
Monitoring Grid Server
Points Events
Tools Analyzers Portlets
Substation Devices
Utility Real Time Data

Web Dashboards,
Pages Scorecards,
OLAP, Reports

figure 10. The basic architecture of the decision support system.

may/june 2007 IEEE power & energy magazine 41

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 Redundancy not only brings the system to a high level of
reliability but also further enables the modularity of design
and increases the benefits of standardization.
integral effort to develop information systems for the staff.
These systems will benefit asset management, system plan-
ning, and operations.
Information dashboards that support each of these busi-
ness areas are currently under development. For example,
operations will have ready access to fault location informa-
tion, SER, lightning strikes, transformer monitors, weather
and video streams, and operating performance information.
Planners will benefit with more accurate system and device
loading information, better system event capture, weather
profiles, and improved system models.
Asset management will have access to an immense
array of information that will be distilled from the
detailed operating data. For example, consider trans-
former monitoring and life management. The transformer
relays report currents and voltages. Oil condition is moni-
tored with installed gas-in-oil detectors. Top-oil tempera-
ture sensors and accessory alarms connect to data-
collection IEDs. A weather station reports ambient condi-
tions. This body of data can support life assessment and
emergency operating decisions. To get these results, the
key data is extracted and analyzed with modeling algo-
rithms and stored as trend results for each transformer.
Self-organizing neural networks preprocess the vast
amounts of operational data for the operating and mainte-
nance management personnel who get prioritized succinct
information on which they can act, quickly if needed.
Also, data or alarms indicating maintenance problems or
repair issues can act as triggers. These can originate in the
substation or with back-office processing functions. These
drive notices to business partners; create work orders and
status tracking, map issues to geographic information sys-
tems, update asset management records, and search for
patterns or issues requiring broad action.
Converting Data into
Business Intelligence
Recently there have been a number of initiatives in the U.S.
power industry around the notion of an intelligent grid.
Along these lines, one of the core elements of this project
is to capture all of the data available at the substation and
stream it to a central location for decision making as well
as operational support.
The substation data sources are the following:
✔ advanced IEDs for circuit breaker operation
✔ digital fault recorder
42 IEEE power & energy magazine
Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
✔ transformer monitoring systems
✔ security system
✔ weather station (not shown on the diagram)
✔ phasor measurement units (PMUs)
The automated substations contain distributed data his-
torian servers that collect data from the substation data
collection system (DCS) and send them to the central data
historian server. Unlike a relational database, data histori-
ans provide an efficient means of storing temporal data
(time-series data) using various algorithms to essentially
compress the data. One needs to use caution in their
selection of historian vendors to assure that the compres-
sion method is adequate for the intended use. We chose to
use a vendor that implements a lossless method. The com-
munications network and quality of service controls are
used to prioritize data traffic from the substations, with
SCADA having the highest priority. The data historian
real-time service receives IED data from the DCS and for-
wards them to the historian server. This is the means by
which real-time data are made available to the centralized
data warehouse providing a means for business analytics
to be performed.
The project identified four levels of analytics:
Level 1 Simple thresholds and alarms: monitoring trans-
former oil temperature.
Level 2 Financial trends, basic system performance met-
rics: budget versus actual, TSAIDI, TSAIFI, etc.
Level 3 Real-time event analysis, interpretation of event
sequences: diagnosing circuit breaker failure
modes from DFR waveform data.
Level 4 Analytics for optimization purposes: prioritiza-
tion of asset maintenance, asset replacement.
The decision support system (Figure 10) implements the
data integration, analytics, and information distribution
functions. The central data repository is a data warehouse
that is structured in compliance with the CIM for utilities.
Use of CIM is central to the concept of open standards.
Information is disseminated from the data warehouse and
analytics stack throughout the enterprise via Web services
and portals.
Figure 11 shows how the analytics provide decision sup-
port for operations and business functions. The Level 1 ana-
lytics (parameter thresholds and notifications) are quite
extensive and require sophisticated management to permit
each authorized user to subscribe to only those notifications
that are of interest in the user’s job role. Any authorized user,
may/june 2007
from maintenance engineer up to chief operations officer, can
subscribe to notifications as desired. This is likewise true for
higher-level analytics, key performance indicator dashboards,
and decision support analyses.
The CIM data warehouse and data historian jointly sup-
ply the data to drive a variety of analytics. The project has
defined a large set of analytics, including 19 system per-
formance metrics and many financial and operational
measures. The analytics architecture supports both opera-
tional metrics and business key performance indicators and
each person in the organization can receive the relevant
analytics and support data and can customize his or her
portal to show preferred information in preferred locations
and formats.
Through careful consideration of the relevant key busi-
ness drivers, the project arrived at a suitable intelligent
grid strategy. Having created a business model that relies
heavily upon outsourcing, advanced automation, and the
use of analytics to support operational and business deci-
sions, the team developed an architecture that uses tech-
nology to support both the outsourcing strategy and
minimal staffing by making maximum use of information
sources and tools.
The business case for this approach shows the value of
advanced automation and decision support tools to be con-
tained in the following:
✔ reduced operation and maintenance expenditures
✔ reduced capital expenditures
✔ low staffing requirements
✔ increased transmission system reliability
✔ preservation of the value of infrastructure through use
of open standards.
Realization of these benefits is achieved by maintaining
a low headcount, using remote monitoring to reduce field
manpower through reduction of both scheduled and
unscheduled visits to their widespread collection of substa-
tions, reducing CAPEX and operation and maintenance
costs through improved information-based asset manage-
ment and reliability-centered maintenance, and using open
standards to guide the selection of equipment, systems, and
architectures that minimize the future impact of changes in
any one system, component, or supplier.

Utility Databases
Financial
DB's Grid Meta Data

Performance
DB's
Source:
Staff Maintenance
Contractors DB Strategic Functions
Suppliers Analytics L.2 Dashboards,
Customers GIS OLAP, Scorecards, Asset Life Cycle Management
DB Grid Expansion Planning
ETL

Field Force Simple Meterics Cube Views,

CIM Reports System Performance Analysis
Regulators CMMS Constraints
Data CAPEX Optimization
DB Warehouse Analytics L.4 Budget Manpower
Asset Cash Flow Regulatory
Fault Mitigation Planning
Incident Data Mining, Clustering,
Normalized Regional Inter-Department Performance Metric Improvement
Models
DB Regression, CART, Post-Fault Analysis
Model Construction

Optimization Tools
Solutions: Operational Functions
Data Integer Programming ML Estimators and Classifiers
Prioritization,
Historian Integral Maximization, Linear and Nonlinear Programming,
Subsetting Work Management
Analytics L.1 Dynamic Programming, ACO/PSO, Simulated Annealing,
Search Techniques Predictive Maintenance
Thresholds Real Time Event Interpretation
and Grid Control
Alarms Asset Utilization Optimization
Monitor Monitor Event-Based Maintenance
Points Events Analytics L.3

Real Time Event Analysis

Calc Calc Advanced Diagnostics
Points Events Utility Front and Back
Office Functions

Utility Real Time Data

Implement Decisions and Control

Transform Data into Information

Collect Low Level Data and Events

figure 11. Analytics support for business functions.

may/june 2007 IEEE power & energy magazine 43

Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
 The many aspects and dimensions of cyber security
for a project like this are like the multiple ugly heads of
the mythical monster called the Hydra.
Future Sight
Investing in a robust communication infrastructure, a nimble
LAN-based operating system and the computing power of
IEDs put into place a system that is flexible enough to be
enhanced in the future, especially in the area of visibility.
PMUs are part of an emerging technology. The project
implementation included the installation of PMUs on all
345-kV buses. This arrangement provided for monitoring of
all lines and transformers associated with the station.
Improved visibility and operator situational analysis were
key factors in implementing this technology. One of the key
findings of the August 2003 blackout was the lack of opera-
tor awareness during the time leading up to the blackout.
PMUs offer substantially improved intelligence not only for
real-time operations but also for postevent analysis and sys-
tem model validation and more.
Farther down the path of substation automation is the
use of IEC 61850 GOOSE messages over the corporate
WAN to other substations. If this method doesn’t reach the
speed of communication over power line carrier or fiber
communications for carrying pilot protection data, then it
might be used as a backup. This might also be a way to
perform inter-substation remedial action schemes and wide
area protection or monitoring schemes.
The Vision of Substation Automation
Recognizing the limitations of the term “substation automa-
tion,” the concept it represents is realizing all the benefits
that digital technology can bring to the substation. The
overall strategy is not only to automate substations but to
optimize them. Substation automation is not only a protec-
tion issue, it’s not a metering issue, and neither is it a super-
visory control or a data acquisition issue. The vision of
substation automation is not only system wide but system
deep. In this case, it not only runs from the Mackinac
Bridge to the “wrist” of the Michigan “mitten,” substation
automation affects the system from the terminal block to the
345-kV circuit breaker. That three-breaker substation east of
the Michigan Dunes is the beginning of the realization of
the vision of substation automation.
Acknowledgments
IEC is a registered trademark of Commission Electrotech-
nique Internationale. Google is a registered trademark of
Google Technology, Inc. SONET is a registered trademark of
SONAT, Inc.
44 IEEE power & energy magazine
Authorized licensed use limited to: IEEE Xplore. Downloaded on January 21, 2009 at 09:13 from IEEE Xplore. Restrictions apply.
For Further Reading
R. Brantley, K. Donahoe, J. Theron, and E. Udren, “The
application of IEC 61850 to replace auxiliary devices includ-
ing lockout relays,” presented at the 60th Annual Georgia
Tech Protective Relaying Conference, Apr. 2006.
U.S.-Canada Power System Outage Task Force, “Final
report on the August 14, 2003 blackout in the United States
and Canada: Causes and recommendations,” Apr. 2004
[Online]. Available: http://www.nerc.com.
R. Krutz and R. Vines, The CISSP Prep Guide: Mastering
the CISSP and ISSEP Exams, 2nd ed. New York: Wiley, Apr.
2004.
Biographies
Paul Myrda has 30 years of experience in electrical power
systems engineering. Most recently he was director of opera-
tions and chief technologist for Trans-Elect Inc. He was
instrumental in developing an overarching strategy in asset
management and championed an innovative protection and
control system upgrade project for the Michigan Electric
Transmission Company, a former affiliate of Trans-Elect. This
project fully leveraged the capability of IEC 61850-based
microprocessor relays, physical security, telecommunications,
and data warehousing technologies using EPRI’s common
information model. His diverse background includes planning,
engineering, information systems, and project management.
He has an M.B.A. from Kellogg Graduate School of Manage-
ment and an M.S.E.E. and a B.S.E.E. from Illinois Institute of
Technology. He is a licensed professional engineer in Illinois,
a member of CIGRE, and a Senior Member of the IEEE.
Kevin Donahoe has spent the last 25 years working in the
electric utility industry. The last 22 of those years have been
spent testing, installing, trouble shooting, specifying, setting,
estimating, designing, reviewing, documenting, and setting
standards for protection and control schemes. He spent
20 years with Commonwealth Edison, an Exelon company,
before moving to GE Energy. Though the majority of his
experience has been with transmission and distribution sub-
stations, he has significant experience with generation protec-
tion and distribution protection with specific experience with
interconnection requirements. He received his B.S.E.E. from
the Illinois Institute of Technology in 1981 and in 1993
received an M.B.A. from Lewis University. Donahoe is a
member of the IEEE Power System Relaying Committee and
the IEEE Standards Advisory. He is a licensed professional
engineer in Illinois, Oklahoma, and Michigan. p&e
may/june 2007