Article

Effective and Efficient Protection Testing

What does it mean, and is it measurable?

Author | Michael Albert

 When testing protection relays, procedures that once were effective and efficient aren’t anymore. A brief look
at the technical development within the field of protection testing over the last 30 years makes this painfully
clear. Which begs the question: What does effective and efficient protection testing look like today, and how
can it be measured?

Let’s take a glance at the past. In the pre-digital era, protection devices were set up with only a few
parameters (Figure 1). These parameters could be tested easily, while the results were documented by
hand. Then, incredible developments in the field of microelectronics completely changed the game. Over
roughly 30 years, ever-new advancements entered our daily lives. It was only a matter of time until they also
became present in the assets of our power grids and ultimately arrived in protection relays.

Figure 1: Circuit diagram of an electromechanical protection device, intervention by means of "parameters".

Electromechanical and static devices’ parameters were limited, making it possible to test them all. The first
generation of digital devices only had a few additional parameters. So, nothing really changed, as checking
every parameter during commissioning or routine testing was still possible. But soon, new technologies and
the possibilities they offered to protection engineers led to many parameters that could no longer be tested
in the field.

Therefore, sending a test engineer into the field to test a substation’s protection devices is no longer
sufficient. Today, there are far too many things to do before testing and to recognize during a test.
Preparation is vital nowadays and has the most significant potential for efficiency gains.

Before we dive in further, one more thing needs to be defined clearly – the scope of the term protection
testing. This article not only considers a single protection testing task but instead tries to cover the long-term
efforts and potential connected to testing assets within substations over their entire lifespan to create a more
realistic and comparable picture of protection testing. For that reason, the term includes testing strategies
and the full scope of the protection testing task, from preparation to execution to documentation, when
considering effectiveness and efficiency.

© OMICRON 2023 Page 2 of 3

1 Ensuring Effectiveness

As mentioned above, testing all the parameters of a modern protection relay in the field is no longer feasible
for testing engineers. Before they even start testing, they need to know which parameters must be tested to
be effective. An effective protection test requires procedures that contain the necessary steps – nothing
more, nothing less. In simple terms, effectiveness can be described as “Doing the right things.”

As each test has its purpose, not every test needs to be performed regularly. Some tests are only relevant
for a specific phase in the life cycle of a protection device. In contrast, others should be performed more
frequently to ensure the reliable operation of the relay.

Regardless of the test’s purpose, before a test case can be run, it must be defined and prepared
accordingly, as seen in Figure 2. All these tests will be effective if performed correctly, but is this approach
also efficient?

Figure 2: Test purpose vs. product life-cycle

2 Improving Efficiency
Efficiency depends on various factors, from the testing approach, state-of-the-art tools supporting it, and the
test engineer’s experience. However, all of them are subject to change. On the one hand, test objects are
advancing and use new technologies. On the other hand, protection testing solutions add new features, and
test engineers are becoming more experienced. Of course, they could also miss out on catching up on new
developments. These factors must be considered when trying to “Do the right things correctly” which is a
basic definition of efficiency. Or, to say it another way – executing the right test procedures, without
mistakes, in the shortest amount of time.

The two most significant factors for efficiency are usually standardization combined with automation. Once
the most effective testing approach is found, further efficiency gains are highly dependent on eliminating
manual interventions in the testing process. While this clearly can be achieved with automation, the level of
automation is essential. This is where standardization comes into play, as it maximizes automation within or
even across testing approaches. Generally, the more standardized an environment is, the more automation
is possible. Whereas single-function tests can be automated relatively easily, a complete test procedure for
a relay is usually done in parts depending on standardization efforts.

© OMICRON 2023 Page 3 of 4

 State-of-the-art tools allow you to combine automated test steps in templates flexibly, capture the results
automatically, and create required legal documentation from them. This is important to mention, as a clear
scope is needed for effectiveness and efficiency to be able to measure it. Preparation and documentation
are part of the protection testing task and must be included to cover the entire picture. However, efficiency
gains are not the only reason automated testing based on pre-defined templates should be favored because
there’s also test quality.

3 Quality aspects of protection testing

Quality is closely connected to the effectiveness of a test. Similar to efficiency, the quality of a test can have
different aspects, which is why we first need to define the properties of a high-quality test. These are some
of the vital quality characteristics of a protection test:
• Test depth: This measurement indicates how thoroughly specific tests are executed, e.g., how many
test shots are performed to derive the characteristic curve of a time overcurrent protection. In-depth
testing positively impacts reliability but can negatively impact efficiency if overdone.
• Test repeatability: It lets you know if a test can easily be repeated with the exact same test values.
This is a prerequisite for high-level testing quality and the basis for a structured analysis of protection
devices’ misbehavior. In addition, test results can easily be reproduced comprehensibly. When
deriving mean values of trip times, relay times, etc., repeatability is also necessary.
• Test coverage: Test coverage indicates how many existing parameters are verified with the test setup
in a settings-based testing approach. For a system-based approach, this characteristic indicates which
parts of the protection system were included in the test. In general, the value of a system test is higher,
as it verifies if the entered parameters match the protection concept and if they are suitable to fulfill
the desired behavior.
• Test performance: If the testing time required for a specific test or sequence of tests can be reduced
while covering the same scope, the value of the test performance increases. Short testing times create
capacities to increase test depth, reduce costs or integrate a wider range of other work tasks. They
may also allow to fulfill the tremendously increasing challenges and workloads brought about by the
energy transition.

Other quality characteristics may also be relevant depending on the purpose of the test, and their
importance can be assessed differently from company to company.

We've now discussed the essential aspects necessary for taking a closer look at key performance indicators
and to answer the question: How can protection testing efficiency be measured?

4 Key Performance Indicators for protection testing

Many discussions about protection testing usually focus on the best way to perform tests and argue either
for automated testing or manual approaches. However, all these arguments and claims only have limited
value if not backed by data. Unfortunately, most of these discussions don’t cover how efficiency can be
described or how it should be measured, let alone discuss any key performance indicators. Nonetheless,
they are the relevant means for assessing different approaches.

“A performance indicator or key performance indicator (KPI) is a type of performance measurement.

KPIs evaluate the success of an organization or a particular activity (such as projects, programs, products,
and other initiatives) in which it engages.” [1].

© OMICRON 2023 Page 4 of 5

 These KPIs must be identified and clearly defined before assessing a protection testing process. If suitable
measures can’t be found to evaluate the overall process, subprocesses or workflows can be evaluated
instead. Here are some dimensions that are usually a source for relevant KPIs:
• Quantity of tests (e.g. number of test shots)
• Quality of tests (e.g. test coverage, test depth)
• Duration (e.g. overall test time)
• Efficiency (e.g. time, costs)
• Costs
• …
The test process must be analyzed to define the exact metrics and KPIs that suit a dedicated company. The
following chapters will cover some examples of metrics and KPIs that could be used to evaluate particular
aspects of a protection testing process:
• Test coverage
• Test repeatability
• Performance of the test execution

4.1 Test coverage

The test coverage for a test procedure can be used as a KPI as follows:

𝐾𝐾𝐾𝐾𝐾𝐾𝑐𝑐𝑐𝑐𝑐𝑐 = (No. of changeable parameters)–

(No. of tested parameters)

Obviously, the KPIcov = 0 is optimal, as zero is the highest reachable value for this KPI. This easy example
shows that having clear references for assessing the measured KPI is essential.
Various steps can be implemented to reach the optimal KPI for the testing process, e.g.:
• Introducing test specifications for each test that must be fulfilled
• Standardizing protection testing tasks by specifying the entire testing procedure to avoid missing steps
• Analyzing the test specifications regularly to monitor the test coverage and minimize system errors

4.2 Test repeatability

Another pillar of high quality in protection testing is the repeatability of the test procedure.
The KPI for repeatability can be defined in a similar way:

𝐾𝐾𝐾𝐾𝐾𝐾𝑟𝑟𝑟𝑟𝑟𝑟𝑟𝑟 = � test values for test #1 –

� test values for test #2

The optimum state for this KPI is also zero. If the KPI is greater than zero, there are two options for
improving the performance of quality aspect repeatability:
• Using test plans to ensure the same test values are used every time the test is performed
• Switching from manual testing to automatic testing

4.3 KPI for test performance and cost efficiency

Measuring the test duration is one of the most significant KPIs directly related to costs. It can be measured
easily and is highly relevant for almost all utilities, e.g., cost reduction targets usually drive process
optimizations. In the example below, we assume that the quality of the work in the protection testing field
and the utilization of the personnel is at a consistently high level (value >90%). That means the work is being
done effectively – tasks are done correctly, and the quality is high. But is this process time efficient?

The following considerations focus on the time needed to fulfill all relevant test tasks. The KPI for testing
process efficiency can be defined as:

© OMICRON 2023 Page 5 of 6

 𝑇𝑇𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 + 𝑇𝑇𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡
𝐾𝐾𝐾𝐾𝐾𝐾𝑒𝑒𝑒𝑒𝑒𝑒 =
𝑇𝑇𝑚𝑚𝑚𝑚𝑚𝑚
Where:
- Tpreparation: overall preparation time
- Ttest: overall test time, including documentation
- Tmin: minimum possible time for preparation and test

Although the minimum possible time is unknown in this equation, we know that the optimal state of the
KPIeff = 1. Therefore, this KPI cannot be used to measure the performance of a specific testing task, but it
can be considered for measuring changes in the testing process. Let’s assume a utility performs manual
testing and wants to change to an automated testing approach. The KPIs for these two testing methods are:

𝑇𝑇𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 + 𝑇𝑇𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡
𝐾𝐾𝐾𝐾𝐾𝐾𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 =
𝑇𝑇𝑚𝑚𝑚𝑚𝑚𝑚
for manual testing and

𝑇𝑇𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 + 𝑇𝑇𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡
𝐾𝐾𝐾𝐾𝐾𝐾𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 =
𝑇𝑇𝑚𝑚𝑚𝑚𝑚𝑚
for automated testing. As Tmin is in both equations equal the quotient of the two KPIs describes the KPI for
improving the testing process

𝐾𝐾𝐾𝐾𝐾𝐾𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚𝑚 𝑇𝑇𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 + 𝑇𝑇𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡

𝐾𝐾𝐾𝐾𝐾𝐾𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = =
𝐾𝐾𝐾𝐾𝐾𝐾𝑎𝑎𝑎𝑎𝑎𝑎𝑎𝑎 𝑇𝑇𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 + 𝑇𝑇𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡

as a measured value. This KPI can be used as a basis for assessing actual cost reductions resulting from an
increase in efficiency due to the testing time reduction.

To make this theoretical discussion more tangible, the last part of this article will focus on a practical
example.

5 Efficiency gains in practice

What possibilities do efficiency gains and cost reductions offer? In the following example, we’ll take potential
actions for turning efficiency gains into numbers. The time when actions are introduced is relevant to the
overall benefit that can be generated. Therefore, in the example, we’ll look at the different phases of a new
substation or the extension of an existing substation. In the following example, we will focus on testing
protection relays, as shown in Figure 3.

© OMICRON 2023 Page 6 of 7

 Figure 3: Protection testing in a substation

Over the lifetime of a substation, the assets installed pass through various phases. The assets must be
tested during each phase with a defined set of tests, as visualized in Figure 4. Preparing and executing the
individual tests can be done separately (as seen in Figure 2) and manually, but that would be the least
efficient approach. Preparation is usually the most time-intensive part of a protection test. It consumes about
50 to 60 percent of the entire test process. In Figure 5, this equals five-time units. Execution accounts for
roughly one-time unit and documentation for another two-time units.

Figure 4: Preparation vs. execution of required tests

© OMICRON 2023 Page 7 of 8

 Figure 5: Calculation example

If test plans and automation are introduced instead of a manual approach, the time required is reduced
drastically. At the same time, test coverage and depth can be increased substantially, while test repeatability
can be easily pushed to an optimum level.

Whenever tests are performed for the first time, future requirements are considered during their thorough
preparation and design. Each test can be repeated without changing individual test modules.

Test procedures stored in repeatable test plans allow tests to be performed one-to-one in later phases of an
asset’s life cycle. This pays off once the assets enter their maintenance phases, and the same tests need to
be performed regularly. Such an approach can reduce the needed time by up to 80%.

Let’s focus on an example. Figure 6 shows a section from an industrial power grid. It contains several motors
which are similar in size. These motors are protected with the same protection scheme with similar
protection settings. In most cases, the protection devices are from the same manufacturer, which means that
the differences between the individual protection devices are minor. Differences may be found in specific
parts of the protection scheme (e.g., pickup values, trip times).

Figure 6: Part of an industrial grid

On average, we have to deal with approximately 30 different parameters from one protection device to the
next. Therefore, nearly every test case in the protection testing procedure must be recalculated and
overworked for testing the next protection device. A procedure with 30 test cases is required for testing
these 30 parameters and ten more essential ones (e.g., CT and VT settings) for a protection device.

© OMICRON 2023 Page 8 of 9

 Table 1. Effort for adapting test cases

Number of calculations Number of changed

needed per test case parameters per test case
Trip time test one stage 2 12
Trip time characteristic test (50/51/67) 6 18
Pick up test 2 9
Drop off test 2 9
Average 3 12

Table 1 displays an average of 3 calculations and 12 altered parameters per test case that can be used to
discuss the efficiency of protection testing.

Hence, preparing the 30 test cases from our example needs 90 calculations and 360 parameters, which
must be changed during the testing of the protection devices used for every motor in Figure 6. If 20 similar
protection devices in this substation need to be tested,1800 calculations must be done, and 7200
parameters must be changed manually.

For automated testing, 90 Calculations, 360 parameters (both steps only need to be done once), and 600
(20 times the 30 varies) parameters have to be entered. For an easier calculation, let’s assume that every
step requires the same amount of time. Using the formula for KPIimprov we get the following:

𝑇𝑇𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 + 𝑇𝑇𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 9000

𝐾𝐾𝐾𝐾𝐾𝐾𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = = = 8,6
𝑇𝑇𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑢𝑢𝑡𝑡𝑡𝑡 + 𝑇𝑇𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡𝑡 1050

By testing 30 similar devices we have an improvement of

13500
𝐾𝐾𝐾𝐾𝐾𝐾𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = = 10
1350

In addition to substantial efficiency improvements, by factors 8,6, and 10 in our examples, two additional
aspects must be highlighted. Firstly, the effort to prepare a manual test procedure is greater than the
preparation time needed to use an automated test plan. Secondly, the manual testing approach increases
the possibility of human error during preparation and execution dramatically. This is why improving efficiency
with automation also enhances the quality of the protection testing process.

6 What does this mean for daily work?

New technologies in one business field lead to changes in other technologies and working processes.
Nowadays, the pressure due to time and cost limitations is becoming more and more severe. On top of that,
the energy transition is a catalyst for these two topics. In this context, validating technical working processes
by analyzing their economic properties is critical.

This article shows why KPIs are vital to the protection testing field and why they should be implemented if
they’re not being used yet. They improve and optimize working processes by making them measurable.

When using a KPI to compare a manual testing approach to an automated one, its time-saving efficiency
shows that state-of-the-art testing methods can leverage significant improvements. Measurable
improvements by, e.g., a factor of 10, is more than enough reason to examine your own processes more
closely.

[1] Performance Indicator, Wikipedia article May 2019, https://en.wikipedia.org/wiki/Performance_indicator

© OMICRON 2023 Page 9 of 10

7 Author
Michael Albert studied general electrical engineering at Saarland University.
After his studies, he worked in power engineering, focusing on protection
technology. He’s worked for OMICRON since 2005 as a product manager and
in the engineering service field.

© OMICRON 2023 Page 10 of 11

OMICRON is an international company serving the electrical power
industry with innovative testing and diagnostic solutions. The application of
OMICRON products allows users to assess the condition of the primary
and secondary equipment on their systems with complete confidence.
Services offered in the area of consulting, commissioning, testing,
diagnosis and training make the product range complete.
Customers in more than 140 countries rely on the company's ability to
supply leading edge technology of excellent quality. Service centers on all
continents provide a broad case of knowledge and extraordinary customer
support. All of this together with our strong network of sales partners is
what has made our company a market leader in the electrical power
industry.

For more information, additional literature, and detailed contact information

of our worldwide offices please visit our website.
© OMICRON 2023 Page 11 of 11
www.omicronenergy.com