Metrics To Prove You CARE About Cybersecurity

Metrics to Prove You CARE About Cybersecurity
Published 15 July 2021 - ID G00743959 - 16 min read
By Analyst(s): Claude Mandy, Sam Olyaei, Paul Proctor

Initiatives: Security and Risk Management Leaders; IT Cost Optimization, Finance, Risk
and Value; Technology, Information and Resilience Risk
Organizations struggle to demonstrate that their cybersecurity

program and controls are consistent, adequate, reasonable and
effective (CARE). Security and risk management leaders can use
the CARE framework to develop metrics to prove the credibility and
defensibility of their cybersecurity program.
Overview
Key Findings
■ Security and risk management leaders struggle to demonstrate and communicate a
minimum standard of due care to customers, regulators, auditors and senior
management.
■ Security and risk management leaders often focus on operational metrics that
provide limited value to business stakeholders due to their technical nature.
Operational metrics are necessary to run the program, but they are not useful or
relevant to business leadership in their raw form.
■ Most metrics used by organizations have been developed in a specific and often
tactical context; they are not being used to prove a desired outcome has been
achieved.
Recommendations
Security and risk management leaders responsible for reporting on the effectiveness of an
organization’s security program should:
■ Develop a catalog of CARE metrics to help prove that cybersecurity controls are
consistent, adequate, reasonable and effective. These outcomes should be
contextualized further to create more effective stakeholder messaging.
Gartner, Inc. | G00743959 Page 1 of 14
This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

■ Use CARE metrics to assess the credibility and defensibility of the organization’s
cybersecurity program and provide meaningful insight into the organization’s desired
control outcomes.
■ Exercise caution when aggregating metrics across different controls to avoid

creating meaningless metrics that have little influence on decision making.
Strategic Planning Assumption

By 2024, 80% of the magnitude of fines imposed by regulators after a cybersecurity
breach will come from failures to prove the duty of due care was met, as opposed to the
impact of the breach.
Introduction
In The CARE Standard for Cybersecurity, Gartner introduced the CARE standard. The CARE
standard focuses on achieving cybersecurity outcomes that are consistent, adequate,
reasonable and effective, as opposed to driving greater investment in tools and processes
(see Figure 1).
Figure 1: The CARE Standard for Cybersecurity
Source: Gartner (July 2021)

The CARE standard was developed in order to provide a framework to assess the
credibility and defensibility of an organization’s cybersecurity program. The CARE
standard builds upon the concept of reasonable steps, often used by regulators and legal
proceedings to determine if the standard of due care has been met (see Note 1).
Security and risk management (SRM) leaders can further increase the credibility and
defensibility of their cybersecurity program by developing metrics based on the CARE
standard.
Analysis
Develop Metrics to Prove You CARE
This research provides SRM leaders with clear guidance on what types of metrics (with
relevant examples) are useful to help prove that cybersecurity controls are consistent,
adequate, reasonable and effective. There is, however, no prescriptive document or set of
metrics that any organization could follow that will give complete assurance that the
standard of due care in a particular circumstance has been met. Each organization must
evaluate its own particular circumstances and take into account a number of factors to
make an informed judgment about what is “good enough.”
With this in mind, the CARE framework should be used as a guide for organizations to
expand the catalog of metrics across all controls by identifying how they would apply
each metric type to a specific control within their environment. Using the CARE framework
to develop and structure metrics enables SRM leaders to translate their operational
metrics into categories that are easily understood by a nontechnical audience.
Although the categories of CARE metrics can also be used to aggregate individual control
metrics across entities and similar controls, SRM leaders should exercise caution.
Aggregating metrics across different controls can create meaningless metrics. It is
important to remember that all metrics must do at least one of the following:
■ Inform and educate the audience about issues that are important to them.
■ Influence a decision that the audience has to make.
■ Change the behavior of the audience or their direct reports.

By aggregating metrics without clear logic, the meaningful insight and intent of the
metrics can be stripped away. For example, aggregating the percentage of antivirus
coverage with the percentage of awareness training coverage is illogical, as these metrics
assess the coverage of different targets (i.e., systems vs people). For every metric, you
should be able to articulate what decision and what action would be taken for a “bad”
result, and who would be responsible for that decision and action.
However, CARE metrics can and should be further contextualized to the audience by
drilling into detail for specific business units and systems. It is not enough to give the
audience data; security and risk management leaders must embed context with the
metrics they report. Figure 2 provides an illustration of how adding context to metrics
creates more effective stakeholder messaging. The arrow indicates a progression toward
more precise and contextualized metrics.
Figure 2: Contextualization of CARE Metrics

Consistency Metrics: Prove Your Controls Are Consistent Over Time
A consistency metric is a high-level measure of the consistency of a security control,

simplified for gathering and review on a weekly, monthly or quarterly basis. Consistency
metrics assess whether the controls are working consistently over time across an
organization.
Consistency metrics are the most popular and frequently used type of metric among SRM
leaders. The frequency of use may be due to a historical focus on deployment of
technologies to address issues. As a result, only a small subset of the operational
outcomes that can be used to assess the consistency of cybersecurity controls are used in
practice, with focus on the coverage and currentness of controls.
Table 1 provides a broader example set of operational outcomes to assess the

consistency of your controls based on a subset of illustrative controls. These metrics
should be continuously updated, measured and reported over time to demonstrate that
they remain consistent.

Table 1: Consistency Metrics
(Enlarged table in Appendix)
Adequacy Metrics: Prove Your Controls Meet Business Needs
An adequacy metric is a high-level measure of the adequacy of a security control against

stakeholder expectations, simplified for gathering and review on a weekly, monthly or
quarterly basis. Adequacy metrics assess whether the controls are satisfactory and
acceptable in line with business needs.

To demonstrate adequacy, SRM leaders must demonstrate that the controls meet the
expectations of stakeholders through a robust governance process. These expectations
may be documented and approved in the form of policy requirements, protection-level
agreements (PLAs), tolerance levels or service-level agreements (SLAs), or they may be
stipulated in the business case for the control.
Table 2 provides a set of operational outcomes to assess the adequacy of your

cybersecurity controls against your stakeholders’ expectations. In practice, these
examples may be based on metrics already being gathered to evidence the consistency,
reasonableness and effectiveness of your controls. These operational outcomes should
be reviewed and updated at least annually to ensure that they remain adequate and are:
■ Understood in a business context.
■ In line with stakeholders’ expectations in terms of balancing the need for protection
with the need to run the business.
■ Not far from what are normally considered acceptable results for similar conditions.

Table 2: Adequacy Metrics
Reasonableness Metrics: Prove Your Controls Are Appropriate, Fair and

Moderate
A reasonableness metric is a high-level measure of the reasonableness of a security

control, gathered by assessing the business impact and friction caused by the control,
and simplified for gathering and review on a weekly, monthly or quarterly basis.
Consistency metrics assess whether the controls are appropriate, fair and moderate.

Demonstrating the reasonableness of cybersecurity controls requires an understanding of
the trade-off between the need to protect and the need to run the business. SRM leaders
should always be guided by credible and defensible business needs, including the value
of the assets being protected, the cost of protection and the potential losses or risk being
mitigated. For most organizations, reasonableness should be measured at the program or
risk level, rather than at the level of an individual control. A balanced scorecard for
security is a well-understood and highly utilized tool reporting tool to ensure the security
program is balanced across the financial, customer, operational and staff aspects of the
program (see Toolkit: Developing a Balanced Scorecard for Security).
Every organization requires the flexibility to meet its unique needs and circumstances and
respond to risks and threats based on the organization’s budget, size and needs. Gartner
recommends adopting a risk, value and cost (RVC) optimization approach. The RVC
approach can demonstrate to key stakeholders that the organization has the right
priorities and investments to balance the need to address risk with the need to achieve
their desired business outcomes. Organizations can demonstrate reasonableness through
an assessment of risk (to both the organization itself and to third parties) and robust
governance in order to reduce the risk to within tolerable levels.
Another common approach is to demonstrate the reasonableness of the program through

benchmarking, such as:
■ The organization’s level of expenditure matches or exceeds the industry average

(such as the figures reported in Gartner’s annual IT Key Metrics Data — see IT Key
Metrics Data 2021: Overview).
■ The organization’s control compliance and/or implementation level expressed within

the context of a cybersecurity framework (such as Gartner’s Controls Maturity
Benchmarking Service or NIST CSF implementation tiers) matches or exceeds its
peers’.
■ The organization’s maturity benchmarks (such as Gartner’s IT Score for Security and
Risk Management) matches or exceeds its peers’.
It is important to note that none of these approaches in isolation is a good way to decide
whether your controls are reasonable. Each is imprecise, highly dependent on
circumstances and subject to change over time, but can identify immediate concerns that
should be assessed in more detail. In addition, they ignore the nonfinancial impact of
excessive or too restrictive security.

At an individual control level, the trade-off between business needs and cybersecurity
needs can be identified more directly by measuring signs of friction from the business.
Table 3 provides a set of operational outcomes to measure the friction created by your
security program and assess the reasonableness of your security controls.
Table 3: Reasonableness Metrics


Effectiveness Metrics: Prove Your Controls Are Successful in Producing the
Desired Outcome
An effectiveness metric is a high-level measure of the effectiveness of a security control,

simplified for gathering and review on a weekly, monthly or quarterly basis. Effectiveness
metrics assess whether the controls are successful and/or efficient in producing a
desired or intended outcome.
Demonstrating that your controls are successful in producing the desired or intended
levels of protection requires an understanding of what the desired levels of protection are.
The levels of protection vary depending on the type of control and its intended purpose.
Gartner recommends developing outcome-driven metrics (ODM) that provide a direct line
of sight to protection levels in a business context. At a high level, this typically translates
to reducing both the number of incidents and their impact by either reducing the time to
detect and respond or being able to effectively recover.
Table 4 provides a set of operational outcomes on which to base the development of your
ODM and to assess the effectiveness of your security controls against the desired levels
of protection.

Table 4: Effectiveness Metrics
Evidence
Gartner received over 2,000 inquiries on the topic of security and risk metrics and reporting
between July 2019 and June 2021.
This analysis is based on Gartner analysts’ experience in working with clients to improve
their security and risk metrics. It is also based on Gartner observations of the most
common security metrics and analysis of the assertions being made by the metrics.

Note 1: The Standard of Due Care
For the purposes of this research, we use “due care” as a generic term that reflects the
requirements of the legislative instruments summarized below in the particular context
that each one applies.
Most countries’ laws have similarly vague language. A further complicating issue is that
the meaning of the term “due care” depends on what country you are in, and where that
country’s law system was derived. The term of art “standard of due care” derives from U.K.
common law, and has specific meaning in countries where the law is based on that
system. It is a judgment made by a court, and is used to allocate or assign liability.
Other legal systems, such as the Napoleonic Code, have very different definitions of the
role of the judiciary, and thus may use the term differently.
Recommended by the Authors

Some documents may not be available as part of your current Gartner subscription.
The CARE Standard for Cybersecurity

A Decision Model to Optimize Risk, Value and Cost
Outcome-Driven Metrics for Cybersecurity in the Digital Era
IT Score for Security and Risk Management

Controls Maturity Benchmarking Service
Developing Metrics for Security Operational Performance

© 2021 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of
Gartner, Inc. and its affiliates. This publication may not be reproduced or distributed in any form
without Gartner's prior written permission. It consists of the opinions of Gartner's research
organization, which should not be construed as statements of fact. While the information contained in
this publication has been obtained from sources believed to be reliable, Gartner disclaims all warranties
as to the accuracy, completeness or adequacy of such information. Although Gartner research may
address legal and financial issues, Gartner does not provide legal or investment advice and its research
should not be construed or used as such. Your access and use of this publication are governed by
Gartner’s Usage Policy. Gartner prides itself on its reputation for independence and objectivity. Its
research is produced independently by its research organization without input or influence from any
third party. For further information, see "Guiding Principles on Independence and Objectivity."

Table 1: Consistency Metrics
Metric Type What It Measures Example Metrics
Control coverage These metrics measure how consistently the Threat and vulnerability management: Percentage
control is deployed throughout the environment of systems scanned for vulnerabilities.
compared to required scope. It demonstrates that
the control covers the areas where it is needed and
intended.
Phishing awareness: Percentage of employees

enrolled in phishing simulation program.
Third-party risk assessment: Percentage of third

parties with risk assessment completed.
Identity management: Percentage of systems

utilizing enterprise directory services.
Endpoint protection: Percentage of systems

configured to approved build.
Business continuity: Percentage of business

processes with approved business continuity
plans.
Detection and response: Percentage of data

sources monitored by enterprise cybersecurity
monitoring services.
Control currentness These metrics measure how consistently the Threat and vulnerability management: Percentage
Gartner, Inc. | G00743959 Page 1A of 9A

control is kept updated throughout the of systems scanned within X days.
environment compared to the latest version. It
demonstrates that the control is maintained.

that received phishing simulation within X months.

parties with risk assessment conducted within X
days.
Identity management: Percentage of user

accounts that have been used within X days.
Endpoint protection: Percentage of endpoints with

anti-malware definitions applied within X hours.
Backup and restore: Percentage of systems with

successful backup and restore test within X days.
Detection and response: Percentage of threats

with detection signatures updated within X days.
Control strength These metrics measure how consistently the Threat and vulnerability management: Percentage
control is configured to provide an expected level. of systems utilizing comprehensive vulnerability
It demonstrates that the control is configured at scanning (authenticated scans).
the highest setting suitable for your organization.

receiving sophisticated phishing simulation
templates.

parties with comprehensive third-party risk
assessment completed.
Identity management: Percentage of user

accounts protected with strong passwords (longer
than 15 characters) and/or multifactor
authentication.

hard drive encryption and preboot authentication.
Backup and restore: Percentage of systems with

comprehensive backup strategy implemented.
Control availability These metrics measure how consistently the All technology-based controls: Percentage of
control can be used. It demonstrates that the hours control was unavailable due to an
control is available when it is needed. unplanned outage.
Control accuracy These metrics measure how consistently the Detection and response: Percentage of false
control detects what it is required to. It positives.
demonstrates that the control has a level of
precision required that can be relied upon.

Table 2: Adequacy Metrics
Type of Operational Outcome What It Measures Example Metrics
Internal policy compliance These metrics measure how adequate the control All compliance-related controls: Percentage of
is when compared to an internal policy. It compliance with a specific policy requirement (e.g.,
demonstrates that the control meets the Percentage of systems in scope compliant to PCI
expectations of stakeholders as defined in an DSS 5.1 — Deploy antivirus software on all systems
approved policy. commonly affected by malicious software
[particularly personal computers and servers]).
Protection-level agreement achievement These metrics measure how adequate the control Threat and vulnerability management: Percentage
is when compared to defined outcomes. It of assets regularly patched within PLA.
demonstrates that the control meets the
expectations of stakeholders as defined in an
agreement.

anti-malware definitions regularly applied within
PLA.
Business continuity: Percentage of DR tests that

achieved their PLA, e.g., recovery point objective
within the recovery time objective.
Application development: Percentage of

projects/products regularly applied within PLA.
Detection and response: Percentage of threats

with detection signatures updated within PLA.

Detection and response: Percentage of incidents
detected within PLA.
Business case benefit realization These metrics measure how adequate the control Depends on the business case.
is when compared to projected benefits usually
documented in a business case. It demonstrates
that the control meets the expectations of
stakeholders when approved a project or business
case.

Table 3: Reasonableness Metrics
Delays These metrics measure how reasonable the control Identity management: Average delay (in hours)
is when compared to the delays caused to the when adding new access.
business as a result. It demonstrates that the
control does not unnecessarily impact the
business.
Identity management: Average latency (in

seconds) in authentication caused by multifactor
authentication.
Third-party risk assessment: Average delay (in

days) when signing contracts with third parties.
Threat and vulnerability management: Average

downtime (in hours) required for patching.
Incidents (security-caused) These metrics measure how reasonable the control All controls: Number of incidents caused by
is when compared to the incidents caused by the security control.
control. It demonstrates that the control does not
unnecessarily impact the business.
1
Complaints These metrics measure how reasonable the control All controls: Number of complaints caused by
is when compared to the complaints about the security control.
control. It demonstrates that the control does not
unnecessarily impact the business.

Exemptions These metrics measure how reasonable the control All controls: Number of policy
is when compared to the exemptions from policy exemptions/exceptions approved by security
for the control. It demonstrates that the business control.
cannot comply reasonably without unnecessarily
impacting the business.
Threat and vulnerability management: Number of

patching exemptions or extensions.
Endpoint protection: Number of endpoints with

anti-malware exemptions
1
Complaints can be extremely subjective in nature and subject to bias and manipulation for other purposes.

Table 4: Effectiveness Metrics
Control timeliness These metrics measure how effective the control is Threat and vulnerability management: Average or
when compared to its objective. It demonstrates maximum number of days required to patch critical
that the control is effective in preventing, detecting security vulnerabilities.
or recovering from impacts to the business.
Cloud security: Average or maximum number of

days required to detect and remediate cloud
security control misconfiguration.
Identity management: Average or maximum

number of days required to disable/remove
access.
Detection and response: Average or maximum

number of hours taken to detect security incidents.
Endpoint protection: Average or maximum number

of hours malware is present before detection.
Incident prevalence These metrics measure how effective the control is Threat and vulnerability management: Number of
when compared to the types of incidents it should incidents per year related to unpatched
address. It demonstrates that the control is vulnerabilities.
effective in reducing the volume of incidents that
could be prevented.
Cloud security: Number of incidents per year

related to cloud configuration issues.
Risk reduction These metrics measure how effective the control is All controls: Demonstrable decrease in risks or
when compared to the inherent risk it is intended to business impact.
reduce. It demonstrates that the control is
effective in reducing the risk or impact to the
business.

Metrics To Prove You CARE About Cybersecurity - 743959 - NDX

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Metrics To Prove You CARE About Cybersecurity - 743959 - NDX

Uploaded by

Copyright:

Available Formats

Published 15 July 2021 - ID G00743959 - 16 min read

By Analyst(s): Claude Mandy, Sam Olyaei, Paul Proctor

Organizations struggle to demonstrate that their cybersecurity

Gartner, Inc. | G00743959 Page 1 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

■ Exercise caution when aggregating metrics across different controls to avoid

Strategic Planning Assumption

Figure 1: The CARE Standard for Cybersecurity

Source: Gartner (July 2021)

Gartner, Inc. | G00743959 Page 2 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

■ Inﬂuence a decision that the audience has to make.

■ Change the behavior of the audience or their direct reports.

Gartner, Inc. | G00743959 Page 3 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Figure 2: Contextualization of CARE Metrics

Source: Gartner (July 2021)

Gartner, Inc. | G00743959 Page 4 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

A consistency metric is a high-level measure of the consistency of a security control,

Table 1 provides a broader example set of operational outcomes to assess the

Gartner, Inc. | G00743959 Page 5 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Adequacy Metrics: Prove Your Controls Meet Business Needs

An adequacy metric is a high-level measure of the adequacy of a security control against

Gartner, Inc. | G00743959 Page 6 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Table 2 provides a set of operational outcomes to assess the adequacy of your

■ Understood in a business context.

Gartner, Inc. | G00743959 Page 7 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Reasonableness Metrics: Prove Your Controls Are Appropriate, Fair and

A reasonableness metric is a high-level measure of the reasonableness of a security

Gartner, Inc. | G00743959 Page 8 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Another common approach is to demonstrate the reasonableness of the program through

■ The organization’s level of expenditure matches or exceeds the industry average

■ The organization’s control compliance and/or implementation level expressed within

Gartner, Inc. | G00743959 Page 9 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Table 3: Reasonableness Metrics

Gartner, Inc. | G00743959 Page 10 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

An effectiveness metric is a high-level measure of the effectiveness of a security control,

Gartner, Inc. | G00743959 Page 11 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Gartner, Inc. | G00743959 Page 12 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Recommended by the Authors

The CARE Standard for Cybersecurity

IT Score for Security and Risk Management

Gartner, Inc. | G00743959 Page 13 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Gartner, Inc. | G00743959 Page 14 of 14

This research note is restricted to the personal use of gr-itgbscc@dpdhl.com.

Metric Type What It Measures Example Metrics

Phishing awareness: Percentage of employees

Third-party risk assessment: Percentage of third

Identity management: Percentage of systems

Endpoint protection: Percentage of systems

Business continuity: Percentage of business

Detection and response: Percentage of data