Adaptive

Design
Continuous innovation
to accelerate utility
customer engagement
  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

What is Adaptive Program Design?

Good experimental designs guiding responsive program optimization

When you receive a catalog from J.Crew or Pottery Barn, or a credit card offer from Capital One bank,
there is a good chance that you are a participant in a test. These and many other customer-facing
companies use subtle changes to their catalogs, offers, phone scripts, and other customer contacts
to conduct thousands of small experiments and collect data on the impacts. The companies then
use this information to rapidly improve the next customer outreach while simultaneously testing new
concepts; this approach is often referred to as “Test and Learn”.

OVER THE PAST FIVE YEARS, Nexant has pioneered the
adaptation of this concept to the evaluation and improvement of
utility energy programs. Our Adaptive Design Process (ADP) utilizes
small scale experiments to test the efficacy of energy efficiency,
demand-response, and conservation program alternatives. The
essence of the ADP is the use of small experiments during the
implementation and evaluation phases of utility energy conservation
programs, continuously testing intervention alternatives and
adjusting the program in response. Thus, program managers can
accelerate the optimization of their processes instead of waiting for
a final evaluation.
Embrace entrepreneurial thinking
if you want to better serve customers
Adaptive design represents a customer-centric approach to
improving utility programs and combines the rigor of scientific
experimentation with recent innovations in product design and
delivery. Nexant adopted this entrepreneurial mindset for utility
programs because we saw that customers, not regulators, are
having more and more influence on program success. As the
energy market changes it is clear that utilities will need a deeper
understanding of their customers’ preferences in order to better
influence their behavior.

ADAPTIVE PROGRAM DESIGN 2  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE
Good design is one that changes
customer behavior toward your
desired outcomes
When taking a thoughtful, customer-driven
design to market, success is dependent on the
ability to keep up with customer intentions.
This means constantly learning how customers
are engaging with the product or service and
then using that knowledge to improve both
their experience and the service. To achieve
this, Nexant’s ADP approach takes effective
tactics from Eric Ries’ method described in
“The Lean Startup,” a scientific approach to
organizational and project management.
A Lean Startup approach rapidly develops
products that perfectly fit the market. Using an
approach dubbed “fast-learning,” entrepreneurs
constantly observe and measure customer
behavior and acceptance and then make rapid
decisions about whether to keep building out
features, kill the ones that are not working, or
introduce new features based on an unexpected
ADAPTIVE PROGRAM DESIGN
CASE STUDY ADVANTAGES SUMMARY
customer behavior. In order to make such
decisions with confidence about what
customers really want, it is crucial to test often,
test well, and test with program adjustments
in mind. The ADP combines the lean startup
concept of fast-learning with a scientific
test-and-learn approach.
The ADP approach also makes use of “design
thinking,” a creative approach to product design
used effectively by companies like IDEO and
thought leaders like Stanford’s d.school.
Design thinking begins with the customer,
focusing on empathy to drive design as opposed
to starting with a solution and forcing customers
to embrace it. Design thinking recognizes that
customers almost never know what they want;
in fact, they usually want something just like the
thing they have - only better. Therefore, “need-
finding” is crucial: look for implicit needs by
observing how people work around problems,
and noticing gaps in usability.
3  20
  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

Lean Startup and design thinking have similar assumptions:

 Both assume extreme uncertainty

 Both assume that the users don’t know what

?

they want and can’t tell you what to make, directly

 Both assume that a rapid cycle of building and

learning is the only process that yields actionable data

ADAPTIVE PROGRAM DESIGN 4  20

  INTRO SERVING CUSTOMERS
How does a utility introduce this cycle of testing, measurement,
fast-learning and need-finding into programs? While it may sound
disruptive, adaptive design allows utilities to introduce continuous
testing without detracting from essential program processes. The
results can be justified because the ADP supplements the innovation
of entrepreneurial thinking with the validity of scientific methods.
The Benefits of Using Experiments
to Measure Program Impacts
There are currently two main sources of energy savings estimates:
those based on engineering models and those based on empirical
evidence or observational studies. Engineering-based estimates
measure the difference of energy use before and after a specific
treatment; these are good for assessing simple savings but are
not able to measure many other types of impacts, such as those
resulting from behavioral changes. Empirical studies document
observable effects such as a bill comparison before and after
receiving a treatment, but are not able to measure unobservable
ADAPTIVE PROGRAM DESIGN
BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY
impacts (e.g. if the number of people living in a house changes or
if a behavior change is due to a new job with a new schedule). As a
result, empirical studies often have poor external validity, which can
create problems when the results are used to inform goals in
other programs.
In order to overcome these shortcomings, more effectively
measure program impacts and validate EE programs, policymakers
and evaluators are increasingly calling for more evidence-based
evaluation and planning of energy efficiency programs and
interventions. The California Public Utility Commission (CPUC), in
particular, has been an early recognizer of the benefits of evidence
based decision making and the use of randomized controlled trials
(RCTs). In addition, the importance of improving the confidence in
efficiency program evaluation through experimental design is well
summarized by Vine et al. (including Nexant’s Dr. Michael Sullivan):
“The use of experimentation, particularly RCTs, has rarely been
applied to rigorously test alternative energy efficiency program
5  20
  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

design features and, more fundamentally, determine the benefits of campaigns and educational reports). This ability to measure
energy efficiency policy initiatives. The resulting absence of a sound the previously unmeasurable, combined with less-assailable
empirical foundation for calculating energy efficiency impacts is methodology and findings, makes experimentation a strong tool for
impeding progress in the development of effective energy efficiency advocating the benefits of energy efficiency.
programs and has led some in the policy community to advocate
non-energy efficiency options with more rigorous foundations Experimental Research for
and less risk of failure in moving the U.S. along a path toward an Energy Efficiency Programs
environmentally more robust energy system.”
Randomized controlled trials (RCTs) are a well-known form of
experimental research. RCTs are the standard among scientists
When applied to energy efficiency programs, evaluating the effectiveness of medical treatments and have been
a well-designed RCT can answer the question used to evaluate federal social programs and global anti-poverty

“What would have happened in the absence efforts. They work by evaluating a treatment group (those who
receive a treatment) against a control group (those who do not).
of the program or specific intervention?”
In an ideal setting the only difference between those groups is the
presence or absence of the treatment.
Experimentation has demonstrated to regulators, stakeholders, and
utilities that they can measure effects of interventions that were
otherwise unquantifiable (such as savings associated with outreach

* Experimentation and the evaluation of energy efficiency programs, Vine, E., M. Sullivan, et. Al., Energy Efficiency (2014) 7:627-640

ADAPTIVE PROGRAM DESIGN 6  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

“ The use of experimentation, particularly randomized controlled trials

(RCTs), has rarely been applied to rigorously test alternative energy
efficiency program design features and, more fundamentally, determine the
benefits of energy efficiency policy initiatives.

The resulting absence of a sound empirical foundation for calculating

energy efficiency impacts is impeding progress in the development of
effective energy efficiency programs and has led some in the policy
community to advocate non-energy efficiency options with more rigorous
foundations and less risk of failure in moving the U.S. along a path toward
an environmentally more robust energy system. ”
— D R. M I C H A E L S U L L I V A N , N E X A N T

ADAPTIVE PROGRAM DESIGN 7  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

When applied to energy efficiency programs, a well-designed RCT can answer

the question “What would have happened in the absence of the program or
specific intervention?” Using this information an evaluator can determine
with high confidence the causal impacts of the program or intervention. A
well designed RCT can provide the highest level of both internal and external
validity as compared to other experimental designs and non-engineering
based evaluation methods.

Sometimes it is not possible to perform a true RCT, such as in situations where

treatment cannot be withheld or arbitrary assignment has already occurred.
In these situations there are a number of quasi-experimental designs which
may be used that utilize the logic of experimentation and are still quite
reliable. They include: randomized encouragement design (RED), regression
discontinuity designs, interrupted time series designs, within subjects designs,
and others.

ADAPTIVE PROGRAM DESIGN 8  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

Accelerating
Program Optimization
and Improving
Cost-Effectiveness
Beyond the benefits to program evaluation, the primary benefit of the ADP is
that programs can immediately reap the benefits of improvements to their
effectiveness and resource utilization. The typical program cycle can take
multiple years to move from the identification of an opportunity, to picking a
solution and piloting or testing it, and finally to improving the program based
on program outcomes. Even if the program is deemed successful, it is done
without knowledge of the opportunity cost or the potential benefits of all the
untested solutions, as well as their relative effectiveness compared to the
chosen option. After this process, the program may be improved but it cannot
be said that it is “optimized.”

ADAPTIVE PROGRAM DESIGN 9  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

The typical program cycle

aims to incrementally improve programs

Choose only one hypothesis Test only once. Get results at the end

Months to years

Result
Program is improved, but improvement
relative to other options is unknown.

ADAPTIVE PROGRAM DESIGN 10  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

The adaptive design process

aims to rapidly optimize programs

N QUICK Pivot
AR LY
LE
Perish
Test
Side-by-Side
Optimize the mix of program
tactics and proceed

Identify all viable

options to test Persevere with next test

Result
Overall cost-effectiveness is maximized; program is
optimally tuned to the needs of each sector and customer.

ADAPTIVE PROGRAM DESIGN 11  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

Informed by data
 Marketing Effectiveness  Customer Propensity  Interval Data
• Incentive structure and level • Engagement in other programs • Load shape cluster
• Types of customer touches • Bills and consumption characteristics • End-use disaggregation
• Number of outreach attempts • Customer and building characteristics • Weather sensitivity
• Pre-announcements • Billing analysis
• Timing of Messaging

 Costs  Benefits  Test Findings

• Equipment and installing • Avoided energy costs • Empirical results from
• Marketing acquisition • Avoided generation capacity Test and Learn Group

• Incentives

ADAPTIVE PROGRAM DESIGN 12  20

  INTRO SERVING CUSTOMERS
Under the ADP, multiple opportunities are identified, viable options
are pulled out, and multiple design concepts are simultaneously
tested. The program is then adjusted in response to the success or
failure of individual design components and the process repeats.
Using this approach, program managers and implementers can
optimize the offering to any chosen combination of program
outcomes, such as increasing per-customer savings, maximizing
participation, improving overall cost-effectiveness, or others.
Applications of the ADP to Utility Programs
There is growing acceptance in the utility industry that all efficiency
programs, in all sectors, have behavioral components which can
have significant impacts on program performance. Consider a
rebate program. What is the optimal rebate amount? Is participation
most increased by higher rebates or marketing in an additional
language? Or consider a direct install program. Could marketing
effectiveness be most improved through improved contractor
training or radio advertisements? Is customer experience
improved most through increased product choice or inclusion of
additional measures?
ADAPTIVE PROGRAM DESIGN
BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY
Answering these questions through a traditional program approach
can require many years of skilled management to fine-tune; well-
designed experiments, however, can quickly measure the related
impacts and precisely provide answers. Experimentation is not
without cost and therefore should only apply to those aspects
of program delivery that can be manipulated to create the most
leverage in changing outcomes.
Nexant’s general approach is to start with a controlled launch
and simultaneously and systematically test various techniques
on random samples of the population. Advanced analytics and
sector segmentation are used to test different outreach strategies,
measure mixes, incentive levels, and other program elements in
order to tune the offering to the customers and optimize for various
desired outcomes. The ADP provides rapid feedback and measured
evidence of the impact and effectiveness of various recruitment
tactics. This in turn allows us to optimize program techniques
and targeting based on what we learned actually works and what
does not. Experimenting with alternative marketing techniques in
particular, such as different messages, channels and strategies, is
relatively easy to do and can lead to significant advances
in effectiveness.
13  20
  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

What are the impacts we are measuring?

Impacts to measure What to vary in tests

(output/results) (dials to fine-tune)

• Information • Form
• Education • Content
1. Impact on • Knowledge • Delivery system
• Experience • Engagement
consumer
• Analytics
• Frequency
• Duration

• Inducements (subsidies, rebates, • Amount

price breaks, ex-post rewards and praise) • Timing
2. Effectiveness of • Costs • Framing
reward structure • Penalties • Recipients
• Prices • Delivery system

• Signage
a) Point-of-sale delivery systems • Advertising
• Packaging with other items/services
3. Market
Interventions
• Education
b) Upstream programs • Inducements
• Service provisions
• Competition

* Supporting and Facilitating Experimentation in Energy Efficiency: Exploring New Validity Constructs to Support Inter- and Intra-Organizational Paths
ADAPTIVE PROGRAM DESIGN to Success. ACEEE, Anne Dougherty, Mary Sutter, and Katherine Randazzo, Opinion Dynamics Corporation. 2012 14  20
  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

CASE STUDY: A LARGE WEST COAST UTILITY

Optimizing
Behavioral
Programs
Residential behavioral information feedback programs, such as
Home Energy Reports (HERs) and Bill Alerts, have been the primary
market for incorporating experimental designs into program design
and evaluation services. Behavioral programs are ideally suited for
optimization through experimental designs because there are no
reliable engineering-based models that could be used to evaluate
cost-effectiveness or savings, so an RCT or quasi-experimental
evaluation model is required to properly evaluate the program.
Incorporating the ADP from the program design stage allows for
program optimization to begin immediately, accelerating program
improvement, and will also aid in the ultimate program evaluation
as better experimental designs can be used.

ADAPTIVE PROGRAM DESIGN 15  20

  INTRO SERVING CUSTOMERS BENEFITS
Nexant’s Evaluation of a 2014 -15
Conservation Campaign
A large west coast utility conducted a 2014-15 conservation
campaign in which they tested the effectiveness of various enhanced
information programs on their customers with recently deployed
advanced meters. The testing included multiple versions and
combinations of Home Energy Reports (HERs) and Bill Tracker Alerts
(BTAs) produced by different vendors, and sought to determine
the effectiveness and savings related to each method. From these
experiments we learned:
• One provider’s HERs were significantly outperforming the other
• Populations in urban areas are more responsive to electronic
HERs and BTAs
• HER related savings persisted through the following winter in the
absence of new reports
• BTA participant savings persisted through the following winter in
the absence of promotional materials
• The most responsive participants were in the top usage quartile,
in areas with lower population density, with lower rates of Latinos,
and not on CARE low-income rates
ADAPTIVE PROGRAM DESIGN
TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY
Using the information gained from these experiments, Nexant
proposed to use the ADP (referred to as “Test & Learn” in the report)
to improve the program. Some of the proposed improvements were:
• Improve the BTAs to include more graphics vs. being text and
data-intensive
• Test effectiveness of BTAs without including costly
informational materials
• Test leveraging additional AMI data for more behavioral savings,
such as weather sensitivity reports
• Continue Opower HERs and test thermostat messaging on new
populations; discontinue Aclara HERs
• Test effectiveness of Spanish language HERs in areas with high
rates of Latino population
This work demonstrates best practices for applying adaptive design
combined with advanced analytics for segmenting customers in
order to discover which behavioral programs work best for
which segments.
16  20
  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

Advantages
Optimizing the Targeting and Recruitment
Efforts of Traditional EE Program Models

As discussed above, all programs have behavioral elements which

can significantly impact program success. The ADP need not
be limited to behavioral programs and can also be used within
traditional program types. For example, within a traditional low
income weatherization program, the ADP can be utilized to identify
the most effective outreach methods and identify specific barriers
to participation. If an experiment can be used within a program
to test the efficacy of program intervention or evaluation options,
then the ADP can be used to accelerate and optimize that process. Fewer than one percent of the potential market takes advantage

For instance, in 2016, Nexant is engaged to use the ADP in its of the EE services offered, but the cost of customer acquisition is

implementation of PG&E’s Light Industrial program, a traditional high, so it is crucial to program success to introduce innovative

EE program offering retrofit services to industrial clients with loads outreach techniques using the test-and-learn and rapid, incremental

under 500kW. improvement that the ADP enables.

ADAPTIVE PROGRAM DESIGN 17  20

  INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

Optimizing Dynamic
Pricing Impacts on Consumption Summer Peak Rate

Experimental designs are also being used to assess the impacts

of dynamic pricing on the timing and magnitude of electricity + Group C
consumption. The experiments include control groups that stay on Control Group
the standard tariff and treatment groups that are placed on new
time varying tariffs or information programs. Applied to dynamic
pricing, the ADP can be used to simultaneously test multiple pricing + Group A
structures, find statistical relationships between energy use by rate
period and prices, measure the effectiveness of various types of Time of Use Rate
informational materials, and discover how dynamic pricing can be
used to further other smart grid related objectives. The process
allows for an accelerated optimization of dynamic pricing as well as + Group B
the discovery of other ways in which dynamic pricing could benefit
grid operations or further other utility objectives.

Simultaneously test multiple pricing structures

ADAPTIVE PROGRAM DESIGN 18  20

 INTRO SERVING CUSTOMERS BENEFITS TYPICAL VS ADAPTIVE CASE STUDY ADVANTAGES SUMMARY

Nexant’s ADP
helps utilities satisfy
customer demands
It is difficult to learn how best to engage with your customers. Utilities and
regulators typically take an engineering-based approach to service and
program improvements which can be at odds with expectations customers
have developed from other consumer-facing industries. Utilities can use
today’s extended program development and evaluation cycles as an
opportunity to accelerate learning about their customers through “Test
& Learn” or Adaptive Design. Not only does this serve customers better,
it also maximizes cost-effectiveness by pursuing the optimal intervention
combinations for each segment.

To create a culture of innovation, utilities can nurture curiosity - about their

customers, about technology, about possibilities. Adaptive design allows
utilities to foster a new perspective on value creation: a human-centered,
co-creation process focused on real end-user needs, ultimately yielding the
highest value to all stake holders.

19  20
 Find out more about Nexant Utility Services 

Nexant offers best-in-class energy consulting and implementation services that transform utility business processes and improve smart grid, clean energy, distributed
energy resources, and customer experience initiatives. Nexant experts help utilities embrace a customer-centric strategic planning model that aligns markets,
products, technology and customer strategies, reducing risk to achieve superior business results.

consulting@nexant.com | 415.369.1198

Adaptive Program Design 20  20