How to prioritize metrics

as an e-commerce CTO
DANNY MILES CTO, Dollar Shave Club
 ABOUT THE AUTHOR
Danny Miles joined Dollar Shave Club as the company’s first Chief
Technology Officer in 2017. Previously, Danny served as the Vice President
of Direct-To-Consumer Technology at Nike, and has more than twenty years
experience in architecting, implementing and launching leading-edge tech-
nical solutions.

As CTO, I am accountable for the reliability and performance

of Dollar Shave Club’s site. When I get to the office in the morn-
ing, I open up several tabs that display my metric dashboards.

This morning ritual usually takes me about 10 sponsibilities, but it’s a foundational area that the
minutes. I do this not to find problems but to whole business depends on. The graphic below
get situational awareness to start my day. Site illustrates why site reliability and performance is
reliability and performance is just one of my re- so important.

02 — 12
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO

1 SITE RELIABILITY AND PERFORMANCE

ON-SITE USER BEHAVIOR METRICS

2
(AKA WEB ANALYTICS)

3 UPPER MARKETING FUNNEL METRICS

4 BUSINESS PERFORMANCE METRICS

If the site isn’t doing well, users won’t be moving
through the on-site sales funnel and converting
(impacting the metrics in ring 2). Then, big driv-
ers of traffic to our site like Google and Facebook
will see that we aren’t converting, because we
report that information back to them (impacting
the metrics in ring 3). In addition, any ad spend
will have been wasted. Then, traffic will dry up
and business metrics (ring 4) will take a big hit.
That’s why site reliability and performance is such
a critical area to the business: any issues get sig-
nificantly amplified beyond just the loss of a sale/
conversion.
in the order shown in the graphic. Therefore, I
check my metrics in this order during my morning
routine. Sometimes I’ll see anomalies and trends
that might not get picked up by alerts. If I do see
something that doesn’t look right or something I
don’t understand, I’ll ping a member of the team.
But, generally, first thing in the morning is not the
time to be discovering performance issues: this is
a level-setting exercise to begin my day.
In the sections below, I dive into each of my met-
ric areas and describe the specific metrics I look
at in each ring during my morning routine.

Incidentally, I’m also accountable for the metrics

03 — 12
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO

Top site reliability and performance metrics

I could get quickly bogged down in system per-
formance data but, as CTO, that’s not my best
use of time. Because I sit between the engineer-
ing team and other business functions, I need
metrics that tell me whether we’re meeting our
obligations to the business and hitting our goals.
Over the years, I’ve zeroed in on some key metrics
in the site reliability and performance space that
matter to me most. Below, I describe the metrics I
check regularly.

01 APPLICATION-LEVEL METRICS
I want to see some key application-level metrics. It’s important to go through the basics of the e-com-
merce application and make sure it’s serving users (and users are behaving on it) as expected.

Active sessions We care a lot about this metric. This is the stateful
part of the site where people are actually logged
in and creating carts. If active sessions drops off
you know something is wrong.

Carts created If you’re not seeing people create carts or orders

for a period of time, that’s a signal that some-
thing’s amiss.

Orders processed This tells me we’re able to make money and also
tells me at what rate we’re making money.

I consider security to be a part of site reliability and performance engineering. While the following metrics
provide indications of site functionality, I also use them to assess security issues.

Logins per day Both failed and successful. This number should be
fairly predictable. Unexpected changes may indi-
cate a security threat or breach.

Payment declines If I get a spike in the number of payment declines, it

could be an outage in our payment processing ser-
vice, a spike in fraud activity, or a code or API break.

04 — 12
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO
Changes to account
While these metrics are important to the business they’re also important to the tech team because they
can signal technical issues. Next, I’ll go a level deeper into system-level metrics.
02 SYSTEM-LEVEL METRICS
Uptime
We have a “Sunglasses Man” who pops up every
once in a while telling customers the site’s not
available. Our goal is to never see Sunglasses
Man. We should be getting alerts about problems
way before Sunglasses Man needs to show up.
My view on site downtime is a bit different than
what I typically hear, but I think it reflects the
reality of our digital business. For me, downtime
isn’t as much about losing new orders and con-
versions, although it does certainly affect those
metrics. In my experience, the biggest impact of
downtime on an e-commerce site is the lost mar-
keting dollars and wasted effort driving traffic to
the site. Marketing campaigns drive our business.
The worst thing that can happen is an outage
during a big campaign we’re running around
March Madness or, when I was at Nike, during the
Olympics. If our site isn’t available or performant
100 percent during those times the impact can
ripple through partners, search engines, and ad
servers. Not only have you wasted your ad spend
05 — 12
This is a big one from a fraud perspective. Account
takeover threats have become fairly common in the
direct-to-consumer e-commerce space. Members
don't frequently change primary account fields like
their email, password, address, or payment infor-
mation. This is especially the case on a subscription
site where members configure most of their settings
at sign-up. If account changes spike above normal
you may have a malicious actor on your hands.
but you are promoting poor quality of your brand
while your digital presence is down.
The impact of downtime through the big ad plat-
forms that drive traffic to our site, like Facebook
and Google, lasts beyond just the time you are
offline. We report back in real time to Facebook,
and their algorithms get tuned based on whether
people are landing on our site and completing our
checkout. If we have an outage on our site or if we
stop reporting back to our ad partners, their al-
gorithms may stop serving impressions and it will
take days to re-tune and scale the traffic back up.
Outside of our programmatic buying platforms,
the lead times for TV and radio spots are not
something you can halt during an outage so you
are literally throwing money away at that point.
So, downtime has a lot of ripple effects. It’s not
just losing new customers and orders—it’s all the
investment you’re putting into marketing, search,
media, and social.
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO

Requests & Traffic

I want to understand the load on the infrastructure and whether it matches up with expectations. I dis-
cussed several traffic metrics above. There are a few more technical traffic metrics that I look at:

Rate limits I want to see if we’re triggering any of our rate

limits on key APIs. For example, if a login endpoint
is spiking over 1,000 times per second from a spe-
cific network or client, that’s potentially a sign of
malicious activity or an application error.

Unique users on the site How many unique users are on the site and does
it match up with expectations? I have week-over-
week and year-over-year comparisons for this
metric, as well as a predicted range.

Requests served from edge I look a lot at traffic served at the edge, via our
versus origin (cache hits CDN, versus our core infrastructure. I
versus cache misses) want to make sure requests are hitting the
cache. If too many requests are coming
all the way to your origin web server for
images and other content, then something may
not be right and your end users could be experi-
ence slow performance. A lot of activity should be
caught at the edge via CDN, which helps if you’re
experiencing a DDoS attack.

Latency
The site needs to be fast, otherwise users will become frustrated and leave the site before they
complete their purchases. I primarily focus on two performance metrics:

Response times For any API or any endpoint, I want to know what
are the response times. Response times can di-
rectly correlate to your conversion rates. In my ex-
perience, when response times start to deteriorate,
they don't usually improve without intervention.
Before you get to a point where your website’s

06 — 12
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO

not up, you’re usually going to see a slowdown

somewhere. It’s very rare that the site just goes
‘click’ and it’s off. More often than not, things start
to slow down and time out. And then it ultimately
results in not being able to load the application.

Page load time I focus a lot on the time it takes for pages to
actually be useful to the user, which is the time it
takes to load and render a piece of content and
for the page to be interactive. Page load time is a
measure of speed from the user’s vantage point,
and this information is captured by monitoring
the client side. The time it takes to transact or
get through an API is more likely to degrade after
we’ve pushed changes into production. This
would in turn have business impact. I recommend
having an “always on” offense when it comes to
speed as the continous deployment of new code,
content, pixels, and tracking can quickly shift
your site from a 2-3 second load time to a 7-8
second load time where you’ll see your conversion
rate go off a cliff.

I like to further break down page load time by device and by browser to ensure that users are having
similar experiences. I’ve definitely experienced situations where certain device types and browsers have
prolonged load times and we’ve had to engineer device-specific fixes, especially now that most brands'
traffic is coming from mobile devices that are less performant than desktops and tablets.

Errors
Errors will always occur on the site, but I really Then I check error message patterns to
need to be aware of any systemic flaws that are see any underlying issues.
blocking users from viewing content, using fea-
I just described the site reliability metrics I pay
tures, and making purchases. Are we getting mal-
the most attention to as CTO. Next, I describe
formed requests or behaviors in the request that
some of the “outer ring” metrics I pay the most
don’t make sense? I check 8-10 error rates which
attention to.
capture key user requests and Tier 1 services.

07 — 12
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO
Metrics the engineering team
supports but does not own
While platform reliability and performance is
my team’s responsibility, there are several other
metric areas where my team has some degree of
shared accountability with other teams. I check
CTO + Digital Product: On-site user behavior metrics
(Web Analytics)
As Dollar Shave Club’s CTO, I manage our soft-
ware development lifecycle. I spend a lot of time
dealing with outstanding tickets, defects, and
building out new features and functionality.
However, the Digital Team, headed up by a peer
of mine, establishes the roadmap and vision for
the content and functionality of the site. My team
implements that vision. So, the look, feel, and
functionality of the site is a partnership between
my team and the Digital Team.
Time on site
Bounce rate
Number of pages visited
Product/content views
Cart adds/removals
08 — 12
these metrics next. This will vary slightly from
one e-commerce firm to another and from one
CTO to another. Here’s a peek at what matters to
me at Dollar Shave Club.
Our collective goal is to move users seamlessly
through the site towards a purchase. This is
all about pulling the right levers through tests,
design, content, and features to increase AOV
[average order value] and conversion rates. Many
people associate this layer of monitoring with
web analytics. Here are some of the metrics that
matter to me in this layer. If any of these metrics
deviate from expectations there may be a techni-
cal issue (but not necessarily).
This is active browsing time on the site.
This is the percentage of visitors who land on a
page and leave the site without proceeding to an-
other page on the site.
This is the number of pages visited per session.
This tracks what users are looking at and how often.
This tracks products added and removed from
carts across experiences.
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO

Time to conversion This is the amount of time it took for the user to
make a purchase after landing on the site.

Abandonment rate This is the percentage of site visitors who left the
site without making a purchase. Also important to
ask is: Are people abandoning the site in particular
places? This will help us reduce leakage and keep
users moving towards a purchase.

A/B Testing Groups We are typically running multiple feature and con-
tent experiments across the site. Comparing the
metrics above by testing group is key.

All of these metrics are a level up from the purely time the company comes up with a new theme or
technical because they involve UX, content, and campaign there’s work to do. Also, new features
design. We’re constantly adding new site instru- that my team is responsible for developing con-
mentation to capture user behavior data. Every stantly impact these metrics.

CTO + Marketing Partnership: Upper marketing funnel metrics

The upper marketing funnel takes place largely supports the Marketing Team in two main ways:
off the site, and is primarily owned by the setting up robust instrumentation to capture data
Marketing Team. However, the Technology Team and marketing analytics platforms.

SETTING UP INSTRUMENTATION AND MAINTAINING DATA PIPELINES

As I explained above, it’s critical to ensure our are operational. Any number of things can disrupt
data pipelines back to referrers like Google, reporting back to ad platforms. The two most
Facebook, and other large drivers of site traffic common issues are:

Misfiring pixels The Javascript we load in the browser from a site

like Facebook isn’t working or running correctly.

Misconfigured pages Deployed pages, for example new landing pages,

on our site aren’t reporting accurately.

09 — 12
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO
The way to find problems with reporting back to
referrers is to set up robust alerts in your web
analytics platform. For example, if I’m serving ad
impressions, but not converting at normal or ex-
pected rates, there’s likely something wrong with
the site or the instrumentation. I would immedi-
ANALYTICS: MULTI-TOUCH ATTRIBUTION
Assuming everything is reporting back as de-
signed, we still have a lot of work to do to help
optimize our ad spend through analytics. One of
the tough things to capture and the most elusive
in the e-commerce space is “multi-touch attribu-
tion” (MTA). This is about understanding what it
costs to get someone to our site. It’s the holy grail
for marketing when we can truly understand what
impressions visitors saw on Facebook, organic
search, and paid search and how they got to the
Referrer tracking
CTO + Operations Partnership: Post-funnel metrics
My job isn’t over when the customer converts on
the site. I also need to make sure we’re perform-
ing after the order is taken and doing a proper
handoff to fulfillment. That means making sure
the orders are processed and flowed to the
warehouse and that credit cards were billed
Orders sent to warehouse
(expected and actual)
10 — 12
ately want to stop spending money on ads until
the problem is resolved. As a last resort, Google
and Facebook will stop serving ads and driving
traffic to the site, but I’d prefer to know well be-
fore that happens.
site. Did this person see a billboard, TV com-
mercial, or Facebook ad? Where did they come
from and how much did we spend to get them to
our site? This is a complicated instrumentation
and analytics exercise. It is the responsibility
of the digital and marketing teams to drive this
work, but my team supports them technically to
stitch all the vendor data together with our onsite
analytics.
I use this to closely monitor where traffic is com-
ing from, such as paid ads, organic (direct) search,
or our social and email campaigns.
properly and on time. For a subscription business,
we process orders for customers automatically
on a recurring basis though large batch jobs that
happen every night. Here are the primary metrics
I track in this area:
I pay close attention to the number of orders I ex-
pect to send to the warehouse, and the number ac-
tually received by the warehouse. If we miss a day,
that’s bad because people are waiting idle at the
warehouse one day and have a huge backlog the
next day (sometimes requiring extra resources to
get everything shipped out).
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO
Payments processed
(expected and actual)
Because we are a subscription business, there are
some unique things I pay attention to. The worst
day of the year for us is February 28 because
we need to process payments and orders for all
subscriptions for the 28th, 29th, 30th, and 31st.
Business performance metrics where technology has an impact
The technology team plays a key role supporting
business performance metrics, but so does virtu-
ally every other team. These metrics are lagging
indicators of technology issues: I do not want
to see technology issues reflected in business
metrics. Technology issues should be detected
Average order value
(AOV)
Order rate/Ship rate
(orders per hour/day)
Conversion rate
11 — 12
As a subscription business, people expect to be
charged and receive email confirmations consis-
tently each month. If we start moving around the
day, that creates a poor customer experience. I
pay close attention to the number of payments I
expect to process, the number actually processed,
and the number that failed to process. Then we
need to dig into the number that failed to process
and find out why.
Similarly, holidays also add complexity to our
operations. Any issues with order or payment pro-
cessing have ripple effects for fulfillment, opera-
tions, and customer service.
well before they materially affect the business. If
we do our part on the areas I described above, the
business performance metrics will follow. Here
are the top business metrics the tech team ulti-
mately supports:
The average dollar value for orders in a given time
period. Numerous technology issues can affect
AOV, from site availability to site speed to errors
to payment processing.
AOV tell us that we’re making money. Orders per
hour/day tells us if we’re making money at the
rate we should be. Again, numerous technology
problems can slow down our order rate.
The percentage of users who complete a certain
action (usually a purchase). In my experience,
conversion rate is tricky to talk about because it
depends on the team and the context. A marketer
might
HOW TO PRIORITIZE METRIC S AS AN E- COMMERCE C TO
Churn/cancellation rate
Return rate
Technology problems create friction that will drive these business metrics in the wrong direction. This is
why I’m obsessive about my platform reliability and performance metrics.
Conclusion
CTOs from e-commerce and retail firms swim
in a sea of business and system health data.
These leaders are ultimately responsible for very
granular technical performance metrics (like
disk space), but they also play a vital role sup-
porting key business performance metrics (like
1 2 — 12
might care about the conversion rate of a single
email. Others might care about the conversion
rate of all site visitors (and even then there are
different methods: some people include bounces,
others do not). The key is to get on the same page
quickly about what version of conversion rate
you’re talking about and what assumptions you’re
making.
We closely monitor the rate of customer cancel-
lations and try to correlate cancellations to root
causes. Technology issues leading to poor cus-
tomer experiences could very well be a cause of
churn if your recurring processes and systems are
not reliable and predictable.
We closely monitor the order return rate and
try to correlate cancellations to root causes.
Technology issues can sometimes interfere with
getting an order right.
conversion rate). That’s why I wrote this article:
to offer what I think is an efficient framework for
thinking about and prioritizing metrics, so that
e-commerce CTOs can check what matters in 10
minutes, and then go about their days.