Class Session 11

Product Metrics
MGT 561 – Product Management
Professor Alex Burnap
LOGISTICS
§ Released (last class): Team Assignment IV: Product
Requirements Document (PRD)

Team Assignment IV: Product Reminder: Your new product feature must
§ What is a Product Requirements Document (PRD)? Requirements Document (PRD) solve the Course Project Problem Statement

§ Primary communication tool between the PM, Engineering, and

Design (product “Triad”)
§ Justifies and Defines {Improve, New, Kill} a Single Product Feature

• Course Project and Team Assignment IV

• Your team will define your new
product feature in the PRD.
• Your team will present contents of
your PRD for Final Presentation

2
PREVIEW OF YOUR PRD: TO COMMUNICATE PROBLEM SPACE TO ENGINEERING AND
DESIGN AND ITERATE ON SOLUTION DEFINITION
§ Product Requirement Document
§ Document for a single product feature
§ Details the problem space, solution space, testing and launch plan.
§ Details the customers we are targeting, and evidence that this actually a
real customer problem/need and business opportunity.
§ Document varies significantly across companies

§ Goal
§ Cross-functional alignment across product, engineering, design,
marketing/PMM/Go-to-Market (GTM)
§ Tool for communication and source of truth for why
§ Iterative, living document across functions
Team Assignment IV: Product
§ “Problem Space” justification with “Solution Space” scoping, not how to Requirements Document (PRD)
implement the feature itself

§ Not the Goal

§ PM “handoff” to engineering/design/marketing/PMM
§ This is how PRDs sometimes get a bad name.
3
PREVIEW OF YOUR PRD: PROBLEM DEFINITION AND SOLUTION DEFINITION
§ Problem Definition
§ Primary responsibility of you as the PM
§ What is the customer’s primary need / problem?
§ How did we validate it was a “real” customer needs?
§ How ”painful” or important is the need?
§ How many customers does the need affect in TAM?
§ What is the business case?
§ How does this align with strategic objectives (OKRs)?
§ Basically, what we have focused on in class until now…

§ Solution Definition
§ Responsibility of PM is to communicate “problem space”
to “solution space" designers and engineers
§ “User Stories” for communicating problem space
§ Afterwards, PM is often just a ”clarification” role

4
“USER STORIES” DEFINED IN THE PRODUCT REQUIREMENT DOCUMENT (PRD)
§ Important: “Customer Needs” are higher-level concepts than “User
Stories” found in “Section 2 - Solution Definition.” Take for example
the customer “restaurant manager” with customer need “food
delivery accountability.” A single customer need is at the same
conceptual level a single associated “Product Feature” and a single
associated Product Requirements Document (PRD), where “Product
Feature” is from the viewpoint of the customer. However, a single
customer need leads to several “User Stories” such as, “As a
restaurant general manager, I want to quickly understand how often
food deliveries are taking longer than quoted to food orderers today,”
which leads to a single “sub-feature” from viewpoint of
engineering/design, “Show Average Overquote Delivery View in
Manager Dashboard.”

§ Example User Story:

Customer Need (in form of “User Story”)
“As a restaurant general manager, I want to quickly understand how
often food pickups are taking longer than quoted to food orderers today,
so that I can make changes to my quoting or restaurant operations.” (Blue): “User Story” written by PM
(Red) : Details solution by Design and Engineering

5
USER STORIES
§ User Stories
§ “Input” to engineering and design development workflow
§ From PM’s “problem definition” and “solution definition”
§ Already some degree of input from initial cross-functional
“sign off” across product/engineering/design/marketing

§ User Story maps to a feature (eng/des viewpoint)

§ are “smaller pieces” of Product Feature (customer view)

§ Note
§ this is one typical “workflow” for product development, and
of course varies across firms
§ We chose this workflow since it minimizes common
problem of “feature factory”

6
WRITING USER STORIES
§ General Structure
§ As a (who)
§ I want to (what)
§ So that (why)

§ Example
§ “As a restaurant general manager, I want to quickly understand
how often food pickups are taking longer than quoted to food
orderers today, so that I can make changes to my quoting or
restaurant operations.”
Iteratio
§ User Stories are written in the “Problem n
Space”
§ focused on the user (the customer), NOT the product
§ not meant to be too detailed Problem Space Solution Space
§ not meant to be a replacement for communication (PM) (not you)
between the engineers and PMs and designers

7
COURSE PROJECT: BEGIN DEFINING YOUR NEW PRODUCT FEATURE TODAY
§ Goal: What new product feature solves the
customer need(s) you identified, segmented, and
targeted in Course Project: Midterm Presentation

§ Note: You do not need to stick with your ”current”

product feature “idea.” But brainstorm solutions Course Project
and refine along the way. Product design is iterative. “Future of Work”

§ You will need to define for your “eventual” feature:

§ Product Metrics
§ Scientific Hypothesis (for your metrics)
§ Did our feature “work”? How well?
§ A/B testing (for your hypotheses, for your metrics)

§ We will cover these topics with a feature in mind! So

begin brainstorming and defining feature(s) now.
8
 LAST CLASS
§ Goals of Product Feature Strategy
§ Product feature strategy has different goals (depends on feature)
§ Key Point: Measuring your feature goal via product metrics is critical.
§ Brief Intuition on Product Feature Strategy vs Product Growth Strategy
§ (we’ll talk today in detail about how products (and companys) “fail”)

§ 3 Feature Decisions: Improve Existing, Add New, Kill Features

§ AARRR Funnel: Key tool for diagnosing product problems and

measuring health of our product. Good Growth, Bad Product (Metrics)

§ Headspace Case Study

§ Identified problem using product metrics and the AARRR framework.
§ Used customer journey as tool to understand customer needs and value
from features. This included different segments of customer needs.
§ Experimentation for evaluation of product feature.
High-Friction, Personalized Activation
§ Feature improved value to customer, increasing Product-Market Fit. 9
TODAY’S LEARNING OBJECTIVES
§ Motivation: You are a PM at Headspace

§ Theory: Product Feature Metrics vs Product Growth Metrics Product vs Growth Metrics
§ Definition of Product Feature and Growth Metrics
§ Why this is important? 2x2 Matrix of “Good and Bad Product/Growth Metrics”

§ Types of Metrics (High-Level Concepts)

§ Leading and Lagging Metrics
2x2 Matrix of Metrics: Success and Failure
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ AARRR Funnel Ratio Metrics

§ Metric Hierarchy (Mid-Level Concepts)

§ Primary, Secondary, Guardrail Metrics Ratio and Cohort Metrics

§ How to Measure Metrics + How are Metrics Tracked? (Low-Level Concepts)

§ Product Instrumentation and Metric Tracking

Product Instrumentation and Metric Tracking

10
RECALL: HEADSPACE - VERY GOOD “ACQUISITION” BUT LOW “RETENTION”
“We are growing very aggressively. In 2017, we grew 75
percent. But when we look at the underlying metrics, we are
buying our growth. We’re building up this bigger and bigger
snowball that could potentially be very hard to control.”
- Robert Viera, Head of Product, Headspace

Less than 20% of users retained after 1 month

(Recall: retention “plateau” is quantitative measure of to product-market fit)

Why is this bad?: “Growth before product-market fit”

11
 RECALL: HEADSPACE - VERY GOOD GROWTH BUT BAD PRODUCT (“RETENTION”)

“We are growing very aggressively. In 2017, we grew 75

percent. But when we look at the underlying metrics, we are
buying our growth. We’re building up this bigger and bigger
snowball that could potentially be very hard to control.”
- Robert Viera, Head of Product, Headspace

Less than 20% of users retained after 1 month

(Recall: retention “plateau” is quantitative measure of to product-market fit)

High-User Growth (Metrics)

12
MOTIVATION: YOU ARE A PM @ HEADSPACE: NEW ONBOARDING FLOW
RECALL: YOU CHANGED “LOW FRICTION” ACTIVATION TO 4X LONGER “HIGH FRICTION BUT
PERSONALIZED” ACTIVATION

Question (1/5): What do you think this change did to the

original activation metric “first meditation”? Why?
13
MOTIVATION: YOU ARE A PM @ HEADSPACE: NEW ONBOARDING FLOW
WHAT IS YOUR “NORTH STAR” <ACTIVATION> METRIC?

Question (2/5): What would you define as the new “activation” metric?
Why? What timescale would you measure this over?
(Old metric: “first meditation”)
14
MOTIVATION: YOU ARE A PM @ HEADSPACE: HIGHER-LEVEL METRICS
PRODUCT GROWTH STRATEGY GOAL - INCREASE MONTHLY ACTIVE USERS (MAU)

Question (3/5): How might your new activation

metric affect Monthly Active Users (MAU)?

PM Concept: “metric hierarchy” – “higher-level”

metrics are made up of lower-level metrics
Mathematical Example: All product growth metrics
are functions of product metrics
15
YOU ARE A PM @ HEADSPACE: WHAT USER “EVENTS” DO YOU TRACK?

Question (4/5): Where in the Headspace app would you measure your new “activation” metric?
What event “trigger”? (e.g., “user clicks on <example> button”)
What event “properties”? (e.g., user id, session id, percent meditation completed, etc.)

16
YOU, THE PM @ HEADSPACE: “GUARDRAIL” METRICS
Context: There are other product teams, including an
“ads team”
§ Ads Team Primary Metric (i.e., “north star” metric)
§ # of ad impressions
§ click-through rate (CTR) of ads.
§ Problem: Their “guardrail” metrics on ad impressions
drops below threshold. No promoted spend on your
new onboarding flow.

§ Question (5/5): What other metrics are affected by

the new change to the Headspace onboarding flow?
(PM Concept: “Guardrail” metrics)
§ Example Answer:
§ Ad Team ”Guardrail” Metric: # of ad impressions drops below 2
per user session

17
TODAY’S LEARNING OBJECTIVES
§ Motivation: You are a PM at Headspace

§ Theory: Product Feature Metrics vs Product Growth Metrics Product vs Growth Metrics
§ Definition of Product Feature and Growth Metrics
§ Why this is important? 2x2 Matrix of “Good and Bad Product/Growth Metrics”

§ Types of Metrics (High-Level Concepts)

§ Leading and Lagging Metrics
2x2 Matrix of Metrics: Success and Failure
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ AARRR Funnel Ratio Metrics

§ Metric Hierarchy (Mid-Level Concepts)

§ Primary, Secondary, Guardrail Metrics Ratio and Cohort Metrics

§ How to Measure Metrics + How are Metrics Tracked? (Low-Level Concepts)

§ Product Instrumentation and Metric Tracking

Product Instrumentation and Metric Tracking

18
RECALL: GOAL OF PRODUCT FEATURE STRATEGY
§ Many possible goals for new product feature.
Generally, multiple goals at the same time:
§ Increase value of product to customers (e.g., increase retention)
§ Alignment with strategic objectives (e.g., increase WAU)

§ Key Question: how do we measure whether we are

achieving our product feature “goal”?
§ Answer 1: Define the “right” product metrics
§ Answer 2: Define the “right” scientific hypothesis (for metric)
§ Answer 3: Define the “right” A/B testing of our hypothesis
Should we focus primarily on increasing
§ Good product metrics value of product for product-market fit?
§ Tells us how inputs change outputs.
§ Account for Scale / Time
§ Actionable for changing product features.
19
RECALL: PRODUCT FEATURE STRATEGY VS PRODUCT GROWTH STRATEGY
§ Product feature strategy focuses on our product features AND how an
average (or cohort’ed) customer behaves, and 3 decisions we can make:
§ Improve existing features
§ Add new features
§ Kill existing features

§ Product growth strategy focuses on our product AND our total new and
existing customers, and how to increase the # of customers or revenue Product Feature
§ Tradeoffs between acquisition, retention, revenue Strategy
§ Market expansion strategy

§ Key Takeaway
§ Product feature strategy (and metrics) typically consider a single customer, and how
changing features affect their individual behavior.
§ Product growth strategy (and metrics) always consider the total customer base
(existing and new), and how changing features affect the overall customer base

§ Corollary: Product Metrics and Growth Metrics are Different Product Growth
§ Feature: Which product feature will help us increase retention? Strategy
§ Growth: Do we tradeoff overall acquisition with revenue in the short term?
20
RECALL: HEADSPACE - VERY GOOD GROWTH BUT BAD “RETENTION”
“We are growing very aggressively. In 2017, we grew 75
percent. But when we look at the underlying metrics, we are
buying our growth. We’re building up this bigger and bigger
snowball that could potentially be very hard to control.”
- Robert Viera, Head of Product, Headspace

Less than 20% of users retained after 1 month

(Recall: retention “plateau” is quantitative measure of to product-market fit)

Metric: Monthly Active Users (MAU) Metric: 30-Day-Retention Rate

Question: Is this a product or growth metric? Why? Question: Is this a product or growth metric? Why?

21
DEFINITION: PRODUCT FEATURE METRICS VS PRODUCT GROWTH METRICS
§ Product Metric: Product metrics characterize how a product takes in new
users, processes them, and produces active users, revenue, IT support
requests, etc.

§ Key Takeaway: Product metrics do not depend on number of users.

§ If you calculate a product metric based on 10,000 new users and then do the Example Product Metric
same based on 1,000,000 newcomers, there will be (very little) difference. Customer Lifetime Value (LTV)

§ Growth Metric: Growth metrics directly depend on the number of new

users. If the number of users changes, the growth metric also changes.

§ Growth metrics are “higher-level” than product metrics

§ Mathematically, Growth Metric = f(product metrics, # of users)

Example Growth Metric

Monthly Recurring Revenue (MRR)

22
IN-CLASS EXERCISE: PRODUCT METRICS VS GROWTH METRICS

23
Credit: GoPractice
RECALL: (1/2) WHY THIS MATTERS? EASY TO BY MEASURE THE “WRONG” METRICS

Example Growth Metric: MAU

Example Product Metric: Retention
Monthly Active Users (DAU)

§ Product-Market fit (PMF) - Product is known by customers in TAM, product’s features

satisfy customer needs to give “value” to customers, and customers choose (or retain)
our product versus competitors.

24
RECALL: (2/2) WHY THIS MATTERS? MANY PRODUCTS “FAIL” BY TRYING TO GROW
BEFORE HAVING ENOUGH PRODUCT-MARKET FIT (I.E., DELIVER ENOUGH “ VALUE”)
Percent of Total Addressable Market (TAM)

Bass Diffusion Curve

Initial and Growth Maturity

Improved
Product-
Market Fit

Time
Key Takeaway: Product Growth without enough Product-Market Fit (PMF)
leads to overwhelming customer churn (i.e., lack of retention).
(i.e., we do not deliver enough “value” from “features” to satisfy ”needs”) 25
 2X2 MATRIX: PRODUCT METRICS VS GROWTH METRICS
Good Growth Metrics Bad Growth Metrics

Good Product
Metrics
Bryan Enriquez Failure: Not transitioning from “enough”
Yale MBA 2020 Product-Market Fit to focus on Growth

Generative AI, Blockchain,

Facebook Metaverse, Google Glass,
Carbon Nanotubes, Cold Fusion,
Bad Product
Metrics Failure: Over 99% of startups never
Failure: Trying for Growth before “enough” Product-Market Fit to get
“enough” Product-Market Fit
initial traction, much less growth.

Takeaway: You can have all 4 cases. Many companies ”fail” due to being
in the bottom left or top right case.
26
 QUESTION: WHICH CASE IN THE 2X2 HAVE WE NOT (YET) SEEN IN CLASS?
(FORESHADOWING: THIS IS A WILD BUT ALSO COMMON CASE)

Example: Grubhub had increasing growth metrics and “good enough” product metrics
Why Fail? Grubhub “failed” by not focusing enough on “growth” after “enough” product-market fit

27
TODAY’S LEARNING OBJECTIVES
§ Motivation: You are a PM at Headspace

§ Theory: Product Feature Metrics vs Product Growth Metrics Product vs Growth Metrics
§ Definition of Product Feature and Growth Metrics
§ Why this is important? 2x2 Matrix of “Good and Bad Product/Growth Metrics”

§ Types of Metrics (High-Level Concepts)

§ Leading and Lagging Metrics
2x2 Matrix of Metrics: Success and Failure
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ AARRR Funnel Ratio Metrics

§ Metric Hierarchy (Mid-Level Concepts)

§ Primary, Secondary, Guardrail Metrics Ratio and Cohort Metrics

§ How to Measure Metrics + How are Metrics Tracked? (Low-Level Concepts)

§ Product Instrumentation and Metric Tracking

Product Instrumentation and Metric Tracking

28
TYPES OF METRICS
§ Types of Metrics
§ Leading and Lagging Metrics
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ Special Case: AARRR Funnel Ratio Metrics
§ Primary, Secondary, Guardrail Metrics

29
LEADING AND LAGGING METRICS

§ Leading Metrics are inputs, lagging metrics are

outputs of product strategy.
Engagement “Nudges” & “Revival”
(Leading Metrics: % revived)

§ Lagging Metrics are often KPIs, but typically

indirectly affected by product features.
Usually: “North star metric” High-Friction, Personalized Activation
Always: Product Growth Metrics (Leading Metrics: onboarding completion rate, avg.
session time)

§ Goal: Try to define leading product metrics that

impact later (important) lagging metrics. “Low 30-Day Retention Rate”
(Lagging Metric)

30
COHORT METRICS
§ Types of cohorts
§ Temporal cohorts (users activated in January)
§ Behavioral cohorts (power users, casual users)
§ Revenue cohorts (repeat buyers, low frequency)
§ Needs cohorts (different needs, customer journeys)

§ Why cohorts?
§ Metrics often average out trends. This can be
misleading.
§ Cohorts help separate out trends.

Croll, A., & Yoskovitz, B. (2013). Lean analytics: Use data to build a better startup faster. O'Reilly Media, Inc.
31
EXAMPLE COHORT METRIC: COHORT RETENTION RATE
§ Example: Retention rate across temporal
cohorts

§ Why? Key metric of product-market fit over time

§ Critical to define retention metric correctly

(i.e., what is core action)
§ Dropbox: upload file
§ Dashlane: login with password

32
ABSOLUTE VS RATIO METRICS
§ Absolute vs Ratio Metrics
§ Absolute: Daily Active Users (DAU)
§ Ratio: Daily Avg. Users (DAU) / Monthly Avg. Users (MAU)

§ Ratios generally more useful than absolute metrics for

tracking changes in product health
§ Normalized to unit of analysis

§ Example Ratio Metrics:

§ Channel attribution %
§ LTV:CAC ratio (Lifetime Value : Customer Acquisition Cost)

33
EXAMPLE RATIO METRIC: CHURN
§ Churn Rate (CR) is the percentage of users who end relationship with firm during
a given period.
§ Churn of users
§ Churn of revenue

§ Opposite of Retention
§ Churn Rate = 100% - Retention Rate

Examples
§ User Churn Rate - (Users beginning of month/day/week – Users end of month/day/week) / Users
beginning of month/day/week
§ Revenue Churn Rate - For a subscription-based business model with monthly recurring revenue
(MRR): [(MRR beginning of month – MRR end of month) – MRR in upgrades/upsells during month]
/ MRR beginning of month.

34
EXAMPLE RATIO METRIC: DAU/MAU (ALSO CALLED “STICKINESS”)

§ DAU/MAU: Daily Active Users (DAU) / Monthly Active Users (MAU)

§ Pros: Common metric for product engagement / retention.
§ Cons: Snapshot in time. Susceptible to “shocks” like marketing campaigns. User
opens email once and never again.
§ Best for: Social apps, messaging apps. Not for: E-commerce, AirBnB 35
AARRR FUNNEL (RATIO) METRICS
§ Acquisition Rate = Targeted Users divided by Acquired User
§ Problems with consumer awareness
§ Problems with product-channel alignment

§ Activation Rate divided by Acquisition Rate

§ Problems with user onboarding friction
§ Problems with user understanding value proposition

§ Retention Rate divided by Activation Rate

§ Problems with value proposition.
§ Problems with user engagement and frequency.

§ Revenue Rate divided by Retention Rate

§ Problems with value proposition.
§ Problems with pricing.

§ Referral Rate divided by Retention Rate

§ Problems with incentivization.

36
EXAMPLE: ACQUISITION PROBLEM AT UBER
§ Problem
§ (2016) 1.7 / 5.0 star rating
§ 50% ratio acquisition -> activation

§ Product Feature
§ Push Notification: In-app rating review
§ Leading vs Lagging Metric

§ Outcome
§ (2017): 4.7 / 5.0 star rating

Credit: Andrew Chen, a16z

37
IN-CLASS EXERCISE: AARRR FUNNEL (RATIO) METRICS
§ Acquisition Rate = Targeted Users divided by Acquired User
§ Problems with consumer awareness
§ Problems with product-channel alignment

§ Activation Rate divided by Acquisition Rate

§ Problems with user onboarding friction
§ Problems with user understanding value proposition

§ Retention Rate divided by Activation Rate

§ Problems with value proposition.
§ Problems with user engagement and frequency.

§ Revenue Rate divided by Retention Rate

§ Problems with value proposition.
§ Problems with pricing.

§ Referral Rate divided by Retention Rate

§ Problems with incentivization.

38
IN-CLASS EXERCISE: YOU ARE A PM @ HEADSPACE: NEW ONBOARDING FLOW

Context: You are a PM measuring

“Headspace Community” conversion
after the change to onboarding flow.
Headspace Community
(In-app running incentive)
39
IN-CLASS EXERCISE: FUNNEL METRICS AT HEADSPACE
§ Context: You are a PM measuring
Headspace Community conversion
after the change to onboarding flow.

§ Key Concept: Funnel “Events”

§ Step A = “First Basic Meditation” (the user
completed at least 25% of the Basic Meditation)
§ Step B = “First Need State Meditation” (the user
completed at least 25% of first <Sleep> Meditation)
Step A Step B Step C
§ Step C = “Headspace Community Click-Through”
(the user clicked on the Headspace Community
product feature)

§ Your Metric: Conversion from Step A

à Step B à Step C within 15 days
§ Note: Funnel’s date range is February 1-28.
Question (1/3): How many users will be counted at the Step 3
of the funnel if conversion window is 15 days?
40
Figure Credit: GoPractice
IN-CLASS EXERCISE: FUNNEL METRICS AT HEADSPACE
§ Context: You are a PM measuring
Headspace Community conversion
after the change to onboarding flow.

§ Funnel “Events”
§ Step A = “First Basic Meditation” (the user
completed at least 25% of the Basic Meditation)
§ Step B = “First Need State Meditation” (the user
completed at least 25% of first <Sleep> Meditation)
Step A Step B Step C
§ Step C = “Headspace Community Click-Through”
(the user clicked on the Headspace Community
product feature) Question (2/3): What is the value of “Your Metric” over the timescale
given above (Feb. 1 -> Feb 28)? (i.e., the percentage of users)
§ Your Metric: Conversion from Step A
à Step B à Step C within 15 days
§ Note: Funnel’s date range is February 1-28.

41
Figure Credit: GoPractice
IN-CLASS EXERCISE: FUNNEL METRICS AT HEADSPACE
§ Context: You are a PM measuring
Headspace Community conversion
after the change to onboarding flow.

§ Funnel “Events”
§ Step A = “First Basic Meditation” (the user
completed at least 25% of the Basic Meditation)
§ Step B = “First Need State Meditation” (the user
completed at least 25% of first <Sleep> Meditation)
Step A Step B Step C
§ Step C = “Headspace Community Click-Through”
(the user clicked on the Headspace Community
product feature) Question (3/3): What percentage of users reached Step B of the
funnel (event “First Need State Meditation”)?
§ Your Metric: Conversion from Step A
à Step B à Step C within 15 days
§ Note: Funnel’s date range is February 1-28.

42
Figure Credit: GoPractice
TODAY’S LEARNING OBJECTIVES
§ Motivation: You are a PM at Headspace

§ Theory: Product Feature Metrics vs Product Growth Metrics Product vs Growth Metrics
§ Definition of Product Feature and Growth Metrics
§ Why this is important? 2x2 Matrix of “Good and Bad Product/Growth Metrics”

§ Types of Metrics (High-Level Concepts)

§ Leading and Lagging Metrics
2x2 Matrix of Metrics: Success and Failure
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ AARRR Funnel Ratio Metrics

§ Metric Hierarchy (Mid-Level Concepts)

§ Primary, Secondary, Guardrail Metrics Ratio and Cohort Metrics

§ How to Measure Metrics + How are Metrics Tracked? (Low-Level Concepts)

§ Product Instrumentation and Metric Tracking

Product Instrumentation and Metric Tracking

43
METRIC HIERARCHY: PRIMARY METRICS, SECONDARY METRICS, GUARDRAIL METRICS
§ Primary Metrics
§ What is the core metric we are aiming to track?
§ “North star” metric

§ Secondary Metrics
§ Which metrics are needed to measure the primary
metric?
§ Mathematically, primary metric = f(secondary metrics)
§ Example: “Monthly Active Users (MAU)” is a function of
“Average Retention Rate”

§ Guardrail Metrics
§ What metrics let us know thing are going not as
intended?
Breaking down “north star” into component metrics
§ Typically leading, not lagging metric
44
EXAMPLE: METRIC HIERARCHY - PRIMARY, SECONDARY, GUARDRAIL METRICS
§ Context: You’re a PM on the Yelp Advertisements Team

§ Primary Metric
§ Yelp Ad Revenue per Hour for Local Business Pages
§ Question: What secondary metrics might we need?
§ Secondary Metrics (go into the primary metric)
§ Average Cost per Click (CPC)
§ Number of users per hour seeing Ad Impression
§ Number of Ad Impressions per “Clicked On” Business IDs

§ Guardrail Metrics (are you affecting other key metrics?)

§ Other Primary Metrics: Weekly Active Users (WAU)
§ Other PM Team Metrics: Searches for local businesses (Consumer Content PM)
§ Other Secondary Metrics: Are users changing? Power vs Casual Users
§ Other PM Team Secondary Metrics: Are different types of local businesses
being clicked on? 45
PM ROLE “SCOPE”: THE METRICS YOU ARE IN CHARGE OF LARGELY DEPEND ON
WHAT LEVEL YOU ARE IN THE PRODUCT ORGANIZATION
§ Stage of Product Career (typically) “Owns” what?

PM, Associate PM (APM) Product Feature

PM Lead, Group Lead Product / Product Component
Product Leader / Director Product Line / Product

§ Corresponding Product Metrics

PM, Associate PM (APM) Product Metrics
PM Lead, Group Lead Product and Growth Metrics
Product Leader / Director Product, Growth, Product Line Metrics
Related Resources
Yale Career Panel: A Candid View of Product Management
PM Levels at Amazon, FB, Google, LinkedIn, Twitter

46
TODAY’S LEARNING OBJECTIVES
§ Motivation: You are a PM at Headspace

§ Theory: Product Feature Metrics vs Product Growth Metrics Product vs Growth Metrics
§ Definition of Product Feature and Growth Metrics
§ Why this is important? 2x2 Matrix of “Good and Bad Product/Growth Metrics”

§ Types of Metrics (High-Level Concepts)

§ Leading and Lagging Metrics
2x2 Matrix of Metrics: Success and Failure
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ AARRR Funnel Ratio Metrics

§ Metric Hierarchy (Mid-Level Concepts)

§ Primary, Secondary, Guardrail Metrics Ratio and Cohort Metrics

§ How to Measure Metrics + How are Metrics Tracked? (Low-Level Concepts)

§ Product Instrumentation and Metric Tracking

Product Instrumentation and Metric Tracking

47
HOW ARE METRICS TRACKED IN TECH PRODUCTS?
§ Question: You are a PM that wanting to track “events” in
our product. You are about to speak to the engineering
manager for metric tracking. What do you need to know?

§ Key Point: The PM generally does not code in metric

tracking, but the PM needs to know what should be tracked.

§ How?
§ Recall: Metrics tracked by defining “events” by our users
§ PM usually defines event “triggers” and event “properties”
§ Engineering “instruments” our product to track “events”

§ Example User Events

§ Amazon: “Clicked on Shopping Cart Checkout Button”
§ Spotify: “Clicked skip track to next song”
48
“EVENTS” ARE TRACKED ON EITHER FRONT OR BACK-END
§ Front-end Front End Back End
§ Client (web or mobile browser)
§ Languages: Javascript

§ Front-end
§ Client (mobile native app)
§ Languages: Swift (iOS), Java (Android)

§ Back-end
§ Language: What backend server code is written in
§ Examples: Python, Ruby, Java
§ Note: Javascript on back-end too (e.g., Node.js)
Database

49
USERS, SESSIONS, EVENTS

User Metrics
User
e.g. days since last
session

Session Session Session Session Metrics

e.g. number of
unique meditation
videos clicked on
Event Event Event

Event Event Event

Event Metrics
e.g. meditation
Event Event video plays

Event
50
EVENTS
§ Event = “User 12” triggered “Clicked on Unique
Meditation Video” with need state “Sleep”

§ 2 Things to Define
§ Event Trigger
§ Event Properties

§ Event Trigger: “Clicked on Unique Meditation Video”

§ Event Properties:
§ User ID: 01245 (from stored “Cookie” or via native app)
§ Meditation Video ID: “04288” (can link to need state “Sleep”)
§ Time: 11:59:12 PM, UTC - 6
§ Percent Video Completed: 0.42
51
4-STEP FRAMEWORK: HOW TO MEASURE A PRODUCT METRIC
1. Identify user “event(s)” to track (i.e., measure):
• Count (e.g. clicks, page views, visits, downloads)
• Time (e.g. minutes per session)
• Value (e.g. revenue, units purchased, ads clicked)
• Key Point: This is the scientific “hypothesis” you want to test
( more very soon in A/B Testing)

2. Choose the unit of analysis

• Per session (e.g. minutes per session)
• Per user (e.g. clicks per user)
• Per page (e.g. revenue per page)
• Per time (e.g. views per month)
3. Define event “triggers” and “properties” for your event
4. Choose the appropriate statistical function
• Boolean (e.g., user “activated” by clicking on first personalized meditation”)
• Average (e.g. average minutes per session)
• Total (e.g. total revenue per month)
• Count (e.g. click count per week)
• Median (e.g. median revenue per user) Credit: Dan Lee 52
EXAMPLE: STREAMING MUSIC APP
Context: You are a PM at Spotify and have
launched a new recommendation engine for
adding songs to Spotify Playlists.

Question: Which of these events would you pick

to track product feature success?
1. Songs clicked
2. Songs added
3. Songs added per user
4. The average number of songs added per user

Credit: Dan Lee 53

EXAMPLE: STREAMING MUSIC APP
Context: You are a PM at Spotify and have launched a new
recommendation engine for adding songs to Spotify Playlists.
Question: Which of these would you pick?
1.Songs clicked
2.Songs added
3.Songs added per user
4.The average number of songs added per user

§ Bad
§ Option 1, songs clicked, misses the purpose of a playlist recommendation, which
helps users find songs to add to their playlist. Merely clicking a song is not
representative of finding a relevant song. Hence, click is not the most meaningful
action to use.
§ Option 2, songs added, is slightly better than songs clicked as it measures the
intended purpose of a playlist recommendation. However, the metric lacks
granularity based on a unit of analysis and statistical function.

§ Better
§ Option 3, songs added per user, is definitely better than the first two, but it’s unclear on the
calculation of the songs added per user. Is it the total or average number of songs added per
user?

§ Best
§ Option 4, the average number of songs added per user, is the best given that it contains all
the three properties required to make a meaningful metric: action, unit of analysis and
statistical function.

Credit: Dan Lee 54

TODAY’S LEARNING OBJECTIVES
§ Motivation: You are a PM at Headspace

§ Theory: Product Feature Metrics vs Product Growth Metrics Product vs Growth Metrics
§ Definition of Product Feature and Growth Metrics
§ Why this is important? 2x2 Matrix of “Good and Bad Product/Growth Metrics”

§ Types of Metrics (High-Level Concepts)

§ Leading and Lagging Metrics
2x2 Matrix of Metrics: Success and Failure
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ AARRR Funnel Ratio Metrics

§ Metric Hierarchy (Mid-Level Concepts)

§ Primary, Secondary, Guardrail Metrics Ratio and Cohort Metrics

§ How to Measure Metrics + How are Metrics Tracked? (Low-Level Concepts)

§ Product Instrumentation and Metric Tracking

Product Instrumentation and Metric Tracking

55
IN-CLASS EXERCISE: PLANT SUBSCRIPTION
§ Context: You are a PM at a plant delivery
subscription product

§ Goal: Track “events” on customer

journey to measure AARRR metrics
§ Churn rate %
§ Needs: “user acquired”, “user activated,”
“view cart,” etc.

Credit: Lex Roman

56
 INSTRUMENTING “EVENTS” OF CUSTOMER JOURNEY

Event: Event: Event: Event:

User views subscription User clicks subscription User views cart User begins checkout

57
EVENTS
Question: Which of the following are “events”?
(Select more than one)
❏ Added to cart
❏ Performed a search
❏ Out of stock
❏ Day of the week
❏ Initiated Cart Checkout
❏ Successful Payment

58
EVENT PROPERTIES
Question: Which of the following are “events”, which are “event properties”?

❏SKU: 2340983 EVENT PROPERTY

❏Add to Cart EVENT
❏Song Played EVENT
❏Genre: Pop EVENT PROPERTY
❏Reservation Booked EVENT
❏Type: Dog Grooming EVENT PROPERTY

59
WHAT DATA DID WE SAVE DURING CUSTOMER JOURNEY?
§ User
§ Initial referrer
§ Email
§ Phone
§ Landing Page
§ Subscription Type

§ Session
§ Session Length
§ Pages Visited

§ Events
§ Viewed Subscription
§ Clicked Subscription
§ Viewed Cart
§ Began Checkout

60
PRODUCT INSTRUMENTATION SPECIFICATION
(WHAT YOU AS A PM MIGHT DEFINE ALONG WITH ENGINEERING)
Event Name (case propertiesToTrack Example Property Values
# Screen Triggers when... Data / Property Type Description
sensitive!) (camelCase) (case sensitive)

See Mixpanel docs for default All properties below should

N/A N/A There are default events that should be tracked by the Mixpanel SDK
events be named in camelCase

Invite
user finishes onboarding and This is how we'll know how many new users are coming in through
Social
presses the "Get Started" Get Started source Event Property - String predefined growth loops — referral invites, received invoices, or
Invoice Request
button content.
Content

Google
user selects a signup method Sign Up Selected Twitter
signupMethod Event Property - String
Facebook
Email
Google
Twitter
the login menu opens Login Selected signupMethod Event Property - String
Facebook
Email

Credit: Crystal Widjaja

Credit: Mixpanel
61
TAKEAWAYS
§ Theory: Product Feature Metrics vs Product Growth Metrics
§ Definition of Product Feature and Growth Metrics (mathematical too)
§ Why this is important? 2x2 Matrix of “Good and Bad Product/Growth Metrics”
Product vs Growth Metrics

§ Types of Metrics (High-Level Concepts)

§ Leading and Lagging Metrics
§ Cohort Metrics
§ Absolute vs Ratio Metrics
§ AARRR Funnel Ratio Metrics 2x2 Matrix of Metrics: Success and Failure
§ In-Class Exercise: Funnel Metrics (from GoPractice Ch. 6.1)

§ Metric Hierarchy (Mid-Level Concepts)

§ “North Star” Metric / Primary, Secondary, Guardrail Metrics
§ How to define a metric hierarchy - Breaking Down High-Level Metrics into Component Ratio and Cohort Metrics
Metrics

§ How to Measure Metrics + How are Metrics Tracked? (Low-Level

Concepts)
§ Product Instrumentation and Metric Tracking
Product Instrumentation and Metric Tracking
62