Product Management

GAME Framework
 GAME FRAMEWORK
The GAME framework is a 4-step process that can define metrics for any feature
or product. For the remainder of the post, I will refer to a hypothetical “product”
but feel free to substitute in “feature” whenever a “product” is discussed.

Step 1: Articulate Your Goals

This is obvious, but you should always start by defining your goals for your
product. Your goals are critical to ensuring that you maintain a purpose-oriented
strategic mindset. They also serve as tangible markers for revisiting and validating
directional correctness throughout the metrics development process. Placing goal-
setting as your first step forces you to be top-down, which is critical for developing
metrics. A bottom-up approach is suboptimal because it relies on intuition and can
often lead to analysis paralysis.

To define your goals, ask yourself the following questions:

User Goals: How will my users benefit from my product? What problems do
my users want my product to solve for them? How will my users interact with
the product? How will my users feel when they use my product? What is my
vision for how this product will integrate in my users’ life?
Business Goals: What are the tactical or strategic business benefits? Increase
revenue? Decrease cost? Be more competitive? Enter a new market? What does
my business look like if my product is successful?

Note that for many of the best products, user and business goals align. These goals
are often two sides of the same coin.
Once you have this set, you are ready to get in on the action.
 GAME FRAMEWORK
Step 2: List the Actions That Matter

The next step is to define your user actions, or more precisely, all the actions you
want your users to take within your product. This should start as a qualitative list.
Don’t worry about the numbers or whether they are trackable yet.

Here are some example questions you may ask yourself using the common ARM
metrics framework. Pick the set of questions that align with your goals:

Acquisition & Activation: How will my prospective users hear about my

product? What actions do my prospective users have to complete to become a
user of my product? What actions do my users have to take to get value from
my product? At what point is my product solving a real problem for my users?
Retention & Engagement: What gets my users to come back to my product?
What do my users do when they are engaged with and interested in my
product? What are the actions that provide my users repeat value? How often
does it make sense for my users to take those actions? Daily? Weekly? Monthly?
Once in a long while?
Monetization & Revenue: What form of “payment” does my user purchase
with, money or time? What actions do my users take for my product to start
charging? If my product is free, what are the reasons my users will opt to pay
for it? Do they pay me directly or does someone else pay me when my users
take action (e.g. ad/affiliate)? Do they pay me for every action (transaction) or
do they pay me periodically (subscription)?

You can also use metrics frameworks like AARRR to flush out the actions. The key
to doing this well is to be comprehensive but not exhaustive: have the broadest
coverage, but don’t bother differentiating the minutiae.
 GAME FRAMEWORK
Step 3: Define Your Metrics

You are now ready to turn each desired user action (qualitative) into a measurable,
trackable value (quantitative). Note that if you haven’t already done so, bring
engineering and data teams in at this stage to vet your metrics and provide technical
guidance on the feasibility of collecting/storing the desired data.

Here are some major decisions for how to count each action:
Direct vs. Proxy: Can the action be directly tracked? Or do you have to use a
proxy to measure the action? For example, you can measure “clicks” directly,
but you may need to proxy “views” with values like scrolling.
Individual vs. Aggregate: Can you group many actions for an overview, then
separate out as slices for later analysis? For example: total revenue → revenue by
product line → revenue by individual product.
Magnitude vs. Ratio: Does it matter more for you to measure the overall
magnitudes of the action? Or should you track as a comparison using a ratio,
whether it be a rate (per time) or a normalizing factor (percentage, per user,
etc)? For example, you can measure “total revenue,” or you can measure
“revenue per day” and “revenue per payer,” which will give you very different
perspectives on the same data.
Intrinsic vs. Heuristic: Can you derive more knowledge from the measure
intrinsically? Or do you have to rely on heuristics for the metric to be valuable?
For example, while “daily active user” (DAU) may be intrinsically useful for
B2C apps, a heuristic like “number of paying users who were active in the last
30 days” may be more appropriate for B2B SaaS products that are expected to
be used monthly.

For most, this is the hardest step in the framework. Some of your decisions are
informed by strong intuition, while others are informed by technical constraints or
further data analysis.
 GAME FRAMEWORK
Step 4: Evaluate your Metrics

The best way to ensure your metrics are providing you the correct insights into
your product is to test and iterate. You simply will not know how a metric behaves
until you start collecting data.
The most import evaluation you need to make is the functional usefulness of the
metric. You can check for usefulness by monitoring the metric for any false-
positives or false-negatives. For example, if the metric drops, does it signal a real
problem in the product? Conversely, if an issue surfaces, is the metric helping you
flag this issue? Of course, you would also expect the metric to flag true-positives
and true-negatives.

Different behavior with your metrics may inform iterations in each of your GAME
steps:
Evaluating Metrics: Is the data trending in the way you expected? Is the
metric stable over time (i.e. not wildly fluctuating)? Is the metric flagging issues
with your product and prompting further analysis? Are these the correct data
points to collect? If you answer “no” to any of these questions, you want to
reconsider how you developed your metrics.
Evaluating Actions: Are these actions reflective of product, user, or business
goals? Are your key actions telling the correct story about user behavior? Does
your action set cover all emerging user behaviors? Are your actions safe from
being “gamed” by your users? If you answer “no” to any of these questions, you
want to redefine the set of key actions.
Evaluating Goals: Is the metric correlating to business or user success? Are
your users happier when your metrics are positive? If you answer “no” to any of
these questions, you want to revaluate how your metrics relate to your goals.

Ask yourself these honest questions and remember that you are looking for your
metrics to measure the vital signs that give you an actionable indication of health
and performance.
APPLYING GAME: FACEBOOK NEWSFEED

Let’s apply the framework step-by-step:

Goals for the Newsfeed can be broken down between user and business goals:
User Goal: I want to be able to see the latest of what my friends are up to
without going to each of their profiles.
Business Goal: Users are only coming to Facebook when they have a reason
to: they are looking to post or they have been messaged/mentioned. We want
to improve engagement by creating an evergreen content discovery experience.

Actions taken on the Newsfeed can be collected into the following qualitative list
when considering “engagement” as the primary goal for both user and business:

Logging into Facebook to look at Newsfeed

Posting to Newsfeed (text, photo/video, photo/video album, live video)
Viewing Newsfeed items, including watching a video
Scrolling through the Newsfeed
Clicking on Newsfeed items (content, name/photo of the poster)
Commenting or replying to a comment
Liking a Newsfeed item or a comment
Sharing a Newsfeed item to your own or someone else’s wall (person, page,
group), or as a private message
Saving a post (yes, you can look at saved posts later… who knew?)
Hiding, unfollowing, reporting, or turning off notifications for a Newsfeed
item

Notice how the list is relatively comprehensive but not exhaustive: it covers all of
the bases but doesn’t try to split hairs between all the different post, click, or
comment types. We can also see how product design around creating or changing
actions can be a major lever for impacting engagement.
APPLYING GAME: FACEBOOK NEWSFEED
Metrics is where this gets interesting. In trying develop the most useful, actionable
metric, we can vet them through our various decision points:

Direct vs. Proxy: Most of the actions above can be tracked directly with clicks
except for viewing a feed item. For viewing, we may use a proxy to count the
number of items viewed, such as “count viewed if 75% of the feed item is in the
device viewport.”
Individual vs. Aggregate: Because our primary goal is measuring
engagement, we care about our users engaging with the Newsfeed in general
more than any specific action, so we will define a metric with aggregate
measures. With that said, some actions (like posting) are more valuable to us, so
we may choose to group them differently. For example, we can aggregate
Newsfeed posting actions into the first bucket, viewing actions into a second
bucket, and interacting actions (inclusive of links, comments, share, blocks) into
a third.
Magnitudes vs. Ratios: Looking at the magnitude of Newsfeed action is
relatively meaningless since any given user can generate a great number of
actions. It is more useful if we consider either actions-per-day or actions-per-
user (or both, i.e. actions-per-user-per-day).
Intrinsic vs. Heuristic: Unfortunately, there’s no easy measure to determine
whether someone is “engaged” or not. Let’s say a hypothetical analysis shows
that people who post at least 1 feed item OR view at least 20 feed items OR
interact with at least 3 feed items are 2x more likely to come back to Facebook
the next day. We can use this data to develop a heuristic around an engagement
metric, whereby if a user crosses any of the three thresholds above, they are
considered “engaged.”

Pulling this all together, our metric for the Facebook Newsfeed will look something
like this…:

…where “engaged user” is a heuristic that’s developed by aggregating both direct

and proxy actions and comparing it as a ratio to “total users” on a daily basis.
APPLYING GAME: FACEBOOK NEWSFEED
Evaluations will come in the form of validating many of the assumptions we made
above by putting your newly defined metric to work. For example:

Honing Heuristics: We may find that the definition around “engaged user” is
overly permissive and that 90% of our users are regularly considered “engaged,”
which doesn’t leave much room for insight or improvement. We dig deeper to
find that of the 90%, a majority became engaged through the “view at least 20
feed items” criteria, so we adjust that threshold up to 50 to create a higher bar
for an engaged user.
Disaggregating Actions: We may find that people who take actions against
negative posts (hiding, unfollowing, reporting, or turning off notifications)
actually correlates inversely to engagement. From this, we can decide to pull
these actions out of our “interacting” bucket (bucket 3) and develop a separate
metric to monitor negative Newsfeed behavior.
Shifting Goals: We may find that we’ve done such a great job on engaging
users with the Newsfeed that engagement is no longer the primary goal.
Instead, we may decide to shift the goal to “monetization,” where we develop
ad-related features into the Newsfeed. While engagement may not be the
primary goal anymore, we want to continue monitoring the engagement
metric to ensure the new monetization features do not have an adverse impact
on engagement.
MBAtrek Private Limited
414 (4th Floor), Suncity Business Tower, Golf
Course Road, Sector-54, Gurugram – 122003

Contact: +91 9821948334/5, +91 9311432447/8