PRODUCT

TYPES OF PRODUCT
Products can be roughly divided into 3 main categories: tangible products (physical goods),
services (intangible value) and software. The borders between those types are flexible, since
software could provided as a service (SAAS), services can be packaged as products and products
might have software integrated. However, there are some important traits of each type that
would impact the approach to Agile transformation

tangible/physical products
Mass production is a complex process that usually doesn't have a lot of
flexibility, so there would be some elements of waterfall.
It is crucial to invest into user research as well as prototyping and user testing
before launching the production
For one-of-the-kind products (like wedding cakes) there are more
opportunities to learn quickly from the customer
Applying principles of Kanban and Lean (will be discussed later) to the
production will help to improve efficiency and reduce waste
When developing new products, build a cross-functional team.

Services
Services might be provided as products (packaged services) or individually (tailored to
customer). In any case there is more flexibility than in case of tangible products.
Customer feedback is very important and any service provider must do everything to shorten
the feedback loop and adjust services to increase the value delivered.
Building cross-functional self-organizing team around service delivery would help to
increase the quality of the outcome.
Ensure to involve the customer throughout the service delivery process.

software
Software allows a lot of flexibility and we can make changes relatively easily, so it is normally
possible to be truly Agile in software development
However, the barriers for entry into software development are relatively low and the
competition is high
(c) Masha Ostroumova 2020
MVP (MINIMUM VIABLE PRODUCT)
Minimum viable product is used to validate hypothesis and understand whether the customers
are willing to pay for the product we want to offer them. Instead of developing a fully functional
product and infrastructure, we build MVP which allows users to get some value out of the
product and give us feedback.

It is common to "fake" parts of MVP in the background, especially when we are validating ideas
which require a huge infrastructure or large investments to build. For example, matching
algorithms can be replaced with a human, or a delivery network can be manually operated. In
big marketplaces where it's important to have presence of 2 parties (buyers and sellers), the
"seller" part can be faked as well to validate the hypothesis that people would be willing to buy
products.

When large companies launch an MVP of a new product or service, they usually call it "beta" in
order to minimize the risks associated (as the customers start having certain expectations for a
new service that might be discontinued). Consult with your legal team before making any
promises to customers.

MVP VS. PROTOTYPE

MVP is often confused with a prototype, but
those are different things. Prototype is a tool
used for ideation and can be anything from
rough sketches to a fully working version of a
product, but it is usually not supposed to be
shared with customers. Prototypes can be used
for user testing, but they are not real products.
On the other hand, MVP is an actually working
version of product which offers customer value
and can be used by real customers. We use MVP
to validate our hypothesis (that customers are
willing to buy or use our product or sign up for
our service).

(c) Masha Ostroumova 2020

PRODUCT METRICS
It is important to define the product metrics before we start measuring them, because otherwise
we will be likely to trick ourselves into thinking that "numbers look good". At the same time,
always remember Goodhart's law: "When a measure becomes a target, it ceases to be a good
measure."

Most businesses would measure their success by profitability, where profit would be revenue
minus the cost. However, it is also important to consider user acquisition cost (the cost of
adding one more user) and the user lifetime value (the average purchase x purchase frequency).

We also have to track conversion rate (percentage of users accomplishing a goal) as increasing it
helps to increase the profitability.

other metrics
Traffic - how many users visit your site/app/service. Traffic should be measured with
conversion. An increase in traffic with no increase in number of conversion is not a good sign.
Daily active users - how many users take actions (not necessarily conversion-related actions)
on your platform. Also should be viewed together with conversion rate.
Time spent on site/platform/service. A useful metric for services selling content and ads.
New/repeated ratio - which percentage of users are new and which are repeated. Depending
on the nature of product, you might need to increase either new or repeated user.

A/B TESTING AND EXPERIMENTING

Whenever we want to validate our idea, we do product experimentation and the most popular
approach is to run A/B tests, when we show to the users 2 versions of the product and track
which group performs better.

rules of a/b testing

Make only one change at a time (e.g. button color) so that we can attribute the user behavior
change to a particular UI/functionality change. It is acceptable though to do A/B/C/D...
testing, when there are more than 2 versions of the same product (e.g. users see either red,
blue, green or grey button, while the rest of UI doesn't change)

(c) Masha Ostroumova 2020

 Test at the same time (i.e. don't test version A on Monday and version B on Tuesday) with the
same population sample in each test group (i.e. don't show version A to French-speaking
users and version B - to Spanish-speaking users).
Set a hypothesis in advance - an assumption about the change in user behavior we are
expecting to observe.

Population and sampling

Population is your target audience, and
the goal of experimentation is to
understand the preferences and the
behavior of the population. Since in
most cases it is impossible to test ALL
target users, we select a sample which
should help us to understand the
population. In any case there will be a
small error in our data, but if the sample
size is too small, we are likely to get a
wrong idea about the population

Statisticians use special formulas to calculate the sample size. You can also refer to this table
from SurveyMonkey or use online A/B test calculators which would help to identify the right
sample size for a given population size and acceptable margin of error.

normal distribution and confidence levels 99%

The results of test will be aligned along a
bell-shaped curve (normal distribution), 95%
where most of the values will be
concentrated around the center (mean),
68%
68% of values will fall within 1 standard
deviation, 95% - within 2 standard
deviations and 99.7% - within 3 standard
deviations.

Confidence levels tell us how confident we are that that true mean of the population falls within
our sample. In most cases we use confidence levels of 95%.

(c) Masha Ostroumova 2020