This article was downloaded by: [202.207.0.

245] On: 15 March 2024, At: 23:55

Publisher: Institute for Operations Research and the Management Sciences (INFORMS)
INFORMS is located in Maryland, USA

Management Science
Publication details, including instructions for authors and subscription information:
http://pubsonline.informs.org

When to Be Agile: Ratings and Version Updates in Mobile

Apps
Gad Allon, Georgios Askalidis, Randall Berry, Nicole Immorlica, Ken Moon, Amandeep Singh

To cite this article:

Gad Allon, Georgios Askalidis, Randall Berry, Nicole Immorlica, Ken Moon, Amandeep Singh (2022) When to Be Agile: Ratings
and Version Updates in Mobile Apps. Management Science 68(6):4261-4278. https://doi.org/10.1287/mnsc.2021.4112

Full terms and conditions of use: https://pubsonline.informs.org/Publications/Librarians-Portal/PubsOnLine-Terms-and-

Conditions

This article may be used only for the purposes of research, teaching, and/or private study. Commercial use
or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher
approval, unless otherwise noted. For more information, contact permissions@informs.org.

The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitness
for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or
inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or
support of claims made of that product, publication, or service.

Copyright © 2021, INFORMS

Please scroll down for article—it is on subsequent pages

With 12,500 members from nearly 90 countries, INFORMS is the largest international association of operations research (O.R.)
and analytics professionals and students. INFORMS provides unique networking and learning opportunities for individual
professionals, and organizations of all types and sizes, to better understand and use O.R. and analytics tools and methods to
transform strategic visions and achieve better outcomes.
For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org
 MANAGEMENT SCIENCE
Vol. 68, No. 6, June 2022, pp. 4261–4278
http://pubsonline.informs.org/journal/mnsc ISSN 0025-1909 (print), ISSN 1526-5501 (online)

When to Be Agile: Ratings and Version Updates in Mobile Apps

Gad Allon,a Georgios Askalidis,b Randall Berry,b Nicole Immorlica,c Ken Moon,a Amandeep Singha
a
The Wharton School, University of Pennsylvania, Philadelphia, Pennsylvania 19104; b Northwestern University, Evanston, Illinois 60208;
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

c
Microsoft Research, Redmond, Washington 98052
Contact: gadallon@wharton.upenn.edu, https://orcid.org/0000-0002-4815-6667 (GaA); gaskalides@gmail.com (GeA); rberry@eecs.
northwestern.edu, https://orcid.org/0000-0002-1861-6722 (RB); nicimm@gmail.com (NI); kenmoon@wharton.upenn.edu,
https://orcid.org/0000-0003-2529-7122 (KM); amansin@wharton.upenn.edu (AS)

Received: June 14, 2018
Revised: September 11, 2019;
November 23, 2020
Accepted: April 5, 2021
Published Online in Articles in Advance:
October 5, 2021
https://doi.org/10.1287/mnsc.2021.4112
Copyright: © 2021 INFORMS
Abstract. Lean and agile models of product development organize the flexible capacity to
rapidly update individual products in response to customer feedback. Although agile op-
erations have been adopted across numerous industries, neither the benefits nor the factors
explaining when firms choose to become agile are validated and understood. We study
these questions using data on the development of mobile apps, which occurs through the
dynamic release of new versions into the mobile app marketplace, and the apps’ customer
ratings. We develop a structural model estimating the dependence of product versioning
on (a) market feedback in the form of customer ratings against (b) project and work-based
considerations, such as development timelines, scale economies, and operational con-
straints. In contrast to when they actually benefit from operational agility, firms become ag-
ile when launching riskier products (in terms of uncertainty in initial customer reception)
and less agile when they are able to exploit scale economies from coordinating develop-
ment over a portfolio of apps. Agile operations increase firm payoffs by margins of 20% to
80%, and interestingly, partial agility is often sufficient to capture the bulk of these returns.
Finally, turning to a question of marketplace design, we study how the mobile app market-
place should design the display of ratings to incentivize quality (increasing app categories’
average user satisfaction rates by as much as 22%).

History: Accepted by Jayashankar Swaminathan, operations management.

Supplemental Material: The online appendices are available at https://doi.org/10.1287/mnsc.2021.4112.

Keywords: agile product development • empirical operations management • mobile apps • online marketplace • product quality and reviews •
product versioning and innovation • structural estimation

1. Introduction Although agile models promise higher product quali-

Under traditional waterfall methods, a product’s de-
velopment is planned as a sequence of stages that are
coordinated to deliver product quality on schedule
while realizing cost efficiencies between a firm’s divi-
sions and multiple products.1 Widely adopted in soft-
ware development practices, lean and agile models of
product development offer a starkly different ap-
proach that has recently gained traction in numerous
industries, ranging from content creation (National
Public Radio) to specialized manufacturing (John
Deere, Saab, and General Electric Appliances) (Rigby
et al. 2016, 2018, Berez and Poindexter 2019). Agile op-
erations call for small, cross-functional teams focusing
on the single product. The team is empowered to
push its product to market, collect customer feedback,
and reorganize workflows and product releases in re-
sponse to customer needs. In addition to industries
newly adopting the agile model of product develop-
ment, the growing reach of technology-based firms
such as Netflix, Spotify, and Amazon has further ac-
celerated the spread of agile practices.
ty and productivity, they have spurred debate (see, e.g.,
Rigby et al. 2018) regarding which firms or which verti-
cals within a firm can benefit from agile operations.
First, carrying out agile operations can be costly and
challenging both in terms of reorganization and in inter-
facing effectively with marketing and other functions
within the firm (Rigby et al. 2018, Sharma et al. 2018).
Second, business reviews (Rigby et al. 2016) and cases
attest that operational agility is most beneficial when
market demand is uncertain and fast-moving, the de-
sign space large, and the product modular such that
rapid, incremental improvements deliver value. Yet, to
the best of our knowledge, no comprehensive study has
examined the empirical evidence for when agile opera-
tions deliver tangible improvements in outcomes nor
for when firms choose to be agile in practice.
The objective of this paper is to investigate the bene-
fits of operational agility. An agile developer possesses
two defining characteristics. First, the developer’s cycle
time is short. Thus, the developer requires the produc-
tive capacity to rapidly update its product.2 Second, the

4261
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4262
agile developer is responsive to customers. Specifically,
when customers are dissatisfied, the developer recog-
nizes their dissatisfaction and utilizes its agile capacity
to expedite an improved product release. In this way,
operational agility lies at the intersection of product de-
velopment and supply operations. Whereas past litera-
ture has called for organizing the supply chain to pro-
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
duce flexibly and responsively when market demand is
substantially uncertain (see, e.g., Fisher 1997), opera-
tional agility mandates that product improvements cor-
respond to customer feedback.
Our empirical study uses data from the marketplace
for mobile apps. This market is economically signifi-
cant in its own right, surpassing $82 billion in 2017
revenues and projecting $120 billion in 2019.3 We col-
lect data for a sample of apps from Apple’s iOS App
Store, which is one of the two major mobile app
stores.4 Across the various genre categories, our panel
dataset records apps’ weekly reviews and ratings, in-
cluding version information, in addition to app-level
characteristics. As a third-party firm separate from the
mobile app store, the developer typically releases an
app into a mobile app store, and then adds, removes,
and edits features of the app in subsequent version
updates. Importantly for our study, the iOS platform
makes its entire selection of products and reviews
available in a single marketplace, therefore allowing
us to observe both when the developer releases new
versions and the concurrent ratings of apps.
Applying econometric and structural empirical
methods to our data, we consider four basic questions.
(1) Can the empirical pattern and timing of a develop-
er’s app versioning be used to characterize and quanti-
fy operational agility? (2) What factors drive the choice
to be agile in practice? (3) By comparing agile and non-
agile models, what is the value of agility both for the
firm and in its broader effect on product quality? How
much agility should a firm pursue knowing that agility
can be expensive to pursue for organizations? (4) Last-
ly, supposing that product quality is improved by agile
versioning, can the marketplace operator (e.g., Apple,
in the role of managing the iOS ecosystem and App
Store) incentivize agility?
1.1. Related Literature
In operations, the most extensive literature regard-
ing versioning addresses the static assortments or
product lines (see, e.g., Shapiro and Varian 1998, Ghose
and Sundararajan 2005, Netessine and Taylor 2007,
Bhargava and Choudhary 2008) produced by firms in
response to market characteristics. A significantly nar-
rower body of papers focuses on intertemporal prod-
uct versioning and upgrades. Both the early modeling
literature (see, e.g., Dhebar 1994, Fudenberg and Tirole
1998) and more recent modeling and empirical work
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
(see, e.g., Zhang and Seidmann 2010, Brecko 2017) in
this area seek to explain pricing and innovation rates
through consumer demand features. However, al-
though the existing literature largely assumes that
firms are essentially agile, in reality, firms institute
organizational constraints around their internal pro-
cesses and development timelines (see, e.g., Krishnan
and Ulrich 2001). Even in the software and app devel-
opment industry, top-down constraints are common-
place; for example, the ridesharing app Lyft announces
a new update every week, whereas the academic
course platform Canvas releases versions every three
weeks. In other cases, firms schedule around develop-
ment “sprints” that can be more flexibly planned.
More closely reflecting these issues, a second stream
of literature explains how firms’ product development
processes should be tailored to the market and produc-
tion uncertainties they face. MacCormack and Verganti
(2003) empirically supported the value of early market
feedback, and Terwiesch and Loch (1999) explained
how the early resolution of uncertainty allows firms to
boost project-level efficiencies by planning overlapping
development activities. Loch et al. (2001), Erat and
Kavadias (2008), and Sommer et al. (2009) showed that
choosing sequential product innovation is more appro-
priate than parallel innovation when feedback is imper-
fect but the innovation space is constrained in its com-
plexity. Krishnan and Ramachandran (2011) considered
implications for modular design, and Nagarajan et al.
(2018) relatedly address the product design of apps.
MacCormack et al. (2001) document that flexibility in
the development process significantly complements ac-
cess to early market response. A broader literature with-
in operations similarly links organizational design and
strategy to the ability to react to information over time
(see, e.g., Mendelson 2000, Petruzzi and Dada 2001,
Lurie and Swaminathan 2009). Some recent literature
(Gupta et al. 2020, Yoo et al. 2020) analytically studies
operationalization of agile methods.
In relation to these literatures, our contribution ad-
dresses operational agility in theory and in practice. In
increasingly many contexts, firms collect continuous
product feedback, placing greater stress on their organi-
zational infrastructures. Using data from the app econ-
omy, we characterize the extent to which agile organi-
zation has been, and moreover should be, adopted.
Methodologically, we study firms’ product version-
ing decisions by developing and estimating a structural
model. We add to the fast-growing structural empirical
literature on consumer-facing operations and services,
including Allon et al. (2011), Akşin et al. (2013), Li et al.
(2014), Yu et al. (2016), and Moon et al. (2018), and on
apps (Mendelson and Moon 2016, 2018).
Our contribution lastly informs the informational de-
sign problem of how marketplaces should convey rat-
ings to incentivize suppliers. The focus on managing
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS 4263

developers’ incentives stands in contrast to the exten- Table 1. Characteristics of the Sampled iOS Apps
sive literature studying ratings from the perspective
Mean Median
of consumer choice (see, e.g., Chevalier and Mayzlin
2006, Li and Hitt 2008, Luca 2016). Our study also dif- Entire sample
fers from Dellarocas (2005, 2006), which focuses on Price $1.81 $0
No. of Text Reviews 49.8 3
robustness to strategic manipulation in designing
App size (MB) 36.7 14
online marketplace ratings. The design problem we Subsample (18%) offering in-app purchases
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

address resembles that of innovative contests, where
development activities are likewise delegated (see,
e.g., Terwiesch and Xu 2008, Boudreau et al. 2011, Ales
et al. 2017). The issues we study extend beyond apps
inhabiting mobile devices, for instance, on emerging
home assistance platforms, such as Amazon’s Alexa
and Google Home.
2. Data and Background
Collected in February 2015, our dataset randomly
samples 2,035 apps from Apple’s iOS App Store.5 For
each app, we record its genre category, its then-
current price, and the size and prices of its in-app pur-
chase assortment. Importantly for our study, we re-
cord apps’ publicly available histories of reviews and
ratings. We describe below how summary ratings are
formed from apps’ histories of individual ratings and
then used prominently on product pages and in
search results to influence future downloads.
We organize an unbalanced long panel dataset,
with an average of 95 weeks observed for each app
over the overall studied period of 349 weeks. The da-
taset covers from the release of each app’s first version
up through the time we collected the data or, if earlier,
the app’s termination, which we define as not having
received any reviews for two months.
In-app purchase assortment size 3.4 2
In-app purchase price $5.04 $1.99
submitted ratings and the written reviews themselves
are made accessible upon further clicking and scrolling
on apps’ product pages. In managing the platform’s
reviews, Apple actively screens for fake reviews and
bars users from leaving multiple reviews. In Online
Appendix A, we further describe how ratings and re-
views have been managed and used by the iOS App
Store, including recent policy changes, and address the
limited instances in which previously submitted re-
views are overwritten.
From the product page, we collected each app’s his-
tory of reviews, including for each the date of submis-
sion, the version of the app addressed, its star rating
and review text, and the submitting username. Table 1
displays the sample’s summary statistics. Up through
the time we collected the data, our apps averaged 50
text reviews, evidencing an upward skew vis-à-vis the
sample’s median of three reviews received. In terms of
prices, the median app is free with no in-app pur-
chases offered. Eighteen percent of the apps offer
in-app purchases, with the median offering compris-
ing two purchase options for $1.99 each.

2.1. Ratings and Reviews 2.2. Versioning

Downloading users can rate apps on an integer scale
of one to five stars, with the option of including a writ-
ten review. Browsing customers encounter prominent-
ly displayed summary statistics about the ratings that
each app has gathered so far — these are the app’s
summary rating, which is the average rating for the
current version, and the ratings volume. Additionally,
detailed information such as the distributions of
App developers release new versions with the mar-
ketplace’s approval. Table 2 summarizes the observed
versioning behavior. On average, developers update
their apps with version releases once every two to
three months. Versioning behavior varies across gen-
res, for instance, with game and entertainment devel-
opers tending to release versions more frequently
compared with those in the education, books, and

Table 2. Age and Frequency of Versioning Overall and for the 5 Largest Genres

Version updates per week Age in weeks

Mean Median Q25 Q75 Mean Median Q25 Q75

Books 0.08 0.04 0.03 0.09 95 79 38 142

Business 0.08 0.05 0.03 0.10 101 81 40 149
Education 0.08 0.04 0.03 0.07 95 80 40 139
Entertainment 0.10 0.05 0.03 0.09 95 79 36 144
Games 0.13 0.05 0.03 0.13 83 69 24 128
Overall 0.10 0.05 0.03 0.09 95 79 38 141
Note. Data observed over 349 weeks for 2,035 sampled apps; reported numbers are summary statistics over
each app’s averaged numbers (giving equal weight to each app).
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4264
Figure 1. Ratings on Days Immediately Prior to and Following Versioning
(a)
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
Note. (a) Ratings distribution; (b) ratings increase.
business categories. Our sample’s apps have an aver-
age age of approximately 1.8 years.
Versioning typically results in improved user ratings.
Across all versioning episodes observed in our data,
Figure 1 compares the ratings received by apps on the
days immediately prior to and subsequent to the new
version’s release. As Figure 1(a) shows, postupdate rat-
ings at 3.62 stars are 2.6% higher (significant at 0.01 lev-
el) than the preupdate 3.53 stars on average. More gran-
ularly, Figure 1(b) shows apps’ postupdate ratings
averages conditional on their preupdate ratings aver-
ages, divided into half-star interval bins. Poorly rated
mobile apps reaped the greatest benefit from version-
ing; for instance, a one-star-rated app, conditional on its
Figure 2. Reviews and Ratings over Version Lifetime
(a)
Note. (a) Daily ratings average; (b) daily reviews volume average.
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
(b)
developer choosing to invest in and release a new ver-
sion, averages three stars upon updating. Reflecting the
difficulty of further improving a high-performing app
and the variability of single-day (preupdate) ratings,
ratings modestly decline on average for versioning
apps already recording four stars or above.
More generally, as shown in Figure 2(a), a version’s
ratings peak shortly postrelease and then trend mildly
downward over its lifetime. Similarly, Figure 2(b) il-
lustrates that the daily volume of reviews logged for a
version drops substantially with the time elapsed
since its release. Across versioning episodes, we find
that the weekly volume of reviews increases on aver-
age by 20% from 3.59 preupdate to 4.33 postupdate.
(b)
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS 4265

Table 3. Average Portfolio Size, Market Risk, and Competition for the 5 Largest App Genres

Initial ratings risk Market competition

Genre Portfolio size Mean SD Mean SD

Books 21 0.38 0.11 0.01 0.01

Business 34 0.41 0.08 0.02 0.01
Education 15 0.40 0.10 0.03 0.01
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

Entertainment 4 0.39 0.09 0.07 0.02

Games 29 0.38 0.11 0.35 0.11
Notes. Weekly for each app, we identify the number of apps in the developer’s portfolio and average over the
app’s lifetime. Portfolios include developers’ apps regardless of genre. We show the average portfolio size by
genre. Initial ratings risk is measured as the standard deviation in apps’ review star ratings in the first 4
weeks. Competition is measured by the number of apps in the genre normalized by the total number of apps
at the time of app launch. We show the competition at app launch averaged across all apps.

2.3. Genres and Portfolios 3. Model

Several genre-centric factors influence developers’
versioning behaviors. First, developers in different
genres realize different levels of cost efficiencies — in
terms of customer acquisition, platform-based devel-
opment, and in cross-utilizing personnel talent —
from maintaining a portfolio of apps. Table 3 reports
the average portfolio size maintained by the develop-
ers of the apps in each of the five largest genres. Clear-
ly, for example, the developers of business and games
apps manage larger app portfolios than those focusing
on the areas of education and entertainment.
Across the app genres, developers face different lev-
els of market risk in terms of the uncertainty about
how an app is received by users. We measure a prod-
uct’s market risk using the standard deviation of the
star ratings it received in its initial four weeks. Table 3
shows the summary statistics of this risk measure
across our sample, and the initial ratings risk is con-
siderable in every genre. The results are similar when
using apps’ initial two or three weeks instead.
Lastly, apps face varying levels of competition with-
in their genres. We measure an app’s market competi-
tion by the number of its competitors active at the
time of launch. An active app is considered a competi-
tor if it belongs to the same genre as the focal app and
is produced by a different developer. For complete-
ness, we rely on a larger sample of 18,000 top apps
sampled over the period of our study to track an
app’s competitors. This helps us to avoid sampling er-
rors and allows us to more comprehensively track
app’s competitors. Lastly, because the size of the iOS
platform evolved significantly over our study, we nor-
malize market competition by the total number of iOS
apps at the time. As Table 3 shows, the normalized
competition levels differ between genres, because
some genres are much larger or smaller. When we
add genre-level fixed effects in our analyses, we con-
sider only the relative levels of competition faced by
apps within a genre, that is, within-genre variation in
competition.
We develop a structural framework that identifies the
economic considerations behind developers’ version-
ing decisions. By explicitly modeling developer pay-
offs, we go beyond a reduced-form estimation of de-
veloper behavior. Although the latter approach validly
estimates developer versioning for the in-sample envi-
ronment, it offers limited insight into the behavior after
developers adapt to different circumstances of interest,
for example, when developers are limited to a choice
of nonagile business models or when the platform’s
summary ratings are differently produced. Online Ap-
pendix B explains this distinction further and also
shows that our payoff-based model compares favor-
ably against a reduced-form model in out-of-sample
performance.
Our structural model also distinguishes between
versioning that is agile and nonagile for the same app.
A completely agile developer should version in re-
sponse to the state of the app’s quality and ratings in
order to optimize its incoming and future stream of
revenue-generating downloads against versioning
costs. Because costs vary from developer to developer,
it is not necessarily true that every agile developer ver-
sions frequently. Therefore, the model accounts for
how a developer times its updates in response to the
evolving state of the app’s quality and ratings. By at-
tributing a level of responsiveness to each app, we dis-
tinguish agile developers from developers that version
frequently but indiscriminately at low cost.
Thus, our structural model’s second benefit is to em-
pirically characterize agility in terms of responsiveness
to customer feedback. (We expect that the most com-
prehensively agile apps will score highly on this front
and exhibit relatively lower costs to version within
genre.) We can then relate apps’ estimated shares of
agile versioning to developer characteristics to shed
light on the motivations underlying agile organization.
3.1. Overview
We model a Markov decision process in which the de-
veloper decides, recurring over time, whether to release
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4266
newly updated versions of its iOS mobile app. Version
updates improve the app’s quality of user experience
by adding or polishing features, addressing bugs and
deficiencies, and meeting users’ evolving needs. At the
same time, prospective new users use the app’s sum-
mary ratings to impute a perceived quality level before
deciding whether to download. Consequently, our
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
model’s Markov state includes both the app’s underly-
ing quality over time, which affects users’ actual experi-
ences, and ratings that affect its perceived quality.
In the remainder of Section 3, we present the devel-
oper’s versioning Markov decision process and its op-
timal solution. It is worth emphasizing that our model
addresses two distinct types of app versioning. The de-
veloper’s optimal solution to its Markov decision pro-
cess captures her agile versioning behavior (namely,
her versioning that is highly responsive to the state of
ratings). Separately, our model allows the same devel-
oper to exhibit a rate of exogenous versioning that is
not responsive. We represent a developer’s agility
based on imputing its share of versioning of each type.
3.2. Definitions and States
Our Markov decision process (MDP) models develop-
er i’s decision to update its mobile app over time. Let
us first introduce notation to define the characteristics
of mobile app i and its time-varying state.
We partition app i’s state in week t into two parts;
the first is (mostly) observed in our data, whereas the
second set of states is unobserved by the researcher
yet needed for our model to fully rationalize i’s
choices seen in the data. The first component of the
state, Xit, consists of the vector
X it : {τit , qit , Dit , D̃ it , Rit } ∈ Z+ × [0, 1] × Z3+ ,
in which τit represents weeks elapsed since i’s last ver-
sioning update, qit represents i’s quality level (which
we describe further below), Dit the number of down-
loads made in the weeks from i’s last versioning update
through week t, D̃ it the number of downloads made
during week t itself, and Rit the number of “satisfied”
ratings submitted since i’s last versioning update
through week t. With the exception of qit, all of Xit is ob-
served directly or by proxy from our data. In particular,
we assume that the frequency of downloads can be ac-
curately proxied by the frequency of feedbacks re-
ceived. See, for example, Cabral and Hortacsu (2010),
which follows a similar approach for eBay sellers.
We defer discussing the unobserved portion of the
state,
it ∈ R2 , until we address the developer’s deci-
sions. Together, the state is denoted sit : {X it ,
it }, and
we use S to denote the state space (i.e., S  Z+ ×
[0, 1] × Z3+ × R2 ).
Finally, a set of time stationary parameters governs
how the app’s state, sit ∈ S, transitions week to week.
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
These are represented by the vector θi : {qi,0 , δi , ci ,
zi , pi }. qi,0 is app i’s initial quality state (as Section 3.4
details, the quality state corresponds to a current
“probability of user satisfaction”), and δi is the weekly
rate at which such quality decays; ci, zi, and pi charac-
terize i’s costs and benefits from versioning, which we
explain below in the appropriate context.
For the remainder of Section 3, which models a sin-
gle app, we typically drop the subscript i. The subse-
quent estimation will allow apps to differ in their
quality and cost parameters θi.
3.3. Weekly Rewards and Actions
The developer collects his or her stream of rewards
from the app, discounted at the weekly rate γ ∈ (0, 1),
over the discrete time horizon t  1, 2, 3: : : . The weekly
reward r(s, a) accrues in state s when updating action a
is taken. This choice is binary, and thus at ∈ {0, 1} is an
indicator variable for the decision to version in week t.
A policy π prescribes in each t-th week a rule deter-
mining i’s action at in week t based on the history of i’s
past actions and past and present states. Developer i
will be interested in choosing the policy π that maxi-
mizes the developer’s expected discounted reward:
 
π π


∞
J (s0 ) : Es0 γ · r(st , at ) ,
t
(1)
t0
where s0 ∈ S is the initial state, and actions at are cho-
sen in accordance with policy π.
The developer’s weekly reward consists first of the
app’s revenue-generating downloads, D̃ t , made in
week t. (Without loss of generality, the reward’s other
components can be viewed as normalized against the
revenue value of a download.) Against this, i incurs the
developer’s cost to version, c, in weeks when the devel-
oper decides to version (at  1). Lastly, as is common in
discrete choice modeling, we provide that certain un-
observed states,
it ∈ R2 , may influence i’s week-t re-
wards. For week t,
t (1) and
t (0) represent the addi-
tively separable variation in i’s choice-conditional
rewards caused by unobserved states, and they are as-
sumed to independently follow the type-1, extreme val-
ue distribution.
Formally, we specify i’s reward function r : S ×
{0, 1} → R as follows:
r(s, a)  D̃ t − c · at +
t (at ): (2)
With relevance for the existence of i’s optimal stationary
policy, the state space S is complete and separable, the
action space is compact, and the reward r is continuous.
3.4. Quality and Ratings
Our study considers how the developer reacts to its
app’s ratings and quality. For this reason, we impor-
tantly explain upfront how our model defines ratings
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
and quality as states and how they are permitted to af-
fect the firm’s rewards. Later, Section 3.5 describes
how quality and ratings fit within our model’s state
transitions.
3.4.1. Ratings. In keeping with prior literature, we
distinguish between satisfied and unsatisfied ratings.
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
Prospective users view, as the public displayed sum-
mary rating on the iOS App Store, the current ver-
sion’s fraction of satisfied ratings, 
Rt:
Rt

R t : ∈ [0, 1], provided that Dt > 0:
Dt
Such visible ratings play an important role in our
study because they shape consumer perceptions. Con-
sequently, an app’s ratings determine the arrival rate
of its new users, that is, its rate of new downloads
made, λt. Empirically, we assume that 4+ stars are
considered satisfactory and 3 and below are unsatis-
factory. We base the threshold for satisfaction on past
literature treating 4- and 5-star reviews as positive
(see, e.g., Bhattacharjee and Goel 2005, Forman et al.
2008, Dang et al. 2010, Mudambi and Schuff 2010, Fu
et al. 2013, Ho-Dac et al. 2013). As in Ho-Dac et al.
(2013), we treat a 3-star review as a negative review
because it falls below the ratings mean of the sample.
See Mudambi et al. (2014) on the overlap of 4-star and
5-star reviews as clustered outputs when predicting
ratings (sentiment) from review text.
The empirical section will describe how we specify
the new user arrivals as Poisson with mean rate pa-
rameter λt, which is a continuous function of the app’s
visible ratings, 
R t . We further allow λt to depend on i’s
genre and the time elapsed since i’s last versioning up-
date, τt, with the mild technical restriction that λt is
continuous and bounded above by some λ̄ < ∞. The
sensitivity of λt to visible ratings, R t , reflects in part
the competitive intensity of i’s genre market.
3.4.2. Quality. Although visible ratings determine the
arrival rate of i’s users, their experiences are shaped
by its actual quality state, qt ∈ [0, 1]. At time t, qt repre-
sents app i’s Bernoulli probability to produce a satis-
factory experience for each downloading customer.
Each customer then records a star rating on the plat-
form, from which we impute his or her (dis)satisfac-
tion level.
3.4.3. Changes in Quality. However, qt evolves over
time. Two types of quality state transitions are possi-
ble, and which one occurs depends on i’s decision
whether to version in week t. Absent versioning, the
quality level qt decays at the weekly rate δ ∈ (0, 1), that
is, qt+1  δ · qt . Quality depreciates in this way as the
app’s features and content become outdated, and its
users’ tastes evolve.
4267
When i chooses to version, the share of dissatisfied
customers is reduced by the fraction z ∈ [0, 1]. Concrete-
ly, suppose that qt  0:7, implying that in expectation
70% of customers are satisfied, and 30% are dissatisfied.
Versioning reduces that 30% group of dissatisfied cus-
tomers by z × 100%, meaning that the fraction z of
the previously dissatisfied group will be satisfied by
the new version. Thus, in expectation after versioning,
only (1 − z) × 30% of customers are dissatisfied for
week t + 1. More generally, upon versioning, qt+1 
1 − (1 − qt ) × (1 − z).
(3)
3.5. State Transitions
We return to the developer’s MDP, where in each
week developer i selects his or her action at to collect
his or her state-dependent reward r(st , at ). The afore-
mentioned transitions in the app’s quality and ratings,
along with their resulting downloads, constitute part
of how the state st transitions into st+1 between the de-
veloper’s weekly decisions.
Recall that the transition probabilities depend in
part on i’s chosen action. Importantly, if he or she ver-
sions, i’s quality increments and its summary ratings
refresh under the iOS App Store’s display policy. If
not, quality decrements. In each case, new downloads
transpire, and each download generates both revenue
and a new positive or negative rating in the App Store.
The following sequence describes the MDP state
transition q : S × {0, 1} → S.
1. Nonagile versioning. First, if i has not selected to ver-
sion (at  0), with probability p a versioning update
will still occur in week t. The developer still incurs the
versioning cost c in this case.
Exogenous versioning rate p flexibly allows for the
real possibility that the firm institutes versioning that is
not responsive to ratings and quality. Many mobile
developers do utilize agile methods that facilitate
“sprints” ahead of their product releases, and many
extend flexibility into later stages of development
(MacCormack et al. 2001). Yet sustained sprints can re-
sult in poor productivity (see, e.g., Zoeller 2013), and
agile operations may be less advantageous for develop-
ers with portfolios of apps. These may benefit from de-
tailed commitment and coordination in timing and fea-
ture planning.
2. New downloads and ratings. New downloads, D̃ t+1 ,
are realized as a function of displayed ratings R̃ t and
interval since versioning τt. If versioning occurs in the
current period, Dt+1  D̃ t+1 (i.e., the number of ratings
used in the displayed ratings average refreshes), and
otherwise, Dt+1  Dt + D̃ t+1 . New satisfied ratings R̃ t+1
are drawn from the Bernoulli (D̃ t+1 , qt ) distribution. Re-
flecting the iOS summary ratings display, Rt+1  R̃ t+1
upon versioning, and otherwise, Rt+1  Rt + R̃ t+1 . Ver-
sioning refreshes τt+1  1, and otherwise τt+1  τt + 1.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4268 Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS

3. Changes in quality. As described previously, qt+1 
δ · qt absent an update, and qt+1  1 − (1 − qt ) × (1 − z)
with an update.
4. Unobserved states. The unobserved states
t+1 are
realized. These are independently drawn and in partic-
ular depend on neither st nor at.
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
where r̃(x, a)  D̃ − c · a, that is, reward r omitting the
unobserved state
(a), and c accounts for the version-
ing cost incurred when at  0 yet exogenous versioning
occurs. We slightly abuse notation by decomposing
state transition q. T is straightforwardly a contraction
mapping, which implies by the Banach fixed point the-
orem a unique fixed point w∗ ∈ C(X × {0, 1}) such that

3.6. Solving the Versioning MDP w∗  Tw∗ .

We show that a stationary Markovian policy optimal- We additionally define the following operator, T̃,
ly solves the developer’s versioning MDP. 
(T̃v)(s)  max r(s, a) + − c(z) + γ × v(z) q(dz | s, a)
Proposition 1. There exists a stationary Markovian policy a∈{0, 1}
π∗ : S → {0, 1} that maximizes Jπ over all policies π and (9)
initial states s0. 
 max r̃(x, a) +
(a) + − c(z)
Proof. We invoke Bhattacharya and Majumdar (1989)’s a∈{0, 1}
theorem 3.3, where all conditions hold by inspection,
other than (A3 )’s stipulation that + γ × v(z) q(dz | s, a) , (10)


∞
R(s) : γt · rt (s) < ∞ for all s ∈ S (4) on the class of real-valued functions v on S such that
t0 the integral in (9) is finite. Defining v∗ (s)  maxa∈{0,1}
where (w∗ (x, a) +
(a)), then v∗  T̃v∗ . Classical arguments im-
ply the optimality of v∗ and of a stationary, Markovian
r0 (s) : sup |r(s, a)| (5) policy π∗ in selecting a ∈ {0, 1}. Final details concern-
a∈{0, 1}
 ing measurability and selection are left to the refer-
rt+1 (s)  sup rt (s )q(ds | s, a): (6) ence cited.
a∈{0, 1}
3.7. Empirical Application
Clearly, r0 (s) ≤ D̃ + c + |
(0)| + |
(1)| < ∞ at their values The preceding sections take as given developer i’s vec-

for state s. Likewise, r1 (s) ≤ supa∈{0,1},s∈S (D̃ + c+ tor of quality and cost characteristics θi : {qi,0 , δi , ci ,
|
(0)| + |
(1)|)q(ds | s, a), where the integrand corre- zi , pi }. The characteristics θi govern the MDP charac-
sponds to the state s ; c is a fixed constant, D̃ is Pois- terizing i’s active versioning behavior.
son with a rate uniformly bounded above by fixed λ̄, It is important to recall that our MDP permits two
and E is conditionally independent. Thus, r1 (s) ≤ types of versioning. First, pi institutes a weekly rate of
λ̄ + c + 2 ∗ E|
| : B̄ < ∞, where B̄ is a uniform bound exogenous versioning. Unlike the episodes of version-
ing initiated by policy π∗ , such versioning is not adap-
on r1 for all s ∈ S. By induction, B̄ is a uniform bound
tive in its timing. Indeed, there exist rationales for
on all rt , t ≥ 1.


∞ firms to embrace nonagile versioning and develop-
Finally, R(s)  ∞ t0 γ · rt (x) ≤ r0 (s) +
t
t1 γ · B̄  r0 (s)+
t

γ·B̄
ment. We study this issue in Section 5 by examining
1−γ < ∞. the developer characteristics that empirically associate
We briefly characterize the optimal solution. For with low shares of agile versioning (i.e., high shares of
this purpose, let X : Z+ × [0, 1] × Z3+ , that is, the state versioning through pi). Second, i’s policy π∗ dictates
space of the state variables {Xt : t  1, 2, : : : }, and use his or her agile versioning. It is worth noting that i’s
C(X × {0, 1}) to denote the metric space (supremum agile versioning behavior depends on i’s costs, ci, and
norm) of real-valued functions on X × {0, 1} that are benefits, zi, from versioning. Thus agile versioning is
bounded and upper semicontinuous. Notably, C(X × not necessarily frequent whenever the rate of exoge-
{0, 1}) is complete. nous versioning pi is low.
Consider the following monotone operator T : C(X × Given θi, the associated policy π∗ attaches to each
{0, 1}) → C(X × {0, 1}): observed state X ∈ X a probability over whether de-
  
veloper i chooses to version, where we account also
(Tw)(x, a)  r̃(x, a) + − c(s ) + γ · max [w(s , a ) for his or her weekly rate of exogenous versioning pi:
a ∈{0, 1}

+
(a )] · q(d
) · q(s | x, a) (7) exp{w∗ (X, 1)} + pi · exp{w∗ (X, 0)}
Pr{Versioning |X}  :
  exp{w∗ (X, 1)} + exp{w∗ (X, 0)}
 r̃(x, a) + − c(s ) + γ · (Γ + log(exp{w(s, 0)} (11)

+ exp{w(s , 1)})) · q(s | x, a) (8) Various methods are available to compute the choice-
conditional value functions w∗ , with contraction
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
mapping T being a candidate algorithm guaranteed to
converge. Then expression (11) provides the probabili-
ty that i releases a new version given the week’s state X.
Our data consist of many developers and apps. We
treat each app’s characteristics θi as independently
drawn from a population distribution F(θ; Θ0 ) (the
discounting rate γ is assumed shared). Distribution F
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
is given by a set of hyperparameters Θ, with their true
values Θ0 being our object of estimation. For example,
we will allow app i’s cost of updating ci to be drawn
from a log-normal distribution, and the parameters of
this distribution are part of Θ0 . In Section 4, we esti-
mate Θ0 from our data; then, any specific θi can be ob-
tained through Bayesian updating.
3.8. Empirical Identification
We briefly outline our model’s empirical identification;
i’s quality, qi,t , is reflected by its observed ratings, each
of which is either positive or negative. The rate at which
quality deteriorates, δi, is captured by the rate of declin-
ing positivity in ratings since versioning. The degree by
which ratings are observed to improve following ver-
sioning episodes pins down zi. Lastly, suppose that two
mobile apps, i and j, are observationally identical in the
above respects and operate in the same genre. However,
i releases version updates substantially more frequently
than j. We then infer that either i’s cost to update, ci, is
lower than j’s, causing frequent agile updating, or that
i’s rate of exogenous updating, pi, is higher. We distin-
guish these two rationales by observing how ratings de-
pendent i’s version timing tends to be.
4. Estimation
Our estimation proceeds in two parts to arrive at a
two-step estimator for the population hyperpara-
meters set forth in Section 3. In the first stage, we non-
parametrically estimate how an app’s rate of new
downloads depends on its public ratings and on the
time elapsed since its last versioning update. Subse-
quently, we estimate our model of version timing
based on observing how developers respond to the in-
centives estimated in the first stage.
4.1. Demand Effect of Ratings
We flexibly estimate an app’s rate of new downloads
in period t as a nonparametric, sieve-based function of
its genre, its displayed fraction of satisfied ratings,

R t−1 ∈ [0, 1], and the time elapsed since its last version-
ing update, τi,t .
More explicitly, period-t downloads are distributed
as Poisson with the rate
⎛ ⎞
⎜⎜⎜ R i,t−1 ⎟⎟
⎟⎟
⎜
λt  gN ⎜⎜⎜⎜⎜ genrei ⎟⎟⎟⎟⎟ β, (12)
⎜⎝ ⎟⎠
τi,t
4269
where gN is a set of sieve-polynomial basis functions,
organized as a vector, that grows with the sample
size, N. In the first stage, we estimate β. Because an
app’s genre is represented in a vector of mutually ex-
clusive dummies, in practical terms, a separate 
β is es-
timated for each genre. For assessing the reliability of
nonparametric prediction of demand (downloads),
we carry out fivefold cross-validation. For each of the
five folds when held out, we estimate the model and
evaluate the R2. Over the five folds, we obtain the de-
cently high average R2 of 0.68.
4.2. Structural Estimation
We estimate our structural model of developer up-
dates. Suppose that we are provided  β and θi, respec-
tively describing a state-dependent distribution over
downloads and a set of app and developer character-
istics. Then, our model in Section 3 characterizes the
developer’s optimal versioning behavior, yielding
state-dependent probabilities of observing versioning
updates made by the developer in the data. Comput-
ing these probabilities, as set out in expression (11),
consists primarily of solving the associated Bellman
Equations (9). For computational efficiency, our esti-
mation procedure is designed to limit the number of
times we are required to resolve such Bellman equa-
tions for different θi.
First, we draw and fix θb , b  1, : : : , B, independent-
ly, from a proposal distribution, F(θ; ΘProposal ). For
each app developer in our sample, i ∈ {1, : : : , N}, we
denote its observed sequence of downloads, ratings,
and versioning events as
Oi : {Ai,t , Di,t , Ri,t : t  0, : : : , Ti }: (13)
After solving the Bellman equations once for each
b ∈ {1, : : : , B}, we can evaluate and store the B × N like-
lihoods, L(Oi | θb ). This precomputing step need only
be performed once, and the B sets of Bellman equa-
tions (and subsequently the B × N likelihood evalua-
tions) can be solved in parallel computations.
As in Ackerberg (2009), we then obtain our maxi-
mum simulated likelihood estimator via importance
sampling as
 

N
B
dF(θ ; Θ)
 SMLE : argmax
Θ log L(Oi | θb ) ·
b
,
Θ i1 b1 dF(θb ; ΘProposal )
(14)
where dF(θb ; Θ) is the only component of the maxi-
mand that cannot be precomputed and fixed. Note
that, by design, F has a common support across alter-
native hyperparameters.
4.3. Results
Our structural hyperparameter estimates are pre-
sented in Table 4. As discussed earlier, our estimated
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4270 Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS

Table 4. Maximum Likelihood Estimates for the Structural quality level, rate of quality depreciation, and the size
Model of quality jumps upon versioning are all chosen by the

model to best fit the ratings path. The timing and fre-

α β 
μ 
σ
quency of versioning then pin down the app’s cost to
∗∗∗
Cost of version update c — — 2:1 0:1∗∗∗ version and rate of nonagile versioning.
(0.45) (0.04) Table 5 presents the average posterior estimates of
Probability of nonagile 0:8∗∗∗ 10:7∗∗∗ — —
apps’ quality and cost parameters for the five largest
versioning p (0.23) (0.14) — —
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

Initial quality q0 8:6∗∗∗ 4:0∗∗∗ — — genres, covering the cost to version, versioning im-
(0.35) (0.17) — — provement in user satisfaction, initial quality, rate of
Quality decay rate δ 67:6∗∗∗ 0:9∗∗∗ — — quality depreciation, and incidence of nonagile version-
(10.12) (0.21) — — ing. For the same genres, Figure 4 displays their shares
Versioning quality boost z 26:9∗∗ 51:4∗∗∗ — —
of agile versioning and the resulting average quality.
(13.56) (7.41) — —
Consider how apps’ outcomes differ across genres in
Notes. Sample of 2,035 apps. Shape parameters α and β characterize Figure 4. Notably, entertainment and games stand out
the beta prior (with mean α=α + β), and μ and σ correspond to the pa-
rameters for the log-normal distribution (with mean exp(μ + σ2 =2)). for their high quality levels (75% to 80% satisfied),
See Online Appendix C for marginal distribution plots. whereas the remaining genres of education, books, and
*p < 0.1; **p < 0.05; ***p < 0.01. business all register below 70% in satisfaction. Interest-
ingly, the genres of entertainment and games differ
structural model delineates a distribution over the app starkly in how their apps’ high quality is achieved. Their
population. The marginal distributions resulting from shares of agile versioning (as opposed to nonresponsive)
the estimated parameters are shown in Online Appen- diverge sharply — 63% versus 39% — to importantly
dix C. Starting from the prior defined by these esti- demonstrate that high quality can be sustained with or
mates, we carry out Markov chain Monte Carlo simu- without being agile. As shown in Table 6, paid and free
lations to estimate posteriors for each individual app. apps do not appear to differ significantly in their shares
Using such posterior estimates, Figure 3 shows how of agile versioning or in quality.
the model estimates a time series path of evolving Turning back to Table 5, we observe that the aver-
latent quality to fit one specific app’s ratings and ver- age rate of quality depreciation is largely uniform
sions data. Consistent with Section 3.8, the initial across the genres. However, we subsequently examine

Figure 3. (Color online) Latent Quality Evolution of a Selected App

Notes. The plot shows the mean evolution path of latent quality for a selected app. The plotted path is generated using the estimated posterior
means for the app’s quality and cost parameters. The plot shows both latent quality and average reviewed ratings weekly, and versioning events
are marked by dashed lines. Ratings and versions are from data.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS 4271

Table 5. Genre Averages in Apps’ Structural Characteristics

Cost to version Nonresponsive Initial quality Quality depreciation Versioning quality

avg

c
avg
versioning 
p
avg
avg
q0 
δ improvement z
avg

Books 8.7 6.8% 0.577 0.994 26.1%

Business 6.8 6.1% 0.612 0.994 26.7%
Education 5.4 5.9% 0.658 0.995 32.9%
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

Entertainment 6.3 4.1% 0.474 0.994 16.8%

Games 7.0 10.9% 0.486 0.993 20.7%
Note. Posterior structural estimates for 2,035 individual apps are obtained through MCMC, with averages
shown for the five largest genres.

whether varying competition may affect quality depre-
ciation rates for apps within a genre. Strikingly, despite
these genres’ high average quality, entertainment and
games versioning events are the least effective at re-
ducing their apps’ shares of dissatisfied customers,
thus entailing more frequent versioning to remain at
the same quality level. Moreover, the average initial
quality of apps released in these two categories starts
substantially lower, signaling that developers experi-
ment after launching apps in the marketplace.
4.4. Predictive Performance and Robustness
We assess predictive accuracy by reserving 20% of the
sampled apps as a test set and estimating the structur-
al model solely on the remaining 80% as the training
set. See Online Appendix B for details on the structur-
al model’s area under the curve (AUC) of 0.77.
Online Appendix D provides a robustness check
showing that our structural estimates remain essen-
tially unchanged when we carry out estimation sepa-
rately for paid and free apps. There, we additionally
estimate a model extension that allows the level of
competition to affect the versioning rate.
Figure 4. (Color online) Versioning Type and Quality by
Genre Bar Plots Denote the Share of Agile Versioning (Left
Axis), and the Dashed Lines Denote the Average Quality
(Right Axis) in Respective Genres
5. Analysis and Prescriptions
In this section, we first study the factors driving devel-
opers’ choices to be agile in practice. We then turn to a
comparison of agile and nonagile models to assess
agility’s value before finally evaluating the market-
place operator’s influence on the developers’ incen-
tives through ratings design.
5.1. When Are Developers Agile?
Regarding when developers choose to organize them-
selves to be agile, we focus on three main hypotheses.
First, cost economies in software development have
long been shown to be significant (Banker and Slaugh-
ter 1997), and in recent years, app developers have cited
efficiencies from building proprietary development
platforms that support features to be shared by their
apps. See, for example, Seif El-Nasr and Magy (2013) re-
garding Zynga’s cloud-based data infrastructure sup-
porting integrated analytics, experimentation, and
cross-promotion. We thus hypothesize that scale econo-
mies can be attained by coordinating development over
a developer’s app portfolio, which undercuts the incen-
tives for agile development of individual products. This
first hypothesis also parallels product development in
automobiles, where lean and agile manufacturers em-
pirically share fewer components across automobile
models than competitors that realize scale economies
by sharing components across multiple products (Clark
et al. 1991, Fisher et al. 1999, Ramdas 2003).
Second, the operations literature (Fisher et al. 1999,
Ramdas 2003) proposes that firms facing higher levels
of market uncertainty should prioritize operational
responsiveness over cost efficiency and reliability.

Table 6. Versioning Type and Quality for Paid and Free

Apps

Average quality % of
Share of agile versioning level sample

Paid 40% 0.709 40.7%

Free 44% 0.737 59.2%
Notes. Ninety-five app-weeks observed for 2,035 apps sampled. Re-
ported numbers are averages over individual apps’ estimates.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4272 Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS

Therefore, we examine the salience of initial market Table 7. Effect of Developer Portfolio Size on Share of
risk in influencing firms’ decisions about whether to Agile Versioning
be agile.
(1) (2) (3)
Lastly, we explore how the competitive landscape
∗∗∗
affects developer agility. We do so in two parts. First, portfolioSize −0.0012** −0:0013 −0.0014***
we hypothesize that when facing greater competition, (0.000) (0.000) (0.000)
inApp 0.017* 0.016** 0.031***
an app must output a higher rate of innovation and (0.010) (0.007) (0.006)
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

content development to satisfy customers. In the con- Constant 0.494*** 0.342*** 0.299***
text of our model, this dynamic manifests as a higher (0.028) (0.067) (0.113)
rate of quality decay for apps facing greater competi- Genre —  
tion. We further hypothesize that, in response, devel- Observations 2,035 2,035 2,035
R2 0.033 0.693 0.376
opers launching apps in more competitive landscapes Adjusted R2 0.014 0.659 0.085
will choose to engage in higher rates of agile and non-
agile versioning. Note. Standard errors in parentheses.
*p < 0.1; **p < 0.05; ***p < 0.01.
The first hypothesis tests whether a developer’s de-
cision to be agile is significantly explained by the size
of its portfolio. We use the following regression speci- Interestingly, cost economies may have genre-level
fication: effects. In discussing Figure 4, we noted that whereas
the entertainment and games genres post high quality
agilityi  genrei + portfolioSizeD(i) + inAppi +
i (15) levels (75% to 80% satisfied), they differ sharply in
An app’s agility is measured by the share of agile ver- their shares of agile versioning. This may be explained
sioning, while portfolioSize refers to the average app by game developers maintaining significantly larger
portfolio size of the app developer D(i) over i s life- app portfolios than entertainment developers.
time. We additionally control for whether the app has The second hypothesis proposes that firms are in-
in-app purchases, inAppi. fluenced by the initial level of market risk. We use the
However, a potential endogeneity arises if better ac- following regression specification,
cess to capital and funding (or a similarly omitted fac- agilityi  genrei + riski +
i , (16)
tor) enables a developer to both organize itself for

greater agility and develop a more expansive portfolio. in which product i s market risk is measured by vari-
In this case, estimating (15) by ordinary least squares ance of initial reviews the app receives.
(OLS) results in a portfolioSize coefficient that is up- Table 8 shows our OLS regression results with
wardly biased; however, this is a conservative bias, be- genre fixed effects. We find that more agile versioning
cause we expect that the effect is negative (a larger associates with higher market risk.
portfolio discourages developer agility). We addition- To test our final hypotheses on competitive effects,
ally apply an instrumental variable: the market share of we use the following regression specifications:
iOS devices six months prior to app i s launch release
δi  genrei + market-competitioni +
(δ)
i , (17)
on iOS. Although the iOS market share should not di-
(agility)
rectly affect agility (exclusion restriction), it plausibly agilityi  genrei + market-competitioni +
i , (18)
influences the number of apps launched by the devel- (p)
pi  genrei + market-competitioni +
i : (19)
oper on the iOS platform specifically (relevance).
We report our OLS estimates in columns (1) and (2) Table 9 shows the estimation results (17) for the com-
of Table 7, where specification (2) adds the genre fixed petition’s effects on the pace of updating required to
effects. We find that the share of agile versioning of an
app is significantly negatively associated with the de- Table 8. Effect of Market Risk on Share of Agile
veloper’s portfolio size, supporting our hypothesis Versioning
that firms with broader portfolios engage in planned
and nonagile development that better preserve econo- (1)
mies of scale. App developers that offer in-app pur- Risk 0.430***
chases tend be more agile in versioning. They may de- (0.007)
pend more heavily on a continuous revenue stream, Constant 0.015***
(0.003)
which requires them to stay relevant and attractive to
Genre 
consumers. Specification (3) in Table 7 shows results Observations 2,035
for our instrument variable regression, which corrobo- R2 0.494
rate our OLS findings. The corresponding first-stage Adjusted R2 0.494
regression F-statistic is 14.07 (P < 0.01), which sup- Note. Standard errors in parentheses.
ports that the instrument is relevant. *p < 0.1; **p < 0.05; ***p < 0.01.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS 4273

Table 9. Effect of Competition on Quality Decay Rate illustration, apps launched at the 75th percentile level
of competition have 34% more levels of nonagile ver-
Dependent variable
δ
sioning versus the 25th percentile level.
(1) (2)
5.2. Gains from Agile Versioning
market-competition(L) −0.0002*** −0.0001**
To next assess the gains from adopting agile versioning,
(0.0000) (0.0000)
Genre —  we compare a firm’s payoffs under the agile and nonag-
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

Intercept 0.9937***
(0.0001)
Observations 2,035
R2 0.0208
Adjusted R2 0.0203
Notes. The table shows impact of competition on app’s quality decay
rate. market-competition(L) refers to the log specification for market
competition being used.
*p < 0.1; **p < 0.05; ***p < 0.01.
maintain an app’s quality. We consistently find that,
within genres, apps facing greater competition face a
faster decay rate δi in quality when not updated, con-
sistent with customers demanding constant innova-
tion and content development to be satisfied. For illus-
tration, compare apps launched at the 75th percentile
level of competition versus the 25th percentile level.
The former’s quality decays by 4% more than the lat-
ter over 10 weeks if not updated.
In columns (1) and (2) of Table 10, we report our
OLS regression results for specification (18), which
tests that firms respond to competition by becoming
more agile. Specification (2) adds genre fixed effects.
Across both specifications, we find that firms
launched into more competitive landscapes engage in
significantly higher shares of agile versioning. For il-
lustration, apps launched at the 75th percentile level
of competition are 22% more agile versus the 25th per-
centile level.
Columns (3) and (4) of Table 10 report our OLS re-
gression results for (19), which tests whether more
competitive landscapes spur developers to also en-
gage in higher rates of nonagile versioning p. We find
that firms situated in more competitive landscapes do
choose higher rates of nonagile versioning. For
0.9932*** ile models. Moreover, this approach permits us to char-
(0.0003) acterize which developers would benefit most from
2,035 agility and whether they are the most agile in practice.
0.1341
0.1242
Specifically, for each app, we compare the NPV (net
present valued) revenue under three different version-
ing approaches. First, we consider nonagile versioning,
where the firm optimizes its frequency of versioning
the product, independent of the customer ratings. We
optimize over a restricted version of the model in Sec-
tion 3 where responsive versioning is disallowed. In-
stead, the developer optimizes over the exogenous ver-
sioning rate, p, which cannot depend on ratings or
quality. Second, we evaluate the firms’ NPV gains un-
der the completely agile model. Using the model of
Section 3, we set p  0 and solve the resulting MDP.
Note that the resulting versioning policy necessarily
dominates nonagile versioning in NPV payoffs. Lastly,
we include the firm’s NPV revenue collected from its
estimated policy as a benchmark for its current deci-
sions in practice.
The panels of Figure 5 present our main findings.
First, as shown in Figure 5(a), agile versioning yields
substantial benefits over nonagile versioning. Even
against the optimal nonagile schedule, agility typical-
ly yields a premium (depicted by the height of the
dashed line over zero) of 20% to 80% in the NPV reve-
nue stream. Second, some of the apps and developers
that could significantly benefit from agility (i.e., those
situated where the dashed line is highest) are not very
agile. Figure 5(a)’s shaded area conveys the density of
apps over their currently chosen shares of agile ver-
sioning, and a healthy share of apps resides close to
where the widest gap separates the solid line from
the dashed line, which represents the NPV revenue

Table 10. Effects of Competition on Share of Agile Versioning and on Rate of Nonagile
Versioning

Dependent variable Dependent variable

Share of agile versioning Nonagile versioning rate p

(1) (2) (3) (4)

market-competition 0.252*** 0.182*** 0.051*** 0.032**

(0.033) (0.04) (0.012) (0.013)
genre —  — 
Intercept 0.406*** — 0.088*** —
(0.004) (0.001)
Observations 2,035 2,035 2,035 2,035
Adjusted R2 0.027 0.500 0.013 0.180
*p < 0.1; **p < 0.05; ***p < 0.01.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4274 Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS

Figure 5. (Color online) Agile Versioning’s Net Present Valued Premium over Optimal Nonagile Versioning
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

Note. (a) All developers; (b) developers with high versioning cost; (c) developers with low versioning cost; (b) and (c) show the top and bottom
quartiles, respectively, of apps in versioning cost c.

gap between apps’ current versioning and fully agile
versioning. Various explanations are plausible; for
example, in light of the previous subsection, some de-
velopers may choose to remain less agile in order to
exploit portfolio scale economies. However, because
the prospective gains from increasing agility are sub-
stantial, managers should carefully consider these
tradeoffs in staying less agile. In Figure 5(b) and (c),
we separately plot the premia and shares of agile ver-
sioning for developers with low and high versioning
cost markets. Developers respond to such costs, with
developers with low versioning costs being significant-
ly more agile. The latter represent the fullest definition
of agility, where responsiveness to customers is cou-
pled with low costs enabling fast development cycles.
Third, apps with costlier versioning receive a relative-
ly high gains from becoming partially agile; that is, the
gap between the dashed and solid lines is comparatively
narrower in Figure 5(b). For these apps, 25% agile ver-
sioning captures approximately 51% of the NPV
achieved by completely agile versioning and approxi-
mately 26% of its NPV premium over the optimal
schedule-based regime. The performance gap is nar-
rower virtually everywhere else. By contrast, for apps
with low versioning costs, 25% agile versioning captures
only about 22% and 12.5% of the agile NPV and NPV
premium, respectively. The explanation is intuitive.
When versioning is costly, it is infrequent, affording an
outsized benefit to being at least somewhat agile and re-
sponsive to customer feedback.
5.3. Marketplace Ratings Design
Even as they surpassed many tens or even hundreds of
billions in downloads, the leading mobile app market-
places have relied on sharply contrasting summary rat-
ing systems. Relevantly for our purposes, the divergent
summary ratings policies differently incentivize devel-
opers in their decisions about whether to engage in ag-
ile development.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS 4275

Before the launch of iOS 11 in September 2017, Figure 6. (Color online) Comparing App Quality Under iOS
Apple’s App Store completely refreshed an app’s sum- and Google Play Summary Ratings
mary ratings whenever the developer released a new
version, meaning that the average rating prominently
displayed to potential customers exclusively addressed
the new version. Google Play’s summary ratings were
instead formed by averaging the individual ratings re-
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.

ceived for all versions in the app’s complete history. In

order to compare the two regimes, we evaluate devel-
oper behavior and NPV revenue outcomes under Goo-
gle’s policy. We use Section 3’s model without resetting
app summary ratings as in Section 3.5, which allows
the summary ratings to continue to evolve as a cumula-
tive average over an app’s lifetime. Importantly for the
marketplace operator, we also evaluate how each rat-
ings design incentivizes developer agility and product
quality. We use our estimated model parameters to
carry out Monte Carlo simulations for each developer
under both regimes. At the end of 5.3, we additionally
address the implications of Apple’s new policy intro-
duced by iOS 11, under which developers can elect
whether to refresh summary ratings.
In presenting our results, we distinguish between
ratings-sensitive and -insensitive markets by genre,
where we define these as the top 25% and bottom 25%
in the marginal sensitivity of downloads to summary
ratings. In ratings-insensitive markets, customers do
not respond to ratings, and hence, the platform’s dis-
play policies cannot significantly affect market out-
comes or firms’ payoffs. Therefore, we focus on the Notes. (a) Ratings-insensitive markets; (b) ratings-sensitive markets.
“Ratings-insensitive” and “ratings-sensitive” marketplaces refer to
ratings-sensitive marketplaces. the apps having sensitivities in the top and bottom 25% quantile
Summarizing our findings, Table 11 shows the re- ranges, respectively.
spective average shares of agile versioning occurring
under the iOS and Google Play regimes. Developers
At the same time, when past ratings have been posi-
are close to three times more agile, that is, in versioning
tive, iOS app developers have conceded that they
responsively as a share of updates, under the iOS re-
sometimes delay updates to avoid refreshing the sum-
gime. Increased agility appears to translate into higher
mary ratings. On the other hand, because Google Play
app quality. To visualize the effect on quality, Figure 6
permanently includes poor or mixed ratings histories
compares the average quality achieved by each of the
in its summary ratings, it both discourages developers
iOS and Google Play policies, with panel Figure 6(b) with these histories from making improvements by
addressing the ratings-sensitive markets. For each level diluting the rewards yet encourages developers to
of agility, the iOS quality level is depicted by the solid pre-emptively version to avoid poor reviews in the
line and Google Play’s by the dashed line. first place. Given the multiple trade-offs in play, it is
The trade-off between the two policies is complex. arguably surprising to find that a single platform poli-
On one hand, Apple’s approach affords developers cy is clearly more effective.
the full marginal benefit of versioning. Especially However, two factors significantly explain the iOS
when past ratings have been poor, Apple’s approach policy’s superior performance in our setting. First,
rewards developers for releasing improved versions. when apps are first launched, they face significant risk
in terms of market reception, as empirically summa-
Table 11. Effect of Ratings Design (Ratings Sensitive rized by Table 3. For example, among apps receiving at
Markets) least five reviews, apps averaged a standard deviation
iOS Google Play
of 2 stars in the ratings they received during their first
three weeks, which translates into middling initial
Share of agile versioning 60.91% 22.69% quality in the context of our model. Thus, enabling a
Average quality 0.80 0.58
process of sequential innovation with exposure to
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4276
market feedback is critical to improving the initially
mixed quality of apps. When poor early ratings remain
permanently attached to new versions, developers can-
not capture the full marginal benefits of quality-
enhancing versioning, and therefore, they underinvest
in such efforts. Second, a significant fraction of apps
launched in ratings-sensitive markets face relatively
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
high versioning costs. Because these apps version infre-
quently, it is very difficult to entirely avoid lapsing into
lower ratings, and the Apple policy incentivizes agility
by forgiving these lapses.
Lastly, thanks to a reviewer suggestion, we address
the recent change in Apple’s summary ratings dis-
played for apps, which occurred in September 2017 af-
ter our data were collected. At this time, Apple granted
developers the option to decide, each time they version,
whether to “reset” their apps’ summary ratings or re-
tain the reviews from past versions as summary rating
inputs (Dillet 2017). We analyzed an additional coun-
terfactual to study the effects of Apple’s revised policy.
We found that developers of high-quality apps signifi-
cantly increased their shares of agile versioning by as
much as 24.1% (pvalue  1e − 6), whereas no statistically
significant change was found for the versioning behav-
ior of developers producing low-quality apps. The ex-
planation appears to be that developers of high-quality
apps were previously reluctant to quickly release their
version updates due to losing the high-quality reviews
from previous versions. However, with iOS 11, these
developers can version more frequently while electing
not to reset apps’ summary ratings.
6. Concluding Remarks
We study operational agility in the versioning-based
development of apps. Using panel data on apps’ ver-
sion releases, reviews and ratings, and developer char-
acteristics, we find that developers become agile when
facing initial risk in the marketplace and when manag-
ing smaller portfolios of apps. As was the case for
manufacturers considering component sharing across
physical products, a smaller product portfolio offers a
developer fewer opportunities for coordinating devel-
opment across apps to realize scale economies. Like-
wise, managing market uncertainty appears to be a
chief motivation in both contexts. Our work draws em-
pirical parallels between these operational settings.
We estimate a structural model that distinguishes
agile versioning (as the solution to a MDP) from non-
agile versioning that is not responsive to customer rat-
ings. Using the model, we find that agility enhances
firm payoffs by 20% to 80%. Moreover, when version-
ing is costly, and hence, infrequent, partial agility per-
forms close to full agility.
Lastly, we show the strength of ratings-based incen-
tives to address a pertinent question of marketplace
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
design. Despite logging hundreds of billions of
downloads, the two predominantly leading app
stores still display their apps’ ratings in opposing
ways. When a consumer views an app, Google Play
displays its average rating over all previous versions,
whereas Apple’s iOS store shows the average rating
received by the current version alone. We find that
the iOS ratings policy better preserves suppliers’ dy-
namic incentives to deliver product quality through
versioning, with up to 22% higher rates of satisfied
ratings in certain markets.
Acknowledgments
The authors thank their editors, three referees, and partici-
pants at 2017-2020 INFORMS Annual Meetings, 2018
MSOM Conference, 2019 POMS Annual Conference, and
2019 MSOM Supply Chain Management SIG Meeting for
their helpful feedback.
Endnotes
1
For example, in software development, Harter et al. (2000) associ-
ate higher product quality with process maturity. The well-known
Capability Maturity Model they cite places emphasis on planning
and a highly repeatable process that can be matured and made pre-
dictable: “Schedules and budgets are based on historical perfor-
mance and are realistic, the expected results for cost, schedule, func-
tionality, and quality of the product are usually achieved
… disciplined process is followed … and the necessary infrastruc-
ture exists to support the process” (Paulk et al. 1993, p. O-7).
2
However, this first factor alone is not sufficient for agility. In par-
ticular, efficiently planned processes can shorten software develop-
ment cycle times (see, e.g., Harter et al. 2000).
3
See https://www.businessofapps.com/data/app-revenues/ (accessed
April 5, 2021).
4
The two largest mobile app stores, Google Play and Apple’s iOS
App Store, offered more than 2.4 and 1.9 million mobile apps for
download onto the Android and iOS platforms, respectively, as of
April 2019. By July 2013, these two mobile app stores had seen
more than 50 billion app downloads each, and Apple announced its
100 billionth download milestone in June 2015.
5
We originally sample 4,938 apps but filter for one or more months
of data since release to carry out estimation.
References
Ackerberg DA (2009) A new use of importance sampling to reduce
computational burden in simulation estimation. Quant. Market-
ing Econom. 7(4):343–376.
Akşin Z, Ata B, Emadi SM, Su C-L (2013) Structural estimation of
callers’ delay sensitivity in call centers. Management Sci. 59(12):
2727–2746.
Ales L, Cho S-H, Körpeoğlu E (2017) Optimal award scheme in in-
novation tournaments. Oper. Res. 65(3):693–702.
Allon G, Federgruen A, Pierson M (2011) How much is a reduction
of your customers’ wait worth? An empirical study of the fast-
food drive-thru industry based on structural estimation meth-
ods. Manufacturing Service Oper. Management 13(4):489–507.
Banker RD, Slaughter SA (1997) A field study of scale economies in
software maintenance. Management Sci. 43(12):1709–1725.
Berez S, Poindexter W (2019) Agile is not enough. MIT Sloan
Management Rev. https://sloanreview.mit.edu/article/agile-is-
not-enough/.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
Bhargava HK, Choudhary V (2008) Research note-when is versioning
optimal for information goods? Management Sci. 54(5):1029–1035.
Bhattacharjee R, Goel A (2005) Avoiding ballot stuffing in eBay-like
reputation systems. Proceedings of the 2005 ACM SIGCOMM
Workshop on Economics of Peer-to-peer Systems, 133–137.
Bhattacharya RN, Majumdar M (1989) Controlled semi-Markov
models – the discounted case. J. Statist. Planning Inference 21(3):
365–381.
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
Boudreau KJ, Lacetera N, Lakhani KR (2011) Incentives and prob-
lem uncertainty in innovation contests: An empirical analysis.
Management Sci. 57(5):843–863.
Brecko K (2017) New features free of charge? Using price to sort
consumers among legacy software versions. Working Paper,
University of Rochster – Simon Business School, Rochester, NY.
Cabral L, Hortacsu A (2010) The dynamics of seller reputation: Evi-
dence from eBay. J. Indust. Econom. 58(1):54–78.
Chevalier JA, Mayzlin D (2006) The effect of word of mouth on
sales: Online book reviews. J. Marketing Res. 43(3):345–354.
Clark KB, Fujimoto T, Cook A (1991) Product Development Perfor-
mance: Strategy, Organization, and Management in the World
Auto Industry (Harvard Business School Press, Boston).
Dang Y, Zhang Y, Chen H (2010) A lexicon-enhanced method for
sentiment classification: An experiment on online product re-
views. IEEE Intelligent Systems 25(4):46–53.
Dellarocas C (2005) Reputation mechanism design in online trading
environments with pure moral hazard. Inform. Systems Res. 16(2):
209–230.
Dellarocas C (2006) Strategic manipulation of Internet opinion fo-
rums: Implications for consumers and firms. Management Sci.
52(10):1577–1593.
Dhebar A (1994) Durable-goods monopolists, rational consumers,
and improving products. Marketing Sci. 13(1):100–120.
Dillet R 2017. The new iOS App Store lets devs choose whether to
reset ratings when updating. Accessed August 2, 2020, https://
techcrunch.com/2017/06/07/ios-app-developers-will-be-able-to
-keep-app-store-ratings-when-releasing-an-update/.
Erat S, Kavadias S (2008) Sequential testing of product designs: Im-
plications for learning. Management Sci. 54(5):956–968.
Fisher ML (1997) What is the right supply chain for your product?
Harvard Bus. Rev. 75:105–117.
Fisher M, Ramdas K, Ulrich K (1999) Component sharing in the
management of product variety: A study of automotive braking
systems. Management Sci. 45(3):297–315.
Forman C, Ghose A, Wiesenfeld B (2008) Examining the relationship
between reviews and sales: The role of reviewer identity disclo-
sure in electronic markets. Inform. Systems Res. 19(3):291–313.
Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N 2013. Why people
hate your app: Making sense of user feedback in a mobile app
store. Proc. 19th ACM SIGKDD Internat. Conf. Knowledge Discov-
ery and Data Mining, 1276–1284.
Fudenberg D, Tirole J (1998) Upgrades, tradeins, and buybacks.
RAND J. Econom. 29(2):235–258.
Ghose A, Sundararajan A (2005) Versioning and quality distortion
in software? evidence from e-commerce panel data. NET Insti-
tute Working Paper (05-14), Orlando, FL.
Gupta S, Agrawal A, Ryan JK (2020) Agile contracting: Managing
incentives under uncertain needs. SSRN Working Paper No.
3674977. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=
3674977.
Harter DE, Krishman MS, Slaughter SA (2000) Effects of process
maturity on quality, cycle time, and effort in software product
development. Management Sci. 46(4):451–466.
Ho-Dac NN, Carson SJ, Moore WL (2013) The effects of positive
and negative online customer reviews: do brand strength and
category maturity matter? J. Market 77(6):37–53.
4277
Krishnan V, Ramachandran K (2011) Integrated product architecture
and pricing for managing sequential innovation. Management
Sci. 57(11):2040–2053.
Krishnan V, Ulrich KT (2001) Product development decisions: A re-
view of the literature. Management Sci. 47(1):1–21.
Li X, Hitt LM (2008) Self-selection and information role of online
product reviews. Inform. Systems Res. 19(4):456–474.
Li J, Granados N, Netessine S (2014) Are consumers strategic? Struc-
tural estimation from the air-travel industry. Management Sci.
60(9):2114–2137.
Loch CH, Terwiesch C, Thomke S (2001) Parallel and sequential
testing of design alternatives. Management Sci. 47(5):663–678.
Luca M (2016) Reviews, reputation, and revenue: The case of Yelp.-
com. Working Paper (12-016), Harvard University - Business
School, Cambridge, MA.
Lurie NH, Swaminathan JM (2009) Is timely information always bet-
ter? the effect of feedback frequency on decision making. Or-
gan. Behav. Human Decisicion Processes 108(2):315–329.
MacCormack A, Verganti R (2003) Managing the sources of uncer-
tainty: Matching process and context in software development.
J. Product Innovation Management 20(3):217–232.
MacCormack A, Verganti R, Iansiti M (2001) Developing products
on “internet time”: The anatomy of a flexible development pro-
cess. Management Sci. 47(1):133–150.
Mendelson H (2000) Organizational architecture and success in
the information technology industry. Management Sci. 46(4):
513–529.
Mendelson H, Moon K (2016) Growth and customer loyalty: Evi-
dence from the app economy. Working Paper (16-24), Stanford
University Graduate School of Business, Stanford, CA.
Mendelson H, Moon K (2018) Modeling success and engagement for
the app economy. Proc. 2018 World Wide Web Conf. (International
World Wide Web Conferences Steering Committee), 569–578.
Moon K, Bimpikis K, Mendelson H (2018) Randomized markdowns
and online monitoring. Management Sci. 64:1271–1290.
Mudambi SM, Schuff D (2010) What makes a helpful review? A
study of customer reviews on Amazon.com. MIS Quart. 34(1):
185–200.
Mudambi SM, Schuff D, Zhang Z (2014) Why aren’t the stars aligned?
An analysis of online review content and star ratings. 2014 47th
Hawaii International Conf. Syst. Sci. (IEEE Explore) 3139–3147.
Nagarajan M, Ryan CT, Sheng L, Cheng Y, Tong C (2018) Incentiv-
ized actions in freemium games. Manufacturing Service Oper.
Management, Articles in Advance.
Netessine S, Taylor TA (2007) Product line design and production
technology. Marketing Sci. 26(1):101–117.
Paulk C, Curtis B, Chrissis MB, Weber CV (1993) Capability maturi-
ty model for software, version 1.1. software engineering insti-
tute. Technical report, CMU/SEI-93-TR-24, DTIC Number
ADA263403.
Petruzzi NC, Dada M (2001) Information and inventory recourse for
a two-market, price-setting retailer. Manufacturing Service Oper.
Management 3(3):242–263.
Ramdas K (2003) Managing product variety: An integrative review
and research directions. Production Oper. Management 12(1):
79–101.
Rigby DK, Sutherland J, Noble A (2018) Agile at scale. Harvard Bus.
Rev. 96(3):88–96.
Rigby DK, Sutherland J, Takeuchi H (2016) Embracing agile. Har-
vard Bus. Rev. 94(5):40–50.
Seif El-Nasr S, Magy AC (2013) Interview with Jim Baer and Daniel
McCaffrey from Zynga. Seir El-Nasr M, Drachen A, Canossa A,
eds. Game Analytics: Maximizing the Valuer of Player Data,
chap. 5 (Springer, London) 73–82.
Shapiro C, Varian HR (1998) Versioning: The smart way to sell in-
formation. Harvard Bus. Rev. 107(6):107.
 Allon et al.: When to Be Agile: Ratings and Version Updates in Mobile Apps
4278
Sharma A, Saboo AR, Kumar V (2018) Investigating the influence of
characteristics of the new product introduction process on firm
value: The case of the pharmaceutical industry. J. Marketing 82(5):
66–85.
Sommer SC, Loch CH, Dong J (2009) Managing complexity and un-
foreseeable uncertainty in startup companies: An empirical
study. Organ. Sci. 20(1):118–133.
Terwiesch C, Loch CH (1999) Measuring the effectiveness of over-
lapping development activities. Management Sci. 45(4):455–465.
Downloaded from informs.org by [202.207.0.245] on 15 March 2024, at 23:55 . For personal use only, all rights reserved.
Terwiesch C, Xu Y (2008) Innovation contests, open innovation, and
multiagent problem solving. Management Sci. 54(9):1529–1543.
Management Science, 2022, vol. 68, no. 6, pp. 4261–4278, © 2021 INFORMS
Yoo OS, Huang T, Arifoglu K (2020) A theoretical analysis of the
lean startup method. Marketing Sci. Articles in Advance.
Yu Q, Allon G, Bassamboo A (2016) How do delay announcements
shape customer behavior? an empirical study. Management Sci.
63(1):1–20.
Zhang J, Seidmann A (2010) Perpetual vs. subscription licensing un-
der quality uncertainty and network externality effects. J. Man-
agement Inform. Systems 27(1):39–68.
Zoeller G (2013) Game development telemetry in production. Seir El-
Nasr M, Drachen A, Canossa A, eds., Game Analytics: Maximiz-
ing the Value of Player Data, chap. 7 (Springer, London), 111–135.