Journal of Survey Statistics and Methodology (2018) 0, 1–25

SIMULTANEOUS ESTIMATION OF MULTIPLE

SOURCES OF ERROR IN A SMARTPHONE-BASED
SURVEY

CHRISTOPHER ANTOUN*
FREDERICK G. CONRAD
MICK P. COUPER
BRADY T. WEST

Although web surveys in which respondents are encouraged to use

smartphones have started to emerge, it is still unclear whether they are a
promising alternative to traditional web surveys in which most respond-
ents use desktop computers. For sample members to participate in
smartphone-based surveys, they need to have access to a smartphone
and agree to use it to complete the survey; this raises concerns about cov-
erage and nonresponse, as well as measurement if those who agree to
participate have any difficulty using smartphones. In an analysis of data
from a smartphone versus desktop (within-subjects) experiment con-
ducted in a probability-based web panel, we compare estimates produced
by the smartphone web survey (one condition) and PC web survey (other
condition). We estimate mode effects and then examine the extent to
which these effects are attributable to coverage, nonresponse, and

CHRISTOPHER ANTOUN is a lecturer in the College of Information Studies and in the Joint Program
in Survey Methodology, University of Maryland, 1218 LeFrak Hall, College Park, MD 20742,
USA. FREDERICK G. CONRAD is research professor in the Survey Research Center, Institute for
Social Research, University of Michigan, 426 Thompson Street, Ann Arbor, MI 48104, and in the
Joint Program in Survey Methodology, University of Maryland. MICK P. COUPER is research pro-
fessor in the Survey Research Center, Institute for Social Research, University of Michigan, and
in the Joint Program in Survey Methodology, University of Maryland. BRADY T. WEST is research
associate professor in the Survey Research Center, Institute for Social Research, University of
Michigan, and in the Joint Program in Survey Methodology, University of Maryland.
The authors thank CentERdata (Tilburg University, Netherlands) for conducting the experiment
reported here in the LISS (Longitudinal Internet Studies for the Social Sciences) panel. The
authors also thank Yazmin Garcia Trejo, Joanne Pascale, Benjamin Reist, and the anonymous
reviewers for their helpful comments on this manuscript.
*Address correspondence to Christopher Antoun, Joint Program in Survey Methodology, 1218
LeFrak Hall, 7251 Preinkert Dr, College Park, MD 20742, USA; E-mail: antoun@umd.edu.
doi: 10.1093/jssam/smy002
C The Author(s) 2018. Published by Oxford University Press on behalf of the American Association for Public Opinion Research.
V
All rights reserved. For Permissions, please email: journals.permissions@oup.com

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 2 Antoun et al.

measurement errors in the smartphone-based survey. While mode

effects were generally small, we find that the smartphone web survey
produced biased estimates relative to PC web for a subset of survey
variables. This was largely due to noncoverage and, to a lesser extent,
nonresponse. We find no evidence of measurement effects. Our find-
ings point to the trade-off of the advanced data collection opportuni-
ties of smartphones and the potential selection errors that such devices
may introduce.
KEYWORDS: Coverage error; Measurement error; Mobile web survey;
Nonresponse error; Total survey error.

1. INTRODUCTION
Mobile web surveys (i.e., surveys in which questions are displayed on
smartphones and sometimes tablets using mobile browsers or dedicated
research apps) are becoming an increasingly important component of the
online research landscape. This is partly because more respondents are
choosing to complete web surveys using mobile devices as opposed to
desktop and laptop computers (hereafter referred to as PCs). It is also be-
cause mobile web surveys in which respondents are encouraged to use
smartphones—referred to here as “smartphone surveys” if smartphones are
the primary survey device and “smartphone web surveys” if the survey is
browser-based—are starting to be implemented in place of traditional web
surveys.
Smartphone surveys have ushered in new opportunities for survey research-
ers to, for example, reach mobile-only users who might not otherwise partici-
pate in online surveys, utilize SMS texting invitations, and leverage the
advanced features of phones (e.g., GPS, camera, and sensors) for collecting
auxiliary data that augments data from survey responses (Link, Murphy,
Schober, Buskirk, Hunter Childs, et al. 2014). However, for sample members
to participate in these surveys, they need to have their own smartphone or be
provided one and agree to use it to complete the survey; this raises concerns
about coverage and nonresponse, as well as measurement if those who agree to
participate have any difficulty using smartphones. Thus, it is still unclear if
such surveys are a promising alternative to traditional web surveys for produc-
ing population estimates.
We use data from a smartphone versus PC (within-subjects) experiment
conducted in a probability-based LISS web panel (Longitudinal Internet
Studies for the Social Sciences) in the Netherlands to examine the quality
of estimates from smartphone surveys. Our analysis has two main steps.
We first estimate smartphone web versus PC web mode effects across sev-
eral survey variables and test whether these effects are statistically

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 3

significant.1 Since mode effects are not just measurement error but rather
the aggregation of all error differences, we then decompose the mode
effects into components attributable to coverage, nonresponse, and mea-
surement error. We do this by estimating each survey variable using differ-
ent subsamples of the full study sample and then taking the differences
between pairs of these estimates (further described in the methods section).

1.1 Background
Researchers are beginning to conduct smartphone surveys in which respond-
ents are encouraged to use smartphones or where participation is restricted to
smartphone users. These surveys, while still mostly limited to feasibility stud-
ies of specific groups (e.g., students, patients) and not as common as web sur-
veys that allow completion by any type of device, have started to emerge for
surveys with larger, more diverse samples that make inferences about target
populations. For example, researchers from the Netherlands Institute for Social
Research and CentERdata (Tilburg University) carried out a study where ap-
proximately two thousand panelists from the LISS panel (the same panel where
the current experiment was conducted) were invited to download a mobile di-
ary app and report their time use on a smartphone over a two-day period
(Scherpenzeel 2017). They found that about 68 percent of the panelists partici-
pated in the survey and that it produced nearly equal time use distributions as a
paper-and-pencil time use survey, except for a few subcategories of activities
(e.g., time spent using computers; time spent eating). The differences were not
due to excluding those panelists who did not have smartphones, as phones
were provided to them, but could have been due to differences between the
panelists who were invited and those who chose to participate, or measurement
effects, or some combination of the two. In another example, researchers at
Research Triangle International (RTI) initiated a smartphone web panel by
recruiting about 750 participants by mail (addressed-based sampling) to down-
load an app and take periodic surveys on their smartphones (Roe, Zhang, and
Keating 2013). This approach, while innovative, did also present several chal-
lenges. For example, the sample members who did not have smartphones were
ineligible to participate, and response rates (conditional on joining the panel)
varied widely depending on the survey topic (they were lower for surveys
about general topics than about current events).
In another study that tried to make inference to a larger population, this one
conducted by the Centers for Disease Control and Prevention (CDC), approxi-
mately 500 smartphone users were recruited over the phone using a cell phone
sampling frame and sent SMS links to two smartphone web surveys on to-
bacco use (Hu, Dayton, Freedner-Maguire, and DuBray 2014). While this

1. We use the phrase “mode effects” to describe the effect of device (smartphone/PC), but one
can make the case that web surveys are a single mode implemented using different devices.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 4 Antoun et al.

approach proved to be feasible—response rates among those who agreed to

participate over the phone were 28 percent to both surveys—the demographic
characteristics of the resulting sample and their reports of tobacco use differed
markedly from national telephone samples. It is unclear what survey error or
combination of errors was responsible. This points to the need for research that
can decompose the total error from a smartphone survey into its components,
as this can inform approaches to reduce or adjust for the largest errors.
Other “conventional” web surveys of the kind that have come to dominate
social and market research where PCs are the primary survey devices have not
encouraged smartphone use, but have a large and growing number of respond-
ents who choose to use smartphones (e.g., Revilla, Toninelli, Ochoa, and
Loewe 2016; Peterson, Griffin, LaFrance, and Li 2017). This trend raises the
possibility that the majority of completed web survey questionnaires may soon
come from mobile users.

1.2 Survey Errors

In this paper, we estimate mode effects in a smartphone web survey against a
PC web reference survey for several survey variables and then decompose the
mode effects into specific error sources. While measurement error (discrepan-
cies on a survey variable between reported values and a “true” value) is often
assumed to be the reason for mode effects, the Total Survey Error (TSE) model
(e.g., Groves 1989) provides a typology of other specific error sources divided
into errors of observation (e.g., measurement error) and errors of nonobserva-
tion (e.g., coverage error, nonresponse error) that can bias survey estimates.
Under this framework, the aggregation of error differences between two modes
is equivalent to the overall mode effect. Thus, we move beyond measurement
error and also examine the effect of smartphone data collection on two other
error sources: coverage and nonresponse. We focus on these errors rather than
sampling, specification, and processing errors (Biemer and Lyberg 2003), be-
cause we think the likelihood of the latter types of errors is the same across
smartphone and PC web surveys.
Coverage error, which in this case refers to differences on survey statistics
between the target population and the population of smartphone owners, is a
concern because smartphone ownership and use is not universal.2 According
to Pew (Poushter 2017), 79 percent of adults in the Netherlands and 77 percent
of adults in the United States own smartphones. Furthermore, there seem to be
differences between smartphone owners and non-owners, with the former

2. Because having a smartphone is necessary to reach a sample member for a smartphone survey,
we focus on it as our indicator of coverage. However, possession may not be sufficient. Those
who have smartphones also have to know how to use their devices and have either easy access to
Wi-Fi or a sufficient amount of data on their data plans.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 5

being younger, better educated, and, in the United States, more likely to be
black or Hispanic (Fuchs and Busse 2009; Metzler and Fuchs 2014; Smith
2015).
Nonresponse error, or discrepancies on survey statistics between the pool of
initial sample members and the pool of respondents to a survey, is of special
concern in smartphone surveys because sample members tend to respond at
lower rates when using smartphones than PCs (De Bruijne and Wijnant 2013a;
Mavletova 2013; Mavletova and Couper 2013; Wells, Bailey, and Link 2014;
Buskirk and Andrus 2014). They are also more likely to break off from web
surveys when using smartphones (Mavletova 2013; Mavletova and Couper
2013; Wells et al. 2014). While nonresponse rates by themselves are not an in-
dicator of bias, differences between respondents and nonrespondents on survey
variables do lead to nonresponse errors, and such differences have been ob-
served in mobile web surveys for education (De Bruijne and Wijnant 2014)
and income (Mavletova and Couper 2013).
The third type of error that we examine is measurement error. Smartphone
web surveys are generally quite similar to PC web surveys in terms of their de-
sign. However, respondents’ use of a small touchscreen and the fact that they
are using their device in a variety of contexts and settings may compromise re-
sponse quality. While measurement error has received the most research inter-
est, we think that it is probably the least prominent error, given the empirical
work showing that respondents using smartphones can provide high-quality
responses as long as they are presented with question formats that are easy to
use on small touchscreens, (i.e., the survey is optimized for smartphone use)
(De Bruijne and Wijnant 2013a; Mavletova 2013; Mavletova and Couper
2013; Buskirk and Andrus 2014; Wells et al. 2014; Lugtig and Toepoel 2015;
Toninelli and Revilla 2016; Antoun, Couper, and Conrad 2017a; Keusch and
Yan 2017).
The sizes of coverage, nonresponse, and measurement errors in smartphone
surveys may vary based on several factors, including the methods used (e.g.,
browser-based versus app-based smartphone survey) and the target population
about which one wants to make inferences. The current study employs a smart-
phone web survey to make inferences about a specific group of online panelists
(described later in more detail). Thus, there is no guarantee the particular
results observed in the current study will generalize to different populations.
One would need a different sample, for example, to understand the impact of
smartphone data collection on errors in a survey of the general population.
Nonetheless, it seems likely that the mechanisms underlying errors in smart-
phone surveys described above (e.g., survey completion on small touchscreens
and lower response rates than PC web surveys) would apply irrespectively of
the target population and that an analysis of errors with our sample might still
yield insights into the viability of smartphone data collection with other
samples.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 6 Antoun et al.

1.3 Research Questions

Based upon our review of this prior research, we developed three research
questions related to smartphone versus PC mode effects for a survey conducted
in an online probability panel.
RQ1) To what extent will smartphone estimates differ from PC estimates?
RQ2) When the mode effect is decomposed into components, which source
of error will generally be the largest contributor to the total error in a given
smartphone survey estimate?
RQ3) Will coverage error and nonresponse error in the smartphone survey
generally move in the same direction or not?
In summary, researchers are beginning to conduct smartphone surveys, but
they may negatively affect the accuracy of survey estimates because of cover-
age, nonresponse, and measurement errors. Therefore, it is important to evalu-
ate the specific error properties of this new method.

2. METHODS
2.1 Study Design and Sample
Our research questions were investigated using data from a two-wave cross-
over experiment used by Antoun, Couper, and Conrad (2017a) to compare
data quality in smartphone and PC web browser–based surveys. The experi-
ment was conducted in the LISS panel, which is a probability-based online
panel administered by CentERdata (Tilburg University, Netherlands) designed
to be representative of the adult (age sixteen and older) population of the
Netherlands; it was started in 2007 by drawing a probability sample of Dutch-
speaking households from the Netherlands population register and recruiting
them by mail, phone, and in-person visits (see Scherpenzeel 2011).3 In the ex-
periment, half of the participants were invited to complete the web survey on
their PC, followed by a smartphone; the other half were invited to complete the
web survey on a smartphone first, followed by a PC. Participants were not in-
vited to use tablets. We refer to the two different waves of the experiment as
“wave one” and “wave two.” Invitations to the surveys were sent by email.
The questionnaire contained forty-six questions (presented on thirty-two sur-
vey pages). The smartphone version was optimized for small devices using a
special style sheet (De Bruijne and Wijnant 2013b). The normal cash incentive

3. The LISS panel has added refreshment samples in an effort to continue to represent the adult
population of the Netherlands. However, the panel itself may have coverage bias because certain
demographic groups—for example, younger adults, single person households, and urban house-
holds—are slightly underrepresented in the panel (LISS 2015).

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 7

Figure 1. Study Design and Group Sizes.

for the LISS panel was provided as payment for participation in both waves
and for both devices.
The experiment was conducted in 2013—from October 7 to October 29 for
wave one and from December 2 to December 31 for wave two. Panelists were
eligible if in a screener survey (conducted two months before the start of the
experiment) they expressed willingness to participate in mobile research and
reported either having their own iPhone or Android phone or expressed will-
ingness to use a smartphone that was provided to them for the study. A total of
5,486 panelists completed the screener, of which 2,263 expressed willingness
to participate. A stratified random sample4 was drawn from this group, result-
ing in a total invited sample of 1,390 panelists (i.e., 695 in each sequence of
modes).
As shown in figure 1, 591 panelists completed the PC web survey in wave
one (of the 695 who were invited), and 536 completed it in wave two (of the
695 who were invited).5 We combine these two groups of panelists across
waves to construct our “reference group” (n ¼ 1,127), which serves as our pop-
ulation of inference for the purposes of this analysis. We use their answers to
the PC web survey to compute benchmark estimates. The 263 nonrespondents
to the PC web survey are excluded because there is no reference information

4. iPhone and Android phone owners were oversampled, and users of other phones or non-users
were undersampled.
5. The PC web response rate (RR1) was 81.1% (American Association for Public Opinion
Research [AAPOR] 2015).

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 8 Antoun et al.

(i.e., responses recorded in PC web) available for them on the specific variables
measured in the study.6
The reference group was quite diverse in terms of age (median ¼ 45 years
old; min ¼ 16 years old, max ¼ 87 years old), gender (49.9 percent male;
50.1 percent female), and education (61.9 percent less than college degree;
38.1 percent college degree or more). However, it cannot be characterized as
representative of the entire panel (nor of the general population of the
Netherlands) because it excludes the panelists who did not express willing-
ness to participate in the experiment and the nonrespondents to the PC web
survey.
Of the 591 panelists who completed the PC web survey in wave one, 430
previously reported owning a smartphone; and of the 536 panelists who com-
pleted the PC web survey in wave two, 370 previously reported owning a
smartphone. We combine these two groups of panelists across waves to con-
struct our “covered group” (n ¼ 800). Although non-owners were technically
“covered” in this experiment since smartphones were provided to those who
did not have them, for our analysis we counted these panelists as not covered
because in a more typical smartphone survey where phones are not provided,
non-owners would be unable to participate.
Of those who were covered, 281 panelists used their smartphones to com-
plete the smartphone survey in wave one, and 308 used their smartphones to
complete it in wave two.7 We combine these two groups of panelists across
waves to construct our “responding group” (n ¼ 589). We use their answers to
the smartphone survey to compute final estimates for that survey.
The three groups—reference, covered, and responding—are overlapping.
The responding group is a subset of the covered group, which is a subset of the
reference group.

2.2 Analysis
For our analysis, we use all survey items, except for those that were formatted
as a slider or spin wheel (Antoun et al. 2017a reported that respondents using
smartphones had difficulty recording their answers for these question formats)
or that were an evaluation of either the device used to complete the survey or
the survey itself (i.e., debriefing questions). The survey did not contain demo-
graphic items. We coded each variable into two categories if it was not already
binary, and we focused on the proportion (expressed as a percentage) of those
providing one answer or the other. Therefore in our analysis, we can express
errors as the difference between two proportions which, unlike the difference

6. These nonrespondents were reliably older (t(1,386) ¼ 9.22, p < 0.01) and more likely to be
male (t(1,386) ¼ 2.92, p < 0.01) than respondents.
7. The devices used by respondents were detected by user-agent string information (Callegaro
2010). Those who used a tablet or PC were counted as nonrespondents.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 9

between two means, can be compared across variables without the need to re-
scale them into relative differences. The full list of variables used is pre-
sented in table 1, with the category used to compute the proportion labeled
as the “relevant response.” We choose the relevant response for each vari-
able to resemble how substantive researchers might analyze the variables.
The variables are divided by topic into three groups: technology, lifestyle,
and politics.
We examine the error associated with estimates based on each of these vari-
ables. To simplify the challenge of isolating error sources, our focus is on de-
scriptive statistics (specifically the proportion of those providing the relevant
response). The process of estimating errors follows several steps. We start by
estimating the total smartphone versus PC web mode effect, which is the ag-
gregation of all error differences.
We estimate this by computing the difference between the relevant propor-
tion measured in smartphone web based on the responding group (n ¼ 589)
and the proportion measured in PC web based on the reference group
(n ¼ 1,127):

Errormode ¼ pSP PC
respond  pref (1)

In this notation, the subscripts refer to the group that the statistic is based on
(reference [“ref ”]; covered [“covered”], and responding [“respond”]), and the
superscript refers to the mode in which the variable is measured (PC web
[“PC”] versus smartphone web [“SP”]).
We decompose the total mode effect into components arising from cover-
age, nonresponse, and measurement error. We do this by estimating each sur-
vey variable based on different subgroups of the reference group. We then take
the differences between different pairs of estimates. Using this approach, the
part attributable to coverage error is estimated by computing the difference be-
tween the relevant proportion measured in PC web based on the covered group
(n ¼ 800) and the proportion measured in PC web based on the reference group
(n ¼ 1,127):

Errorcoverage ¼ pPC PC
covered  pref : (2)

The part attributable to nonresponse error is estimated by computing the

difference between the relevant proportion measured in PC web for the respond-
ing group (n ¼ 589) and the proportion measured in PC web for the covered
group (n ¼ 800):

Errornonresponse ¼ pPC PC
respond  pcovered : (3)

The part attributable to measurement error is estimated by computing the

difference between the relevant proportion measured in smartphone web based

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 10 Antoun et al.

Table 1. Items Used in this Analysis and Their Wording

Relevant response Question Response options

(if applicable)

Technology
Prefer using tablet to What is your preferred device for going a mobile phone/
go online online? smartphone;
Prefer using tablet to If you could choose, which device a personal
fill out would you prefer to use to fill out computer;
questionnaires your next questionnaire? a tablet
Hours watching TV On an average weekday, how much
(>3 hours) time, in total, do you spend watching
television?
Lifestyle
Satisfied with social How satisfied are you with your social not at all satisfied;
life life? not too satisfied;
Satisfied with family How satisfied are you with your family somewhat satisfied;
life life? very satisfied
Satisfied with your How satisfied are you with your pace of
pace of life life?
Satisfied with your How satisfied are you with the feeling
safety from crime of safety where you live?
Exercise less than In a typical week, about how often do less than 1 time per
once per week you exercise? week;
1 or 2 times per
week;
3 times per week;
4 or more times per
week
Binge drank in past On how many days did you have 5 or
30 days more drinks of an alcoholic beverage
on the same occasion?
Ever driven while Have you ever, in your entire life, yes;
intoxicated driven a car (or other motor vehicle) no
when you were (at least a little)
intoxicated?
Eat in restaurants at During the past 12 months, how many
least once per times did you eat in restaurants?
month
Go shopping at least During the past month, how many times
once a week did you go shopping?
Continued

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 11

Table 1. Continued

Relevant response Question Response options

(if applicable)

Politics
Immigrants make In your opinion are the Netherlands 1 ¼ much worse to
country a worse made a worse or better place to live 10 ¼ much better;
place by immigrants coming to live here? dichotomized as
0–4 (worse) and
5–10 (better)
Feel favorably Please indicate how favorable or unfa- 0 to 100;
towards George vorable you feel toward the following dichotomized as
W. Bush person or organization. . . Former 0–49 (unfavor-
U.S. President George W. Bush able) and 50–100
Feel favorably U.S. President Barack Obama (favorable)
towards Barack
Obama
Feel favorably Mark Rutte (current Prime Minister of
towards Mark the Netherlands)
Rutte
Feel favorably Jan Peter Balkenende (former Prime
towards Jan Peter Minister of the Netherlands)
Balkenende
Feel favorably Christian Democratic Appeal (Political
towards CDA party in the Netherlands on the left to
left-right of the political spectrum)
Feel favorably People’s Party for Freedom and
towards VVD Democracy (Political party in the
Netherlands that is considered so-
cially liberal and economically
conservative)

NOTE.—All of the questions were asked in both waves and for both modes.

on the responding group (n ¼ 589) and the proportion measured in PC web

based on the same group (n ¼ 589):

Errormeasurement ¼ pSP PC
respond  prespond : (4)

2.3 Estimating Standard Errors and Confidence Intervals

While estimates from the PC web survey are not free from error, we use them as
a benchmark because we expect the PC web survey to provide more accurate
estimates than the smartphone survey. This is because of its preferred coverage
properties and response properties: in the LISS panel, internet access is universal
because computers and Internet access were provided to those who did not have

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 12 Antoun et al.

them upon joining the panel, and PC web response rates are generally quite
high.8 Thus, statistically significant deviations between the two surveys are inter-
preted as a reflection of reduced data quality in the smartphone survey.
We need to distinguish statistically significant deviations from those that
were not statistically significant, since the latter could be merely the result of
sampling variation rather than true bias. We consider the deviations to be sta-
tistically significant if their 95 percent confidence intervals, computed by esti-
mating standard errors of the deviations and then adding and subtracting 1.96
standard errors from the deviations, do not contain zero.
The fact that the three groups are overlapping, and hence not independent,
requires special estimation of standard errors. For overall error estimates, stan-
dard errors are estimated using the following formula adapted from Lee (2006,
p. 465):
 
se pSP  pPC
respond  ref rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
   
nref  nrespond
¼ var pSP respond þ var p PC
notrespond; notcovered ;
nref
(5)

where there are nref panelists in the reference group and nrespond panelists in the
responding group, and pPC notrespond; notcovered is the estimated proportion from
PC web for those who do not own smartphones and did not respond to the
smartphone survey. The variances of the proportions are estimated using the
variance formula for a simple random sample.
For coverage error and nonresponse error estimates, standard errors were es-
timated using the same framework (Appendix A). These formulas depend on
two assumptions. First, all of the observed group sizes are assumed to be fixed
quantities rather than random quantities. Second, the covariance between esti-
mates in each equation is assumed to equal zero.
For measurement error estimates, standard errors are estimated using the for-
mula associated with McNemar’s test for comparing dependent proportions
(McNemar 1947):
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
   
  pPC PC
þ pPC PC PC PC
SP PC 12 1  p12 21 1  p21 þ 2ðp12  p21 Þ
se prespond  prespond ¼ ;
nrespond
(6)

where pPC PC PC
12 and p21 are measures of response change: p12 is the proportion of
respondents who recorded a yes answer in PC web after recording a no answer

8. Futhermore, since all panelists can go online using a computer, there is no smartphone-only
sample that can be reached with a smartphone survey but not a PC Web survey.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 13

in smartphone web, and pPC 21 is the proportion of respondents who recorded a

no answer in PC web after recording a yes answer in smartphone web.9
We compare the resulting confidence intervals from the closed-form formulas
to those obtained using bootstrap resampling, which involved the steps of draw-
ing 1,000 independent with-replacement samples of the same size as the full
group (n ¼ 1,127) from that group, repeatedly computing the error estimate for
the relevant survey variable, and then assessing the distribution of these error esti-
mates (with the 2.5th and 97.5th percentiles of this distribution serving as the up-
per and lower limits of the bootstrapped confidence interval). We focused this
comparison on two variables that were found to have either relatively large cover-
age or nonresponse error. The formulas and the bootstrap approach yield nearly
identical results across all error estimates (Appendix Table), suggesting that the
assumptions that underlie the formulas are reasonable. For reasons of parsimony,
we opt to present only the standard errors based on the formulas.

2.4 Multivariate Models

For the deviations that are statistically significant, another issue is whether the
bias can be repaired by postsurvey statistical adjustments using demographic
variables. To assess this, we fit two multivariate models. The first one predicts
coverage (having a smartphone versus not having a smartphone) with the vari-
ables that we find to be biased by coverage and the following demographic var-
iables obtained from the LISS data archive: age, gender, education (primary
school; junior high school; high school degree; vocational degree; higher voca-
tional degree; and university degree or more), marital status, presence of chil-
dren living at home, renting status, and urbanization level (not urban; slightly
urban; moderately urban; very urban; and extremely urban.). The second
model predicts nonresponse (responding to the smartphone survey versus not
responding) with the variables that we find to be biased by nonresponse and
the same demographic variables. If the association between these survey varia-
bles and selection into the survey is eliminated in the multivariate models, it
suggests that weighting adjustments may be effective; if the association be-
tween these survey variables and selection into the survey persists, then such
adjustments may not be effective.

2.5 Advantages of Using a Crossover Design for Estimating Errors

Our approach of estimating errors with two sequences of modes in a cross-
over design does not rely on the same assumptions as the traditional

9. The subscripts refer to the off-diagonal elements in a 2  2 frequency table containing

responses in smartphone web and responses in PC web for a binary variable for the same
participants.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 14 Antoun et al.

repeated-measurements approach with a single sequence of modes where

researchers compare responses received from an initial survey and those from
a subsequent survey conducted in a preferred data collection mode (e.g., Fecso
and Pafford 1988; Schouten, van den Brakel, Buelens, van der Laan, and
Klausch 2013). The assumptions that underlie that approach are violated if
respondents’ circumstances change between surveys or if they remember their
previous responses. By contrast, the assumptions that underlie our crossover
design are that the attribute being measured changes in the same way over time
for both sequences of modes and that to the extent memory of the first survey
affects responses or the decision to participate in the second survey, it does so
in the same way for both sequences. We tested these assumptions by looking
at the effects of sequence on responses (means) and response rates, and we
found no consistent violations.10

3. RESULTS
As shown in table 2, proportions for each variable of interest (expressed as per-
centages) are estimated in four different ways: the first column presents the
proportions measured in PC web based on the reference group; the next col-
umn presents proportions measured in PC web based on the covered group;
the third column presents proportions measured in PC web based on the
responding group; and the fourth column presents proportions measured in
smartphone web based on the responding group.
Errors are estimated by taking the differences between different columns.
The overall mode effect, presented in the eighth and final column, is estimated
by comparing columns one and four. The impact of noncoverage, which is pre-
sented in the fifth column, is derived by comparing columns one and two. The
effect of nonresponse, shown in the sixth column, is estimated by taking the
difference between columns two and three. Finally, the impact of measurement
error, which is presented in the seventh column, is estimated by comparing col-
umns three and four.
For example, according to the reference group, 41.6 percent of all PC web
respondents report binge drinking in the past thirty days. The corresponding
estimate from the smartphone survey is 47.7 percent, resulting in an overall
mode effect of 6.1 percentage points (47.7–41.6). We can then decompose this
into the parts attributable to coverage, nonresponse, and measurement errors.
The estimate increases from 41.6 percent to 46.3 percent among the panelists
who own smartphones, resulting in a coverage deviation of 4.7 percentage
points (46.3–41.6). The estimate comes back down to 44.8 percent among the

10. Out of the nineteen variables (described in table 1), two yielded significant effects. Estimates
of TV viewing and support for the VVD party changed more in the PC-smartphone sequence than
the smartphone-PC sequence. We found no significant effect of sequence on response rates.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
on 16 March 2018
Table 2. Percentage Distributions of Survey Variables across Groups (and Standard Errors), and Total Mode Effect Estimates Decomposed into
Coverage, Nonresponse, and Measurement Components (and Standard Errors, Based on the Closed Formula Approach)

by Tufts University user

(1) Benchmark (2) Covered: (3) Responded: (4) Responded: (5) Estimated (6) Estimated (7) Estimated (8) Estimated
(PC web) PC answers PC answers smartphone coverage nonresponse measurement total mode
n ¼ 1,127 n ¼ 800 n ¼ 589 answers error error error effect
n ¼ 589
(SE) (SE) (SE) (SE) (SE) (SE) (SE) (SE)

Technology
Prefer using tablet to go 17.7 19.1 18.0 20.2 1.4 1.1 2.2 2.5
online (1.1) (1.4) (1.6) (1.7) (0.7) (0.9) (1.4) (1.6)
Prefer using tablet to fill 15.4 16.3 15.8 16.3 0.9 0.5 0.5 0.9
out questionnaires (1.1) (1.3) (1.5) (1.5) (0.7) (0.8) (1.5) (1.4)
Hours watching TV 45.9 42.0 42.1 43.1 3.9 0.1 1.0 2.8
(>3 hours) (1.5) (1.7) (2.0) (2.0) (0.9) (1.0) (1.7) (1.9)
Lifestyle
Satisfied with social life 92.6 93.5 94.4 94.6 0.9 0.9 0.2 2.0
(0.8) (0.9) (0.9) (0.9) (0.5) (0.6) (1.1) (1.0)
Estimating Errors in a Smartphone-Based Survey

Satisfied with family 93.2 93.3 93.2 93.7 0.1 0.1 0.5 0.5
life (0.8) (0.9) (1.0) (1.0) (0.5) (0.5) (1.0) (0.9)
Satisfied with your pace 85.8 86.6 87.1 86.9 0.8 0.5 0.2 1.1
of life (1.0) (1.2) (1.4) (1.4) (0.7) (0.7) (1.4) (1.3)
Satisfied with your 94.9 95.0 95.1 94.9 0.1 0.1 0.2 0.0
safety from crime (0.7) (0.8) (0.9) (0.9) (0.4) (0.5) (0.9) (0.9)
Exercise less than once 82.2 81.8 81.7 82.3 0.4 0.1 0.6 0.1
per week (1.1) (1.4) (1.6) (1.6) (0.7) (0.8) (1.4) (1.5)
Continued
15

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

on 16 March 2018
Table 2. Continued
16

(1) Benchmark (2) Covered: (3) Responded: (4) Responded: (5) Estimated (6) Estimated (7) Estimated (8) Estimated

by Tufts University user

(PC web) PC answers PC answers smartphone coverage nonresponse measurement total mode
n ¼ 1,127 n ¼ 800 n ¼ 589 answers error error error effect
n ¼ 589
(SE) (SE) (SE) (SE) (SE) (SE) (SE) (SE)

Binge drank in past 41.6 46.3 44.8 47.7 4.7 1.5 2.9 6.1
30 days (1.5) (1.8) (2.1) (2.1) (0.9) (1.1) (1.8) (1.9)
Ever driven while 42.1 42.5 45.0 45.7 0.4 2.5 0.7 3.6
intoxicated (1.5) (1.7) (2.1) (2.1) (0.9) (1.0) (1.6) (1.9)
Eat in restaurants at 32.9 36.6 36.5 36.2 3.7 0.1 0.3 3.3
least once per month (1.4) (1.7) (2.0) (2.0) (0.8) (1.0) (1.9) (1.9)
Go shopping at least 32.6 30.3 30.2 33.8 2.3 0.1 3.6 1.2
once a week (1.4) (1.6) (1.9) (2.0) (0.9) (1.0) (2.2) (1.8)
Politics
Immigrants make 67.3 68.0 65.2 64.2 0.7 2.8 1.0 3.1
country a worse place (1.4) (1.7) (2.0) (2.0) (0.9) (0.9) (1.6) (1.7)
Feel favorably towards 13.2 14.6 14.8 13.8 1.4 0.2 1.0 0.6
G.W. Bush (1.0) (1.3) (1.5) (1.4) (0.6) (0.7) (1.4) (1.3)
Feel favorably towards 63.4 64.0 66.6 66.4 0.6 2.6 0.2 3.0
B. Obama (1.4) (1.7) (1.9) (1.9) (0.9) (1.0) (2.0) (1.9)
Feel favorably towards 35.7 38.3 39.6 39.6 2.6 1.3 0.0 3.9
M. Rutte (1.4) (1.7) (2.0) (2.0) (0.9) (1.0) (1.9) (1.8)
Antoun et al.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

on 16 March 2018
Feel favorably towards 31.9 32.5 35.1 33.4 0.6 2.6 1.7 1.5

by Tufts University user

J.P. Balkenende (1.4) (1.7) (2.0) (1.9) (0.9) (0.9) (1.9) (1.7)
Feel favorably towards 22.6 22.3 23.3 23.9 0.3 1.0 0.6 1.3
CDA (1.2) (1.5) (1.7) (1.8) (0.8) (0.9) (1.6) (1.6)
Feel favorably towards 27.2 30.1 30.9 34.0 2.9 0.8 3.1 6.8
VVD (1.3) (1.6) (1.9) (2.0) (0.8) (1.0) (1.8) (1.7)
Summary measures
Number of significant – – – – 8 4 0 3
error estimates
Average absolute error – – – – 1.5 1.0 1.1 2.3

NOTE .— Boldface indicates values that are significantly different from zero (p < 0.05). The error estimates presented in columns 5–8 are based on
the rounded values from columns 1–4.
Estimating Errors in a Smartphone-Based Survey
17

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

 18 Antoun et al.

panelists with smartphones who participated in the smartphone survey, result-

ing in a nonresponse deviation of 1.5 percentage points (44.8–46.3). The fi-
nal estimate moves back up to 47.7 percent when it is based on respondents’
answers recorded on smartphones rather than PCs, producing a measurement
deviation of 2.9 percentage points (47.7–44.8). Thus, coverage error is the
largest contributor to the mode effect for this particular variable.

3.1 Total Errors

We first focus on the overall mode effects (addressing RQ1). They varied in
size, ranging from 0.0 percent to 6.8 percentage points. The average mode ef-
fect for all variables (bottom row of column eight of table 2) is 2.3 percentage
points. Therefore, we would not consider the relative size of the mode effects
to be large. Moreover, the magnitude of some mode effects are quite small
(e.g., estimates of exercise and attitudes toward crime in the smartphone web
survey deviate from the benchmark by 0.1 percentage points or less).
However, smartphone web produces biased estimates relative to PC web for a
subset of survey variables. Out of the nineteen variables, three (or 16 percent
of them) yield significant effects. Estimates of binge drinking, support for the
current prime minister (Mark Rutte), and support for the VVD party are in-
flated in smartphone web relative to the benchmark.

3.2 Decomposition of Total Error

When the total error is decomposed into three parts (addressing RQ2), cover-
age error seems to be the most prominent. The average absolute coverage devi-
ation is 1.5 percentage points, compared to 1.0 for nonresponse, and 1.1 for
measurement (bottom row of table 2). Further evidence of coverage bias comes
from the number of deviations that are significantly different from zero.
According to the significance tests, eight of the estimates (or 42 percent of
them) suffer from significant coverage bias, whereas four of the estimates (or
21 percent of them) are biased by nonresponse, and none of the estimates are
biased by measurement effects.

3.3 Directions of Coverage and Nonresponse Errors

We next evaluated the direction of selection errors (addressing RQ3). For elev-
en out of the nineteen variables (or 58 percent of them), the error from
noncoverage is compounded by the error from nonresponse. This does not oc-
cur for any of the technology use variables but is more common for the other
groups of variables (e.g., political views, lifestyle choices), suggesting that a
wide variety of variables may be associated with the decision both to use a
smartphone and to participate in a smartphone survey. Significant coverage

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 19

error and compounding nonresponse error appears to be the cause of the over-
all mode effect for two of the three variables that yield an overall mode effect
(support for the current prime minister and support for the VVD party).

3.4 Multivariate Models

For some survey variables, their association with selection into the survey
might persist in multivariate models. To test this, we fit two models (shown in
table 3). The first one predicts coverage (having a smartphone versus not hav-
ing a smartphone) with the eight variables (based on PC web responses) that
showed statistically significant coverage bias and several demographic varia-
bles commonly used in postsurvey weighting adjustments. The second model
predicts nonresponse (responding to the smartphone survey versus not respond-
ing) with the four variables (based on PC web responses) that showed statisti-
cally significant nonresponse bias and the same demographic variables.11
Out of the eight variables with significant coverage bias, two of them con-
tinue to have a significant association with coverage in the multivariate model
over and above the demographic controls. These variables are related to tablet
use and eating out. The affected variables could conceivably influence smart-
phone usage or at least be influenced by the same factors that influence smart-
phone ownership.
For the four variables with significant nonresponse bias, two of them—
attitudes about immigrants and support for former Prime Minister Jan Peter
Balkenende—continue to have a significant association with nonresponse after
controlling for demographic differences. It is unclear why these variables
would influence the decision to participate directly. They may instead be corre-
lated with auxiliary variables that influence participation but that were not in-
cluded in the model. This possibility is supported by the model’s small R2
value.

4. DISCUSSION
While smartphone-based surveys provide new opportunities for researchers,
our analysis demonstrated that their use can affect estimates relative to a PC
web survey because of selection effects, at least for our particular population
of members of a probability-based online panel and for some of the studied
items. We found that a smartphone web survey produced biased estimates rela-
tive to the PC web reference survey for a subset of the survey variables. The
average size of the mode effects—2.3 percentage points—was not large.

11. We also entered the survey variables into the models one at a time, but the pattern of results
was unchanged.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 20 Antoun et al.

Table 3. Estimated Coefficients in Logistic Regression Models Predicting

Smartphone Ownership (Model 1) and Participation in a Smartphone Survey
Conditional on Smartphone Ownership (Model 2)

Model 1: coverage Model 2: survey

(yes ¼ 1; no ¼ 0) response, conditional
on coverage
(yes ¼ 1; no ¼ 0)

Est Est
(SE) (SE)

Intercept 3.926** 0.700*

(0.463) (0.298)
Demographic characteristics
Age (continuous) 0.083** 0.150
(0.007) (0.080)
Gender: male vs. female 0.211 0.180
(0.160) (0.171)
Highest degree (low to high) 0.096 0.302
(0.055) (0.179)
Married: yes vs. no 0.208 0.141
(0.178) (0.231)
Children in household: yes vs. no 0.286 0.126
(0.183) (0.168)
Renter: yes vs. no 0.076 0.150
(0.195) (0.080)
Urbanization (low to high) 0.083 0.180
(0.062) (0.171)
Survey variables with significant
coverage bias
Prefer using tablet to go online 0.480*
(0.207)
Hours watching TV (>3 hours) 0.007
(0.160)
Binge drank in past 30 days 0.268
(0.170)
Eat in restaurants at least once per 0.448*
month (0.179)
Go shopping at least once a week 0.134
(0.163)
Feel favorably towards George W. Bush 0.122
(0.262)
Feel favorably towards Mark Rutte 0.061
(0.203)
Feel favorably towards VVD 0.350
(0.230)
Continued

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 21

Table 3. Continued

Model 1: coverage Model 2: survey

(yes ¼ 1; no ¼ 0) response, conditional
on coverage
(yes ¼ 1; no ¼ 0)

Est Est
(SE) (SE)

Survey variables with significant

nonresponse bias
Binge drank in past 30 days 0.215
(0.174)
Immigrants make country a worse place 0.418*
(0.190)
Feel favorably towards Barack Obama 0.197
(0.180)
Feel favorably towards Jan Peter 0.391*
Balkenende (0.197)
n 1,126 799
R2 0.226 0.038
Max-rescaled R2 0.323 0.056

NOTE.— *p < 0.05, **p < 0.01. One case has a missing value for age and was ex-
cluded from both models. All variables were based on PC Web responses.

When we decomposed the total mode effect, we found coverage error to be

the largest contributor of error in spite of relatively high coverage rates (71 per-
cent). Coverage error moved in the same direction as nonresponse error for a
subset of survey variables, including two of the variables that showed an over-
all mode effect that was statistically significant. For some variables, we found
that their association with selection errors (coverage and nonresponse) per-
sisted in multivariate models that account for the demographic variables,
suggesting that postsurvey weighting adjustments would not remove all bias
from these estimates.
By contrast, we found no evidence of measurement effects, at least for the
types of questions we examined. In other words, conditional on coverage and
response to the survey, respondents gave similar responses on smartphones as
they did on PCs. This finding of measurement equivalence between smart-
phones and PCs is largely consistent with previous research (see Couper,
Antoun, and Mavletova 2017), though this may not hold for complicated ques-
tion formats (e.g., large grids, sliders) or non-optimized surveys (for a review,
see Antoun, Katz, Argueta, and Wang 2017b).

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 22 Antoun et al.

As mentioned earlier, we assume that the sizes of the errors in other smart-
phone surveys will differ from those reported here because of study-specific
factors. The levels of coverage in the population (e.g., college students may
have relatively low noncoverage rates, while a general population sample from
outside an online panel may have relatively high noncoverage rates) and levels
of participation (e.g., an app-based survey where sample members need to
agree to download a research app may produce relatively high nonresponse
rates) might increase or decrease the sizes of the selection errors. Similarly,
respondents’ levels of familiarity with smartphones and the design and com-
plexity of the survey might affect the sizes of the measurement errors.
Nevertheless, our findings make it clear that selection errors are a potentially
large contributor to error for descriptive statistics in smartphone surveys.
Are smartphone surveys viable for making inference to a target population?
Much like with PC web surveys, our results suggest that they might be viable
in situations where the problem of noncoverage can be overcome. These situa-
tions include a cross-sectional survey of a specialized population with espe-
cially high smartphone ownership rates or a web panel or longitudinal study in
which researchers provide phones to those who do not already have them (e.g.,
Fernee and Sonck 2013). Smartphone surveys may also be appealing in coun-
tries where large numbers of smartphone owners do not have PCs or broad-
band internet access (Smith 2015) or where ownership rates of mobile devices
are actually surpassing those of PCs (Revilla, Toninelli, Ochoa, and Loewe
2015). Still, our findings suggest that until mobile ownership rates increase,
estimates from smartphone surveys should be interpreted with caution, espe-
cially for attributes that directly influence one’s likelihood of owning a
smartphone.
This study has several limitations. First, the significant deviations are inter-
preted as bias because of the preferred coverage and response properties of PC
web surveys, though it is possible that some deviations actually reflect im-
proved data quality in the smartphone survey. Second, the power to detect
such differences is enhanced at the coverage stage (because of larger sample
sizes) compared with the subsequent stages (because of smaller sample sizes).
Finally, some of the selected variables were positively correlated (e.g., general
tablet use and tablet use to fill out questionnaires; satisfaction with social life
and satisfaction with family life). There would be value in future research that
considers a wider range of topics and then attempts to categorize them into
groups based on their relationship with different error sources.
Because of these limitations, our study should be viewed as a preliminary
evaluation of the likely errors associated with surveys restricted to smartphone
users. We encourage more research on the impact of smartphone surveys on
data quality, perhaps in a different setting and with a focus on mean-square er-
ror or multivariate statistics (e.g., regression coefficients).
The promise of smartphone-based research is that researchers can leverage
the advanced features of phones for collecting auxiliary data that augments

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 23

data from survey responses. However, our findings point to the trade-off of the
enhanced data collection opportunities of smartphones and the coverage and
nonresponse errors that such devices may introduce. Finding ways to reduce
selection errors in smartphone surveys is a critical challenge moving forward.

REFERENCES
American Association for Public Opinion Research (AAPOR) (2015), “Standard Definitions: Final
Dispositions of Case Codes and Outcome Rates for Surveys,” available at http://www.aapor.org/
AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf, last accessed
February 9, 2018.
Antoun, C., M. P. Couper, and F. G. Conrad (2017a), “Effects of Mobile versus PC Web on
Survey Response Quality: A Crossover Experiment in a Probability Web Panel,” Public
Opinion Quarterly, 81, 280–306.
Antoun, C., J. Katz, J. Argueta, and L. Wang (2017b), “Design Heuristics for Effective
Smartphone Questionnaires,” Social Science Computer Review, DOI: 0894439317727072.
Biemer, P. P., and L. E. Lyberg (2003), Introduction to Survey Quality, New York: John Wiley &
Sons.
Buskirk, T. D., and C. Andrus (2014), “Making Mobile Browser Surveys Smarter: Results from a
Randomized Experiment Comparing Online Surveys Completed via Computer or Smartphone,”
Field Methods, 26, 322–342.
Callegaro, M. (2010), “Do You Know Which Device Your Respondent Has Used to Take Your
Online Survey? Using Paradata to Collect Information on Device Type,” Survey Practice, 3,
1–17.
Couper, M. P., C. Antoun, and A. Mavletova (2017), “Mobile Web Surveys: A Total Survey
Error Perspective,” in Total Survey Error in Practice, eds. P. Biemer, E. de Leeuw, S. Eckman,
B. Edwards, F. Kreuter, L. E. Lyberg, N. C. Tucker, and B. T. West, pp. 133–154, New York:
Wiley.
De Bruijne, M., and A. Wijnant (2013a), “Comparing Survey Results Obtained via Mobile
Devices and Computers: An Experiment with a Mobile Web Survey on a Heterogeneous Group
of Mobile Devices versus a Computer-Assisted Web Survey,” Social Science Computer Review,
31, 482–504.
———— (2013b), “Can Mobile Web Surveys Be Taken on Computers? A Discussion on a Multi-
Device Survey Design,” Survey Practice, 6, 1–8.
———— (2014), “Improving Response Rates and Questionnaire Design for Mobile Web Surveys,”
Public Opinion Quarterly, 78, 951–962.
Fecso, R., and B. Pafford (1988), “Response Errors in Establishment Surveys with an Example
from an Agribusiness Survey,” Proceedings of the Section on Survey Research Methods,
American Statistical Association, pp. 315–320.
Fernee, H., and N. Sonck (2013), “Is Everyone Able to Use a Smartphone in Survey Research?,”
Survey Practice, 6, 1–7.
Fuchs, M., and B. Busse (2009), “The Coverage Bias of Mobile Web Surveys across European
Countries,” International Journal of Internet Science, 4, 21–33.
Groves, R. M. (1989), Survey Errors and Survey Costs, New York: John Wiley & Sons.
Hu, S., J. Dayton, N. Freedner-Maguire, and P. DuBray (2014), “Using Smartphones to Collect
Information about Health Behaviors: A Formative and Process Evaluation,” paper presented at
18th Annual Federal Computer Assisted Survey Information Collection, Washington, DC.,
available at http://slideplayer.com/slide/4691506/, last accessed February 9, 2018.
Keusch, F., and T. Yan (2017), “Web versus Mobile Web: An Experimental Study of Device
Effects and Self-Selection Effects,” Social Science Computer Review, 35, 751–769.
Lee, S. (2006), “An Evaluation of Nonresponse and Coverage Errors in a Prerecruited Probability
Web Panel Survey,” Social Science Computer Review, 24, 460–475.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 24 Antoun et al.

Link, M. W., J. Murphy, M. F. Schober, T. D. Buskirk, J. Hunter Childs, and C. Langer Tesfaye
(2014), “Mobile Technologies for Conducting, Augmenting and Potentially Replacing Surveys:
Executive Summary of the AAPOR Task Force on Emerging Technologies in Public Opinion
Research,” Public Opinion Quarterly, 78, 779–787.
LISS (2015), “Representativeness of the LISS Panel 2015,” available at https://www.lissdata.nl/
about-panel/composition-and-response, last accessed February 9, 2018.
Lugtig, P., and V. Toepoel (2015), “The Use of PCs, Smartphones, and Tablets in a Probability-
Based Panel Survey Effects on Survey Measurement Error,” Social Science Computer Review,
34, 78–94.
Mavletova, A. (2013), “Data Quality in PC and Mobile Web Surveys,” Social Science Computer
Review, 31, 725–743.
Mavletova, A., and M. P. Couper (2013), “Sensitive Topics in PC Web and Mobile Web Surveys:
Is There a Difference?,” Survey Research Method, 7, 191–205.
McNemar, Q. (1947), “Note on the Sampling Error of the Difference between Correlated
Proportions or Percentages,” Psychometrika, 12, 153–157.
Metzler, A., and M. Fuchs (2014), “Coverage Error in Mobile Web Surveys Across European
Countries,” paper presented at the Internet Survey Methodology Workshop, Bozen-Bolzano,
Italy.
Peterson, G., J. Griffin, J. LaFrance, and J. Li (2017), “Smartphone Participation in Web Surveys,”
in Total Survey Error in Practice, eds. P. Biemer, E. de Leeuw, S. Eckman, B. Edwards,
F. Kreuter, L. E. Lyberg, N. C. Tucker, and B. T. West, pp. 203–233, New York: Wiley.
Poushter, J. (2017), “Smartphones are Common in Advanced Economies, but Digital Divides
Remain,” Pew Research Center, available at http://www.pewresearch.org/fact-tank/2017/04/21/
smartphones-are-common-in-advanced-economies-but-digital-divides-remain/, last accessed
February 9, 2018.
Roe, D., Y. Zhang, and M. Keating (2013), “Decisions, Observations, and Considerations for
Developing a Mobile Survey App and Panel,” in Social Media, Sociality, and Survey Research,
eds. C. A. Hill, E. Dean and J. Murphy. pp. 149–178, New York: Wiley.
Revilla, M., D. Toninelli, C. Ochoa, and G. Loewe (2015), “Who Has Access to Mobile Devices
in an Online Opt-in Panel? An Analysis of Potential Respondents for Mobile Surveys,” in
Mobile Research Methods, ed. D. Toninelli, R. Pinter, and P. de Pedraza. pp. 119–139, London:
Ubiquity Press.
———— (2016), “Do Online Access Panels Need to Adapt Surveys for Mobile Devices?,” Internet
Research, 26, 1209–1227.
Schouten, B., J. van den Brakel, B. Buelens, J. van der Laan, and T. Klausch (2013),
“Disentangling Mode-Specific Selection and Measurement Bias in Social Surveys,” Social
Science Research, 42, 1555–1570.
Scherpenzeel, A. (2011), “Data Collection in a Probability-Based Internet Panel: How the LISS
Panel Was Built and How It Can Be Used,” Bulletin of Sociological Methodology, 109, 56–61.
———— (2017), “Mixing Online Panel Data Collection with Innovative Methods,” in Methodische
Probleme von Mixed-Mode-Ans€ atzen in der Umfrageforschung, eds. S. Eifler and F. Faulbaum,
pp. 27–49, Wiesbaden: Springer.
Smith, A. (2015), “US Smartphone Use in 2015,” Pew Internet & American Life Project, available
at http://assets.pewresearch.org/wp-content/uploads/sites/14/2015/03/PI_Smartphones_0401151.
pdf, last accessed February 9, 2018.
Toninelli, D., and M. Revilla (2016), “Smartphones Versus PCs: Does the Device Affect the Web
Survey Experience and the Measurement Error for Sensitive Topics? A Replication of
Mavletova & Couper’s 2013 Experiment,” Survey Research Methods, 10, 153–169.
Wells, T., J. T. Bailey, and M. W. Link (2014), “Comparison of Smartphone and Online Computer
Survey Administration,” Social Science Computer Review, 32, 238–255.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018
 Estimating Errors in a Smartphone-Based Survey 25

Appendix
For coverage error estimates, standard errors are estimated as follows:
  n  n qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
   
ref covered
se pPC
covered  pPC
ref ¼ var pPC covered þ var pPC notcovered ;
nref
(1A)

where ncovered panelists own a smartphone, and pPC notcovered is the estimated
proportion from PC web for those who do not own smartphones.
Similarly, for nonresponse error estimates, standard errors are estimated as
follows:
 
se pPC  pPC
respond covered
 rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
   
ncovered  nrespond
¼ var pPC respond þ var p PC
notrespond ;
ncovered
(2A)

where pPC
notrespond is the estimated proportion from PC web for those who own
smartphones but did not respond to the smartphone survey.

Appendix Table. Estimated 95% Confidence Intervals for Different Error

Components using the Closed-form Formula Approach and Bootstrap Approach

Survey variable Approach Confidence Confidence Confidence Confidence

for interval for interval for interval for interval for
estimating coverage nonresponse measurement total
SE deviation deviation deviation deviation

Binge drank in formula 2.9–6.4 3.5–0.6 0.6–6.4 2.3–9.9

past bootstrap 2.9–6.3 3.5–0.7 0.5–6.8 2.6–10.1
30 days (%)
Immigrants formula 1.0–2.5 4.6–0.1 4.2–2.2 6.4–0.2
make bootstrap 1.3–2.5 4.6–0.1 4.4–2.2 6.6–0.3
country a
worse
place (%)

NOTE.— Bolded intervals do not contain zero.

Downloaded from https://academic.oup.com/jssam/advance-article-abstract/doi/10.1093/jssam/smy002/4924615

by Tufts University user
on 16 March 2018