See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/318018670

Modeling the Weekly Data Collection Efficiency of Face-

to-Face Surveys: Six Rounds of the European Social
Survey

Article  in  Journal of Survey Statistics and Methodology · June 2017

DOI: 10.1093/jssam/smw034

CITATIONS READS

7 55

2 authors:

Geert Loosveldt Caroline Vandenplas

KU Leuven KU Leuven
117 PUBLICATIONS   2,448 CITATIONS    41 PUBLICATIONS   229 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Mixed mode View project

LIVES - Surveying vulnerabilities across the life course View project

All content following this page was uploaded by Geert Loosveldt on 17 August 2017.

The user has requested enhancement of the downloaded file.

Journal of Survey Statistics and Methodology (2016) 00, 1–21

MODELING THE WEEKLY DATA COLLECTION

EFFICIENCY OF FACE-TO-FACE SURVEYS: SIX
ROUNDS OF THE EUROPEAN SOCIAL SURVEY

CAROLINE VANDENPLAS 5
GEERT LOOSVELDT

Understanding the dynamics of the data collection efficiency is key to

developing strategies to increase response rates. In this article, we model 10
data collection efficiency, specifically the number of completed inter-
views, number of contacts, ratio of completed interviews and contact at-
tempts, and ratio of completed interviews and refusals per time unit in
the field. We apply this concept to the first six rounds of the European
Social Survey. The time course of efficiency for the surveys—148 15
country-round combinations—is analyzed using a multilevel repeated
measurement model in which the weekly data collection efficiency mea-
sures are repeated measurements nested in the surveys. The results show
that the data collection has four main characteristics over time: the initial
efficiency, the initial increase in efficiency (speed), the initial decrease in 20
speed, and the start of a tail when the efficiency levels off. The values of
these characteristics seem to be linked with the length of the data collec-
tion period. Moreover, across all surveys, the analysis suggests that al-
though most interviews are completed and most contacts established in
the first weeks, the fieldwork becomes more productive at the end, with 25
fewer but more successful contact attempts. The covariance parameters,
however, show large differences between the surveys in terms of field-
work dynamics. The results from a model with explanatory variables
show that more interviewers, a good survey climate, and a nonindividual
frame type lead to a higher efficiency, while higher percentages of re- 30
fusal conversion slow down its decrease.
KEYWORDS: Data collection efficiency; Fieldwork process;
Response-enhancement techniques; Response rates.

The authors thank the associate editor and the reviewers for their helpful comments.
Address correspondence to Caroline Vandenplas, Centre for Sociological Research Parkstraat 45 -
box 3601 3000 Leuven, Belgium; E-mail: caroline.vandenplas@kuleuven.be
doi: 10.1093/jssam/smw034
C The Author 2017. Published by Oxford University Press on behalf of the American Association for Public Opinion Research.
V
All rights reserved. For Permissions, please email: journals.permissions@oup.com
2 Vandenplas et al.

1. INTRODUCTION
Over the last twenty years, two phenomena have been observed by survey re-
searchers. First, response rates have been decreasing, as shown in the much cited
paper by de Leeuw and de Heer (2002) and supported by more recent publica-
tions (Dixon and Tucker 2010; Bethlehem, Cobben, and Schouten 2011; Brick 5
and Williams 2013; Kreuter 2013). Lower response rates are undesirable because
they increase the potential for nonresponse bias and reduce the precision of sur-
vey estimates, both of which lead to less accurate survey outcomes. Second, in an
attempt to control the decline in response rates, fieldwork efforts have been ex-
panded and fieldwork strategies have been improved (Koch, Fitzgerald, Stoop, 10
Widdop, and Halbherr 2012). These efforts are aimed at increasing the number of
completed interviews—and hence the response rate—as well as balancing the re-
sponding sample so that a lower response rate does not necessarily produce
higher nonresponse bias (for example, Groves and Couper 1998; Peytchev,
Baxter, and Carley-Baxter 2009; Stoop, Billiet, Koch, and Fitzgerald 2010). 15
To develop an efficient fieldwork strategy, it is important to understand how
the fieldwork yield progresses during the data collection period.
In the current article, we model data collection efficiency per time unit in-
stead of after the data collection period is over. Such an analysis should offer a
deeper understanding of the mechanisms involved: when is the data collection 20
efficiency at its highest, when is it at its lowest, when is it increasing or de-
creasing, and what can we expect in the next weeks, days, hours (depending
on the time unit chosen)? Analyzing the efficiency per time unit should allow
for a more efficient management of resources during the data collection period,
with a view to improving the response rate. A sign of decrease in efficiency 25
could generate a reaction: a switch of mode, a retraining of the interviewers, an
increase in the offered incentive. Modeling the data collection efficiency per
time unit might also help to explain some nonintuitive findings, such as the
negative association between response rates on the one hand, and the data col-
lection duration or the percentage of refusal conversion attempts on the other 30
hand (Vandenplas, Loosveldt, and Beullens 2015).
Modeling of the data collection efficiency also allows us to assess the impact
of different survey characteristics—such as the sampling frame, number of inter-
viewers, mode of contact, mode of data collection, intensity of refusal conver-
sion effort, increase in incentive, or survey climate—on the time course of the 35
fieldwork. In the next section, we introduce and define the key concepts that we
use to model the progress of the data collection: data collection efficiency.

2. DATA COLLECTION EFFICIENCY

To study the data collection yield per time unit, we need to consider the field-
work as a machine, designed to produce completed interviews (or returned 40
Modeling Data Collection Efficiency 3

questionnaires), contacts, refusals, etc. Although data collection efficiency

could be defined in numerous ways, we concentrate on four specifications.
These relate to the potential of the fieldwork per time unit: The weekly number
of completed interviews, the weekly number of contacts (independent of the
outcome), the weekly ratio of the number of completed interviews to the num- 5
ber of contact attempts (productivity), and the weekly ratio of the number of
completed interviews to the number of refusals (performance).
The time unit can also take different forms: hours, days, or weeks. The most
appropriate form depends on the conditions of the fieldwork (such as the con-
tact and data collection mode): the measurement points should be frequent 10
enough to analyze the dynamics but spaced enough to be able to gather the
necessary information and to avoid reporting unimportant fluctuations. The
definition of the time unit is also linked to the mode of contact and data collec-
tion: web surveys have almost continuous information on who is logged in or
has completed the surveys, while face-to-face surveys may have a longer delay 15
to have summary information on the progress of the fieldwork, relying on the
interviewer timely reports (although technology is progressing here). The
European Social Survey, on which we based our analysis, is a face-to-face sur-
vey. We will measure time in number of weeks. From now on, we will concen-
trate on face-to-face surveys and more specifically on the European Social 20
Survey, although all the concepts and analyses presented in this paper could be
adapted to surveys with other data collection designs and modes.
As we want to compare surveys with different sample sizes, we standardize
the number of completed interviews, contacts, and refusals to 100 sampling
units. Thus, we replace the number of completed interviews, contacts, and re- 25
fusals with the corresponding standardized numbers, dividing the original val-
ues by the sample size and multiplying the outcome by 100.
The time course of the data collection efficiency can be presented as a graph,
with the weeks of data collection on the x-axis and the data collection effi-
ciency on the y-axis. 30
In the next section, we describe the data used in our analysis. Subsequently,
we specify the model for the weekly data collection efficiency and discuss the
factors that can have an effect on this.

3. DATA
To explore the data collection efficiency introduced in the previous section and 35
to model its time course, we use data from the European Social Survey (ESS).1
For each round of this academically driven survey, run biennially since 2002, a
random sample is selected in each participating country. Data is then collected

1. More information about the ESS and the specifications for its implementation is available on
the website http://www.europeansocialsurvey.org/
4 Vandenplas et al.

through standardized computer-assisted or paper personal interviews lasting

about 60 minutes.
From the first six rounds, a total of 148 surveys—round-country combina-
tions—can be analyzed. Although each survey follows the ESS guidelines,
they each have their own particularities, which could influence the time course 5
of data collection efficiency. These properties include, among others, the sam-
pling procedure, the contact procedure, whether or not a refusal conversion
procedure was applied, and the duration of the fieldwork. Appendix 1 contains
tables showing the different fieldwork characteristics of the various surveys, as
well as their fieldwork outcomes in terms of response rates, contact rates, and 10
percentages of ineligibles.

4. MODELING THE WEEKLY DATA COLLECTION

EFFICIENCY OVER THE DATA COLLECTION PERIOD
We analyze the weekly efficiency of the data collection and its temporal dynam-
ics for the 148 ESS surveys in two steps. Our aim in the first step is to describe 15
a general time course of efficiency for the ESS over all surveys. We therefore
use a repeated-measurement multilevel model with the 148 surveys as macro
units and the weekly efficiency as repeated measurements at the lowest level
(Model 1). Each survey contributes to the estimation of the model parameters to
a different extent, depending on the duration of their data collection. Indeed, 20
some surveys lasted only two weeks and others almost a year. From this analy-
sis, we obtain a description of the shape of the weekly data collection efficiency
in ESS (determined by the fixed effects) from which each survey is going to de-
viate (random effects). Some surveys have a higher efficiency in the first week,
which drops sooner than the general ESS shape, while others have lower effi- 25
ciency at the start, which is maintained higher for a longer period. Given the
large range of the data collection period, we repeat the overall analysis, group-
ing the surveys by their duration into four groups: very short (fewer than 9
weeks), short (between 9 and 18 weeks), long (between 18 and 27 weeks), and
very long (more than 27 weeks) surveys. We comment on the summary shapes 30
of the data collection efficiency of the four different types of surveys.
In the next step, we expand this model by adding some survey design char-
acteristics that can influence the weekly efficiency (Model 2). In doing so, we
partly explain why there are such large differences in the way the data collec-
tion evolves in term of efficiency for the different considered ESS surveys. 35

4.1 The Basic Model

First, the shape of the weekly efficiency over the data collection period is ex-
amined. For each survey, we calculate the data collection duration, the sample
Modeling Data Collection Efficiency 5

size, the weekly number of interviews completed, the weekly number of con-
tacts, the weekly ratio of the number of completed interviews to the number of
contact attempts (productivity), and the weekly ratio of the number of com-
pleted interviews to the number of refusals (performance). These statistics are
based on data in the contact forms (the short questionnaire that interviewers 5
are asked to complete for each contact attempt). Among other things, the form
records the date, time, and outcome of the contact attempt (contact, noncontact,
refusal, ineligible, appointment, interview, etc.).
The curves, based on the weekly data collection efficiency over the data col-
lection period of all the surveys in round 6, are presented in the following fig- 10
ures. Figure 1 displays the number of weekly completed interviews, figure 2
the weekly number of contacts, figure 3 the weekly productivity, and figure 4
the weekly performance, each week for all countries in round 6.
The information in the figures hints at two general shapes of the data collec-
tion efficiency. These two shapes are illustrated in figure 5. The first shape 15
(bold line) is a monotonic decline in efficiency that levels off toward the end of
the data collection period (the specific week of that leveling off is survey de-
pendent). The second shape (dashed line) is an increase in efficiency followed
by a monotonic decline that levels off. These shapes have four distinguishing
features: the initial efficiency, the initial speed (amplitude of the decrease or in- 20
crease of the efficiency), the initial acceleration (how fast the decrease becomes
larger or the increase becomes smaller), and the start of the tail, which is

BE BG CH CY CZ DE
800
400
0
DK EE ES FI FR GB
Number of completed interviews

800
400
0
HU IE IL IT LT NL
800
400
0
NO PL PT RU SE SI
800
400
0
SK UA XK
800
400
0

0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30
Fieldwork weeks

Figure 1. Evolution of the Weekly Number of Completed Interviews.

6 Vandenplas et al.

BE BG CH CY CZ DE

2,000
1,000
0
DK EE ES FI FR GB

2,000
1,000
Number of contacts

0
HU IE IL IT LT NL

2,000
1,000
0
NO PL PT RU SE SI

2,000
1,000
0
SK UA XK

2,000
1,000
0

0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30
Fieldwork weeks

Figure 2. Evolution of the Weekly Number of Contacts.

BE BG CH CY CZ DE
Number of completed interviews/number of contact attempts

0.8
0.4
0.0
DK EE ES FI FR GB

0.8
0.4
0.0
HU IE IL IT LT NL

0.8
0.4
0.0
NO PL PT RU SE SI

0.8
0.4
0.0
SK UA XK

0.8
0.4
0.0

0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30
Fieldwork weeks

Figure 3. Evolution of the Productivity.

Modeling Data Collection Efficiency 7

BE BG CH CY CZ DE
Number of completed interviews/number of refusals

20
10
0
DK EE ES FI FR GB
20
10
0
HU IE IL IT LT NL
20
10
0
NO PL RU SE SI SK
20
10
0
UA XK
20
10
0

0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30 0 10 20 30

Fieldwork weeks

Figure 4. Evolution of the Performance.

45
40
35
30
25
Efficiency

20
15
10
5
0

Weeks

Figure 5. The Theoretical Shapes of the Weekly Data Collection Efficiency.

defined as the point when the final decrease levels off. These shapes also coin-
cide with the general experience of surveys in ESS. A large portion of the in-
terviews are completed in the first five to seven weeks (or sooner) in most
surveys, followed by a less productive period that can last almost a year to
reach the minimal required responding sample size. 5
Because we distinguish four important features for the weekly data collec-
tion efficiency, a cubic formulation is needed in the multilevel model. The
model studied is as follows:
8 Vandenplas et al.

Pðs; wÞ ¼ b0 þ b1 w þ b2 w2 þ b3 w3 þ s;w ; (Model 1)

b0 ¼ c00 þ u0s ;

b1 ¼ c10 þ u1s ;

b2 ¼ c20 þ u2s ;
b3 ¼ c30 ;

where w runs from 0 to 32 (with 0 representing week 1),2 s stands for the surveys,
and the residual term is normally distributed, s;w  N ð0; r2 Þ: The weeks are
counted from the first contact attempt of the specific surveys. We do not consider
the data collection period for each sampling unit, which would be defined as the
number of weeks since the address has been released. This decision is supported 5
by our aim to describe the data collection as a whole and not the data collection
for the particular sampling units. Spreading the release of the addresses over time
is, however, a fieldwork strategy that may influence the general shape of the
weekly efficiency. The random effects u0s ; u1s; and u2s ; are the survey-specific
intercept, linear, and quadratic parameters that express the survey-specific devia- 10
tions from the overall evolution. The random effects are from multivariate normal
distributions with zero expectations. As we are mainly interested in the general
shape of the weekly efficiency and deviations during the first weeks, we consider
the cubic term as fixed effect (no random part). Adding a random cubic effect
was almost never significant and, for some models, 2 did2 not lead to convergence.
3 15
r0 r0;1 r0;2
6 7
The level 2 covariance matrix is given by 6 2 7
4 r0;1 r1 r1;2 5. In total,
r0;2 r1;2 r22
2,580 measurements were entered in the model.
The variances of the survey-specific parameters (intercept, linear regression 20
parameter, and quadratic regression parameter) are displayed on the diagonal
of the matrix. These parameters are useful to test the differences between sur-
veys in the characteristics of the data collection efficiency evolution. The co-
variances (off-diagonal elements of the matrix) are informative to assess the
relationship between different characteristics of the data collection efficiency 25
evolution (e.g., the initial efficiency and speed).

4.1.1 Distribution shape of the weekly efficiency over the data collection
period—the general ESS curve.
In table 1, we show the model parameters’ estimates for the time course of
data collection efficiency. This is the weekly number of completed interviews, 30

2. For the model to converge, we limit the analyses to 32 weeks. Only Ireland in round 6 (46
weeks) and the Netherlands in round 4 (39 weeks) exceeded 32 weeks of fieldwork.
Modeling Data Collection Efficiency 9

the weekly number of contacts, the productivity (the ratio of the number of
completed interviews to the number of contact attempts), and the performance
(the ratio of the number of completed interviews to the number of refusals).
The main result is that the fixed effects of the multilevel model support the
hypothesized overall distribution of the data collection efficiency in figure 5. 5
The overall ESS shape of the weekly efficiency can be described as follows.
Looking at the intercept values: on average over all the surveys, in the first
week of the fieldwork 4.83 interviews out of 100 are completed, 13.77 contacts
with sampling units are established (possibly the same sample unit more than
once), 22.4 percent of the contact attempts are converted into completed inter- 10
views, and 2.59 more completed interviews are achieved than refusals.
Moreover, the positive sign for the linear terms points to an overall increase in
efficiency (positive speed) at the beginning of the fieldwork (dashed line
shape). The negative quadratic coefficients indicate a leveling off of the in-
crease in efficiency after a few weeks. The significant positive regression cubic 15
coefficients for all interpretations of the data collection efficiency aspects sup-
port the necessity for a tail in the shape.
If we consider the efficiency as being the number of completed interviews
in a week, we can also conclude from the coefficient in the first column of table
1 that, averaged over all the surveys, 34 percent of the total sample become 20
completed interviews after six weeks, about 50 percent after 11 weeks, and 57
percent after 16 weeks, as shown in figure 6. To obtain this result, we calculate
the weekly number of completed interviews for each of the first six weeks and
sum them. The overall weekly number of completed interviews in a specific
week is given by 4.83 þ 0.23 w -0.05 w2 þ 1E-3 w3. Interestingly, the standard- 25
ized number of completed interviews is closely related to the response rate:
without ineligible cases in the sample, both values would be the same.

Table 1. Fixed Effects for the Multilevel Model Describing the Weekly Efficiency
of Data Collection (Model 1)

Effect Completed Contacts Productivity Performance

interviews (SE) (SE) (SE) (SE)

c00 – intercept 4.83 (0.35)*** 13.77 (0.80)*** 0.224 (0.012) *** 2.59 (0.17)***
c10 – linear term 0.23 (0.07)** 0.42 (0.18)* 0.014 (.003)*** 0.18 (0.08)*
c20 – quadratic 0.05 (4E-3)*** 0.13 (0.01)*** 0.001 (2E-4)*** 0.03 (0.01)***
term
c30 – cubic term 1E-3 (1E-4)*** 3E-3 (3E-4)*** 1E-3 (6E-6)*** 1E-3 (2E-4)***

*p < 0.05,
**p < 0.01,
***p < 0.001.
10 Vandenplas et al.

6
Chart Title
6
5
Standardized number of completed

5
4
4
4
interviews

3
3
3
2 34%
2
2
1 16%
1
1 8%
0
1 2 3 4 5 6 7 8 9 10 11 12
0
11 2 2 3 34 54 6 5 7Series2
68 97 Series1
10 8 11 912 13
10 14 11 15 16
12
Weeks

Figure 6. The Time Course of the Weekly Number of Completed Interviews.

16
Standardized number of contacts

14

12

10

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Weeks

Figure 7. The Time Course of the Weekly Number of Established Contacts.

Figure 7 illustrates the general evolution of the weekly number of contacts

for the ESS. The area under the curve in this figure cannot be interpreted as
representing the numbers of the full sample that have been contacted because
an individual (or household) can be contacted more than once. An alternative
specification for the efficiency of the data collection, which would be closely 5
related to the contact rate, would be the weekly number of newly contacted in-
dividuals, but it is not considered here. Overall, around 14 contacts for a sam-
ple of 100 units are established each week in the first five weeks of the
fieldwork. Subsequently, the number of established contacts rapidly declines,
showing once more that the core of the data collection production occurs dur- 10
ing the first weeks of the field period. However, one should bear in mind that
Modeling Data Collection Efficiency 11

50
45
Productivity (%) 40
35
30
25
20
15
10
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Weeks

Figure 8. Evolution of the Weekly Productivity.

the number of contacts established each week will always decline because dur-
ing the fieldwork some cases are classified as final and are not visited anymore
(interview completed, individual recorded as ineligible or a hard refusal who
should not be re-approached, etc.). Finally, in general at around 14 weeks, the
decline starts to level off to form the tail. 5
Figure 8 shows the time course of the weekly productivity (number of com-
pleted interviews/number of contact attempts). Surprisingly, although the
model supports the hypothesized shapes, the productivity appears to increase
over the data collection period. The increase in productivity initially levels off
at around five weeks and then starts to rise again to reach a relatively steep 10
level of growth at around eight weeks. This means that approaching the end of
the fieldwork contact attempts are more successful. This probably reflects the
evolving knowledge of interviewers, together with the organization of the sur-
vey during the data collection period. This allows for a more targeted ap-
proach, such as prioritizing appointments made at a previous contact or by 15
telephone, avoiding recontacting hard refusals, or redistributing unfinished
cases to the better, more experienced interviewers toward the end of the field-
work. It also may be that at this point in time the survey managers choose to re-
work the cases that are most likely to become completed interviews. However,
these are only hypotheses and further research should explore the possible 20
explanations.
Figure 9 illustrates the evolution of the weekly performance (number of
completed interviews/number of refusals) over the field period. Across all ESS
surveys, the performance grows until week 6—where it reaches almost three
times more completed interviews than refusals. It then reaches its minimum of 25
2.6 more completed interviews than refusals in the 15th week. However, it
should be noticed that the scale, and hence the difference, is quite narrow,
12 Vandenplas et al.

2.9

2.8
Perfomance

2.7

2.6

2.5

2.4

2.3
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Weeks

Figure 9. Evolution of the Weekly Performance.

ranging from 2.5 to a maximum of 3. The increase after week 15 is most proba-
bly an artifact of the model and its cubic term, as well as a consequence of the
very few active cases left at this point, when many countries have nearly com-
pleted the fieldwork so that most cases must have already been finalized.
Having learned a lot from the overall ESS curves through studying the fixed 5
parameters, we can now turn to the covariance parameters (see appendix 2).
The first important observation is that all variances—of the intercepts, the lin-
ear, and the quadratic terms—for all efficiency specifications are significantly
(at a 0.001 level) different from zero. These results hint at different approaches
to the allocation of the fieldwork efforts and their success in different countries 10
and different rounds of the ESS. Some surveys start off with many completed
interviews and contacts, and hence high productivity and performance,
whereas others have a lower yield in the first week. Moreover, the residual var-
iance of the model with the random slopes decreases by 36 percent compared
with the residual variance of the only random-intercept model (8.94 percent). 15
The covariance terms (see appendix 2) have the same sign across the differ-
ent interpretations of the data collection efficiency and are almost all signifi-
cantly different from zero (exceptions being the covariance between the
intercept and linear as quadratic coefficients for the productivity and
performance). 20
Examining the scenario represented by the dashed line in figure 5, the nega-
tive covariance between the intercept and the linear term implies that the higher
the intercept (which represents the efficiency in the first week), the lower the
increase in efficiency is at the start of the fieldwork. This can be interpreted as
follows. The better the start of the data collection period in terms of completed 25
interviews or contacts established, the harder it is to improve the yield in the
subsequent weeks. This is also true for the productivity and the performance,
but the covariance is much smaller and not significant. Moreover, the positive
Modeling Data Collection Efficiency 13

covariance between the intercept and the quadratic coefficient indicates that
the higher the intercept, the faster the efficiency increase levels off. This means
that the higher the number of completed interviews, the number of contacts,
the efficiency, or the performance at the outset, the more difficult it is to main-
tain a high efficiency. Last, the positive covariance between the linear and qua- 5
dratic coefficient entails that the steeper the slope is at the beginning of the
data collection period, the faster the efficiency increase levels off: the larger
the increase in the number of completed interviews or contacts, the productiv-
ity, and the performance, the faster these increases level off.

4.1.2 Distribution shape of the weekly efficiency over the data collection 10
period—ESS curves for different survey types.
Although the model for the ESS general curve already brought us some in-
sight about the data collection process of these face-to-face surveys, modeling
so many diverse surveys together may not be optimal. The surveys differ in
many ways but one of the most relevant aspects here is the duration of the data 15
collection period. Therefore, we report in table 2 the parameter estimates of
Model 1 when the model is estimated for surveys with very short (fewer than 9
weeks), short (between 9 and 18 weeks), long (between 18 and 27 weeks), and
very long (more than 27 weeks) data collection periods.
For the weekly number of completed interviews and contacts established for 20
all types of surveys, a shape similar to the general ESS curve appears (see the
dashed line in figure 5). The only exception is that for very long surveys the
number of contacts seems to stay constant over the data collection period. The
performance among very short surveys and the productivity among very short,
short, and long surveys seem to not vary much during the data collection period 25
as the linear, quadratic, and cubic coefficients are not significantly different from
0 at the 0.05 level. Even though the largest intercepts are to be found among
short (and not very short surveys), there is a general trend indicating that longer
surveys are also less efficient in the first week. Similarly, very short surveys
have a large increase in efficiency in the first week, while short, long, and very 30
longer surveys have a decline in this increase in efficiency in the first week.
In summary, surveys with shorter data collection periods have a narrow
high peak in efficiency at the very beginning of the data collection period and
surveys with longer data collection periods have an extended lower “peak” or
bulge in efficiency. 35

4.2. The Elaborated Model

Having examined the general shape of the time course of data collection effi-
ciency, we examine the fieldwork particularities that can influence this time
course, focusing in particular on the weekly number of completed interviews.
In general we assume three possible effects: (1) effects on initial efficiency, (2) 40
14 Vandenplas et al.

Table 2. Fixed Effects for the Multilevel Model Describing of the Weekly Data
Collection Efficiency (Model 1), by Groups of Surveys with Different Durations

Effect Completed Contacts Productivity Performance

interviews (SE) (SE) (SE) (SE)

Very short (15 surveys)

c00 – intercept 3.91 (1.86) 7.74 (4.24) 0.37 (0.04)*** 2.81 (1.23)*
c10 – linear term 10.29 (2.10)*** 22.55 (4.13)*** 0.01 (0.05) 0.03 (3.11)
c20 – quadratic term 2.78 (075)** 6.83 (1.37)*** 0.01 (0.02) 0.03 (1.04)
c30 – cubic term 0.17 (0.08)* 0.50 (0.13)*** 1E-3 (2E-3) 0.05 (0.09)
Short (57 surveys)
c00 – intercept 3.97 (0.56)*** 11.40 (1.39)*** 0.27 (0.02)*** 3.42 (0.46)***
c10 – linear term 1.96 (0.22) 4.25 (0.50)*** 0.01 (0.01) 0.09 (0.17)
c20 – quadratic term 0.36 (0.03)*** 0.84 (0.06)*** 2E-3 (1E-3)* 0.03 (0.02)
c30 – cubic term 0.01 (0.00)*** 0.04 (2E-3)*** 2E-4 (4E-5)*** 2E-3 (1E-3)
Long (51 surveys)
c00 – intercept 2.34 (0.34)*** 9.35 (1.06)*** 0.16 (0.02)*** 2.40 (0.47)***
c10 – linear term 0.71 (0.08)*** 1.72 (0.23)*** 0.02 (4E-3)*** 2E-3 (0.14)
c20 – quadratic term 0.09 (0.01)*** 0.23 (0.02)*** 2E-3 (4E-4)*** 5E-3 (0.01)
c30 – cubic term 0.002 (2E-4)*** 0.006 (5E-4)*** 7E-5 (1E-5)*** 3E-4 (4E-4)
Very long (11 surveys)
c00 – intercept 1.74 (0.42)** 11.44 (2.84)*** 0.12 (0.02)*** 1.66 (0.76)
c10 – linear term 0.24 (0.08)* 0.50 (0.35) 0.02 (0.01)** 0.36 (0.16)*
c20 – quadratic term 0.02 (0.00)*** 0.02 (0.02) 2E-3 (4E-4)*** 0.04 (0.01)*
c30 – cubic term 4E-4 (8E-6)*** 2E-4 (4E-4) 4E-5 (8E-6)*** 9E-4 (3E-4)***

*p < 0.05,
**p < 0.01,
***p < 0.001.

effects on the efficiency each week (the height of the shape), and (3) effects on
maintaining high efficiency during the data collection (the width of the shape).
The ideal would be for the shape of data collection efficiency curve over time
to be as high and/or as wide as possible because the result that needs to be
maximized (the final number of completed interviews, contacted units, etc.) 5
corresponds to the area under this shape. The shape of the data collection effi-
ciency curve can be affected by many things; we concentrate on three: the type
of sampling frame, fieldwork strategies, and survey climate.
First, the type of frame is expected to have an effect on the efficiency in the
first week of the data collection period. Research concerning the impact of the 10
sampling frame on nonresponse bias indicates that interviewers can play an im-
portant role in the sample selection procedure when a sample of households/
addresses is used. There seems to be a tendency to select less reluctant sample
units when interviewers are responsible for the selection (Koch, Halbherr,
Modeling Data Collection Efficiency 15

Stoop, and Kappelhof 2014). Therefore, we expect a negative effect of a sam-

pling frame of individuals on the initial data collection efficiency.
Second, fieldwork strategies can be very diverse. Since its first round in
2002, the ESS has made great efforts to collect high-quality data to ensure
cross-national and cross-cultural comparability. Much of this effort aims at ob- 5
taining a high response rate (directly linked to the standardized total number of
completed interviews) and includes contact procedure protocols as well as ad-
vice on refusal conversion procedures. To increase the chance of contact, the
ESS requires at least four contact attempts, at least one of which should take
place in the evening and one at the weekend; the contact attempts should be 10
spread over at least two weeks. The guidelines on refusal conversion proce-
dures are less strict, recommending re-issuing all soft refusals and as many
hard refusals as possible (Billiet, Koch, and Philippens 2007), as well as as-
signing a different, more experienced interviewer for refusal conversion
(European Social Survey2011; Koch et al. 2012). Interviewers classify refusals 15
as hard (will definitely/probably not cooperate in the future) or soft (may possi-
bly/will cooperate in the future). We analyze the effect of compliance with the
ESS guidelines in terms of contact attempts and refusal conversion, as well as
the effect of the percentage of refusal conversions on the weekly data collec-
tion efficiency. The contact procedure and refusal conversion strategy are 20
expected to influence the width of the curve and the leveling off of the increase
or decrease in the weekly number of completed interviews. Therefore, for each
survey a contact score and a refusal score is calculated, as described in
appendix 3. Furthermore, the percentage of conversion attempts is calculated
as the percentage of sampled people who initially refused but were subse- 25
quently re-approached.
The number of interviewers active each week is considered as a fieldwork
strategy factor that may influence the shape of the weekly data collection effi-
ciency curve in terms of its height. A standardized version of the number of in-
terviewers is used, with the weekly number of active interviewers divided by 30
the sample size and multiplied by 100.
Third, the survey climate is generally defined as the ease of contact and the
ease of obtaining cooperation, or the general propensity of the target popula-
tion to participate in surveys. It is influenced by many factors intrinsic to each
specific survey, but also by external factors such as the overall level of respon- 35
dent burden resulting from market research. Although no indicator exists for
the survey climate, it is measured here as the response rate for the first contact
attempt. It is clear that this specific measurement of the survey climate will
have a direct influence on the initial efficiency (intercept).
Clearly, this list of possible factors influencing the shape of the weekly data 40
collection efficiency is not exhaustive. Many other factors probably play a role
in the weekly efficiency of the data collection: mode of contact used (although
in ESS other mode for the first contact than face-to-face is exceptional), experi-
ence of the interviewers, incentives, geographical constraints, or specific
16 Vandenplas et al.

country legislation, to name a few. Although we strongly believe that one of

the strengths of our analyses is the large number of surveys, which permits us
to evaluate the model quantitatively, it is also a weakness. ESS surveys are
conducted in different years and different countries; it is therefore not straight-
forward to gather information about these additional survey characteristics 5
(e.g., the use of incentives) in a comparable way. We will therefore concentrate
on these factors, acknowledging that other factors could influence the weekly
efficiency.
To test our hypotheses, we elaborate on Model 1 and analyze the following
model (Model 2): 10

Pðs; wÞ ¼ b0 þ b1 w þ b2 w2 þ b3 w3 þ b4  standardized no: of interviewers

þ s;w ;

b0 ¼ c00 þ c01 % interviews completed after 1st contact attempt

þ c02 frame type þ u0 ;
b1 ¼ c10 þ u1 ;
b2 ¼ c20 þ c21 contact score þ c22 refusal score
þ c23 conversion attempts þ u2 ;
b3 ¼ c30 ;

where w runs from 0 to 32 (with 0 representing week 1), s stands for the
surveys, and s;w  N ð0; r2 Þ.
The same covariance matrix is specified as for Model 1.
The model parameters’ estimates are shown in table 3 (fixed effects) and ta-
ble 4 (covariance). Due to missing information in the contact forms, only 118
surveys and 2,173 measurements remain in the analysis. 15
The standardized number of interviewers has a highly significant effect on
the standardized weekly number of completed interviews. In general, per 100
sample units, each extra interviewer generates 2.6 additional completed inter-
views each week over all the surveys. Moreover, the survey climate and the
type of frame influence the shape of the data collection efficiency curve signifi- 20
cantly. Surveys with a better survey climate and a household or address frame
have a larger intercept and, hence, more completed interviews per 100 sam-
pling units in the first week. For 1 percent more completed interviews at the
first attempt, the standardized number of completed interviews in the first week
goes up by 0.06, while surveys with a nonindividual sampling frame produce 25
1.2 more interviews per 100 sampling units in the first week. The last result is
in line with our expectation that surveys with an individual sampling frame
will have lower efficiency at the beginning because the interviewers have
fewer "selection opportunities." Third, although the refusal score and the con-
tact score have no significant effect on the quadratic term, the percentage of 30
Modeling Data Collection Efficiency 17

Table 3. Fixed Effects for the Multilevel Model Testing the Effects of Some
Fieldwork Characteristics on the Weekly Data Collection Efficiency (Model 2)

Effect Estimate (SE)

c00 – intercept 2.28 (0.46)***

c40 – standardized no. of interviewer active in w 2.64 (0.04)***
c01 – survey climate 0.06 (0.01)***
c02 – sampling frame (not individual) 1.16 (0.28)***
c10 – linear term 0.20 (0.05)***
c20 – quadratic term 0.01 (0.00)
c21 – contact score 4E-6 2E-5)
c22 – refusal score 2E-5 (4E-5)
c23 – percentage of refusal conversion attempts 2E-4 (5E-5)***
c30 – cubic term 1E-5 (1E-4)

*p < 0.05,
**p < 0.01,
***p < 0.001.

Table 4. Coefficient Parameters

Covariance parameters Estimate (SE)

r20 – variance of intercept 5.22 (0.08)***

r21 –variance linear term 0.11 (0.03)***
r22 –variance quadratic term 2E-4 (8E-5)**
r0;1 – covariance intercept/linear 0.28 (0.12)*
r0;2 – covariance intercept/quadratic 9E-4 (6E-3)
r1;2 – covariance linear/quadratic 2E-4 (8E-5)*
rc;w –residual variance 2.11 (0.07)***

*p < 0.05,
**p < 0.01,
***p < 0.001.

refusal conversion attempts helps to maintain the efficiency at a higher level

for a longer period.
The variances are still all significantly different than zero (alpha ¼ 0.01), im-
plying that the differences between surveys are only partially explained by the
fieldwork characteristics being examined. Comparing the base model (with 5
the same 118 surveys) with the elaborated model holds the following R2 at the
different levels (see Bryk and Raudenbush 1992, p. 65),
R21 ¼ 0:64; R22 ¼ 0:65; R2b1 ¼ 0:70, and R2b2 ¼ 0.16. This yields the results
18 Vandenplas et al.

that 65 percent of the residual variance, 64 percent of the intercept variance,

70 percent of the variance of the linear regression parameter, and 16 percent
of the variance of the quadratic parameter are explained in the elaborated
model compared with the base model (Model 1). The covariance parameters
between the intercept/quadratic term and linear term remain significant, sug- 5
gesting that the number of interviews completed in the first week influences
its increase/decrease at the start of the fieldwork. In turn, the amplitude of this
increase/decrease influences how fast the efficiency levels off or how fast the
decrease becomes steeper.

5. DISCUSSION 10

The present study introduces the data collection efficiency per time unit as a
tool to analyze the dynamics of the fieldwork process. We considered four
specifications for the efficiency: the weekly number of completed interviews,
the weekly number of contacts, the weekly ratio of the number of completed
interviews to contact attempts (productivity), and the weekly ratio of the num- 15
ber of completed interviews to refusals (performance). We use a multilevel
model with repeated measurements to apply this concept to the first six rounds
of the European Social Survey. In this model, the macro-level units are defined
as the 148 survey-round combinations, and within each survey the second-
level units are the weekly measures of efficiency. 20
The main finding of our analysis is that the weekly data collection efficiency
follows the hypothesized cubic shape. The shape has four important character-
istics: The initial efficiency, the initial rate of the increase in efficiency (posi-
tive linear term in the model), a leveling off of the initial increase (negative
quadratic term in the model), and a tail (significant positive cubic term). The 25
results show that for all the specifications of the data collection efficiency, the
overall curves for the ESS increase at the start of the fieldwork. In the case of
completed interviews and contacts, the curve then reaches a maximum, subse-
quently decreases, and finally levels off to form a tail. The weekly productivity
and performance both show an increase toward the end of the fieldwork, which 30
can be hypothesized to be due to a reduction of the number of cases to be
worked and a more tailored approach to the remaining cases based on greater
knowledge about these cases. From the study of these curves, it also seems that
fieldwork reaches a turning point around five to six weeks after the start, with a
third of the sampling units being converted to completed interviews, the pro- 35
ductivity reaching its maximum and the performance its minimum. The results
clearly indicate that the model can be considered a good general description of
the data collection efficiency in a large number of face-to-face surveys.
A more detailed analysis of the ESS surveys, which were classified by the
duration of their data collection period, reveals that the surveys with shorter 40
field periods have higher efficiency at the start and a steeper increase of
Modeling Data Collection Efficiency 19

efficiency than the surveys with longer field periods, but also that the efficiency
drops sooner when the data collection period is shorter.
We further explain differences between surveys in the time course of the
weekly number of completed interviews. The main findings are that the type of
sampling frame, the survey climate, and the number of active interviewers 5
have an impact on the number of interviews each week (the height of the
curve), whereas fieldwork strategies mostly have an impact on the period dur-
ing which the number of interviews completed each week remains high (the
width of the curve). Indeed, to increase the number of completed interviews
before the start of the tail, the area under the curve needs to be amplified by in- 10
creasing the weekly efficiency or maintaining this efficiency at a high level for
a longer time. Two strategies can be exploited: (1) involving as many active in-
terviewers as possible every week, thereby implicitly increasing the number of
contact attempts, and (2) re-approaching as many refusals as possible. Both
these strategies may have an influence on the survey costs but highlight the im- 15
portance of having enough interviewer power to conduct a face-to-face survey.
Furthermore, the number of interviews completed in the first week is influ-
enced by the survey climate—as measured by the response rate after the first
contact attempt—and the type of sampling frame. The number of completed
interviews in the first weeks increases with a better survey climate and a sam- 20
pling frame that does not consist of individuals. The first finding is, partly due
to the operationalization of the survey climate, as expected. The second is in
line with previous research about the impact of sampling design on nonre-
sponse bias and the role played by the interviewer therein.
The basic and elaborated models show how useful the concept of data col- 25
lection efficiency per time unit can be in the analysis of the fieldwork dynamic.
Such analyses can assist decision-making in the design and the budget prepara-
tion for future surveys. There are, however, several limitations to this research.
The large number of surveys involved in the model is a strength but also a
drawback because specific deviations from the general hypothesized cubic 30
shape cannot easily be considered. One characteristic of the time course of
fieldwork that is not modeled in this research is a possible double-peak in data
collection efficiency (two increases in the efficiency, such as those observed in
Belgium, Germany, and Ireland during round 6 of the ESS); this might occur
when a new set of interviewers is activated or if the sample units are released 35
in two steps. The very general approach also involves a lack of precision or
knowledge regarding the fieldwork strategies applied. One example is the use
of incentives, which are difficult to include because their nature and value vary
substantially across countries and rounds.
Moreover, the analysis relies on the quality of the contact forms. Especially 40
in the first rounds, we cannot be certain that all the contact attempts were prop-
erly recorded in a timely manner. A number of surveys had to be removed
from our analysis due to missing information in the contact forms, such as
date, time, or interviewers, reducing the strength of the analysis. However,
20 Vandenplas et al.

great efforts have been made to improve the quality of this kind of paradata,
which are important in survey methodology research.
The new concepts raise new research questions, one of which is the impact
of differences in the weekly data collection efficiency on data quality. Future
research should also investigate whether the increase in efficiency and perfor- 5
mance at the end of the fieldwork reflects the increasing knowledge of inter-
viewers and better survey strategies regarding the remaining cases—leading to
more tailored fieldwork—or if it could be instead due to less strict application
of some rules intended to guarantee data quality.
Researchers and survey managers might consider weekly efficiency as a tool 10
to monitor and manage the fieldwork process and progress. A weekly compari-
son of the time course of data collection efficiency with the evolution in previ-
ous years could permit the early detection of problems that could lead to a low
final response rate, and early interventions to increase the weekly data collection
efficiency could be undertaken. Finally, the concept of data collection efficiency 15
per time unit is not limited to computer-assisted personal interviews. The tem-
poral dynamics of Internet surveys could also be analyzed in a similar way to
the approach taken here. An interesting question is whether the shape of the ef-
ficiency curve would be similar to the one found for face-to-face surveys here.

Supplementary Materials 20

Supplementary materials are available online at http://www.oxfordjournals.

org/our_journals/jssam/.

References
Bethlehem, J., F. Cobben, and B. Schouten (2011), Handbook of Nonresponse in Household
Surveys, Hoboken, NJ: John Wiley & Sons. 25
Billiet, J., A. Koch, and M. Philippens (2007), "Chapter 9: Understanding and Improving
Response Rates," in Measuring Attitudes Cross-nationally. Lessons from the European Social
Survey, eds. R. Jowell, C. Roberts, R. Fitzgerald, and G. Eva, London: Sage Publications.
Brick, J., and D. Williams (2013), “Explaining Rising Nonresponse Rates in Cross-Sectional
Surveys,” The ANNALS of the American Academy of Political and Social Science, 645, 36–59. 30
Bryk, A. S., and S. W. Raudenbush (1992), Hierarchical Linear Models: Applications and Data
Analysis Methods, Newbury Park, CA: Sage.
de Leeuw, E., and W. de Heer (2002), "Trends in Household Survey Nonresponse: A Longitudinal
and International Comparison," in Survey Nonresponse, eds. R. Groves, D. Dillman, J. Eltinge,
and R. J. Little, New York: Wiley. 35
Dixon, J., and C. Tucker (2010), "Survey Nonresponse," in Handbook of Survey Research (2nd
ed.), eds. P. Marsden and J. Wright, Bingley, UK: Emerald.
European Social Survey (2011), Round 6 Specification for Participating Countries, London:
Centre for Comparative Social Surveys, City University London.
Groves, R., and M. Couper (1998), Nonresponse in Household Interview Surveys, New York: 40
Wiley.
 Modeling Data Collection Efficiency 21

Kreuter, F. (2013), “Facing the Nonresponse Challenge,” The ANNALS of the American Academy
of Political and Social Science, 645, 23–25.
Koch, A., R. Fitzgerald, I. Stoop, S. Widdop, and V. Halbherr (2012), Field Procedures in the
European Social Survey Round 6: Enhancing Response Rates, Mannheim, Germany: European
Social Survey, GESIS. Available at http://www.europeansocialsurvey.org/docs/round6/methods/ 5
ESS6_response_enhancement_guidelines.pdf.
Koch, A., V. Halbherr, I. A. L. Stoop, and J. W. S. Kappelhof (2014), Assessing ESS Sample
Quality by using External and Internal Criteria, Mannheim, Germany: European Social Survey,
GESIS. Available at https://www.europeansocialsurvey.org/docs/round5/methods/ESS5_sam
ple_composition_assessment.pdf. 10
Peytchev, A., K. B. Baxter, and L. R. Carley-Baxter (2009), “Not all Survey Effort Is Equal:
Reduction of Nonresponse Bias and Nonresponse Error,” Public Opinion Quarterly, 73,
785–806.
Stoop, I., J. Billiet, A. Koch, and R. Fitzgerald (2010), Improving Survey Response. Lessons
Learned from the European Social Survey, Chichester, UK: John Breedveld Wiley & Sons Ltd. 15
Vandenplas, C., G. Loosveldt, and K. Beullens (2015), Better or Longer? The Evolution of
Weekly Number of Completed Interviews over the Fieldwork Period in the European Social
Survey, paper presented at the Nonresponse Workshop.

View publication stats