AIDS Behav DOI 10.

1007/s10461-012-0394-8

ORIGINAL PAPER

An Empirical Examination of Respondent Driven Sampling Design Effects Among HIV Risk Groups from Studies Conducted Around the World
Lisa G. Johnston Yea-Hung Chen Alfonso Silva-Santisteban H. Fisher Raymond

Springer Science+Business Media New York 2013

Abstract For studies using respondent driven sampling (RDS), the current practice of collecting a sample twice as large as that used in simple random sampling (SRS) (i.e. design effect of 2.00) may not be sufcient. This paper provides empirical evidence of sample-to-sample variability in design effects using data from nine studies in six countries among injecting drug users, female sex workers, men who have sex with men and male-to-female transgender (MTF) persons. We computed the design effect as the variance under RDS divided by the variance under SRS for a broad range of demographic and behavioral variables in each study. We also estimated several measures for each variable in each study that we hypothesized might be related to design effect: the number of waves needed for equilibrium, homophily, and mean network size. Design effects for all studies ranged from 1.20 to 5.90. Mean design effects among all studies ranged from 1.50 to 3.70. A particularly high design effect was found for employment status (design effect of 5.90) of MTF in Peru. This

may be explained by a bottleneckdened as the occurrence of a relatively small number of recruitment ties between two groups in the population. A design effect of two for RDS studies may not be sufcient. Since the mean design effect across all studies was 2.33, an effect slightly above 2.00 may be adequate; however, an effect closer to 3.00 or 4.00 might be more appropriate. Keywords Respondent driven sampling Design effects HIV/AIDS surveillance Hard-to-reach populations

Introduction Simple random sampling (SRS) is often unfeasible for sampling hidden and hard-to-reach populations since lists or sampling frames for these populations rarely exist. Common non-SRS methods to sample these populations include targeted sampling, cluster sampling, time-location sampling, and respondent-driven sampling (RDS) [1, 2]. Estimates produced from non-SRS data have non-SRS variances with important consequences, since appropriate variance estimates must be incorporated into data analysis, survey planning, and specic sample-size estimations. Sample sizes for non-SRS surveys can be effectively computed by taking the required SRS sample size, and multiplying by the estimated ratio of the non-SRS variance over the SRS variance. This ratio is referred to as the design effect. In studies using cluster sampling, for example, the convention is to multiply the required SRS sample size by a design effect of 2.00 [3]. However, there are no well-established recommendations for setting an appropriate design effect for newer adaptive sampling methods, such as RDS. This has resulted in ongoing discussions among statisticians and researchers about which

L. G. Johnston University of California, San Francisco, Global Health Sciences, San Francisco, CA, USA L. G. Johnston (&) Nieuwezijds Voorburgwal 64F, 1012 SC Amsterdam, The Netherlands e-mail: lsjohnston@comcast.net; lsjohnston.global@gmail.com URL: www.lisagjohnston.com Y.-H. Chen H. F. Raymond San Francisco Department of Public Health, San Francisco, CA, USA A. Silva-Santisteban Unit of Health, Sexuality and Human Development, Cayetano Heredia University School of Public Health, Lima, Peru

123

AIDS Behav

design effect is most appropriate for studies using RDS [4, 5]. RDS, rst introduced in the literature by Heckathorn [68] has since been used worldwide as an effective methodology to sample hidden and hard-to-reach populations, such as injection drug users (IDU), female sex workers (FSW) and men who have sex with men (MSM). RDS data are collected using a chain referral sampling design, whereby the sample population recruits peers through uniquely numbered coupons to monitor recruitment connections. When key assumptions are met and recruitment procedures are followed, RDS has been shown to produce unbiased estimates and reasonable condence intervals of the sampled population [9]. It is essential that studies using RDS are properly planned and implemented, given that they provide major organizations with data for key funding and programmatic decisions (e.g. UNAIDS WHO and national ministries of health). Similar to cluster sampling, the most recent recommendation for RDS studies is to set the design effect at 2.00, which is higher than the originally suggested design effect of 1.25 [5]. This recent recommendation was based on estimates from several real population RDS samples in which design effects were found to range from as high as 10.00 to less than 1.00 and simulations using 85 nonhidden and networked populations in which design effects were found to be close to 5.00 [4]. A more recent paper described design effects across 43 RDS samples of IDU in the United States and recommended a design effect closer to 4.00 [10]. Despite the growing body of literature on design effects in RDS studies, the current recommended practice in RDS is still to collect a sample twice as large as would be needed with SRS (i.e. design effect of 2.00). At the time of this recommendation, few RDS studies of realworld populations were available to evaluate the sampleto-sample design effect variability. Thus, researchers are now questioning whether a design effect of 2.00 is adequate for most RDS studies. In addition, to incorporate the adjustment impact when deriving an appropriate sample size, researchers must rely on estimates and condence bounds from previous studies to calculate an accurate design effect. More evidence is needed to ensure that studies are undertaken with sample sizes large enough to meet RDS study objectives. This paper provides real-world evidence of sampleto-sample variability in estimated design effects using data from nine studies in six countries among IDU, FSW, MSM and male-to-female transgender persons (MTF). The ndings from this paper provide updated guidance and improved parameters for calculating sample sizes to accurately measure much-needed estimates of HIV prevalence, as well as injecting and sexual risk behaviors among hidden and hard-to reach populations.

Methods The studies selected for our analysis were performed from 2008 to 2010 and include the following. Two studies of MTF, one from Lima, Peru and one from San Francisco, US; three MSM studies from Jinan, China [11], Soweto, South Africa [12] and Mauritius [13, 14]; two FSW studies, one from Kiev, Ukraine and one from Mauritius [13]; and, two IDU studies, one from Kiev, Ukraine [15] and one from Mauritius [13]. These studies are summarized in detail in Table 1. We endeavored to include studies with diverse populations and geography to maximize the value of our analysis for others in the eld. Each study conducted face-to-face interviews, followed standard RDS methodology, adhered to established RDS assumptions [16, 17] and received approval from appropriate ethical review boards. We computed design effects for a broad range of demographic and behavioral variables in each study. The variables available for all studies included education (primary, secondary, or higher), employment (yes/no), civil status (married/unmarried), and condom use (yes/no). Condom use was dened as follows by group: for IDU and MTFmost recent episode of sexual intercourse, for MSMat most recent episode of anal intercourse, and for FSWat most recent episode of sexual intercourse with a client. Other variables included history of HIV testing (yes/ no) and HIV status (positive/other). For the IDU studies, the following variables were also included: gender, shared injecting equipment at last injection (yes/no), and knowledge of HIV testing locations (yes/no). The study of Mauritian MSM included knowledge of HIV-testing location (yes/no). Mauritian FSW included receipt of free condoms via governmental and non-governmental service providers in the last 12 months (yes/no), injection drug use in the last 12 months (yes/no), and knowledge of HIVtesting locations (yes/no). We dened design effect as the variance under RDS divided by the variance under SRS. Note that the true variances are unknown, since the populations are not fully observed. We estimated each of the RDS variances by computing a 95 % RDS condence interval, dividing the widths of the interval by two, dividing again by 1.96 (the standard normal value corresponding to a central area of 95 %), and squaring the nal number. The RDS condence intervals were computed in RDSAT 6.0 (http://respondent drivensampling.org), using the multiplicity estimate [18], enhanced data smoothing and 15,000 bootstrap samples per interval estimate. We estimated each of the SRS variances by multiplying the RDS-estimated proportion by 1 minus the proportion, and dividing by the sample size. We chose to use the RDS-estimated proportions rather than sample estimates because we believed that the RDS estimates were

123

AIDS Behav Table 1 Data sources, recruitment statistics, nine RDS studies from around the world Location Peru, Lima USA, San Francisco China, Jinan South Africa, Soweto Ukraine, Kiev Ukraine, Kiev Mauritius Mauritius Mauritius
a

Population MTF MTF MSM MSM IDU FSW IDU FSW MSM

Year 2009 2010 2008 2008 2008 2009 2009 2010 2010

Na

No. of seeds 11 11 5 15 8 4 6 5 9

Mean waves 7 6 7 5.0 7.3 10.7 9.2 5.8 6.9

Max. waves 12 15 15 17 9 13 12 7 13

No. of weeks 14 20 12 30 *8 *6 9 2 6

Coupon return (%) 56.0 30.9 38.3 43.0 35.1 [33.0 83.4 61.8 44.4

450 314 495 363 400 256 505 299 362

excluding seeds

more appropriate estimates for the true proportions (given that the data were sampled using RDS). In addition to the design effects, variances, and RDS point estimates, we estimated several measures for each variable in each study that we hypothesized might be related to design effect: the number of waves needed for equilibrium, homophily, mean network size, and mean network size for the complementary group. All these measures were computed in RDSAT. Equilibrium refers to the independence of samples drawn via RDS from the selection of the seeds (non-randomly selected participants who initiate the sampling chains). After a number of waves (i.e., the number of layers in the recruitment process) and assuming recruitment follows a Markov process, a respondent-driven sample reaches equilibrium, meaning it has a stable distribution of key demographic and risk characteristics that are needed for the RDS estimators to perform properly. Homophily refers to the tendency of individuals to recruit from within their own groups. The measure for homophily used in RDSAT has a value ranging from -1 to 1; values between -1 and 0 indicate the tendency of individuals to recruit from outside their own groups, while values between 0 and 1 indicate the tendency to recruit from inside their own groups [6, 19]. To explore factors possibly related to design effect variability, we plotted the design effects versus study and versus the demographic and behavioral variables. Additionally, we plotted design effects versus the measures identied above: waves needed for equilibrium, homophily, mean network size, ratio of mean network size (the mean network size in the yes group divided by the mean network size in the no group), the RDS point estimate, the RDS standard error, and the SRS standard error. We stratied these plots by study. We used box plots for categorical measures and scatter plots for continuous

measures, supplementing each of the scatter plots with a simple bivariate regression line.

Results The number of seeds used in the studies ranged from 4 to 11 (low among FSW in Kiev, Ukraine and high among MTF in Lima, Peru and San Francisco, US). Equilibrium was reached within 25 waves for all studies, which was not higher than the mean number of waves in each sample (range of mean waves5 waves for MSM in Soweto to 10.7 for FSW in Kiev) and in advance of the maximum number of waves in each of the studies (range of mean waves7 waves for FSW in Mauritius to 17 for MSM in Soweto). Time to reach a nal sample ranged from 2 weeks among FSW in Mauritius to 30 weeks among MSM in Soweto. The coupon return rate was above 30 % for all studies, with the highest rate for IDU in Mauritius (83.4 %). Table 2 displays key statistical output for a range of risk and demographic variables typically of interest to researchers measuring HIV prevalence and risk. Among the variables analyzed for the IDU study in Kiev, differences between all crude and adjusted estimates ranged from 1 to 10 %, homophily ranged from 0.025 to 0.283, equilibrium was reached within 2 to 3 waves and design effects ranged from 1.56 to 3.46 (mean design effect of 1.95). In Mauritius, differences between all crude and adjusted estimates among IDU ranged from 0 to 6.6 %, homophily ranged from 0.004 to 0.153, equilibrium was reached by the second wave for all variables analyzed (with the exception of civil status which was reached by the third wave) and design effects ranged from 1.52 to 3.77 (mean design effect of 2.79).

123

AIDS Behav Table 2 Estimated design effects for specic questions in nine RDS studies from around the world Indicators by population/country Crude (%) Adjusted (%)a Adjusted 95 % CI Mult MNS for Yes Mult MNS for no Homophily (Hx) Estimated Deff SE Waves needed for equilibrium

Intravenous Drug Users Ukraine, Kiev Gender (female) Education (Csecondary) Employed Married Condom use most recent Sexual intercourse (yes) Shared injecting equipment at last injection (yes) Knows where to go to get an HIV test (yes) Ever HIV test (yes) HIV (yes) (missing 14) Mauritius Gender (female) Education ([secondary) Employed Married Condom use most recent sexual intercourse (yes, any partner) Shared injecting equipment at last injection (yes) Knows where to go to get an HIV test (yes) Ever HIV test (yes) HIV (yes) (missing 0) Men who have Sex with Men South Africa, Soweto Education (Csecondary) Employed Married Condom use most recent anal sexual intercourse (yes) Ever HIV test (yes) HIV (yes) (missing 70) China, Jinan Education (Csecondary) Employed Married Condom use most recent anal sexual intercourse (yes) Ever HIV test (yes) HIV (yes) 71.6 80.2 25.6 67.4 26.0 2.4 70.8 81.4 27.4 69.7 19.9 3.2 64.976.4 75.986.4 21.134.0 65.074.7 15.625.2 1.25.9 4.426 4.281 4.227 4.165 4.14 3.71 4.1 4.532 4.364 4.71 4.96 4.35 0.31 0.29 0.39 -0.012 0.245 0.262 2.70 2.20 2.70 1.50 1.80 2.30 2.20 0.030 0.026 0.034 0.025 0.024 0.012 4 4 5 2 3 3 96.8 35.3 50.0 43.5 25.1 95.5 34.1 65.2 37.6 13.7 92.398.1 27.440.4 57.272.7 31.145.8 7.456 7.63 6.045 8.152 5.369 7.228 9.434 6.828 6.468 0.293 0.036 0.131 0.233 0.384 1.80 1.80 2.50 2.20 3.20 2.30 0.015 0.033 0.040 0.037 0.032 2 2 4 3 4 9.0 7.4 23.5 23.4 28.2 81.3 59.5 52.9 9.0 5.3 18.9 23.8 28.6 79.4 57.2 46.3 5.313.4 13.12 3.07.9 14.224.3 13.231.6 21.835.6 72.285.5 49.665.2 37.654.8 18.20 16.2 12.12 13.02 13.31 13.40 13.40 12.94 12.27 12.95 13.10 12.36 12.66 12.62 0.092 0.027 0.062 0.016 0.004 0.147 0.089 0.153 2.54 1.52 2.11 2.67 3.06 3.55 3.12 3.77 2.79 0.021 0.013 0.026 0.047 0.035 0.034 0.040 0.044 2 2 3 2 2 2 2 2 31.8 23.0 56.7 47.3 52.9 18.2 98.0 52.4 40.0 32.8 27.9 59.5 44.4 59.0 12.6 98.2 44.2 30.7 27.239.1 22.733.8 50.468.5 37.951.1 53.667.4 8.716.9 96.299.7 36.550.2 24.036.5 4.4 3.6 4.4 4.4 4.1 6.5 4.6 5.4 5.9 4.7 4.7 5.4 4.3 4.6 3.9 4.0 0.025 -0.211 -0.023 0.115 -0.039 0.103 0.073 0.283 0.274 1.64 1.56 3.46 1.80 2.00 1.59 1.80 1.94 1.79 1.95 0.030 0.028 0.046 0.034 0.035 0.021 0.009 0.035 0.032 2 3 3 3 3 2 2 3 3

Mean estimated Deff

Mean estimated Deff

9.121.8 11.183

Mean estimated Deff

Mean estimated Deff

123

AIDS Behav Table 2 continued Indicators by population/country Crude (%) Adjusted (%)a Adjusted 95 % CI Mult MNS for Yes Mult MNS for no Homophily (Hx) Estimated Deff SE Waves needed for equilibrium

Mauritius Education (Csecondary) Employed Married Condom use most recent anal sexual intercourse (yes) Knows where to go to get an HIV test (yes) Ever HIV test (yes) HIV (yes) Male to Female Transgenders USA, San Francisco Education (Csecondary) Employed Married Condom use most recent sexual intercourse (yes) Ever HIV test (yes) HIV (yes) Peru, Lima Education (Csecondary) Employed Married Condom use most recent sexual intercourse (yes) Ever HIV test (yes) HIV (yes) Female Sex Workers Ukraine, Kiev Education (Csecondary) Employed (other than sex work) Married Condom use most recent client (yes) Received condoms free of charge in the last 12 months (via service providers) Injected drugs in the past 12 months Knows where to go to get an HIV test (yes) Ever HIV test (yes) HIV (yes) (missing 0) 81.1 45.4 33.8 89.6 6.9 82.5 47.9 33.8 88.1 6.3 76.587.8 40.255.8 26.242.0 82.293.3 2.510.6 6.01 4.72 5.06 5.07 6.47 4.90 4.95 4.36 4.91 -0.010 0.057 0.170 0.069 0.314 1.49 1.64 1.88 1.98 1.85 0.029 0.040 0.040 0.028 0.021 2 3 3 3 4 61.5 96.8 22.7 77.4 96.1 29.8 58.5 95.1 21.5 75.1 95.1 29.6 52.367.5 89.698.9 15.029.5 68.681.9 92.198.0 23.338.4 9.017 9.378 9.475 9.386 9.441 8.427 9.102 6.133 8.658 8.299 8.041 8.728 0.024 0.404 0.064 0.109 0.124 0.126 3.10 5.90 4.00 3.00 2.40 3.50 3.70 0.039 0.024 0.037 0.034 0.015 0.038 5 5 4 5 4 5 73.0 26.8 13.7 98.6 40.8 69.3 21.7 14.0 98.4 39.2 63.476.1 16.828.2 10.119.5 96.799.9 31.747.6 4.319 4.517 4.011 4.173 4.316 3.809 4.062 4.190 3.636 4.035 0.091 0.082 0.016 0.115 0.289 1.40 1.50 1.40 1.20 2.00 1.50 0.032 0.029 0.024 0.008 0.040 1 2 2 1 4 45.0 74.0 4.4 52.2 63.3 47.0 10.6 43.7 73.6 6.7 46.0 59.2 40.8 7.4 34.652.9 66.180.6 2.612.0 38.154.1 52.267.2 32.448.9 3.411.7 7.35 6.98 7.80 7.22 7.60 9.50 6.12 6.35 6.37 6.71 0.323 0.144 0.071 0.165 0.093 0.216 0.239 3.20 2.54 3.25 2.42 2.20 2.65 2.54 2.67 0.047 0.037 0.024 0.041 0.038 0.042 0.022 3 5 3 2 2 3 3

Mean estimated Deff

Mean estimated Deff

Mean estimated Deff

28.5 95.8 80.8 25.0

28.8 94.8 79.3 24.8

21.536.1 91.197.7 72.085.7 18.832.5

5.24 5.00 5.13 4.79

4.89 4.60 4.56 5.05

0.187 0.233 0.247 0.131

1.73 2.30 1.93 1.69 1.83

0.037 0.017 0.035 0.035

4 2 3 3

Mean estimated Deff

123

AIDS Behav Table 2 continued Indicators by population/country Crude (%) Adjusted (%)a Adjusted 95 % CI Mult MNS for Yes Mult MNS for no Homophily (Hx) Estimated Deff SE Waves needed for equilibrium

Mauritius Education (Csecondary) Employed (other than sex work) Married Condom use most recent client Received condoms free of charge in the last 12 months (via service providers) Injected drugs in the past 3 months Knows where to go to get an HIV test (yes) Ever HIV test (yes) HIV (yes) (missing 0) 6.5 19.1 38.5 88.0 93.3 4.8 17.0 38.0 86.7 92.1 1.88.5 12.122.3 29.645.9 79.792.2 88.295.7 17.89 21.56 20.87 19.93 19.93 19.32 18.24 17.10 -0.244 -0.061 0.073 0.091 0.068 1.91 1.52 2.19 2.63 1.50 0.017 0.027 0.042 0.032 0.019 3 2 2 2 2

33.1 81.6 64.9 32.2

29.6 79.6 57.7 29.2

22.337.0 73.385.7 48.666.4 21.536.3

21.10 20.08 21.26 20.76

19.10 18.27 17.31 19.25

0.150 0.152 0.253 0.092

2.02 1.84 2.52 2.11 2.03

0.038 0.032 0.045 0.038

2 2 3 2

Mean estimated Deff = not available

a

Salganik-Heckathorn estimator

Mult MNS multiplicity method, mean network size, Deff design effect, SE standard error

In the study of MSM in Soweto, differences between all crude and adjusted estimates ranged from 1.2 to 15.2 %, homophily ranged from 0.036 to 0.384, equilibrium was within 24 waves and design effects ranged from 1.80 to 3.20 (mean design effect of 2.30). For the study of MSM in Jinan, differences between crude and adjusted estimates ranged from 0.8 to 6.1 %, homophily ranged from -0.012 to 0.039, equilibrium was within two to ve waves and design effects ranged from 1.50 to 2.70 (mean design effect of 3.20). For the study of MSM in Mauritius, differences between crude and adjusted estimates ranged from 0.4 to 6.2 %, homophily ranged from 0.071 to 0.323, equilibrium was within 25 waves and design effects ranged from 2.20 to 3.25 (mean design effect of 2.67). In the study of MTF in San Francisco, differences between crude and adjusted estimates ranged from 0.3 to 5.1 %, homophily ranged from 0.016 to 0.289, equilibrium was within 14 waves and design effects ranged from 1.20 to 2.00 (mean design effect of 1.50). In the study of MTF in Lima, differences between crude and adjusted estimates ranged from 0.2 to 3.0 %, homophily ranged from 0.024 to 0.404, equilibrium was within 4 to 5 waves and design effects ranged from 2.40 to 5.90 (mean design effect of 3.70). In the study of FSW in Kiev, differences between crude and adjusted estimates ranged from 0 to 2.5 %, homophily ranged from 0.010 to 0.314, equilibrium was within 24 waves and design effects ranged from 1.49 to 2.30 (mean design effect of 1.83). In the study of FSW in Mauritius,

differences between crude and adjusted estimates ranged from 0.5 to 7.2 %, homophily ranged from -0.006 to 0.253, equilibrium was within 2 to 3 waves and design effects ranged from 1.50 to 2.63 (mean design effect of 2.03). The mean design effect across all studies was 2.33. Figure 1a, b, respectively, show design effect versus each study, and design effect versus waves needed for equilibrium. Figure 2a, b show design effect versus homophily and design effect versus mean network size, respectively. Design effect was clearly related to study (Fig. 1a): it was relatively high in the Lima MTF study and the Mauritius IDU study, for example, and relatively low in the San Francisco MTF study. Moreover, design effect appeared to be strongly related to the number of waves needed for equilibrium (Fig. 1b); the design effects generally increased with the number of waves needed for equilibrium. Design effect was related to homophily (Fig. 2a), as the design effects generally appeared to increase with increased homophily. However, design effect did not appear to be related to mean network size (Fig. 2b).

Discussion Our analysis found that design effects varied across countries and populations but were concentrated in a fairly narrow range from 1.20 to 5.90. Mean design effects among all studies ranged from 1.50 to 3.70 (mean 2.33)

123

AIDS Behav

Fig. 1 Estimated design effect by study and waves needed for equilibrium. a Study. b Waves needed for equilibrium

Fig. 2 Estimated design effect by homophily and mean network size. a Homophily. b Mean network size

indicating that a design effect of 2.00 may not be sufcient for RDS studies. However, given that the mean design effect across all studies was 2.33, indicates that although a design effect slightly above 2.00 might be adequate for calculating sample sizes for RDS studies, an effect closer to 3.00 or 4.00 might be more appropriate. Based on this limited group of samples and the importance to have clear guidance on which design effect to use in planning studies, we suggest that when using RDS, researchers collect a sample at least 2.30 times as large as would be needed under SRS. Our ndings are similar to other recent ndings

of design effects in aggregated datasets of FSW in Brazil where the design effect for HIV was 2.63 and in datasets of IDU in the US where the design effects were between 2.00 and 4.00 among most variables [10, 20]. Fortunately, data from existing studies suggest that so far, RDS has been used in situations where design effects of 2.00 (the level used in most studies) have been reasonably precise. Our and other published ndings imply that because of the possibility of high design effects and variation across populations care should be taken in designing RDS studies. Factors that could impact design effects are the estimator

123

AIDS Behav

variances which, in RDS are inuenced by network structures and the length of recruitment chains. In some extreme network structures, such as those with high homophily and those for which many waves of recruitment are needed to achieve equilibrium, the prevalence estimates could be so variable that even though they are unbiased, they might not be useful for surveillance purposes. For example, a particularly high design effect was found for employment status (design effect of 5.90) of MTF in Peru. This may be explained by a bottleneckdened as the occurrence of a relatively small number of recruitment ties between two groups in the population. Bottlenecks affect the design effect since the variance is not simply a function of homophily, but also of the network structure. Long recruitment chains result in low design effects while short recruitment chains result in high design effects [4]. Researchers can attempt to control the length of recruitment chains by reducing the number of coupons provided for participants to recruit their peers and by limiting the number of seeds used to initiate recruitment. By using fewer coupons for recruitment, chains are allowed to grow longer rather than wider, before the sample size is exhausted [21]. Similarly, a relatively small number of seeds would likely result in fewer but longer recruitment chains rather than many and shorter recruitment chains. Decisions to control the growth of chains should be considered only when recruitment moves quickly and is sustainable; otherwise the calculated sample size may not be attained when reducing the number of coupons or using a small number of seeds. The effect of number of seeds is shown in Fig. 1a and Table 1. Because each study has just one value for the number of seeds, it is possible to see whether, as hypothesized above, the studies with a small number of seeds (Table 1) had lower design effects (Fig. 1a), and whether the studies with a large number of seeds had larger design effects. The studies of MSM in Jinan, FSW in Kiev, and FSW in Mauritius used a small number of seeds and tended to have lower design effects compared to the studies of MSM in Soweto and MTF in Lima which used a large number of seeds and tended to have larger design effects. However, the study of MTF in San Francisco, which used a relatively large number of seeds, had relatively low design effects which may be a result of other factors such as characteristics of the underlying network structure. Our analysis has several limitations. First, we relied on the 95 % condence intervals that result from the standard errors produced with the multiplicity estimate which employs simulations of a rst-order Markov process using a bootstrap process to estimate design effects. It is commonly accepted that the 95 % condence intervals produced with this estimation process may not capture additional information inuencing sample estimates

including variations in the recruitment dynamics of survey participants or subtle underlying network structures in the population [4, 22, 23]. New estimators and variance procedures are being developed and tested, however these methods are not yet widely accepted and used [20, 24, 25]. Nonetheless, in the absence of a more robust sampling scheme to sample hidden and hard-to-reach populations or improved methods for deriving condence intervals for RDS data, we believe that the ndings here provide enough evidence to demonstrate the need for design effect adjustments in future studies of these populations in most countries. In addition, it would be useful for future published ndings from studies to report design effects, now that repeated studies of the same populations in the same cities using RDS are being conducted, and it would be helpful to see more specic reporting of potential variations in design effects within a population over time.
Acknowledgments We would like to thank our colleagues, Shiman Ruan, Tim Lane, Jenna Rapues and Tetyana Saluk, who graciously shared their data with us.

References
1. Carlson RG, Wang J, Siegal HA, Falck RS, Guo J. An ethnographic approach to targeted sampling: problems and solutions in AIDS prevention research among injection drug and crackcocaine users. Hum Organ. 1994;53(3):27986. 2. Magnani R, Sabin K, Saidel T, Heckathorn D. Review of sampling hard-to-reach populations for HIV surveillance. AIDS. 2005;19(Suppl 2):S6772. 3. Family Health International. Behavioral surveillance surveys: guidelines for repeated behavioral surveys in populations at risk of HIV. Arlington: Family Health International. 2000. http://www. fhi.org/en/HIVAIDS/pub/guide/bssguidelines.htm. Accessed 15 Nov 2012. 4. Goel S, Salganik MJ. Assessing respondent-driven sampling. Proc Natl Acad Sci USA. 2010;107:67437. 5. Salganik MJ. Variance estimation, design effects and sample size calculations for respondent driven sampling. J Urban Health. 2006;83(Suppl 7):98112. doi:10.1007/s11524-006-9106-x. 6. Heckathorn D. Respondent driven sampling: a new approach to the study of hidden populations. Soc Probl. 1997;44(2):17499. 7. Malekinejad M, Johnston LG, Kendall C, Kerr L, Rifkin M, Rutherford G. Using respondent-driven sampling methodology for HIV biological and behavioral surveillance in international settings: a systematic review. AIDS Behav. 2008;12(suppl 1):10530. 8. Johnston LG, Malekinejad M, Rifkin MR, Rutherford GW, Kendall C. Implementation challenges to using respondent-driven sampling methodology for HIV biological and behavioral surveillance: eld experiences in international settings. AIDS Behav. 2008;12(Suppl 1):13141. 9. Heckathorn D. Respondent driven sampling II: deriving valid population estimates from Chain-referral samples of hidden populations. Soc Probl. 2002;49(1):1134. 10. Wejnert C, Pham H, Krishna N, Le B, DiNenno E. Estimating design effect and calculating sample size for respondent-driven sampling studies of injection drug users in the United States. AIDS Behav. 2012;16(4):797806.

123

AIDS Behav 11. Ruan S, Yang H, Zhu Y, et al. HIV prevalence and correlates of unprotected anal intercourse among men who have sex with men, Jinan. China. AIDS Behav. 2008;12(3):46975. 12. Lane T, Raymond HF, Dladla S, et al. High HIV prevalence among men who have sex with men in Soweto, south Africa: results from the Soweto mens study. AIDS Behav. 2011;15(3): 62634. 13. Johnston LG, Saumtally A, Corceal S, Mahadoo I, Oodally F. High HIV and hepatitis C prevalence amongst injecting drug users in Mauritius: ndings from a population size estimation and respondent driven sampling survey. Int J Drug Policy. 2011;22(4):2528. Epub 2011 Jun 22. 14. Johnston LG.: Integrated behavioral and biological surveillance survey among injecting drug users in Mauritius, 2009; Mauritius Ministry of Health, Port Louis. 2009. www.gov.mu/portal/ sites/sida/idu.pdf. Accessed 15 Nov 2012. 15. Taran YS, Johnston LG, Pohorilac NB, Saliuk T. Correlates of HIV risk among injecting drug users in sixteen Ukrainian cities. AIDS Behav. 2011;15(1):6574. 16. Heckathorn DD. Extensions of respondent-driven sampling: analyzing continuous variables and controlling for differential recruitment. Sociol Methodol. 2007;37:151207. 17. Volz E, Heckathorn DD. Probability-based estimation theory for respondent-driven sampling. J Off Stat. 2008;24(Suppl 1):7997. 18. Salganik MJ, Heckathorn DD. Sampling and estimation in hidden populations using respondent driven sampling. Sociol Methodol. 2004;34:193239. 19. McPherson M, Smith-Lovin L, Cook JM. Birds of a feather: homophily in social networks. Ann Rev Sociol. 2011;27:41544. 20. Szwarcwald CL, de Souza Junior PR, Damacena GN, Junior AB, Kendall C. Analysis of data collected by RDS among sex workers in 10 Brazilian cities, 2009: estimation of the prevalence of HIV, variance, and design effect. J Acquir Immune Dec Syndr. 2011;57(Suppl 3):S12935. 21. Johnston LG, Khanam R, Reza M, et al. The effectiveness of respondent driven sampling for recruiting males who have sex with males in Dhaka, Bangladesh: a pilot study. AIDS Behav. 2007;2(2):294304. 22. Goel S, Salganik MJ. Respondent-driven sampling as Markov chain Monte Carlo. Stat Med. 2009;28:220229. 23. Poon AFY, Brouwer KC, Strathdee SA, et al. Parsing social network survey data from hidden populations using stochastic context-free grammars. PLoS ONE. 2009;4(9):e6777. doi: 10.1371/journal.pone.0006777. 24. Gile K J. Improved inference for Respondent-Driven sampling data with application to HIV prevalence estimation. J Am Stat Assoc. 2011;106(493):135146. 25. Gile KJ, Handcock MS. Respondent-driven sampling: an assessment of current methodology. Sociol Methodol. 2010;40:285327.

123