Journal of Sports Sciences

ISSN: 0264-0414 (Print) 1466-447X (Online) Journal homepage: http://www.tandfonline.com/loi/rjsp20

Reliability of meta-analyses to evaluate resistance

training programmes

Antonio Arruda, Daniel Souza, James Steele, James Fisher, Jürgen Giessing &
Paulo Gentil

To cite this article: Antonio Arruda, Daniel Souza, James Steele, James Fisher, Jürgen Giessing
& Paulo Gentil (2016): Reliability of meta-analyses to evaluate resistance training programmes,
Journal of Sports Sciences

To link to this article: http://dx.doi.org/10.1080/02640414.2016.1243799

Published online: 04 Nov 2016.

Submit your article to this journal

View related articles

View Crossmark data

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=rjsp20

Download by: [Ryerson University Library] Date: 04 November 2016, At: 04:12
JOURNAL OF SPORTS SCIENCES, 2016
http://dx.doi.org/10.1080/02640414.2016.1243799
LETTER
Reliability of meta-analyses to evaluate resistance training programmes
We read with great interest the study by Schoenfeld, Ogborn,
and Krieger (2016) and would, first of all, like to congratulate
the authors for the well-written paper and valorous effort to
treat the topic in an objective and straightforward manner.
However, to improve understanding of the topic, we offer
some comments based on theoretical evidence and our prac-
tical experience.
A common criticism of meta-analyses is that they usually
combine studies that have important methodological differ-
ences and, consequently, the summary effect can be largely
influenced by these differences across studies (Field, 2015). For
this reason, an important stage before the decision to use meta-
analysis involves consideration of variables that might explain
variations in calculated effect sizes. However, the power of meta-
analyses to test moderators depends on the number of studies
available and the sample sizes used in studies.
Resistance training outcomes are influenced by many vari-
ables that interact with each other (Paoli, 2012). Hence,
attempts to estimate the impact of 1 variable when others
are not controlled has a high risk of bias. Although the authors
made an effort to control for confounding variables by using
regression models, these did not include other resistance
training variables, and the small number of studies may also
have reduced the power to detect interactions. This seems to
be the case when analysing different body parts. Of the 5
studies that have analysed both upper and lower body muscle
size, 4 of them (Bottaro, Veloso, Wagner, & Gentil, 2011;
McBride, Blaak, & Triplett-McBride, 2003; Ostrowski, Wilson,
Weatherby, Murphy, & Lyttle, 1997; Radaelli et al., 2014)
showed different patterns of muscle hypertrophy for upper
and lower body muscles, with upper body, but not lower body
muscles, generally responding better to higher volumes of
resistance raining. Only the study of Rønnestad et al. (2007)
showed similar behaviour among different body parts.
However, Rønnestad et al. (2007) evaluated the trapezius
muscle, while the other studies evaluated additional arm mus-
cles (biceps and/or triceps brachii). Therefore, including stu-
dies that measured legs and arm muscles in the same analyses
could be misleading.
Another possible caveat concerns analysis of the number of
sets per exercise, instead of the number of sets in which the
muscle/s was/were involved. For example, the participants in
the study by Correa et al. (2015) performed both leg press and
knee extensions. Although the number of weekly sets per
exercise was 3 or 9, the number of sets involving knee exten-
sors were 6 or 18, respectively. With regard to arm muscles,
multijoint exercises (e.g., lat pull downs) promote the same
increases in muscle size as single-joint exercises (e.g., biceps
curls) (Gentil, Soares, & Bottaro, 2015). This suggests that upper
body multijoint exercises should be included when counting
the number of sets for arm muscles. This was not performed in
© 2016 Informa UK Limited, trading as Taylor & Francis Group
the studies of Correa et al. (2015), Ostrowski et al. (1997),
Radaelli et al. (2015) Radaelli et al. (2014) Radaelli et al. (2014),
and Ribeiro et al. (2015), and does not appear to have been
accounted for in the analyses of Schoenfeld et al.
The poor control and inadequate reporting of effort is
another point of concern. Most people that advocate training
at low volume, suggest that exercises should be performed to
momentary concentric failure. Training to momentary failure
may be critical during low-volume resistance training (Giessing
et al., 2014), making it important to control for intensity of
effort in the studies analysed (Fisher & Smith, 2012; Steele,
2014). The inadequate reporting of training effort is evident in
the study of Radaelli et al. (2015) who stated that participants
performed 8–12 repetitions to concentric failure, but in addi-
tion, stated that loads were increased only when participants
could perform more than 12 repetitions in all 3 sets. This
seems implausible. Having reached true momentary failure in
the first set, with a rest of only 90–120 s, there would be either
a decrease in repetitions or it would be necessary to reduce
the load used in subsequent sets, thus making it near impos-
sible to keep participants within the 8–12 repetition range
(Willardson & Burkett, 2005, 2006). Furthermore, increasing
load for 1 exercise (e.g., leg press) would almost certainly
reduce repetitions on subsequent exercises using similar mus-
cle groups (e.g., leg extension and leg curl). Whether partici-
pants in this study did indeed train to momentary failure as
suggested is unclear from the description in the methods
offered by the authors. While, sensitivity analyses did not
reduce the effect size estimated to result in statistical non-
significance, we think it is important to note that the effect
size and confidence intervals generated from the study by
Radaelli et al. (2015) is the only to suggest convincingly that
higher volumes are more beneficial.
Inspection of the methods used in included studies included
suggests it is unlikely that participants reached momentary
concentric failure in most cases. For example, Sooneste,
Tanimoto, Kakigi, Saga, and Katamoto (2013) reported that the
participants performed sets at 80% of 1 RM reaching momen-
tary failure or until 10 RM was completed. The study of Cannon
and Marino (2010) involved knee flexion and extension at 50%
of 1 RM during week 1 and 75% of the 1 RM for weeks 2–10,
and the participants were instructed to perform either 1 or 3
sets of 10 repetitions. Considering previous reports of number
of repetitions performed at different percentages of 1 RM
(Hoeger, Hopkins, Barette, & Hale, 1990), it seems likely that
many participants in both studies did not reach momentary
failure in the initial sets. Support for this is that participants
performed the same number of repetitions in sets 1 and 2 in
the study of Sooneste et al. (2013).
Based on these observations, it is possible that in many of
these studies, groups that performed single sets per exercise
2 A. ARRUDA ET AL.
were training at submaximal efforts while the multiple sets
groups, though also using submaximal efforts, were accumu-
lating fatigue from set to set and thus were at a closer proxi-
mity to momentary failure in later sets. This has been seen in
previous studies with volume-matched training to failure and
not to failure using rating of perceived exertion scales. So, in
addition to comparing volume, these studies might be com-
paring groups that trained at differing intensities of effort,
which can produce different outcomes (Giessing et al., 2014).
We acknowledge that it is not the responsibility of
Schoenfeld et al. to “police” studies in this way, however,
inclusion of such studies serves only to reduce meta-analyses
outcomes to a series of numerical values with limited real-
world validity. A wider question on use of meta-analyses
arises: Is it possible to pool data from studies using disparate
methods to generate an overall conclusion on the effects of
manipulation of a variable in an intervention when that vari-
able also interacts with the manipulation of other variables?
Considering the above points, it is impossible that higher
volumes are more beneficial when intensity of effort is not
maximal (i.e., repetitions are not performed to momentary
failure). However, with the poor reporting of set endpoints in
many resistance training studies, it is often difficult to know
whether 2 interventions have appropriately controlled for this
variable. Indeed, many meta-analyses of resistance training
variables pool studies that have compared them under vary-
ing conditions. From a practical perspective, one is left won-
dering whether higher or lower set volumes are better when
performing repetitions to momentary failure, under heavy
and/or light load conditions, with long or short repetitions
durations, etc. The role of interactions among other resistance
training variables and set volume was apparently not consid-
ered in the meta-analysis of Schoenfeld et al. Indeed, even if
they were to have included meta-regressions for these, some
have argued that even with this type of consideration, a meta-
analysis is not the right tool to tease apart aspects of inter-
ventions that work and those that do not (Field, 2015).
With these considerations in mind, we opine that funda-
mental inadequacies in the primary studies could have been
carried over to the meta-analysis. Additionally, benefits of
increasing training volume could have arisen as compensation
for a low intensity of effort in training. The influence of inten-
sity of effort could be one of the reasons why most studies
have shown that lower body muscles respond better to high-
volume training, since participants trained to, or closer to,
momentary failure more frequently in upper body than
lower body exercises (Gentil & Bottaro, 2010).
We also note inclusion of studies with participants of
different training history in the same analysis. The already
well trained have reduced muscle hypertrophy response to
training (Ahtiainen, Pakarinen, Alen, Kraemer, & Hakkinen,
2003). Other issues that could have influenced the results,
and were not considered, were the sex of participants and
their age. The dissimilarities of participants, methods, muscle
groups, and types of exercise among the studies analysed
brings into question if it is really possible to suggest the
existence of a dose–response relationship. We wonder
whether increasing number of sets leads to greater muscle
gains or whether the conclusion drawn by Schoenfeld et al.
was “contaminated” by methodological differences in the
studies included. This seems to be an important issue, espe-
cially because if we analyse the studies that used more than
2 training groups, for most there is no clear sign of graded
results with increasing number of sets. For example, in the
study of Ostrowski et al. (1997) increase in triceps muscle
thickness was 2.7%, 4.6%, and 4.7%, whereas in rectus
femoris muscle thickness was 6.7%, 5.0%, and 13.1% for 1,
2, and 4 sets per exercise, respectively. Radaelli et al. (2015)
reported a graded response in effects of 1, 3, and 5 sets per
exercise for elbow flexor but not for elbow extensors muscle
thickness. This means that the meta-analysis presents 2
studies in a total of 4 comparisons of more than 2 volumes,
where only 1 analysis showed a graded response for increas-
ing the number of sets. Additionally, it is important to note
that Ostrowski et al. (1997) suggested a possibility of over-
training as the number of sets increased, because of nega-
tive alterations in testosterone/cortisol ratio. Therefore, one
should be cautious before adopting “the more, the better”
as an approach to choosing the optimal number of sets.
The practical applications suggested by Schoenfeld et al.
suggest a greater response to higher volumes, but do not
consider how this information might be applied in a training
programme. Consider that if 10+ sets per muscle group/week
produce the greatest adaptation, can a person self-select how
they divide these exercises throughout the week (i.e., the neces-
sary volume performed in only 1 workout, or across 2, 3, 4, or 5
progressively lower volume workouts)?. The results of this
meta-analysis could be considered as supportive of practical
recommendations from proponents of low-volume resistance
training approaches (Fisher, Steele, & Smith, 2013) if one were
to perform ~3 workouts a week consisting of ~4 exercises per
muscle group, where a single set of each exercise is performed.
In conclusion, considering the large number of variables
involved in resistance training and the methodological incon-
sistencies in the current literature, it seems impossible to make
comparisons of different studies or include different studies in
the same analysis. For a meta-analysis to be valid, a large
amount of data on homogeneous subgroups should accumu-
late for topics where there is strong consensus about which
variables have theoretical importance, and this does not seem
to be the case for resistance training studies. Because of this,
the generalisation of meta-analyses should be viewed with
caution until we have a large number of studies providing
adequate control of variables. Rather than prematurely per-
form meta-analyses on differing resistance training variables,
which are all hindered by the inherent limitations of meta-
analyses (Shapiro, 1994) including low study numbers and
study heterogeneity (Field, 2015), and serve only to reduce
the complexity of resistance training variables to a single
statistic, greater value can be obtained by designing and
conducting studies of larger and homogenous samples that
can adequately address the topics considered. Otherwise, we
can be comparing oranges with apples or, worse, we can be
assuming that oranges and apples are the same.
Disclosure statement
No potential conflict of interest was reported by the authors.

References
Ahtiainen, J. P., Pakarinen, A., Alen, M., Kraemer, W. J., & Hakkinen, K.
(2003). Muscle hypertrophy, hormonal adaptations and strength devel-
opment during strength training in strength-trained and untrained
men. European Journal of Applied Physiology, 89(6), 555–563.
doi:10.1007/s00421-003-0833-3
Bottaro, M., Veloso, J., Wagner, D., & Gentil, P. (2011). Resistance training
for strength and muscle thickness: Effect of number of sets and muscle
group trained. Science & Sports, 26(5), 259–264. doi:10.1016/j.
scispo.2010.09.009
Cannon, J., & Marino, F. E. (2010). Early-phase neuromuscular adaptations
to high- and low-volume resistance training in untrained young and
older women. Journal of Sports Sciences, 28(14), 1505–1514.
doi:10.1080/02640414.2010.517544
Correa, C. S., Teixeira, B. C., Cobos, R. C., Macedo, R. C., Kruger, R. L., Carteri,
R. B., … Reischak-Oliveira, A. (2015). High-volume resistance training
reduces postprandial lipaemia in postmenopausal women. Journal of
Sports Sciences, 33(18), 1890–1901. doi:10.1080/02640414.2015.1017732
Field, A. P. (2015). Dread returns to Mega-Silly One. Health Psychology
Review, 9(1), 15–20. doi:10.1080/17437199.2013.879198
Fisher, J., & Smith, D. (2012). Attempting to better define “intensity” for
muscular performance: Is it all wasted effort? European Journal of
Applied Physiology, 112(12), 4183–4185; author reply 4187–4188.
doi:10.1007/s00421-012-2463-0
Fisher, J., Steele, J., & Smith, D. (2013). Evidence-based resistance training
recommendations for muscular hypertrophy. Medicina Sportiva, 17(4),
217–235.
Gentil, P., & Bottaro, M. (2010). Influence of supervision ratio on muscle
adaptations to resistance training in nontrained subjects. Journal of
Strength and Conditioning Research, 24(3), 639–643. doi:10.1519/
JSC.0b013e3181ad3373
Gentil, P., Soares, S., & Bottaro, M. (2015). Single vs. multi-joint resistance
exercises: Effects on muscle strength and hypertrophy. Asian Journal of
Sports Medicine, 6(1), e24057. doi:10.5812/asjsm.24057
Giessing, J., Fisher, J., Steele, J., Rothe, F., Raubold, K., & Eichmann, B.
(2014). The effects of low volume resistance training with and without
advanced techniques in trained participants. The Journal of Sports
Medicine and Physical Fitness. doi:R40Y9999N00A140123[pii]
Hoeger, W. W. K., Hopkins, D. R., Barette, S. L., & Hale, D. F. (1990).
Relashionship between repetitions and selected percentages of one
repetition maximum: A comparison between untrained and trained
males and females. Journal of Strength and Conditioning Research, 4
(2), 47–54.
McBride, J. M., Blaak, J. B., & Triplett-McBride, T. (2003). Effect of resistance
exercise volume and complexity on EMG, strength, and regional body
composition. European Journal of Applied Physiology, 90(5–6), 626–632.
doi:10.1007/s00421-003-0930-3
Ostrowski, K. J., Wilson, G. J., Weatherby, R., Murphy, P. W., & Lyttle, A. D.
(1997). The effect of weight training volume on hormonal output and
muscular size and function. Journal of Strength and Conditioning
Research, 11(3), 148–154.
Paoli, A. (2012). Resistance training: The multifaceted side of exercise.
American Journal of PhysiologyEndocrinoogy and Metabolism, 302(3),
E387. doi:10.1152/ajpendo.00541.2011
Radaelli, R., Botton, C. E., Wilhelm, E. N., Bottaro, M., Brown, L. E., Lacerda,
F., … Pinto, R. S. (2014). Time course of low- and high-volume strength
training on neuromuscular adaptations and muscle quality in older
women. Age (Dordrecht), 36(2), 881–892. doi:10.1007/s11357-013-9611-2
Radaelli, R., Fleck, S. J., Leite, T., Leite, R. D., Pinto, R. S., Fernandes, L., &
Simão, R. (2015). Dose-response of 1, 3, and 5 sets of resistance exercise
on strength, local muscular endurance, and hypertrophy. Journal of
JOURNAL OF SPORTS SCIENCES 3
Strength and Conditioning Research, 29(5), 1349–1358. doi:10.1519/
JSC.0000000000000758
Radaelli, R., Wilhelm, E. N., Botton, C. E., Rech, A., Bottaro, M., Brown, L. E., &
Pinto, R. S. (2014). Effects of single vs. multiple-set short-term strength
training in elderly women. Age (Dordrecht), 36(6), 9720. doi:10.1007/
s11357-014-9720-6
Ribeiro, A. S., Schoenfeld, B. J., Pina, F. L. C., Souza, M. F., Nascimento, M. A.,
Dos Santos, L., … Cyrino, E. S. (2015). Resistance training in older
women: Comparison of single vs. multiple sets on muscle strength
and body composition. Isokinetics and Exercise Science, 23(1), 53–60.
Rønnestad, B. R., Egeland, W., Kvamme, N. H., Refsnes, P. E., Kadi, F., &
Raastad, T. (2007). Dissimilar effects of one- and three-set strength
training on strength and muscle mass gains in upper and lower body
in untrained subjects. Journal of Strength and Conditioning Research, 21
(1), 157–163. doi:10.1519/00124278-200702000-00028
Schoenfeld, B. J., Ogborn, D., & Krieger, J. W. (2016). Dose-response rela-
tionship between weekly resistance training volume and increases in
muscle mass: A systematic review and meta-analysis. Journal of Sports
Sciences, 1–10. doi:10.1080/02640414.2016.1210197
Shapiro, S. (1994). Meta-analysis/Shmeta-analysis. American Journal of
Epidemiology, 140(9), 771–778.
Sooneste, H., Tanimoto, M., Kakigi, R., Saga, N., & Katamoto, S. (2013).
Effects of training volume on strength and hypertrophy in young
men. Journal of Strength and Conditioning Research, 27(1), 8–13.
doi:10.1519/JSC.0b013e3182679215
Steele, J. (2014). Intensity; in-ten-si-ty; noun. 1. Often used ambiguously
within resistance training. 2. Is it time to drop the term altogether?.
British Journal of Sports Medicine, 48(22), 1586–1588. doi:10.1136/
bjsports-2012-092127
Willardson, J. M., & Burkett, L. N. (2005). A comparison of 3 different rest
intervals on the exercise volume completed during a workout. Journal
of Strength and Conditioning Research, 19(1), 23–26.
Willardson, J. M., & Burkett, L. N. (2006). The effect of rest interval length
on the sustainbility of squat and bench press repetitions. Journal of
Strength and Conditioning Research, 19(1), 23–26.
Antonio Arruda
Human Performance Research Laboratory (LAPEDH), University
of Pernambuco, Petrolina, Brazil
arrudabq@hotmail.com
Daniel Souza
Federal University of Rio Grande do Norte, Post Graduate
Program in Physical Education, Natal, Brazil
James Steele and James Fisher
Sport Science Laboratory/Centre for Health, Exercise & Sport
Science, Southampton Solent University, Southampton, UK
Jürgen Giessing
Institute of Sport Science, University of Koblenz-Landau,
Landau, German
Paulo Gentil
College of Physical Education, Federal University of Goias,
Goiania, Brazil
Accepted 28 September 2016