M E T H O D O L O G Y PA P E R

Validation of a new tool for the assessment of study

quality and reporting in exercise training studies:
TESTEX
Neil A. Smart, 1 Mark Waldron, 1 Hashbullah Ismail, 1 Francesco Giallauria, 1,2 Carlo Vigorito, 2
Veronique Cornelissen 1,3 and Gudrun Dieberg 1
1
University of New England, School of Science and Technology, Armidale, New South Wales, Australia, 2Department of Translational Medical
Sciences, Division of Internal Medicine and Cardiac Rehabilitation, Federico II University of Naples, Naples, Italy, and 3University of Leuven, KU
Leuven, Department of Rehabilitation Sciences, Leuven, Belgium

ABSTRACT

Introduction: Several established tools are available to assess study quality and reporting of randomized controlled
trials; however, these tools were designed with clinical intervention trials in mind. In exercise training intervention
trials some of the traditional study quality criteria, such as participant or researcher blinding, are extremely difficult to
implement.
Methods: We developed the Tool for the assEssment of Study qualiTy and reporting in EXercise (TESTEX) – a study
quality and reporting assessment tool, designed specifically for use in exercise training studies. Our tool is a 15-point
scale (5 points for study quality and 10 points for reporting) and addresses previously unmentioned quality
assessment criteria specific to exercise training studies.
Results: There were no systematic differences between the summated TESTEX scores of each observer [H(2) ¼ 0.392,
P ¼ 0.822]. There was a significant association between the summated TESTEX scores of the three observers, with
almost perfect agreement between observers 1 and 2 [intra-class correlation coefficient (ICC) ¼ 0.93, 95% confidence
interval (CI) 0.82–0.97, P < 0.001], observers 1 and 3 (ICC ¼ 0.96, 95% CI 0.89–0.98, P < 0.001) and observers 2 and
3 (ICC ¼ 0.91, 95% CI 0.75–0.96, P < 0.001).
Conclusions: The TESTEX scale is a new, reliable tool, specific to exercise scientists, that facilitates a comprehensive
review of exercise training trials.
Key words: assessment tool, exercise training, study quality, study reporting
Int J Evid Based Healthc 2015; 13:9–18.

Introduction adhere to an intervention, be it medication or exercise,

E xercise training study designs and reporting, especi-

ally in clinical populations, are increasingly complex.
The benefits of exercise training for improving the
clinical status of patients are widely recognized, but
research and service provision funding may be withheld
in certain circumstances owing to the growing concerns
over poor exercise compliance. If a person refuses to
Correspondence: Neil A. Smart, School of Science and Technology,
University of New England, Armidale, NSW 2351, Australia. E-mail:
nsmart2@une.edu.au
DOI: 10.1097/XEB.0000000000000020
then the potential to improve is likely to be less and, for
this reason, interest in translational research into exercise
training is burgeoning. Poor study designs and incom-
plete reporting in the exercise sciences may limit the
usefulness of study findings in terms of translation to
clinical servicing.
The most robust individual study design is the
randomized controlled trial, but results from two differ-
ent exercise training interventions are often conflicting.
In order to determine which trial is more believable, one
could rank studies in terms of the standards of meth-
odological and reporting quality. The reason one may

International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute 9

©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
 NA Smart et al.
employ this ranking strategy is to identify sources of
bias. One common source of bias is measurement error
where researchers who deliver the intervention are also
responsible for taking outcome measurements – which
may be biased in favour of the exercise over control
group; blinding of outcome assessors may therefore
eliminate this form of bias. Outcome reporting from
trials may be deficient in a number of ways and this is
easily rectified. Selective outcome reporting is com-
mon, where post-intervention change in initially stipu-
lated outcome measures are withheld from publication
because results are undesirable. Reliability, or the con-
sistency of a measurement or the absence of measure-
ment error,1 and degree of observer error or validity are
two other sources of error. In exercise training studies
health practitioners are almost always concerned with
safety, yet adverse events are not always reported.
Related to this, one may report a certain type of
exercise intervention to be more beneficial to a group
than another intervention; yet this is of secondary
importance if adverse events rates, withdrawal or
adherence rates are worse. For these reasons, we have
focussed on the specifics of methodology and report-
ing in exercise training studies with respect to these
shortcomings.
Various tools are available to assess the quality of
methodology and reporting in randomized controlled
trials. The Consolidated Standards of Reporting Trials
statement2 is a general tool to guide study reporting,
and while the JADAD score3 was for some time the
preferred quality and reporting assessment tool fav-
oured by the Cochrane Collaboration,4 a more recent
risk of bias (ROB) scale has been developed by Cochrane,
although the reliability of ROB appears low.5 Adopting a
general tool, such as ROB, for exercise training studies, is
extremely likely to lack the specificity and certain criteria
(such as participant blinding) will be redundant. Within
the field of physical therapy, the Physiotherapy Evidence
Database (PEDro) tool has become widely used by
physiotherapists and is currently also the tool of choice
for exercise physiologists. Some of the criteria included
in the PEDro scale, many of which were adapted from the
Delphi list,6 are often redundant for exercise training
studies. Examples of this are the blinding criteria utilized
by the PEDro scale; blinding of exercise training partici-
pants is not feasible, as is blinding of the investigators
directly supervising the training. Therefore, these criteria
are redundant for exercise training studies. On the con-
trary, other important methodological and reporting
criteria determining the effectiveness and the risks of
an exercise intervention are not adequately addressed.
Examples of these are reporting of withdrawals/adverse
10 International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute
©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
events, session attendance, exercise adherence, and
exercise programme characteristics, which are important
in exercise training studies and are not included in the
PEDro scale.7 The authors of this work have experience
assessing study quality for numerous published meta-
analyses.8–15 The PEDro scale is perhaps the tool that
comes closest to meeting the methodological and
reporting requirements of exercise studies; however,
several shortcomings remain.
The primary objective of this study was to develop
an exercise science-specific scale, designed for use by
exercise specialists, to assess the quality and reporting of
exercise training trials. The secondary objective was to
assess the validity and reliability of this scale. It is
intended that the scale will be used by researchers
conducting systematic review and meta-analyses, so
they can quantify the strength of individual study
designs and reporting; and by exercise science prac-
titioners seeking to establish whether a particular inter-
vention is beneficial or safe in the face of conflicting
evidence. While developing the Tool for the assEssment
of Study qualiTy and reporting in EXercise (TESTEX) scale,
we aimed to avoid using redundant criteria, which we
feel are not applicable for exercise training studies,
and to include new criteria, which we think are most
relevant to study design, quality and reporting in
exercise sciences.
Methods
The authorship group consisted of members of an
existing collaborative with previous experience in pro-
ducing meta-analyses and therefore assessing study
quality. All members were asked to list the difficulties
they had encountered in using PEDro in assessing
exercise training studies. Members were also asked
which items they thought were redundant and which
should be included in an exercise training-specific
assessment tool. The group used the PEDro scale as a
template for the development of a new scale. A series of
meetings were organized, during which newly pro-
posed and existing (PEDro) criteria were assessed for
inclusion in the new scale. Newly proposed criteria were
based upon difficulties experienced when conducting
study quality assessment for meta-analyses, which is
crucial for information accuracy and translation into
clinical practice. Most items were unanimously included
in the protocol, but two items were debated at three
meetings. At the third meeting, a consensus was
reached and a draft protocol was circulated for com-
ment. Three drafts were edited before a final version was
reached. Once the draft had been finalized, a reliability
study was conducted.

Reliability of TESTEX
Study selection and quality assessments
Three reviewers/observers (M.W., N.S., and G.D.) inde-
pendently evaluated the quality of 19 published exercise
training studies using the TESTEX criteria. The studies
were randomly selected from a list of randomized con-
trolled trials of exercise training in people with chronic
disease. This search intentionally produced a broad
variety of exercise studies (randomized controlled trials),
administering exercise interventions to a range of
patients. It was not deemed necessary to search for
studies with any greater scrutiny, given the intended
broad application of the TESTEX scale. Nineteen studies
were selected for the reliability analysis since this equa-
ted to the average number of studies included in meta-
analyses conducted by the current authors. In total, there
were 15 points available (5 points for study quality and
10 points for reporting) in the TESTEX scale that were
each assigned either a ‘1’ or ‘0’ by the observers. All
observers had experience in conducting exercise train-
ing studies and varying levels of expertise in assessing
study quality of exercise intervention trials. Each
observer was provided with a copy of the TESTEX pro-
tocol, the 19 research papers and an Excel spreadsheet
on which to record the data.
Statistical analysis of reliability
The inter-observer agreement between each observer
(n ¼ 3) was assessed for each individual point available
on the TESTEX scale (15 in total) using the Cohen Kappa
statistic (K). The Kappa statistic is based on the following
calculation: K ¼ (PO
PC)/(1
PC), where PO is the num-
ber of observed agreements and PC is the number of
agreements expected by chance. The Kappa statistic and
intra-class correlation coefficients (ICCs) are appropriate
for measuring agreement between individuals when the
data are nominal (i.e. ‘1’ or ‘0’).1,16 The use of these
statistics is consistent with previous studies assessing
the inter-observer agreement of quality assessment
tools.7
A secondary analysis was also performed, based
on the same data-set, to evaluate the inter-observer
reliability of the summated TESTEX scale score (i.e. score
out of 15). Following checks for normality using the
Shapiro–Wilk statistic, systematic differences between
the three observers was evaluated using non-parametric
Kruskal–Wallis tests. The reliability of the total score of
each observer was assessed using an ICC (2, k) and the
associated 95% confidence intervals (95% CIs). For all
analyses, we described the level of agreement between
the observers according to Landis and Koch17: (>0.81
‘almost perfect’; 0.61–0.80 ‘substantial’; 0.41–0.60
International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute
©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
METHODOLOGY PAPER
‘moderate’; 0.21–0.40 ‘fair’; 0.00–0.20 ‘slight’ and 0.00
‘poor’). Data analyses were performed using SPSS ver-
sion 20 and statistical significance was set at P less than
0.05 throughout.
Results
TESTEX is a study quality assessment tool for exercise
training studies that addresses previously unmentioned
quality assessment criteria such as crossover from
sedentary control to exercise, periodic adjustment of
exercise training intensity in respect to physical train-
ing adaptation, and reporting of exercise programme
characteristics. The TESTEX scale uses 12 criteria with
some criteria having more than one possible point, for a
maximum score of 15 points (5 points for study quality
and 10 points for reporting). A concise summary of the
TESTEX scale is provided in Table 1. A detailed descrip-
tion of each TESTEX criterion and justification for
inclusion is included below. A comparison between
the TESTEX criteria and those used by the PEDro scale
was conducted and is included in Table 2.
TESTEX criteria
Common study quality criteria
Eligibility criteria specified – TESTEX Criterion 1
In exercise training studies, specific diagnostic criteria
must fall within a certain range for the condition to exist,
e.g. to be considered hypertensive a person’s systolic
blood pressure should be greater than 139mmHg and/or
diastolic blood pressure greater than 89mmHg or being
treated with antihypertensive medication. However,
often the mean and standard deviation (SD) values for
a study group indicate that some values fall outside the
specified range and, hence, eligibility criteria ‘are not
met’ by all participants. We note that in the PEDro
document it is stated that this criterion influences exter-
nal validity, but not the internal or statistical validity of
the trial. We also note that the PEDro scale suggests that
eligibility criteria should be specified but this criterion is
not used to calculate PEDro score. Indeed, the PEDro
scale was specifically designed to avoid the assessment
of external validity. We feel that eligibility criteria
are more precise in the exercise sciences for three
reasons. First, eligibility is less reliant on clinical judge-
ment (e.g. physical examination) and more dependent
on numerical values from objective diagnostic tests
in exercise science. Second, the scope of practice for
exercise scientists excludes diagnosis, which is not
always the case for physiotherapists. Third, manuscript
and grant reviewer feedback over the years lead us to
unequivocally believe that some eligibility criteria are
extremely likely to affect the statistical validity of exercise
11
 12

Table 1. ‘Detailed TESTEX scale’ (maximum score 15)
Criterion
Study quality
1 – Eligibility criteria specified
2 – Randomization specified
3 – Allocation concealment
4 – Groups similar at baseline
International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute
Explanation
Eligibility criteria should be specified and fulfilled and specific diagnostic test values should be provided
for all participants.
A description of the method used to allocate patients into treatment groups should be provided.
It should be stated if group allocation was concealed; meaning if a patient was eligible for inclusion in
the trial was unaware (when this decision was made) of which group the patient would be allocated
to.
Baseline data of all participants who were randomized should be presented. There should be no
NA Smart et al.
Scoring
1 Point – if eligibility criteria are clearly stated and fulfilled
1 Point – if methods are described and they are truly random e.g.
coin-tossing, sequence of randomly generated numbers
1 Point – if group allocation was concealed from patients eligible
for inclusion in the trial (e.g. consent should be given before
randomization)
1 Point – if baseline data are separated by group allocation,

Blinding of all participants
Blinding of all therapists
5 – Blinding of assessor (for at least
one key outcome)
Study reporting
6 – Outcome measures assessed in
85% of patients
significant difference in the measure of the severity of the treated condition between treatment presented and no differences are apparent
groups.
This item is not scored. No point
This item is not scored. No point
It is not always possible to blind patients and/or therapists; however, blinding of assessors is reasonable. 1 Point – if it is stated unambiguously that the assessor of at least
If assessors of primary outcome measures are blinded to the intervention allocation of the patients, 1 primary outcome measure was blinded to group allocation
this should be stated clearly.
The percentage of patients completing the study in both groups should be reported. No point – if withdrawals are >15%
Any adverse events (serious medical events, deaths, hospitalizations etc.) should be reported for each 1 Point – if adherence>85%
intervention group. 1 Point – if adverse events are reported
The percentage of exercise sessions completed by the exercise patients who did not withdraw from the 1 Point – if exercise attendance is reported
study should be reported. Total possible – 3 points

7 – Intention-to-treat analysis
8 – Between-group statistical
comparisons reported
9 – Point measures and measures of
variability for all reported outcome
measures
10 – Activity monitoring in control
groups
11 – Relative exercise intensity
remained constant
12 – Exercise volume and energy
expenditure
When a patient withdraws, this analysis is conducted by using either the last value obtained for each of 1 Point – if intention to treat analysis was performed on outcomes
the outcome measures as a post-intervention value, or by using the baseline value as a post value. of interest
This analysis should be added to the data of those that did complete the study and an analysis
conducted.
Comparison of exercise vs. comparator (control) group for the primary and at least one secondary 1 Point – if between-group statistical comparisons are reported for
outcome should be performed. the primary outcome measure of interest
1 Point – if between-group statistical comparisons are reported for
at least one secondary outcome measure
Total possible – 2 points
Point estimates should be provided for all outcomes, otherwise this could be deemed selective 1 Point – if all outcomes are reported with point estimates
outcome reporting.
Between-group differences may be diluted if control patients crossover to intervention. As many as one 1 Point – if control patients are asked to report their levels of
third of patients do this, so some measure e.g. exercise diary or activity monitoring should be physical activity and data are presented
supplied so this effect can be measured and quantified.
Exercise intensity is considered by many to be the best stimulus for adaptation. Once patients begin an 1 Point – if exercise load is titrated to keep relative intensity
exercise programme at a set intensity they will begin to adapt. Throughout the study duration the constant
relative intensity will fall in those that do adapt. Therefore, periodic assessment of exercise capacity
should be conducted and the intensity titrated up (or in those that lose fitness, titrated down) so
that exercise intensity remains constant.
Exercise parameters; session and programme duration, session frequency, exercise training intensity and 1 Point – if exercise volume and energy expenditure can be
modality should be clearly reported. calculated
Total out of a possible 15 points

©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.

Table 2. Comparison between PEDro scale and TESTEX scale
PEDro
Eligibility criteria specified
Random allocation of patients
Allocation concealed
Groups similar at baseline
Blinding of all participants
Blinding of all therapists
Blinding of all assessors
Outcome measures assessed in 85% of
patients
Intention-to-treat analysis
Reporting of between-group statistical
comparisons
Point measures and measures of variability
reported
Total points possible
PEDro, Physiotherapy Evidence Database.
a
When a patient withdraws, this analysis is conducted by using either the last value obtained for each of the outcome measures as a post-intervention value, or by
using the baseline value as a post value.
training trials. For these reasons, we award 1 point
for studies that report and fulfil included eligibility
criteria.
Randomization specified – TESTEX Criterion 2
Consistent with the second criterion of the PEDro scale,
we feel it is insufficient to consider that a study has
utilized a random allocation if the published manuscript
merely states that allocation was random. The precise
method of randomization (e.g. computer-generated ran-
dom numbers, coin-tossing and dice-rolling) should be
specified. Quasi-randomized allocation procedures do
not satisfy this criterion. Random allocation ensures that
(within the constraints provided by chance) treatment
and control groups are comparable; this is especially
important in exercise training studies as sample size is
often less than 50 patients and study withdrawal is more
than 15%. We award 1 point for studies that stipulate the
method of randomization.
International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute
©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
METHODOLOGY PAPER
Points TESTEX Points
TESTEX study quality (total 5 points)
None Eligibility criteria specified 1
1 Randomization specified 1
1 Allocation concealment of all patients at the time of randomization 1
1 Groups similar at baseline 1
1 Blinding of all participants None
1 Blinding of all therapists None
1 Blinding of assessor (for at least one key outcome) 1
TESTEX study reporting (total 10 points)
1 Outcome measures assessed in 85% of patients
Study withdrawals reported (>15% – no point; <15% – 1 point) 1
Adverse events reported 1
Session attendance reported 1
1 Intention-to-treat analysisa 1
1 Reporting of between-group statistical comparisons
Primary outcome reported 1
Secondary outcome(s) reported 1
1 Point measures and measures of variability for all reported outcome 1
measures reported
Activity monitoring in control groups
To avoid/measure crossover to exercise by sedentary control 1
patients; method of activity monitoring is reported
Relative exercise intensity remained constant
Periodic evidence-based adjustment of exercise intensity is 1
reported
Exercise volume and exercise expenditure
Information on all exercise characteristics (intensity, duration, 1
frequency, mode is provided to calculate exercise volume and
expenditure
10 Study quality – 5 points: Study reporting –10 points 15
Allocation concealment – TESTEX Criterion 3
In addition to eligibility criteria, the TESTEX scale awards
1 point for the concealment of allocation. A study is
considered to have provided allocation concealment if
the potential patients were unaware of which group they
would be allocated to, at the time the patients give
their consent.
Groups similar at baseline – TESTEX Criterion 4
Studies of exercise training interventions should report
at least one measure of the severity of the condition
being treated and at least one (different) key outcome
measure at baseline. The rater must be satisfied that the
groups’ outcomes would not be expected to differ (on
the basis of baseline differences in prognostic variables
alone) by a statistically significant amount. Discrepancies
at baseline between groups may be indicative of
inadequate randomization procedures. One point is
awarded if baseline data are presented by group
13
 NA Smart et al.
allocation, and there is no significant difference between
study groups in the key outcome(s) of interest and
the measure of the severity of the condition being
treated.
Blinding of all participants
Blinding involves ensuring that participants were unable
to discriminate whether they had or had not received the
treatment. It is acknowledged that in some physiother-
apy studies it is possible to provide ‘sham’ interventions
that could be perceived by participants to mimic actual
interventions. In exercise training studies participant
blinding is difficult to achieve, with just a few notable
exceptions. For example, there have been studies that
have compared exercise training to: cycling at zero load;
‘functional electrical stimulation (FES)’ or ‘sham’ and
inspiratory muscle training (IMT), whereby participants
can be subjected to interventions that are likely to be
below the stimulus threshold to elicit physiological
adaptation. Despite these attempts at ‘sham’ training,
participants are usually aware of the groups to which
they have been randomized (based on information that
they receive when giving their consent to participate), so
true blinding is almost impossible. Studies that have
allocated participants to either FES or IMT intervention
groups could be considered to have employed shams or
placebos. We acknowledge that the reason for using
sham treatments is that they can be manipulated to fall
above or below the therapeutic threshold expected to
elicit a physical adaptation, but feel that this is in general
not feasible in exercise training studies. For this reason,
unlike the PEDro scale, we do not award a point for
this criterion.
Blinding of all therapists
Like in criterion 3, it is our contention that exercise
training interventions do not lend themselves to blind-
ing the administering therapist. When therapists have
been blinded, the reader can be satisfied that the appa-
rent effect (or lack of effect) of treatment was not due to
the therapists’ enthusiasm or lack of enthusiasm for the
treatment or control conditions. In exercise studies per-
formed on patients with chronic disease, it is of funda-
mental importance that the administering therapist is
fully aware of the possible effects of a given treatment.
Indeed, in studies administering exercise interventions
to moderate-high risk patients, it would be considered
negligent on behalf of the therapist not to have obtained
an a priori record of the patient’s medical history and the
scope of effects that might occur in that individual as
a result of the treatment. Furthermore, part of the
therapist’s role is to provide motivation for patients to
14 International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute
©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
continue to exercise, inherently attaching the prac-
titioner to the treatment. Collectively, we feel that the
above reasons render this criterion as an inappropriate
scoring criterion. For this reason, unlike the PEDro scale
we do not award a point for this criterion.
Blinding of assessor (for at least one key
outcome) – TESTEX Criterion 5
Whereas blinding of patients and therapists is very
difficult to implement in exercise training studies, it is
reasonable to expect to blind assessors (those people
that conduct outcome data measurements) to the inter-
vention allocation of the participants. When assessors
have been blinded, the reader can be satisfied that the
apparent effect (or lack of effect) of treatment was not
due to the assessors’ biases impinging on their measures
of outcomes. One point is awarded if it is stated, unam-
biguously, that the assessor of the primary outcome
measure was blinded to group allocation. An exception
to this is where studies state that measurements are
completely automated, for example, measurements of
blood analyses in which case the potential for human
bias has been removed, 1 point can be awarded.
Common study reporting criteria
Outcome measures assessed in 85% of
patients – TESTEX Criterion 6
The volume and duration of exercise training required
to elicit adaptations varies with different outcome
measures, exercise prescriptions and patient character-
istics. However, it is generally accepted that significant
changes in some measures, such as cardio-respiratory
fitness, cardiac function, lipids or glycaemic control, are
not immediate and that at least 1 month is required to
detect changes. Due to the extended intervention
periods typical of exercise training, the proportion of
patients who complete the study is often less than 85%
and not all of those who complete the study attend all
exercise sessions (we deal with high withdrawal rates in
criterion 7 on ITT). It is therefore important in exercise
training studies to distinguish between exercise adher-
ence and exercise attendance as both are relevant. For
the purposes of this document we will define exercise
adherence as the number of withdrawals and com-
pletions in both the study’s intervention and control
groups. Exercise attendance is defined here as the per-
centage of target sessions completed by each individual
who completes the study. Quite often more than 15% of
people will withdraw from an exercise training study
during the stipulated study period. Moreover, exercise
attendance is less than 85% in some of the people who
do not withdraw from the study. It is therefore desirable

that both the number of withdrawals in each allocation
group, and also the mean and SD of the percentage
exercise session attendance are reported for interven-
tion groups. The one caveat to this would be where an
alternative target, for example, kilocalories per week of
exercise energy expenditure was successfully achieved.
In this case a point would be awarded regardless of the
number of exercise sessions completed.
We also feel that physical activity in control group
patients should be monitored but we address this, and
attribute a score, in criterion 10. We award 1 point if
studies report exercise training adherence of at least
85%; no point will be awarded if adherence is less
than 85%. We award 1 point for reporting adverse
events (deaths, hospitalizations, etc. are reported). We
award a point for this as the uptake of exercise therapy is
almost always evaluated in terms of a balance between
expected benefits and the risk of adverse events. We also
award 1 point for reporting session attendance for the
exercise group(s).
‘Intention-to-treat’ analysis – TESTEX
Criterion 7
In this criterion, we assess if all patients for whom out-
come measures were available received the treatment or
control condition as allocated or, where this was not the
case, data for at least one key outcome in which one is
interested was analysed by ITT analysis. We actually
propose that, when possible, an ITT analysis is conducted
so either the last value obtained for each of the outcome
measures is used as the post-intervention value, or the
baseline value is used as the post-value, when a patient
withdraws. The inclusion of this criterion aims to estab-
lish if certain patient demographics, clinical status, medi-
cations and so on predispose patients to withdraw. The
overall aim here is to help future research and clinical
services to improve exercise adherence by identifying
predisposing factors that lead to study withdrawal. One
point is awarded if an ITT is conducted. ITT analysis is
performed by substituting the last measurement (which
may be the baseline measurement) in those that did not
complete the study and these data are included in
analyses of those that did complete.
Between-group statistical comparisons
reported – TESTEX Criterion 8
To score in this criterion, between-group statistical com-
parisons should be reported for the primary and at least
one secondary outcome measure. If all outcomes are
not reported, this would deem to be selective outcome
reporting. We are primarily interested in whether inter-
vention groups improve, but a comparison with a control
International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute
©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
METHODOLOGY PAPER
group provides two things. Firstly, this will indicate
whether there is regression to the mean, that is patients
improve without exercise, perhaps because their medi-
cations are optimized or other reasons unrelated to
the exercise intervention. Secondly, it may be that
some patients who should have exercised did not
and some who should not have received the interven-
tion did. Although we attribute a score to these issues in
other criteria, the between-group comparison alerts us
to the fact that regression to the mean or inappropriate
treatment allocations may have occurred and also
whether the difference between groups is greater than
can plausibly be attributed to chance. One point is
awarded if the primary outcome of interest is reported,
with another point awarded for the reporting of at least
one secondary outcome for a total of 2 points.
Point measures and measures of variability for
all reported outcome measures – TESTEX
Criterion 9
Point estimates (often P values) of treatment effect
only provide limited information about the outcomes
of treatment and control groups. A more comprehensive
approach is to also provide measures of variability. We
suggest, however, that this is extended to all reported
outcome measures to avoid selective outcome report-
ing. We award 1 point for this criterion.
Activity monitoring in control groups – TESTEX
Criterion 10
The largest trial of exercise training in heart failure to
date (HF-ACTION)18 suggested that one of the reasons
for a lower than expected post-intervention difference
between groups occurred because approximately 30%
of patients allocated to sedentary control undertook
exercise training privately. It is therefore recommended
that a robust study design quantifies this by making
some provision for measuring activity levels in control
patients to avoid crossover to exercise. This may be a
simple method, such as providing patients with an
activity diary or, more advanced, by providing acceler-
ometry or heart rate monitoring devices to assess poten-
tial contamination of control group by monitoring
physical activity behaviour. We award 1 point for any
reporting on results of activity monitoring in sedentary
control participants.
Relative exercise intensity remained constant –
TESTEX Criterion 11
As patients adapt to exercise training, if the workload is
kept constant, then the relative exercise intensity will
continually fall as patients improve their physical work
15
 NA Smart et al.

capacity. Therefore, periodic assessments of work Reliability study

capacity (using either rate of perceived exertion, heart
rate reserve or percentage maximum heart or peak VO2)
should be undertaken at least once during the training
programme and the exercise load titrated accordingly.
Any drift in exercise intensity means that adaptations will
be attenuated or even plateau after 3–4 weeks and also
that any conclusions based on expected magnitude of
adaptation in relation to intensity will be difficult to
examine. We award 1 point for periodic adjustment
for the purpose of keeping relative exercise intensity
constant.
Exercise volume characteristics and energy
expenditure – TESTEX Criterion 12
Often there is a dose-response relationship between
exercise and clinical improvement although clinical
markers respond differently to different types and doses
of exercise. Adequate reporting should allow the reader
to calculate, rather than estimate, the volume of exercise
in terms of energy expended during the programme of
exercise. To this end, the exercise parameters; type
(aerobic/resistance/combined), session and programme
duration, session frequency, exercise training intensity
and modality should be clearly reported so each exercise
prescription can be evaluated if it meets the therapeutic
threshold or not. One point is awarded if all exercise
characteristics are reported adequately (i.e. intensity,
frequency, mode, duration of session and duration of
the intervention) and the exercise volume and energy
expenditure can be evaluated.
As presented in Table 3, the agreement (K) between
observers 1 and 2 ranged between 0.44 (moderate) and
1.00 (almost perfect). The moderate agreements
occurred in the categories of ‘Groups similar at baseline’
and ‘Study withdrawals below 15%’. The almost perfect
(or constant values 100% agreement) agreements
occurred in 8 of the 15 categories (53.3%) and the
substantial agreements in 5 (33.3%). Observers 2 and
3 achieved almost perfect agreement (or constant agree-
ment) in 7 of the 15 categories (46.6%), substantial
agreement in 5 of the 15 categories (33.3%), with
3 categories (20%) reaching moderate agreement.
Observers 1 and 3 achieved almost perfect agreement (or
constant agreement) in 11 of the 15 categories (73.3%),
substantial agreement in 2 categories (13.3%), and
moderate agreement in the remainder (13.3%).
There were no systematic differences between the
summated TESTEX scores of each observer [H(2) ¼ 0.392,
P ¼ 0.822]. There was a significant association between
the summated TESTEX scores of the three reviewers, with
almost perfect agreement between observers 1 and 2
(ICC ¼ 0.93, 95% CI 0.82–0.97, P < 0.001), observers 1 and
3 (ICC ¼ 0.96, 95% CI 0.89–0.98, P < 0.001) and observers
2 and 3 (ICC ¼ 0.91, 95% CI 0.75–0.96, P < 0.001).
Discussion
Building on the widely used PEDro and JADAD scales, we
have developed the TESTEX scale as an exercise science-
specific scale, designed for use by exercise specialists to
assess the quality of randomized controlled trials of

Table 3. Inter-observer reliability (Kappa  SE) between the three expert reviewers using the 15-point
TESTEX criteria
TESTEX criteria Observer 1 vs. Observer 2 vs. Observer 1 vs.
observer 2 observer 3 observer 3
Eligibility criteria included Constant Constant Constant
Randomization method stated 1.00 (0.00)f 1.00 (0.00)f 1.00 (0.00)f
Allocation concealment Constant Constant Constant
Groups similar at baseline 0.48 (0.24)d 0.61 (0.23)e 0.32 (0.30)d
Assessor blinded 0.69 (0.16)e 0.69 (0.16)e 1.00 (0.00)f
Study withdrawals <15% 0.44 (0.33)d 0.41 (0.21)d 0.41 (0.21)d
Adverse events reported 0.65 (0.18)e 0.77 (0.15)e 0.88 (0.11)f
Session attendance reported 0.63 (0.18)e 0.52 (0.20)d 0.86 (0.14)f
Intention-to-treat analysis 1.00 (0.00)f 0.61 (0.24)e 0.61 (0.24)e
Between-group primary analysis Constant Constant Constant
Between-group secondary analysis Constant Constant Constant
Point measures for all outcomes Constant Constant Constant
Activity monitoring controls 0.77 (0.14)e 0.51 (0.21)d 0.77 (0.14)e
Relative exercise intensity adjusted 0.73 (0.17)e 0.73 (0.17)e 1.00 (0.00)f
Exercise energy expenditure information reported 0.83 (0.17)f 0.83 (0.17)f 1.00 (0.00)f
17
Superscript letters denote the following level of agreement between observers : a, poor; b, slight; c, fair; d, moderate; e, substantial; f, almost perfect.
Constant ¼ 100% agreement between observers, preventing kappa analysis.

16 International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute

©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.

exercise training in clinical populations. The TESTEX scale
uses 12 criteria, with some criteria scoring more than one
possible point, for a maximum score of 15 points, as
compared to the PEDro scale, which uses 11 criteria for a
maximum score of 10 points.
In contrast to the PEDro scale, the TESTEX scale takes
eligibility criteria into account. Both scales award 1 point
for the concealment of allocation. Subsequent blinding
of patients and therapists is nearly always unachievable
in exercise training studies and does not attract any
points. The TESTEX scale expands on outcome measures
including reports on study withdrawals, session attend-
ance, intention-to-treat (ITT) analyses, reporting seden-
tary control crossover to exercise, periodic adjustment of
exercise load so that intensity remains constant, adverse
events and measurement errors, as well as description of
exercise characteristics which allows a calculation of
exercise volume and energy expenditure. We acknowl-
edge that exercise training studies do not lend them-
selves to participant blinding and this may introduce
measurement error.
The inter-observer reliability of the 15 different
TESTEX items ranged between moderate and almost
perfect agreement, suggesting that observers, varying
in experience, can reach an acceptable level of reliability
without specific training or familiarization. These find-
ings reflect the clarity of our descriptions for each item of
the TESTEX and are consistent with those reported
for the PEDro scale.7 The moderate agreements occurred
consistently for the criteria of ‘Study withdrawals less
than 15%’. Follow-up interviews revealed some minor
oversights among the observers, whereby points were
awarded to studies reporting withdrawals of only the
treatment groups, and not the control groups. Given that
this is clearly stated in the TESTEX criteria, and only minor
disagreements were observed, there appears to be
limited threat to the reliability of each of the TESTEX
items.
The observers also agreed, almost perfectly, on the
summated TESTEX score, with ICCs ranging from 0.91
to 0.96. In a sample of 19 studies, the typical difference
between observers’ summated TESTEX scores ranged
from 1 to 2 points and was not systematically differ-
ent. We applied the TESTEX criteria to a previous meta-
analysis performed by some of the current authors in
the area of exercise intervention, accounting for a
worst-case error of 2 points. An error of 2 points
would not have resulted in the exclusion of research
papers from the meta-analysis and would not have
altered the conclusions of the study. On this basis,
the worst case error of the summated TESTEX can be
tolerated.
International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute
©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.
METHODOLOGY PAPER
We feel that the TESTEX scale with the newly intro-
duced criteria addresses common shortcomings in study
design, quality and reporting in the exercise sciences. We
are therefore confident the TESTEX scale will improve
study design and reporting, thus qualifying inferences
and conclusions in the exercise sciences.
Conclusion
The TESTEX scale is a new reliable tool, specific to
exercise scientists, that facilitates a comprehensive
review of exercise training trials.
Acknowledgements
The authors would like to acknowledge the researchers
who assisted with the review procedures.
The authors report no conflicts of interest.
References
1. Batterham A, George KP. Reliability in evidence-based
clinical practice: a primer for allied health professionals.
Phys Ther Sport 2003; 4: 122–8.
2. Antes G. The new CONSORT statement. Br Med J 2010; 340:
c1432.
3. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of
reports of randomized clinical trials: is blinding necessary?
Control Clin Trials 1996; 17: 1–12.
4. Higgins JPT, Green S (Eds): Cochrane handbook for system-
atic reviews of interventions Version 5.1.0 [updated March
2011]. The Cochrane Collaboration; 2011. www.cochrane-
handbook.org. [Accessed 31 July 2014]
5. Hartling L, Hamm MP, Milne A, et al. Testing the risk of bias
tool showed low reliability between individual reviewers
and across consensus assessments of reviewer pairs. J Clin
Epidemiol 2012; 66: 973–81.
6. Verhagen AP, de Vet HC, de Bie RA, et al. The Delphi list: a
criteria list for quality assessment of randomized clinical
trials for conducting systematic reviews developed by
Delphi consensus. J Clin Epidemiol 1995; 51: 1235–41.
7. Maher CG, Sherrington C, Herbert RD, et al. Reliability of the
PEDro scale for rating quality of randomized controlled
trials. Phys Ther 2003; 83: 713–21.
8. Cornelissen VA, Buys R, Smart NA. Endurance exercise
beneficially affects ambulatory blood pressure: a syste-
matic review and meta-analysis. J Hypertens 2013; 31:
639–48.
9. Smart N, Meyer T, Butterfield J, et al. Individual patient
meta-analysis of exercise training effects on systemic brain
natriuretic peptide expression in heart failure. Eur J Prev
Cardiol 2012; 19: 428–35.
10. Smart NA, Dieberg G, Giallauria F. Functional electrical
stimulation for chronic heart failure: a meta-analysis. Int J
Cardiol 2013; 167: 80–6.
11. Cornelissen VA, Smart NA. Exercise training for blood
pressure: a systematic review and meta-analysis. J Am Heart
Assoc 2013; 2: e004473.
17
 NA Smart et al.

12. Ismail H, McFarlane JR, Nojoumian AH, et al. Clinical out-
comes and cardiovascular responses to different exercise
training intensities in patients with heart failure: a system-
atic review and meta-analysis. J Am Coll Cardiol Heart Fail
2013; 1: 514–22.
13. Ismail H, McFarlane J, Smart NA. Is exercise training
beneficial for heart failure patients taking beta-adrenergic
blockers? A systematic review and meta-analysis. Congest
Heart Fail 2013; 19: 61–9.
14. Smart NA, Dieberg G, Giallauria F. Intermittent versus
continuous exercise training in chronic heart failure: a
meta-analysis. Int J Cardiol 2013; 166: 352–8.
15. Smart NA, Giallauria F, Dieberg G. Efficacy of inspiratory
muscle training in chronic heart failure patients: a systematic
review and meta-analysis. Int J Cardiol 2013; 167: 1502–7.
16. Fleiss J, Cohen J. The equivalence of weighted kappa and
the interclass correlation coefficient as measures of
reliability. Educ Psychol Measurement 1973; 33: 613–9.
17. Landis JR, Koch GG. The measurement of observer agree-
ment for categorical data. Biometrics 1977; 33: 159–74.
18. O’Connor CM, Whellan DJ, Lee KL, et al. Efficacy and safety
of exercise training in patients with chronic heart failure:
HF-ACTION randomized controlled trial. J Am Med Assoc
2009; 301: 1439–50.

18 International Journal of Evidence-Based Healthcare ß 2015 University of Adelaide, Joanna Briggs Institute

©2015 University of Adelaide, Joanna Briggs Institute. Unauthorized reproduction of this article is prohibited.