STATISTICS AND RESEARCH DESIGN
Does more or new data mean that a
nonsignificant result will become
significant?
Nikolaos Pandis, Associate Editor of Statistics and Research Design
Bern, Switzerland
I
t is common in the orthodontic literature to interpret
study findings based solely on the P value even
though this is an arbitrary threshold. The P value is
a surprise index and indicates the probability of
observing the results at hand under the null hypothesis.
For example, under the assumption that there is no dif-
ference between 2 methods in the time required to finish
orthodontic alignment, the P value gives the probability
of observing a difference of, say, 1, 2, or 5 months (or a
more extreme one) between treatment A and B. Depend-
ing on the difference in time to complete the alignment
under the 2 interventions, the number of patients, and
the variability of the data, a P value is calculated that in-
dicates how likely the observed results would be under
the assumption of no difference. A large difference
such as 5 months is more likely to command a smaller
P value and thus a statistically significant result,
compared with a smaller difference such as 1 or
2 months, with everything else being equal.
If the P value is very small or very large, there is less
controversy in stating a significant or a nonsignificant
result; however, caution should be exercised because
significance does not necessarily indicate clinically
important.1 The distinction between statistical signifi-
cance and no significance becomes less clear if the calcu-
lated P value is just above the 0.05 P value threshold, as
it allows room for exploitation during the interpretation
and by assuming that the P value is moving to the direc-
tion of statistical significance (toward P \0.05).
In the literature, it is common to encounter misinter-
pretation and/or overinterpretation of study results in
the presence of nonstatistical significance, and this has
been considered as a form of research spin.2 For
example, we can see statements such as “Statistical sig-
nificance was not reached, but there was a trend for A to
be better than B,” or “., we do not know whether a
larger sample size would have affected the statistically
nonsignificant findings differently.”3
Am J Orthod Dentofacial Orthop 2020;158:150-1
0889-5406/$36.00
Ó 2020 by the American Association of Orthodontists. All rights reserved.
https://doi.org/10.1016/j.ajodo.2020.04.015
150
Are the above statements correct? No, because there
is no guarantee that if we add more patients, the differ-
ence between treatments A and B will stay the same, and
it will not become smaller and, in turn, less significant. In
cases where the difference remains the same or even be-
comes larger, adding more patients will decrease the P
value if the variability of the data does not change; how-
ever, this is often not how it works out.
In addition, planning a new study to repeat a previ-
ous smaller study (with a borderline significant result)
with a similar design and a larger sample size will not
guarantee the same result as the smaller study and/or
it will not guarantee a P value smaller than 0.05.
To determine if claims like the above are justifiable,
Wood et al4 calculated the probability of the P value
moving away from the statistical significance threshold
(0.05) and toward nonsignificance under different sce-
narios. They found that in a study in which the P value
was 0.06, the percentages of times that we would expect
a P value larger than 0.06 (less significant) were 38.6 %,
34.4%, 27.5 %, and 21.8% when increasing the sample
size by 10%, 20%, 50%, and 100%, respectively. There-
fore, the probability of not reaching statistical signifi-
cance by adding more patients given the P value of
0.06 is substantial and does not corroborate with the
claims. In addition, they examined the probability to
expect a nonsignificant result of a new repeat experi-
ment, which is independently analyzed, with the same
number of patients. If the first experiment found statis-
tical significance anywhere from 0.001 to 0.05, the per-
centage of times for the repeat experiment to find
nonsignificant results was expected to range from
17.3% to 50.0%, respectively. For P values between
0.06 and 0.15 in the first experiment, the percentage
of times for the repeat experiment to find nonsignificant
results was expected to be from 50.2% to 64.4%, respec-
tively. Therefore, there is a high probability that the
repeat experiment will not confirm the claims of signif-
icance that was supposedly missed in the first, smaller
study.
In conclusion, caution should be exercised when in-
terpreting results close to the threshold of statistical
Statistics and Research Design
significance because additional patients or a larger
repeat study are not a guarantee for a statistically signif-
icant result.
REFERENCES
1. Wasserstein RL, Lazar NA. The ASA statement on p-values: context,
process, and purpose. Am Stat 2016;70:129-33.
2. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and
interpretation of randomized controlled trials with statistically
American Journal of Orthodontics and Dentofacial Orthopedics
151
nonsignificant results for primary outcomes. JAMA 2010;303:
2058-64.
3. Bj€orksved M, Arnrup K, Lindsten R, Magnusson A, Sundell AL,
Gustafsson A, et al. Closed vs open surgical exposure of palatally
displaced canines: surgery time, postoperative complications, and
patients' perceptions: a multicentre, randomized, controlled trial.
Eur J Orthod 2018;40:626-35.
4. Wood J, Freemantle N, King M, Nazareth I. Trap of trends to statis-
tical significance: likelihood of near significant P value becoming
more significant with extra data. BMJ 2014;348:g2215.
July 2020  Vol 158  Issue 1