ISSN 1062-3590, Biology Bulletin, 2019, Vol. 46, No. 11, pp. 1449–1457. © Pleiades Publishing, Inc., 2019.

Russian Text © The Author(s), 2018, published in Radiatsionnaya Biologiya. Radioekologiya, 2018, Vol. 58, No. 5, pp. 453–462.

METHODOLOGY
OF SCIENTIFIC SEARCH

Redefining the Critical Value of Significance Level

(0.005 instead of 0.05): The Bayes Trace
A. V. Rubanovich*
Vavilov Institute of General Genetics, Russian Academy of Sciences, Moscow, 119991 Russia
*e-mail: rubanovich@vigg.ru
Received April 2, 2018

Abstract—In 2017, a group of the leading mathematical statisticians published a paper-manifesto having an
extremely simple sense: the common critical level of p-values should be decreased by an order of magnitude
(0.005 instead of 0.05) (Benjamin, et al., 2017). In this review, the arguments of proponents and opponents
of this proposal are discussed. Moreover, the problems related to the “reproducibility crisis” of the scientific
results are considered. The corresponding argumentation cannot be understood without consideration of the
fundamentals of the theory of statistical derivation. In this connection, the precise sense of some concepts,
such as p-value, the Bayes factor, and the minimum a posteriori probability of the zero hypothesis are dis-
cussed in the review. This is made mainly with the examples related to the comparison of frequencies. It was
shown that, when using p-values, particular attention should be paid to the comparison of low frequencies on
the highly abundant samples. Some practical recommendations on application of the Bayes analysis are
given.

Keywords: p-value, critical level of significance, Bayes factor, reproducibility crisis

DOI: 10.1134/S1062359019110086

INTRODUCTION ments of proponents and opponents of lowering the

In summer 2017, a large group of leading mathemati-
cal statisticians (D.J. Benjamin, J. Berger, S. Goodman,
J. Ioannidis, D. Moore, T. Sellke, et al.) published a
small preprint-manifesto entitled “Redefine statistical
significance” (RSS) on the site PsyArXiv.com. The
sense of this paper is extremely simple: the common
critical level of p-values should be reduced by an order
of magnitude (0.005 instead of 0.05). In addition to
statisticians, the group of the authors of RSS includes
the well-known researchers in various fields, including
economics, sociology, physiology, anthropology,
medicine, epidemiology, ecology, and philosophy
(a total of 72 persons). This indicates a strong support
of this innovation by the scientific community. In Jan-
uary 2018, this document was reprinted in Nature
Human Behaviour [1].
It is not surprising that the manifesto of RSS
caused a strong response in scientific journals, online
publications, and mass media. The revision of the crit-
ical level of significance affects the interests of all
researchers, including radiobiologists, whose discus-
sions around the “low-doses,” “thresholds,” or, say,
the effects of electromagnetic radiation are largely
reduced to arguments about the statistical significance
of discovered phenomena.
In this review, we present a compilation of
responses to the RSS manifesto and discuss the argu-
critical significance level. The corresponding argu-
ment cannot be understood without considering the
foundations of the statistical inference theory. In this
connection, we will discuss the precise meaning of
concepts such as the p-value, the Bayes factor, and the
minimum a posteriori probability of the null hypothe-
sis. In general, this will be done using the examples
associated with the comparison of frequencies. We
demonstrate that, when using p-values, special atten-
tion should be given to the comparison of low frequen-
cies at large samples. Some practical recommenda-
tions on the use of Bayesian analysis will be given.
1. ARGUMENTS OF THE AUTHORS
OF THE RSS MANIFESTO
Pros of the revision of the critical level of signifi-
cance in the RSS manifesto can be mostly reduced to
the following two provisions.
1. Lowering the critical level of p-value will signifi-
cantly reduce the proportion of publications with false
results. This provision is based on the simple reasoning
that first appeared in the famous essay by John Ioan-
nidis “Why most published research findings are
false” [2]. Suppose we analyze 1000 hypotheses, of
which only 10 are valid (1%) and 990 are wrong. At
a standard test power (80%), we have 10 × 0.8 = 8 con-
firmed correct hypotheses. At the same time, at a sig-

1449
1450 RUBANOVICH
Table 1. Minimum Bayes factor (local minimum for the pri-
ors in H0) and the minimum probability H0 corresponding
to the p-value from the “gray zone”
p-value min H 0 BF min(H0|data)
0.05 0.407 0.289
0.04 0.350 0.259
0.03 0.286 0.222
0.02 0.213 0.175
0.01 0.125 0.111
0.005 0.072 0.067
nificance level of 0.05, the number of cases of confir-
mation of false hypotheses will be 990 × 0.05 ≈ 50.
Thus, the percentage of published false results will be
50/(8 + 50) = 86.2%. It is clear that, by lowering the
critical level of p-value by an order of magnitude, we
can achieve a significant reduction in the proportion
of false positives: 5/(8 + 5) = 38.5%.
2. Effects at p-values close to 0.05 cannot be statis-
tically significant because this is contrary to the results
of “Bayesian analysis,” which will be discussed in Sec-
tion 5. Now we will only mention the following fairly
general statement. Suppose that, before the experi-
ment, we assumed that our chances of success were 50
to 50. In other words, the a priori probability of the
null hypothesis was equal to 50%. Then, after the
experiment, which showed a p-value = 0.05, the prob-
ability of the null hypothesis was at least 29% (see
Table 1). This rigorous mathematical statement,
obtained by using the Bayesian approach, killed
enthusiasm for considering the effects at p-value =
0.05 as statistically significant. Indeed, our experiment
did not make the null hypothesis improbable (say, at a
level of 5%, as it is often considered by a naive user). It
still holds at least 29% or higher. For comparison, in
the same situation at p-value = 0.005, the minimum a
posteriori probability of the null hypothesis is 6.7%
(Table 1).
The authors of RSS claimed that their performance
was dictated primarily by the concern of the extremely
low reproducibility of biological, medical, and other
scientific research. Indeed, the “reproducibility cri-
sis” literally amazed the science of the 21st century,
which was repeatedly discussed in mass media and sci-
entific publications (see, e.g., [2, 3]). By proposing to
lower the critical level of p-values by an order of mag-
nitude, the authors of RSS postulate that this simple
step will immediately improve the reproducibility of
the results of research in many fields. Subsequently,
this phrase has become the main target of a flurry of
critical attacks by statisticians and active experiment-
ers, although the authors of RSS emphasized that the
dichotomy of p < 0.05 is not the only cause of the low
reproducibility of research.
It should also be noted that the authors of RSS pro-
posed to use the threshold value of 0.005 only for the
newly discovered effects, maintaining the critical level
of 0.05 for repeated (verifying) tests. In general, the
authors of RSS propose to no longer consider the
results with p-values from the range (0.005–0.05) sta-
tistically significant and define them as “suggestive”
(setting one thinking).
Of course, the manifesto of RSS is not the first
attempt to revise Fisher’s critical significance level.
Even in the middle of the last century, academician
A.N. Kolmogorov, referring to the rule of “three sig-
mas,” repeatedly proposed to use the critical level of
0.003 or even 0.001 [4]. In the 1960s, the lowering of
the critical level of p-values to 0.01 was passionately
advocated by A. Melton, Chief Editor of Journal of
Experimental Social Psychology [5]. A notable step in
rethinking the role of p-values in the scientific search
was the statement of the Board of American Statistical
Association (ASA) in 2016 [6]. However, the publica-
tion of RSS had the greatest response in the scientific
world. In contrast to the earlier statements, RSS lays
down specific and very significant changes that can be
easily implemented by editors of scientific journals
and funding institutions.
2. “ALPHA-WARS” IN 2017
The RSS manifesto has caused an unprecedented
(in terms of scale) discussion in blogs and scientific
publications, which was called the “alpha-war.”
Immediately after the publication of the manifesto,
a young Dutch physiologist Daniel Lakens in his blog
announced a subscription among the opponents of the
revision of the critical level of p < 0.05. As a result, in
September 2017, the preprint “Justify your alpha” was
published, which was signed by 87 coworkers [7].
According to the authors of this publication, the
threshold p < 0.005 is as arbitrary as p < 0.05. The
threshold cannot be generally fixed and should depend
on what is already known about the subject of research
and the risks associated with obtaining an incorrect
answer. A relatively high probability of false positive
results can be taken in a preliminary study, whereas the
final test of a drug may require lower p-values.
In addition, the authors of [7] first figured out the
cost of the lowering of the critical p-value level by an
order of magnitude: to maintain the accepted testing
power, the size of samples should be increased, on
average, by 70%. Such a requirement may be too much
for budgets of many research groups.
With respect to overcoming the “reproducibility
crisis,” the authors of [7] noted that, according to [8],
among the results at p-values from the range (0.005–
0.05), the proportion of the confirmed results was
24%. However, among the results at p-value < 0.005,
the proportion of the confirmed results was also not
BIOLOGY BULLETIN Vol. 46 No. 11 2019
 REDEFINING THE CRITICAL VALUE OF SIGNIFICANCE LEVEL
too high (49%), which does not correspond to the
expectations of the authors of RSS.
The next prominent paper entitled “Abandon sta-
tistical significance,” one of the authors of which is
Andre Gelman, a well-known expert in Bayesian anal-
ysis, was published in September 2017 [9]. The authors
of this article recommend to completely abandon test-
ing the null hypothesis significance and consider the
p-value as only one of many information indices with-
out a privileged role in making decisions about the sig-
nificance of a phenomenon and the possibility of its
publishing. With regard to classifying research into
new (p < 0.005) and repeated (p < 0.05) ones, the
authors of [9] believe that this recommendation is
quite impractical, especially in the fields where
research is gradual and cumulative. And further, since
they (the authors of RSS) are not able to determine
what a new effect is, the proposed policy will lead to
inconsistency in the practice of reproducing results.
Similarly to many subsequent commentators [10–
14], the authors of [9] believe that RSS virtually has no
evidence that p < 0.05 is one of the leading causes of
the poor reproducibility of scientific research. In opin-
ion of the authors of [9], the leading cause is the
absence of correction for multiple comparisons (both
actual and potential), which has become the norm in
applied research. In [13], it is also emphasized that
lowering the threshold of p-values will lead to a drastic
increase in the publication bias, i.e., to a shift of the
pool of published works towards the greater effects.
The harshest statements against RSS were in the work
by Harry Crane “Why “redefining statistical signifi-
cance” will not improve reproducibility and could
make the replication crisis worse,” which was pub-
lished in November 2017 [14]. Crane regards the pro-
posal of RSS as extremely erroneous, presented under
false pretenses, and supported by flawed analysis and
suggests that it should not be accepted. H. Crane also
reports that, according to his calculations, there are
several possible scenarios in which the cutoff p < 0.005
will make the situation with reproducibility even
worse.
3. RSS AND REPRODUCIBILITY CRISIS:
p-HACKING
Deborah Mayo, a well-known statistician and phi-
losopher of science, wrote in her blog errorstatis-
tics.com that almost everyone knows what is the real
cause of nonreproducibility—selective publication of
the most effective results, ignoring the multiplicity of
tests, and sequential testing of hypotheses until one of
them would be significant, as well as selective presen-
tation of methods and results of their application.
In modern biostatistics, the tendency described by
Deborah Mayo is called the “p-hacking” [15]. This
term covers a wide range of variants of artificial over-
BIOLOGY BULLETIN Vol. 46 No. 11 2019
1451
estimation of the statistical significance of results.
These include, for example,
1. A posteriori formulation of hypotheses. A researcher
puts forward hypotheses after obtaining data, stating
that all assumptions preceded the experiment.
2. Incomplete representation of experimental data.
Publication of the most favorable results.
3. Data editing by eliminating the outliers and
grouping subsamples. Persistent fragmentation of
a sample in search for stratification providing “signif-
icant” differences of subsamples.
4. The use of various statistical tests with further
publication of the most favorable results.
5. The use of the stopping-rule effect, i.e., gradu-
ally increasing the sample size until obtaining a signif-
icant result at a level of p < 0.05 [15].
6. Incorrect or incomplete account of the multi-
plicity of comparisons. Incorrect construction of per-
mutation tests.
7. Transforming and normalizing data instead of
using nonparametric statistics.
8. The use of multivariate statistical analysis with-
out proper validation of statistical significance, e.g.,
the selection of predictors using the “Stepwise”
regression analysis algorithm [16].
9. The use of total estimates of risk factors selected
from a large number of predictors [16].
10. The use of a test sample for “clarification” of
the results obtained using a training sample, which is a
very common mistake in a two-stage (discovery set +
validation set) search for effective predictors [16, 17].
The “effectiveness” of all these approaches,
indeed, little depends on the critical level of signifi-
cance. For example, manipulations using items 8–10
make it possible to overcome thresholds p < 10–5 or
even 10–10 [16]. For this reason, the majority of critics
of RSS believe that the revision of the threshold p-value
will not result in a significant improvement to the
reproducibility of results. The author of these lines
basically shares this belief. Throughout the past cen-
tury, the threshold value of 0.05 was used everywhere,
but there were no talks about the “reproducibility cri-
sis.” The situation has changed at the turn of the cen-
tury with the advent of new technologies, leading to
experimental plans with a huge number of indepen-
dent, predictor, or grouping variables (e.g., microar-
rays, GWAS, RNA-seq, various omics-technologies,
etc.). Processing corresponding data inevitably leads
to frequent misuse of points 6–10 of the above list of
p-hacking manifestations and, ultimately, to a low repro-
ducibility of announced effects. According to some data,
the false discovery rate (FDR) due to p-hacking is cur-
rently no less than 60% [6, 15].
1452 RUBANOVICH
4. LOGICS OF p-VALUES
Next, we proceed to the discussion of the key con-
cepts of the statistical inference theory, which are
absolutely necessary to fully understand the argu-
ments put forward by the authors of RSS.
The largest number of errors and false interpreta-
tions is directly related to the concept of “p-value”
itself [18]. What a p-value actually is and what it is not?
Unfortunately, many users believe that a p-value is
equal to the probability of the null hypothesis of no
effects. Of course, this is not valid: p-value ≠ P(H0).
Moreover, p-value ≠ P(H0|data); i.e., p-value is not
equal to the conditional probability of the null hypoth-
esis for the observed scenario for data (data). In the
definition of p-value, the “transposed” conditional
probability appears:
p-value = P ( data H 0 ) . (1)
Strictly speaking, definition (1) is also not quite
correct. In defining p-value, data should be under-
stood not only as the obtained scenario of data but also
as the aggregate of all scenarios with an even more
extreme deviation from the null hypothesis. However,
we will omit this fact so as not to clutter the presenta-
tion.
Thus, the p-value is the conditional probability of
the observed data, provided that the null hypothesis is
valid, rather than the conditional probability of the
null hypothesis itself. This probability is very indi-
rectly related to the null hypothesis. We compute
p-value = P(data|H0); however, to make a decision
about the statistical significance, we need the trans-
posed probability P(H0|data). This situation is similar
to that which occurs constantly in the studies with the
“cases–controls” design: we estimate a marker fre-
quency among patients but need primarily the esti-
mates of the frequency of patients among the marker
carriers.
Typically, judging by small p-values, an experi-
menter concludes that the alternative hypothesis Н1
(the presence of effect) is valid. This conclusion is not
always correct. The fact is that р-value = P(data|H0)
may be small, but the P(data|H1) value may be even
smaller. This possibility is well illustrated by the
famous anecdotal example. Let the statement “You
are a woman” be the null hypothesis Н0 and the state-
ment “You are a man” be the alternative hypothesis
Н1. Then, Р(Н0) = Р(Н1) = 1/2. It is known that about
3% of all women living at the moment are pregnant.
That is, assuming data = “Pregnancy,” we have:
P(data|H0) = 0.03. In this case, P(data|H1) ≅ 0 !
P(data|H0). By transposing the common practice of
using p-values to this situation, we have:
p-value = P ( data H 0 ) = 0.03 < 0.05 → H 0
is rejected and H1 is valid, which in this case means:
“Are you pregnant? So you are a man!”
This example clearly illustrates a number of simple
but important provisions to keep in mind when using
p-values:
1. A low p-value is a necessary but not sufficient con-
dition for the validity of the alternative hypothesis Н1.
2. If the probability of a certain event data is small,
this does not necessarily mean that the probability of
an event that is generated by data is also small.
3. In some cases, the problem of adopting the null
hypothesis or the alternative hypotheses should be
solved by comparing the conditional probabilities
P(data|H0) and P(data|H1).
The last provision is fully implemented in the frame-
work of the so-called “Bayesian approach,” in which the
decision about adopting the null hypothesis is made on
the basis of estimates of the Bayes factor (BF):
P (data H 0 )
BF = . (2)
P (data H1)
It is generally accepted that the null hypothesis is
confidently adopted at BF > 3 and is rejected in favor
of the alternative hypothesis at BF < 1/3. The key
argument for an immediate revision of the critical sig-
nificance level of 0.05 is the following observation: at
such p-values, the data obtained are more probable
under the null hypothesis than at the alternative
hypothesis (BF > 1). It is clear that, in this case, it is
quite unacceptable to consider the observed effect sta-
tistically significant. Relevant examples are discussed
in detail in the next section.
5. BAYESIAN APPROACH:
ADVANTAGES, DISADVANTAGES,
AND PRINCIPAL LIMITATIONS
The advantages of the Bayesian approach are quite
obvious and can be reduced to the following provi-
sions.
1. In contrast to the p-value, the Bayes factor BF
has a simple and clear meaning and shows how many
times the observed data are more probable under the
null hypothesis than in the presence of differences.
2. The estimation of BF makes it possible to esti-
mate the a posteriori probability of the null hypothe-
sis, which is impossible in principle when using p-val-
ues. Indeed, according to the formula of Bayes, we
have
P (H 0 data) P (data H 0 ) P (H 0 ) P (H 0 )
= = BF .
P (H1 data) P (data H1) P (H1) P (H1)
This ratio makes it possible to perform a highly desir-
able “transposition” of the conditional probabilities,
i.e., to proceed from P(data|H0) to P(H0|data). For this
purpose, it is usually assumed that, a priori, P(H0) =
P(H1) = 1/2. Then, keeping in mind that P(H1|data) =
1 – P(H0|data), we obtain:
BIOLOGY BULLETIN Vol. 46 No. 11 2019
 REDEFINING THE CRITICAL VALUE OF SIGNIFICANCE LEVEL
BF .
P (H 0 data) = (3)
BF + 1
3. In contrast to p-values, BF is symmetrical with
respect to hypotheses Н0 and Н1. This means that BF >
3 is an argument in favor of hypothesis Н0 and BF <
1/3 is an argument against H0 in favor of H1. For com-
parison: p-value = 0.002 is an argument against H0;
however, p-value = 0.2 is not an argument in favor of
the null hypothesis.
4. BF does not require test power estimates because,
in some sense, it takes into account the type 2 errors. It
can be easily seen that BF has an order of p-value/test
power. At a low test power, the BF value will be obvi-
ously high [19].
5. Numerical experiments showed that the use of BF
almost completely eliminates the stopping rule [19].
6. BF allows naturally taking into account the mul-
tiplicity of comparisons [19, 20].
To implement the Bayesian approach, it is neces-
sary to estimate the ratio of two conditional probabili-
ties—P(data|H0) and P(data|H1). The former is almost
always calculated simply. The corresponding calculations,
in fact, are similar to the calculations of the p-value. The
main difficulties are associated with the estimation of
the conditional probability P(data|H1), for which it is
necessary to assume the distribution pattern of data
provided that the alternative hypothesis H1 is valid.
This a priori and usually unknown distribution is
called prior.
The main highlights associated with calculating the
Bayes factors are illustrated in Fig. 1 using the analysis
of coin tossing (“heads and tails”) as an example. The
null hypothesis is that the frequency of “heads” corre-
sponds to 1/2 (“proper” coin) against the assumption
H1 that the coin is asymmetrical. In case of validity of
hypothesis H1, the probability of the “head” (p) is con-
sidered a random variable with “noninformative”
(e.g., uniform) distribution (all p-values are equiprob-
able). Earlier, this distribution was used by Bayes and
Laplace. Nowadays, the “objectively noninformative”
Jeffreys prior (the so-called arcsine distribution) is
considered more appropriate [21].
Figure 1 shows that the p-values from the “gray
zone” (0.01–0.05) may correspond to relatively high
values of the Bayes factor. For example, the variant
39/100 corresponds to the p-value = 0.035; however,
the Bayesian analysis shows that such data is more
probable under the null hypothesis (BF = 1.1 > 1 for
the Jeffreys prior). If we then proceed to the probabil-
ity of the null hypothesis validity using formula (3), we
obtain P(H0|39/100) = 1.1/2.1 ≈ 0.52. Meanwhile, the
traditional analysis assures us that the differences of
39/100 from 1/2 are statistically significant (p-value =
0.035).
There are more striking examples of such situa-
tions. For example, at a large number of coin tosses,
BIOLOGY BULLETIN Vol. 46 No. 11 2019
1453
the variant 4870/10000 corresponds to the p-value =
0.01. However, the Bayesian analysis shows that, in
this case, the null hypothesis is much more probable:
BF = 4.27 > 3, which corresponds to P(H0|4870/10000) ≈
0.81. The discrepancies between the results of tradi-
tional and Bayesian analyses (Jeffreys–Lindley para-
dox) are particularly noticeable for large samples.
As expected, the estimates of the Bayes factor
essentially depend on the selection of priors. Figure 2
shows the density distributions of five priors differing
in the position of the mean μ value (localization) and
the standard deviation σ. The corresponding BF esti-
mates are shown next in the table. Priors 1 and 2 are
nearly Gaussian distributions for which the mean val-
ues are 0.39 and 1/2, respectively. In the first case, we
will say that the prior is localized in data, whereas in
the second case it is localized in H0. Priors 3 and 4 are
located in H0 and are “noninformative” (see Fig. 1).
The other priors bear certain a priori information
about the distribution of p under condition H1.
Figure 2 shows that, depending on the prior selec-
tion, one may come to the opposite conclusions: prior 1
allows adopting hypothesis H1 (BF = 0.17 < 1/3),
whereas prior 5 leads to the conclusion that the null
hypothesis is valid (BF = 3.85 > 3). It is quite another
matter that both of these priors clearly do not corre-
spond to our situation, for which it is more natural to
use the “noninformative” priors 3 and 4. Neverthe-
less, in many cases the dependence of the Bayes factor
on the selected prior creates principal difficulties. It
can be shown [22] that, irrespective of the prior, the
Bayes factor for large samples (n) increases as
BF ~ nσ. (4)
In essence, this means that, by postulating a fairly
“smeared” prior (largeσ), it is always possible to
obtain high BF values and then confidently adopt the
null hypothesis.
It may seem that this negates any possibility to use
Bayesian analysis in the cases when we do not have a
sufficient basis for prior selection (e.g., on the basis of
preliminary experiments). However, the situation is
partially improved by the fact that, by selecting the
prior, the Bayes factor can be made as large as desired
but cannot be made arbitrarily small. The BF value is
always limited below.
6. MINIMUM BAYES FACTOR
It can be easily seen that the minimum Bayes factor
is achieved for an “acute” prior (σ = 0), which is local-
ized in data. For example, in the case of coin tosses
(Fig. 1), we obtain
min BF = 2−n1 − n2 , (5)
p n1 (1 − p)n2
1454 RUBANOVICH

Bayesian coin toss analysis

H0: Frequency of “heads” = 1/2 data = {n , n }—the number of “heads” and “tails”
1 2
H1: Frequency of “heads” ≠ 1/2
n n n2
data probability at H0 P(data|H0) = C n11 + n2(1/2) 1 (1/2)

n n n2
data probability at H1 P(data|H1) = C n11+ n2 p 1(1 – p)

2(n1 + n2) Unknown random variable

BF = 1 with noninformative distribution
∫0 pn1(1 – p)n2 f(p)dp f(p) (the so-called prior)

100 tosses: n1/100 vs. 1/2 comparison

Noninformative priors
BF
1. Homogeneous prior n1 p-value
(all p values are equiprobable) homogeneous Jeffreys
prior prior
35 0.004 0.087 0.130
f(p) = 1
36 0.007 0.158 0.236
0 1 37 0.012 0.272 0.411

2. Jeffreys objectively informative prior 38 0.021 0.452 0.686

39 0.035 0.718 1.095
40 0.057 1.095 1.678
1
f(p) = 41 0.089 1.603 2.465
π√p(1 – p)
42 0.133 2.252 3.474
0 1
Fig. 1. The simplest case of estimation of the Bayes factor—testing a coin for symmetry. The algorithm for calculating BF for two
types of “noninformative” priors is shown. In the table, BF values are compared to the p-values that were estimated using two-
tailed exact binomial test. The cases where the Bayesian analysis did not confirm the significance of differences from 1/2 are high-
lighted.

Estimates of the Bayes factor significantly depend on the prior selection

100 coin tosses: five priors when comparing 39/100 vs. 1/2, p-value = 0.035
f(p)
5

4 1 2 Prior Prior
Prior BF localization smearing (σ)
3 1 0.17 data 0.08
2 0.32 H0 0.08
2 3 0.72 H0 0.29
4 1.09 H0 0.35
4
3 5 3.85 H0 0.46
1
5

0 0.2 0.4 0.6 0.8 1.0

data H0 p

Fig. 2. Dependence of BF on the prior selection. Bayes factor estimates for five priors differing in the mean value position (local-
ization) and variance (σ2) are shown.

BIOLOGY BULLETIN Vol. 46 No. 11 2019

 REDEFINING THE CRITICAL VALUE OF SIGNIFICANCE LEVEL
Dependence of BF on the fraction of priors localized in H0 (1)
and priors localized in data (2)
BF
1.5
1.0
0.5
–eplnp
Global
min BF
0.1 0.2
Position of local min BF
Fig. 3. BF dependence on the degree of prior “smearing” (σ2—prior variance): (1) priors localized in H0 and (2) priors localized
in data.
where the p value corresponding to our data is
selected: p = n1/(n1 + n2). A similar formula for com-
parison of the mean values for Student’s t test has a
very simple form [22]:
−1t
2
min BF = e 2
. −ep ln p
This minimum, corresponding to the “acute” prior in
data, we will call the global minimum, because any
other prior will yield a higher BF value.
For some classes of priors, there may be local min-
ima, which are naturally always higher than the global
min BF. For example, for unimodal priors localized in
H0, there is always a local minimum BF, which can be
found numerically. For this minimum, a good approx-
imation is known, which was proposed by T. Sellke
[22]: for a broad class of unimodal priors in which the
mean value is localized in H0, the local minimum of
the Bayes factor is approximately equal to
min H 0 BF ≈ −ep ln p.
Here (and only here), p ≡ p-value < 1/e, and e is the
base of the natural logarithm.
The pattern of the dependence of BF on the degree
of “smearing” of prior (σ) depends on the position of
the mean value (i.e., on the prior localization) (Fig. 3).
For the priors localized in data, BF increases mono-
tonically with increasing σ starting from the point cor-
responding to the global minimum. A characteristic
feature of the priors localized in H0 is the existence of
a local minimum that is defined by the formula (6). In
this case, for an “acute” prior localized in H0, always
BF = 1. At large σ, for both classes of priors, BF
increases with increasing σ according to (4).
BIOLOGY BULLETIN Vol. 46 No. 11
1455
Comparison of 39/100 vs. 1/2
p-value 0.035
Global min BF 0.087
1
Local min BF 0.317
2
Local min BF according
0.320
to Sellke: –eplnp
BF for Jeffreys prior 1.095
0.3 0.4
σ
The min BF estimate provides a unique opportu-
nity to assess the minimum probability of realization
of the null hypothesis regardless of the selected prior.
It suffices to substitute the lower limit of BF (6) into
the formula (3). As a result, we obtain
P (H 0 data) ≥ , (7)
1 − ep ln p
where p ≡ p-value. This value can be called the “mini-
mum a posteriori probability of the null hypothesis,”
corresponding to a given p-value.
As above, we will consider the comparison 39/100
vs. 1/2 as an example (Fig. 3). The “noninformative”
Jeffreys prior indicates that, in this case, the data are
more probable under the null hypothesis, although
p-value = 0.035. A conservative experimenter may not
agree with this conclusion since BF depends on the
prior. However, according to (7), the minimal a poste-
riori probability of the null hypothesis is min
P(H0|39/100) = 0.24, and this conclusion almost does
(6) not depend on the selected prior.
The recalculation of p-values into the minimum
probabilities of the null hypothesis is shown in Table 1.
In essence, this table is the main argument of the sup-
porters of the revision of the critical significance level.
As can be seen in Table 1, only at p-value = 0.005 an
acceptable level of the minimum a posteriori probabil-
ity of the null hypothesis can be achieved.
7. RISK ZONES IN USING p-VALUES: LOW
FREQUENCIES AND LARGE SAMPLES
In conclusion, we will discuss situation in which
the use of p-values from the “gray zone” (0.01–0.05)
is especially dangerous. Most of these situations are
2019
1456
Table 2. Traditional and Bayesian analyses and compari-
sons 1/n vs. 9/n at different sample sizes (n)
Comparison 1/n vs. 9/n
n hypothesis probability equal to 42% (at an a priori
p-value BF
50 0.016 0.116
100 0.018 0.197
500 0.021 0.504
1000 0.021 0.724
5000 0.021 1.639
10000 0.021 2.322
50000 0.021 5.198
p-Values for two-tailed Fisher’s exact test and Bayes factors for
Jeffreys prior are shown. The cases of rejection of the alternative
hypothesis H1 about the existence of differences are highlighted.
characterized by the formula “low frequencies and
large samples.” In Section 5, we already mentioned
that, for large samples, the results of the traditional
analysis, based on p-values, may contradict the con-
clusions made on the basis of the Bayesian analysis
(Jeffreys–Lindley paradox).
Suppose we compare the frequencies of a certain
event in two samples of equal size, e.g., 1/n vs. 9/n,
where n is the sample size. Then, in calculating the
Bayes factor, it is appropriate to use the “noninforma-
tive” Jeffreys prior, as it was done in coin flipping.
Indeed, under the null hypothesis, each of 1 + 9 = 10
events randomly gets into one of the two samples with
a probability of 1/2. Hence, it is clear that the bilateral
p-value for the comparison of 1/n vs. 9/n is practically
independent of n and asymptotically approaches the
doubled probability of getting less than two “heads” in
ten coin tosses. However, the BF value always
increases with increasing n, as is generally described by
the formula (4).
Table 2 shows the comparison of p-values and
Bayes factors at different sample sizes n. In all cases
presented in Table 2, the frequency of events in the
second group is 9 times higher than in the first group,
and p-value ≈ 0.02. However, at n = 100, the Bayesian
analysis confirms the traditional approach (BF = 0.2 <
1/3), and at n = 50000 is confidently rejects it (BF =
5.2 > 3). Thus, when comparing low frequencies in
large samples, the use of p-values is associated with an
increased risk of obtaining false-positive results.
The situations that are described in Table 2 often
occur in cytogenetics and radiation epidemiology.
Suppose, for example, the comparison of 1000 per-
sons exposed to radiation with a control sample of the
same size showed nine cases of leukemia in the
exposed persons and only one case in the control
group. The relative risk is RR = 9. As a rule, an epide-
miologist in such cases reports a substantial and signif-
icant increase in the incidence of the disease in the
RUBANOVICH
exposed individuals. However, the Bayesian analysis
does not confirm this conclusion: BF = 0.72 > 1/3,
which, according to (3), corresponds to the null
probability of 50%).
In cytogenetic studies, low frequencies for huge
samples are also often compared. This is due to the
malpractice of pooling all viewed metaphases for each
of compared group. As a result, there occur situations
that are described in Table 2. For example, one dicen-
tric per 5000 metaphases was detected in the control
group and nine such chromosomal aberrations per the
same number of cells were detected in the exposed
individuals. In this case, p-value ≈ 0.02 and BF = 1.6;
that is, the observed variant is more probable in the
absence of differences.
CONCLUSIONS
The proposals formulated in the RSS are, probably,
timely, but the scientific community is clearly not ready
to accept them, as follows from the ongoing heated dis-
cussion and the results of the survey conducted in Twitter
by the journal Nature News & Comment. The question
“Should we lower the critical level of p-values” received
562 “yes” and 540 “no.” https://twitter.com/nature-
news/status/890530105554087936.
Of course, lowering the threshold p-value will not
lead to a significant improvement of reproducibility, as
is promised by the authors of RSS. Nevertheless, we
believe that the following recommendations are rele-
vant.
1. It is mandatory to calculate the Bayes factor
when p-values get into the “gray zone” (0.01–0.05).
This can be simply done by using the JASP freeware
[11] or on-line calculators:
https://jasp-stats.org,
http://www.stat.umn.edu/geyer/5102/examp/bayes.
html,
http://pcl.missouri.edu/bf-binomial.
All calculations that are listed in this review can be
reproduced using the specified software.
2. In addition to the BF estimates that are offered by
calculators, the min BF value (according to Sellke) and
the minimum a posteriori probability of the null hypoth-
esis should be estimated using the formulas ((6), (7)).
3. Particular attention should be given to the com-
parison of low frequencies at large samples. In these
situations, the p-values at the level of 0.02–0.05 mean
nothing and lead to false results.
FUNDING
This work was supported by the Russian Foundation for
Basic Research (project no. 16-06-0046517).
BIOLOGY BULLETIN Vol. 46 No. 11 2019
 REDEFINING THE CRITICAL VALUE OF SIGNIFICANCE LEVEL
COMPLIANCE WITH ETHICAL STANDARDS
The author declares that he has no conflict of interest.
This article does not contain any studies involving animals
or human participants performed by the author.
REFERENCES
1. Benjamin, D.J., Berger, J., Johannesson, M., Nosek, B.,
Wagenmakers, E., Berk, R., et al., Redefine statistical
significance, Nat. Hum. Behav., 2018, no. 2, pp. 6–10.
2. Ioannidis, J., Why most published research findings are
false, PLoS Med., 2005, vol. 2. e124.
3. Buck, S., Solving reproducibility, Science, 2015,
vol. 348, no. 6242, p. 1403.
4. Kolmogorov, A.N., Probability theory, in Veroyatnost’ i
matematicheskaya statistika. Entsiklopediya (Probability
and Mathematical Statistics. Encyclopedia), Prokhor-
ov, Yu.V., Editor-in-Chief, Moscow: Bol’shaya Rossi-
iskaya Entsiklopediya, 1999; Moscow: Drofa, 2003,
pp. 874–875.
5. Melton, A.W., Editorial, J. Exp. Psychol., 1962, vol. 64,
pp. 553–557.
6. Wasserstein, R.L. and Lazar, N.A., The ASA’s state-
ment on p-values: context, process, and purpose, Am.
Statistician, 2016, vol. 70, no. 2, pp. 129–133.
7. Lakens, D., Adolfi, F.G., Albers, C.J., Anvari, F.,
Apps, M.A., et al., Justify Your Alpha, 2018. psyarx-
iv.com/9s3y6.
8. Open Science Collaboration, Science. 2015, vol. 349,
no. 6251, pp. 1–8.
9. McShane, B.B., Gal, D., Gelman, A., Robert, C., and
Tackett, J.L., Abandon statistical significance, 2017.
arXiv:1709.07588 [stat.ME].
10. Trafimow, D., Amrhein, V., Areshenkoff, C.N., et al.,
Manipulating the alpha level cannot 1 cure significance
testing. comments on “Redefine statistical signifi-
cance,” PeerJ. Preprints, 2017, vol. 5. e3411v1.
https://peerj.com/preprints/3411/.
BIOLOGY BULLETIN Vol. 46 No. 11 2019
1457
11. Perezgonzalez, J.D. and Frías-Navarro, M.D., Retract
p < 0.005 and propose using JASP, instead, F1000Re-
search, 2017, vol. 6, p. 2122.
12. Amrhein, V. and Greenland, S., Remove, rather than
redefine, statistical significance, Nat. Hum. Behav.,
2018, vol. 2, p. 4.
13. Esarey, J., Replication data for: lowering the threshold
of statistical significance to p < 0.005 to encourage en-
riched theories of politics, Polit. Methodologist, 2017,
vol. 24, no. 2, pp. 13–20. https://thepoliticalmethodol-
ogist.com/v24-n2-fix/.
14. Crane, H., Why “redefining statistical significance”
will not improve reproducibility and could make the
replication crisis worse, 2017. arXiv:1711.07801v1
[stat.AP].
15. Head, M.L., Holman, L., Lanfear, R., Kahn, A.T., and
Jennions, M.D., The extent and consequences of
p-hacking in science, PLoS Biol., 2015, vol. 13, no. 3.
e1002106.
16. Rubanovich, A.V. and Khromov-Borisov, N.N., Ge-
netic risk assessment of the joint effect of several genes:
critical appraisal, Russ. J. Genet., 2016, vol. 52, no. 7,
pp. 757–769.
17. Wray, N.R., Yang, J., Hayes, B.J., et al., Pitfalls of pre-
dicting complex traits from SNPs, Nat. Rev. Genet.,
2013, vol. 14, no. 7, pp. 507–515.
18. Goodman, S., A dirty dozen: twelve p-value miscon-
ceptions, Semin. Hematol., 2008, vol. 45, pp. 135–140.
19. Dienes, Z., How Bayes factors change scientific prac-
tice, J. Math. Psychol., 2016, vol. 72, pp. 78–89.
20. Held, L. and Ott, M., On p-values and Bayes factors,
Annu. Rev. Stat. Appl., 2018, vol. 5, pp. 393–419.
21. Jeffreys, H., An invariant form for the prior probability
in estimation “problems,” Proc. R. Soc. London, Ser. A,
1946, vol. 186, no. 1007, pp. 453–461.
22. Sellke, T., Bayarri, M.J., and Berger, J.O., Calibration
of p values for testing precise null hypotheses, Am. Statist.,
2001, vol. 55, pp. 62–71.
Translated by M. Batrukova