See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/329320009

The nature of the biological material and the irreproducibility problem in

biomedical research

Article  in  The EMBO Journal · January 2019

DOI: 10.15252/embj.2018101011

CITATIONS READS

3 368

1 author:

George Papamokos
University of Ioannina
35 PUBLICATIONS   398 CITATIONS   

SEE PROFILE

All content following this page was uploaded by George Papamokos on 11 June 2019.

The user has requested enhancement of the downloaded file.

Commentary
The nature of the biological material and
the irreproducibility problem in
biomedical research
George V Papamokos1,2,3,4
Biomedical research has a reproducibility
problem since various crucial landmark
papers could not be independently repro-
duced. While there are many causes related
to statistical analysis, methodology or insuf-
ficient reporting of experimental details,
this commentary argues that the complexity
of biological material itself is, until now, a
largely ignored source of irreproducibility.
By discussing examples from evolutionary
biology, intrinsically disordered proteins and
current biomedical research, it contends
that some results are irreproducible because
we do not have the knowledge, the tools or
the analytical ability to understand biologi-
cal complexity and how it can give rise to
different results. Instead of casting irrepro-
ducible research out as bad or sloppy
science, they should serve as an inspiration
for pioneering research not just to develop
such tools but also to attempt to explore
what lies beneath our current inability to
deal with complexity.
The EMBO Journal (2019) e101011
R
eproducibility is a central dogma of
science as expressed by Karl Popper:
“. . . for we have seen that non-repro-
ducible single occurrences are of no signifi-
cance to science”. However, the life sciences
do have an irreproducibility problem for
published research articles: during the past
years, evidence of irreproducible results has
been accumulating, and various method-
ological causes have been identified (Eisner,
2018) and extensively reviewed (see Eisner,
2018 and references therein).
1 Biomedical Division, The Institute of Molecular Biology and Biotechnology, FORTH-ITE, Heraklion, Crete, Greece E-mail: george.papamokos@gmail.com
2 Laboratory of Biological Chemistry, Medical School, University of Ioannina, Ioannina, Greece
3 Laboratory of Soft Matter, Department of Physics, University of Ioannina, Ioannina, Greece
4 Department of Physics and School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA
DOI 10.15252/embj.2018101011
ª 2019 The Author
Yet, the role of the unique nature of
biological material itself has not been
discussed analytically for its potential role in
the irreproducibility problem. I suggest that
irreproducibility may also be a combined
result of the complexity of the biological
material and of our finite ability to deal with
it, provided that all other factors are
addressed (Eisner, 2018). Irreproducibility in
carefully selected experiments can also be
interpreted to the end that our current meth-
ods, theories and techniques are either
applied ipsilaterally or are insufficient to
analyse and describe functions, properties
and unknown or missing parts of the biolog-
ical material.
To support this argument, I will first
present the perception of the science of
biology by Ernst Mayr, one of the leading
evolutionary biologists of the past century
who described the evolutionary process as
unpredictable or, at best, only statistically
predictable (Mayr, 1961). Subsequent recent
advancements either challenge the impossi-
ble or probabilistic approach of repeatability
and predictability at certain levels, or
suggest that predictable biochemical
changes in evolutionary biology do not have
a predictable molecular basis. A second
argument is the discovery of a whole new
kind of proteins, intrinsically disordered
proteins (IDPs) (Dyson & Wright, 2005), that
are key factors in health, disease, evolution
and now possible drug targets. Lastly, two
key examples at the level of eukaryotic cells
(Hines William et al, 2014) and at the organ-
ismal level (Lithgow et al, 2017) that were
unable to account for different states
adopted by biological material upon
infinitesimal changes in the microenviron-
ment which support that this is an opportu-
nity to produce new science. I will also
introduce methodology bias as a new term
for the discussion of reproducibility.
Perception of biology by Ernst Mayr
Ernst Mayr, in his classic work “The Growth
of Biological Thought”, epitomized the
philosophy of research in biology. He argued
that the complexity of biological systems
inevitably places biology beyond the limits
of the natural sciences as the diversity of
biological systems is based on rules that are
substantially different from the laws of
physics. Mayr believed that these rules
cannot be described by methodological
reductionism and can only approached prob-
abilistically, and that mathematics, although
fundamental in physics, can contribute less
to biology.
In his landmark paper Cause and Effect
in Biology (Mayr, 1961), Mayr described
biology as consisting of two different schools
of thought: functional biology aims to
answer the question “How?” while evolu-
tionary biology attempts to answer “Why?”
To this end, he used a warbler on his
summer place in New Hampshire as an
example to distinguish between proximate
causes that are linked to functional biology
and ultimate causes linked to evolutionary
biology. Thus, the physiological condition of
the bird—its sensing of photoperiodicity and
temperature—is the proximate cause of
migration, while the reduced availability of
The EMBO Journal e101011 | 2019 1 of 4
The EMBO Journal
food during winter and its genetic disposi-
tion are the ultimate causes of migration.
In the same paper, Mayr concluded that
“. . . causality in biological systems is not
predictive or at best is only statistically
predictive”.
Two years earlier, at the 1959 Cold
Spring Harbor Symposium on Quantitative
Biology, he had questioned not only the
contribution of mathematics to biology, but
also argued that applying mathematics in
evolution is fundamentally and methodolog-
ically wrong: “I am impressed by the
uniqueness, by the unpredictability, and by
the unrepeatability of evolutionary events.
Let me end this discussion with the provoca-
tive question: Is it not perhaps a basic error
of methodology to apply such a generalizing
technique as mathematics to a field of
unique events, as is organic evolution?”
Eventually, his claim that the “generalizing
technique” of mathematics is not applicable to
biology and evolution was refuted by other
scientists (Crow, 2009). Today, fields such as
bioinformatics, computational biology or
molecular modelling indeed show that mathe-
matics can be applied to analyse and under-
stand biological evolution; during Mayr’s time,
it was the human inability to perform highly
complex calculations and the lack of suffi-
ciently powerful computers that prevented the
application of sophisticated mathematical
modelling in biology. By way of example,
computational phylogenetics, which is a part
of systematics, Mayr’s field, demonstrates
how human limitations were enhanced by
computer science—software and hardware.
More generally, computational science
enhanced our ability to solve complex prob-
lems by orders of magnitude and encouraged
scientists to develop new methodologies that
none would have conceived before, simply
because the technology was not available.
Last, but not least, we have entered the era of
big data which requires mathematics and
bioinformatics to make sense of the huge
amount of information.
In general, back in the 1950s, as biology
was maturing into an autonomous research
field, many scientists and philosophers, such
as Michael Scriven, accepted that, in evolu-
tionary biology, “satisfactory explanation of
the past is possible even when prediction of
the future is impossible”. Indeed, applying
the laws and theories on complex problems
in biology, psychology, anthropology,
history, cosmology, economics and quantum
physics increasingly required statistical
2 of 4 The EMBO Journal e101011 | 2019
Irreproducibility and complexity
analysis that inevitably yielded errors. For
Mayr, unpredictability and the unrepeatabil-
ity in biology were therefore almost a
dogma.
Recent findings in molecular biology
challenge this dogma as well as the various
examples of convergent evolution in biol-
ogy. Adaptive convergent evolution entails
the independent emergence of similar or
identical traits in multiple lineages that
undergo the same ecological stimulus (Zou
& Zhang, 2015), At the molecular level,
proteins exhibit convergence too. When dif-
ferent ancestral amino acids are substituted
by the same descendant amino acid along
independent evolutionary lineages, conver-
gent substitution can be identified. When
the same ancestral amino acid is changed to
a newer amino acid along independent
lineages, a parallel substitution takes place
(Zou & Zhang, 2015). Another example is
the observation that many insects that feed
on Apocynaceae plants evolved indepen-
dently the ability to avoid the toxic effects of
cardenolides, chemicals produced by the
same plants. Zhen et al analysed the alpha
subunit of the sodium pump, Na+, K+-
ATPase (ATPa) which is the protein target
for cardenolides in a broad range of taxa
and found evidence of parallel changes and
duplications that explain the shift to avoid
toxicity. Their findings support the hypothe-
sis that adaptation follows evolutionary
paths that minimize negative pleiotropy (See
Zhen et al, 2012; Zou & Zhang, 2015).
This raises an important question in
evolutionary biology: Is evolution indeed
unpredictable and unrepeatable or are our
knowledge, theories and methods still insuf-
ficient? There is a fundamental difference
though: if unpredictability and unrepeatabil-
ity are properties of the evolutionary
process, then these properties must be
systematically verified. If these are unveri-
fied properties that appear occasionally and
in conflict, we must examine whether unpre-
dictability and unrepeatability fall into our
current human inability to predict. This
inability might be a temporary and can be
revoked in the future. The latter would be a
clear sign that more research is needed.
So far, recent works support that evolu-
tion is unpredictable at the molecular level.
For instance, Natarajan et al studied the
predictability of genetic adaptation by exam-
ining the molecular basis of convergence in
haemoglobin function of 56 avian taxa with
different altitudinal range limits—more
George V Papamokos
specifically, they tested whether high-alti-
tude taxa have convergently evolved
increases in Hb-O2 affinity and whether this
evolutionary process is linked to parallel
amino acid substitutions (Natarajan et al,
2016). The authors showed that predictable
changes in biochemical phenotype do not
have a predictable molecular basis, since
only a few changes were attributable to
parallel amino acid substitutions at key resi-
dues, while the majority of them were attri-
butable to nonreplicated substitutions and/
or parallel substitutions at sites that are not
considered “key residues”.
While Mayr’s claim about the value of
mathematics proved wrong, his observation
(Mayr, 1988) about the complexity of biolog-
ical systems is critical: at the molecular
level, biomacromolecules and macro-
molecules of inanimate material do not dif-
fer and show similar physicochemical
behaviour—this is evident from various
studies especially for simple models. But
biomacromolecules possess unique proper-
ties. In a living organism, they are hierarchi-
cally ordered to form new entities:
biomolecules to cells, cells to tissues, tissues
to organs and organs to fully functional
systems. When a fully functional system is
assembled, new unique functions and prop-
erties appear that can only be found in living
organisms: response to external stimuli,
metabolism, growth, differentiation and
replication (Mayr, 1988).
The last experiment described above
(Natarajan et al, 2016), along with Mayr’s
critical observation, show that adaptive
changes may be consequences of various
paths at the molecular level (different muta-
tions with different mechanisms), which
converge to a common endpoint. They show
that the biological material has the neces-
sary complexity to create different processes
upon the same stimulus for reasons not yet
fully understood. It raises a fascinating ques-
tion: Is there a way to simulate reversible
routes from the endpoint to the initial states
of living organisms that evolved?
Intrinsically disordered proteins
Intrinsically disordered proteins (IDPs) are a
class of proteins that do not adopt a defined
secondary structure; they possess flat energy
landscapes; and their spatial representation
resembles a polypeptidic sequence fluctuating
between various conformations such as
extended coils or collapsed globules. (Dyson &
ª 2019 The Author
George V Papamokos Irreproducibility and complexity
Wright, 2005). Despite their dynamic nature,
they are multifunctional and can exhibit vari-
ous interactions depending on cell-signalling
and regulatory networks. Their primary struc-
ture is dominated by charged, polar, small
hydrophilic amino acids, and they are either
entirely disordered or a part of the sequence
constitutes an intrinsically disordered region
(IDR).
IDPs are subject to post-translational modi-
fications, which adds more complexity to IDPs’
functionality. Interestingly, IDPs are highly
abundant in nature and their abundance
increases with the complexity of the organism
from bacteria to archaea to eukaryota: more
than half of all eukaryotic proteins are esti-
mated to embody at least regions of intrinsic
disorder (IDRs). IDP functions play a central
role in many diseases: cancer, neurogenerative
diseases, cardiovascular diseases, type II
diabetes and acquired immunodeficiency
syndrome, while there are numerous findings
that they are also part of the evolutionary
process (Uversky et al, 2014). For example,
the IDP protein a-synuclein is linked to
Parkinson’s disease, dementia with Lewy
bodies, Alzheimer’s disease and Down’s
syndrome.
Unfortunately, a lot of information about
IDPs was lost or ignored. The standard
method to determine the structure of a
protein has been X-ray crystallography, but
many proteins were not investigated
because they failed to crystallize; since these
were considered negative results, this infor-
mation was not published. Moreover, many
proteins had parts missing in their crystal
structure because these were comprised of
intrinsically disordered regions that failed to
crystallize. These parts were ignored until
recently, which is an example of methodol-
ogy bias: the use of a method that can give
reliable results (X-ray crystallography)
provided that the material under study
adopts a specific state or condition (well-
folded protein). However, if the material
cannot adopt this state or condition owing to
unknown properties (IDPs, IDRs), it leads to
methodology bias and misleading results,
wrong analysis and lost information.
Another example of methodology bias is
irreproducibility of experiments involving
enzymes that are caused by differences in
purification procedures and assay methods.
Moreover, molecules that inhibit a specific
enzyme may initiate multiple downstream
effects, because they can also inhibit other
enzymes the activity of which is not
ª 2019 The Author
monitored (The last two sentences are corro-
borating comments of an unknown
reviewer). Mayr’s view that mathematics
cannot be applied to biology can also be
interpreted as a methodology bias.
The result of this methodology bias is
that we are missing a lot of information on
IDPs, which has a severe effect on our abil-
ity to gain knowledge in living organisms.
Immanuel Kant, in his “The Critique of
Judgment” written in 1790, described an
organism as “. . . a whole which result from
the functioning of the parts, while the parts,
in turn, depend on the functioning of the
whole”. In fact, the removal of a part may
be lethal for the organism and for part. Our
lack of knowledge on IDPs is similarly a lack
of knowledge on the functionality of an
important part that prevents us from under-
standing and predicting the function of the
whole living system.
Unexplained irreproducibility in cells
and living organisms
From the wealth of literature on irrepro-
ducibility, I here discuss two papers that
highlight and emphasize the sensitivity of
the biological material to environmental
factors no matter how minute these are (Bis-
sell, 2013). The first example describes how
cells from the same human breast cell line
but from different sources respond dif-
ferently to the same assay. Second, studying
the role of glucose uptake in cancer progres-
sion, Bissell and collaborators showed that
either changes in media glucose levels or
the cells’ shape, when media was kept
constant, changed the nature of the metabo-
lites and the metabolic pathways. A third
example from a collaborator of hers showed
that they could not reproduce their own
experiments using nonmalignant human
breast cell lines obtained from an investiga-
tor with cells obtained from a cell bank.
Subsequent analysis revealed that the
cultured cells had drifted and revealed
crucial information in cell cycle regulation
of the drifted cells.
The same researcher and her collabora-
tors provided another excellent example of
how to track down and eliminate causes of
irreproducibility: working side by side on
the same tumour biopsy, they found that
even small differences in cell isolation—
vigorous stirring versus prolonged gentle
rocking—resulted in irreproducible research
(Hines William et al, 2014).
The EMBO Journal
Gordon J. Lithgow was one of the
authors of a 2000 paper describing that a
drug-like molecule could extend an animal’s
lifespan (Lithgow et al, 2017 and refs 1,2
therein). This finding could not be repro-
duced by other laboratories, and the reason
is still unknown (Lithgow et al, 2017). The
authors spent 4 years and studied more than
100,000 worms to systematically test ageing
interventions in the nematode Caenorhabdi-
tis elegans and eliminated many causes of
variability. Yet, even when performing iden-
tical experiments in a single laboratory, they
observed that some cohorts of worms could
fall into one of two modes of ageing: short-
lived or long-lived. The reason is not under-
stood, and the team is focusing on molecular
differences that may account for this dif-
ference within the same strain. They state
that this phenomenon would have been
undiscoverable if they had not eliminated all
other sources of variability.
In both examples, irreproducibility
results spurred deeper analysis at the molec-
ular level that proved or may prove very
fruitful. Both examples also show that
biological material from cells to living organ-
isms has the property to exhibit different
states at almost identical conditions.
What do cases A, B and C share?
These cases highlight the complexity of the
biological material and a critical conse-
quence: biological material can either
produce the same result following various
molecular routes, or the same stimulus may
have different results. How can this conse-
quence affect science and irreproducible
research? Suppose that a living system,
examined at the molecular level, can exhibit
various states in health, disease or evolution:
S1, S2 and S3. Starting from either of these
states, it can converge to a state denoted as
S4. Suppose now that three independent
laboratories, starting from state S4, explore
the reasons for this reaction and reveal states
S1, S2 and S3, respectively. Although these
results may appear as irreproducible, they
manifest a chance for further research, new
knowledge and the missing factor of biologi-
cal complexity.
The current inability of our methods and
approaches to deal with the complexity of
the biological material and its functions
should indeed inspire scientific progress to
replace earlier views based on ignorance
with solid knowledge based on new
The EMBO Journal e101011 | 2019 3 of 4
The EMBO Journal
findings, methods and approaches. Failures
to reproduce experiments should therefore
not be considered a catastrophe for science
and purged from the literature. Science has
made enormous progress through failures
and errors: for example, the Rayleigh–Jeans
law, the classical approach to describe
black-body radiation, resulted in the ultravi-
olet catastrophe. As the law predicted that
cool objects should radiate in the visible and
ultraviolet regions of the electromagnetic
spectrum, darkness could not exist. Max
Planck eventually brought darkness back
with the introduction of the quantization of
energy.
Envoy
Many methods and tools that are applied to
analyse biological material were originally
designed to study materials of the inanimate
world. The biological material, however, has
unique properties that are not predictable
from its constituting parts. Are these meth-
ods and techniques sufficient to reveal the
secrets of life in health and disease?
Do selected irreproducible results imply
the need to improve existing and to invent
new methods, techniques, theories and
approaches? This commentary presents
evidence and argues that we need to look
out of our current scientific toolbox. This
must not be perceived as an excuse for
4 of 4 The EMBO Journal e101011 | 2019
View publication stats
Irreproducibility and complexity
irreproducible research. On the contrary, it
shows that when dealing with complex
material, we need to share data; describe
our work in detail; improve our methods,
practices and protocols; increase our collab-
oration and our interdisciplinary research;
and deepen our perception of the field.
Questionable practices on a material we only
know it partially, on a material that can
produce multiple results by default, do not
serve science; they just serve career goals.
Instead, they demonstrate how sloppy or
biased research incubates irreproducible
research and builds expensive castles in the
sand. This kind of research is fundamentally
different from the research and its results I
described in this article. We must distin-
guish between bad science and honest fail-
ures that indicate the need for further
research.
Conflict of interest
The authors declare that they have no conflict of
interest.
References
Bissell M (2013) Reproducibility: the risks of the
replication drive. Nature 503: 333 – 334
Crow JF (2009) Mayr, mathematics and the study
of evolution. J Biol 8: 13
Dyson HJ, Wright PE (2005) Intrinsically
unstructured proteins and their functions. Nat
Rev Mol Cell Biol 6: 197 – 208
George V Papamokos
Eisner DA (2018) Reproducibility of science: fraud,
impact factors and carelessness. J Mol Cell
Cardiol 114: 364 – 368
Hines William C, Su Y, Kuhn I, Polyak K, Bissell
Mina J (2014) Sorting out the FACS: a devil in
the details. Cell Rep 6: 779 – 781
Lithgow G, Driscoll M, Phillips P (2017) A long
journey toward reproducible results. Nature
548: 387 – 388
Mayr E (1961) Cause and effect in biology. Science
134: 1501 – 1506
Mayr E (1988) Toward a new philosophy of biology:
observations of an evolutionist. Cambridge, MA:
Harvard University Press
Natarajan C, Hoffmann FG, Weber RE, Fago A,
Witt CC, Storz JF (2016) Predictable
convergence in hemoglobin function has
unpredictable molecular underpinnings. Science
354: 336 – 339
Uversky VN, Dave V, Iakoucheva LM, Malaney P,
Metallo SJ, Pathak RR, Joerger AC (2014)
Pathological unfoldomics of
uncontrolled chaos: intrinsically disordered
proteins and human diseases. Chem Rev 114:
6844 – 6879
Zhen Y, Aardema ML, Medina EM, Schumer M,
Andolfatto P (2012) Parallel Molecular
Evolution in an Herbivore Community. Science
337: 1634 – 1637
Zou Z, Zhang J (2015) Are convergent and parallel
amino acid substitutions in protein evolution
more prevalent than neutral expectations? Mol
Biol Evol 32: 2085 – 2096
ª 2019 The Author