Digital Investigation 9 (2013) 167–169
Contents lists available at SciVerse ScienceDirect
Digital Investigation
journal homepage:
Editorial
Experimental design challenges in digital forensics
Applied research is a major theme in this issue of the
Journal. The included papers cover a range of evidential
areas: Windows Mobile 7, Frostwire, Digsby, JumpLists,
IconCache, and Stegdetect. In addition to providing a
deeper understanding of specific digital artifacts, these
contributions exhibit various approaches to conducting
experiments and interpreting their results.
Designing good experiments is hardly a trivial under-
taking, and has been the focus of brilliant minds at least
since the scientific revolution. Along with the intricacies of
scientific experimentation in general, experiments in digi-
tal forensics pose novel challenges, some of which are
discussed further below. The applied research papers in
this issue attempt to tackle many of these challenges in
order to avoid mistakes and increase scientific rigor in
digital forensics. Read these papers with this in mind: give
experimental design in digital forensics the attention it
deserves.
1. Scientific experiments
Applied research is a fundamentally important aspect of
digital forensics, providing reliable knowledge of computer
processes to support conclusions concerning what activ-
ities occurred on a device or network. In digital forensics,
flawed test results can lead to incorrect decisions, poten-
tially resulting in the loss of a person’s livelihood, liberty or
life. To reduce the risk of incorrect conclusions, attention
must be given not just to the results of applied research, but
also to the underlying experimental design to ensure that
results are reliable and repeatable.
The scientific method is heavily dependent on the
ability to gather accurate observations and to test working
hypotheses. In digital forensics, accurate observation and
testing can be undermined by the inextricable linkage
between experimentation and technology. Even when
conducting studies of the physical world, there can be
intrinsic interactions between the experiment and what is
being studied. Undoubtedly, the most famous example of
this is the Heisenberg uncertainty principle of quantum
mechanics. Conducting experiments in digital forensics
often has the unavoidable complexity of having to make
1742-2876/$ – see front matter ª 2013 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.diin.2013.02.002
measurements from within the environment being stud-
ied. Although we strive for certainty, as a practical matter
we have to accept a high degree of probability as a sub-
stitute for unequivocal proof, and we must work to miti-
gate the factors that can cloud our view of the underlying
truth.
To increase the chances of success, when formulating an
experiment to determine the cause or meaning of digital
artifacts, thoughtful planning is required to eliminate
irrelevant phenomena, and to evaluate individual causes
separately. Planning includes the arrangement of the
experimental environment to eliminate unwanted in-
fluences, figuring out how to control and assess each vari-
able separately, and developing scientific protocols to
govern experimentation.
2. Complex influences
The creation of an experiment and the design of appa-
ratus to measure results is a form of intervention into what
is being studied. For instance, the presence of security
measures such as a firewall can inhibit certain aspects of
the process being studied. The use of a system monitoring
tool to collect data during experimentation can introduce
errors into an experiment. Specific configuration settings of
a computer program or operating system can influence the
findings. The version of the operating system can
completely change the outcome of an experiment. In one
case, digital investigators concluded that file deletion and
disk wiping had occurred on the basis of experiments they
performed using Windows NT, when in fact the arrange-
ment of data on the evidential disk was caused by normal
usage of Windows 98. To reduce the risk of mistakes in
applied research, care must be taken to create an experi-
mental environment that does not bias observations.
In order to obtain the clearest view of cause and effect
when conducting experiments, it is also desirable to isolate
each significant variable and test it individually, while
holding the other variables fixed. In some circumstances, it
may even be necessary to create the equivalent of a control
group for a given experiment by conducting tests that help
distinguish normal usage of the computer system from the
particular process being studied. In digital forensics,
168 Editorial / Digital Investigation 9 (2013) 167–169
important variables can include operating system version
and configuration, file system version, program version and
configuration, client–server interactions, and specific op-
erations performed during an experiment. Although it can
be troublesome to set up separate experiments to test one
variable at a time, in digital forensics, the trouble is usually
worthwhile when weighed against the dangers of report-
ing incorrect results.
To complicate matters, experiments in digital forensics
can encounter unforeseen situations that make it more
difficult to maintain control of the process. A program may
crash or certain triggering events may only occur under
specific circumstances, leading to variations in functionality
under different experimental conditions. In some situations,
a process will only exhibit certain behavior after a given
period of time or when a certain trigger occurs. As a result,
experiments conducted over a period of days may exhibit
different results from those conducted over several months.
In order to control such influences as much as feasible,
when designing an experiment, careful thought must be
given to which interactions are necessary, which influences
are irrelevant, and which factors could create a disturbance
that might distort the results.
3. Reverse engineering
When designing an experiment, a methodical approach
to uncovering conditions and triggers that influence the
operation of a process is preferable. Therefore, in some
situations, applied researchers may wish to become
familiar with the programmatic logic of the code, since it is
this that fundamentally drives the process.
In digital forensics, when a particular computer pro-
gram or operating system feature is being studied, it may
be possible to gain insight into the inner workings. In-
depth analysis of the program being studied can reveal
data structures and encryption keys that help unlock the
content and meaning of experimental results. Under such
circumstances, malware forensics can provide a framework
for applied research in digital forensics. Malware analysts
commonly reverse engineer computer programs to deter-
mine their functionality and to uncover special conditions,
encryption and other nuances in the code.
However, such in-depth dissection of a program or
operating system feature is not always feasible. The time or
skills that are readily available may not be sufficient to
support such an analysis. In some cases, the full application
may be inaccessible, such as client server or obfuscated/
protected code.
4. Reproducibility
Digital investigators survive on their reputations, and
must make every effort to verify experimental findings for
themselves. An error in a forensic report or expert testi-
mony will be attributed to the digital investigator, regard-
less of whether the error was actually due to a mistake in
someone else’s work. Therefore, to be useful in digital fo-
rensics, experiments need to be reproducible. One must
run the same experiment multiple times to verify results,
and to exclude errors and extraneous variations. Keep in
mind that systematic error or bias in an experiment can be
difficult to detect, even between multiple runs of the same
experiment. Ideally, different people will take different
experimental approaches to test a theory – the more ways a
hypothesis is tested, the higher will be our confidence in its
veracity.
Inevitably, there will be differences between multiple
runs of an experiment. Some of these differences may be
important, others irrelevant. One run of an experiment may
encounter unique errors and data corruption that only
become apparent when the experiment is repeated.
Random errors are always to be expected when conducting
experiments and can generally be excluded as a causal
factor by running an experiment multiple times and per-
forming statistical analysis on the results. Running the
same experiment at different times may result in differ-
ences in date–time stamps that are materially irrelevant to
the overall results. However, running multiple experiments
over a long period of time may lead to new findings, such as
automatic resetting of counters, trimming of file contents,
or automatic deletion of data. Therefore, it is also necessary
to define what falls within the parameters of reproduc-
ibility and which differences are to be expected.
5. Reporting
The ability to apply and independently verify research
findings depends heavily on adequate reporting of the
experimental setup and results. Not knowing the version of
application or operating system that was used can make it
difficult to assess whether the findings are associated with
the current version, an older one, or both. Simply providing
a list of digital artifacts and summary of their meaning can
lead to confusion, particularly when some information may
be wrong. To ensure that results are verifiable by others,
applied research reports should detail how experiments
were set up and conducted. Although it may not be
necessary to publish the complete written protocols used to
conduct experiments, it is a good practice to keep them on
file for future reference.
There is also a need for completeness when reporting
outcomes of experimentation. In addition to describing the
successes, report the failures, because these can reveal
crucial characteristics of the process or weaknesses in the
experiment. When strange errors or peculiar outcomes
occur during an experiment that has not been seen before,
such events may turn out to be significant and should be
documented accordingly. These events may be a clue that
further research and testing is required to determine their
cause and meaning.
6. Interpretation of findings
A degree of scientific skill is also required when inter-
preting the results of an experiment. An interpretation
cannot be considered sound until an honest attempt has
been made to eliminate other plausible explanations. This
process is called falsification and is a crucial component of
the scientific method.
Given all of the challenges associated with experi-
mental design, it is necessary to think critically about
 Editorial / Digital Investigation 9 (2013) 167–169
what can and cannot be concluded. Stating the limita-
tions of one’s experimental setup provides digital in-
vestigators with a realistic expectation of the results and
their application. In addition, clearly stating these limi-
tations can motivate other researchers to conduct further
experiments to deepen our understanding of digital
evidence.
7. Conclusions
The usefulness of applied research is greatly increased
by solid experimental design and documentation. Provided
the experimental setup is defined clearly enough to be
169
reproduced, the results can be applied and validated within
the defined context. However, a growing number of cases
have had flawed conclusions presented in court based on
incorrect or misinterpreted experimental results. There-
fore, it is necessary for those conducting applied research in
digital forensics to ensure that their experiments are reli-
able and reproducible, and to clearly communicate the
limitations of their findings.
Eoghan Casey
Johns Hopkins University Information Security Institute,
216 Maryland Hall, Baltimore, MD 21218, United States
E-mail address: eoghan@jhu.edu