Professional Documents
Culture Documents
Common Flaws in Research
Common Flaws in Research
Common Flaws
John Koetsier
Abstract
Research studies are difficult to do right and easy to do wrong. There are
many potholes to avoid, and many factors can impact a study’s validity and
reliability. To find and understand some of the common problems, I’m going
to look at three different types of studies, see what the researchers did, how
they did it, and what problems they encountered. The studies are Beck and
enrichment (2003).
Common flaws 3
In the first study, Natalie Beck and Tony Fetherston studied the effects of
teaching writing with a word processor in primary grades. For six weeks, they
studied both how students felt about using word processing technology
versus paper and pencil, and what effects technology had on the quality of
their writing. As a result, they concluded that students who used word
processors wrote significantly better than students using pencil and paper.
settings and can be counted upon when creating programs and curricula.
In brief, the problems with the study include a very small sampling size -
only seven students – which basically eliminates any opportunity for external
that it matters that much with such a small sample - the researchers used
In addition, the short six-week study ensured that researchers could not
sheer newness galvanizes student attention and effort. Oddly, in what must
be a rare problem for a study with a novelty issue, maturation was also a
problem, since the students apparently used the word processing software
Common flaws 4
There was no control group receiving a placebo and equal but different
treatment. The sample group essentially was its own control group.
students in Texas middle schools, and used the results to argue that
The conclusions are valid and supported by subsequent research, but the
weighted in favor of urban students and against rural students, who were
entirely excluded. In addition, ethnicity was a factor that was not addressed
at all in the study, even though the schools from which students were drawn
were in a city and state that over-represented certain racial groups. A final
sampling complication was the fact that schools subjects were drawn from
Common flaws 5
variables. Finally, high mortality in the course of the study due to data
In the third study, teacher Karen Hayse engaged in action research to guide
resources most of the time, with web resources being the clear leader
initially, trailing off halfway through the 10 weeks, and then regaining
popularity in the final few weeks. Hayse also noticed, anecdotally, that giving
There are a number of concerns with this study, starting with sampling.
before the study started there was already a student preference for
A concern I have is that there may have been a novelty effect … that
activities in the classroom may have chosen web activities simply due to their
newness. A longer study would have reduced any novelty effects that might
be operating.
Hayse mentions that she has controlled for a number of factors, including
types of activities and opportunities to work with peers, but I wonder if the
resource usage came after a student asked friends to play a trivia game; was
a similar thing possible with the web resources? It’s difficult to say without
unlikely that over 10 weeks these options were never chosen. Also, students
self-reported use of activities at the end of the day: this may not be the most
time and effort on this specific facet of their studies, since without a good
critical. Knowing that students had used the particular type of word
processing software that they were testing should have impelled Beck and
Fetherston to find other subjects. And while I can’t prove it without access to
the resources that Hayse used, I suspect that while equivalent on the
surface, and in terms of topic, they may not have been equivalent in terms of
exist at every stage. These three studies illuminate some of the common
issues, and provide insight for researchers about what to avoid and minimize
References
Retrieved from
http://www.smsd.org/custom/curriculum/ActionResearch2003/Hayse.h
tm on March 5, 2008.
Miller, L.M., Schweingruber, H., and Brandenburg, C.L. (2001). Middle School
125-140.