INFO 4080/5080 Research Methods

Final Exam Article Evaluation

STEP-BY-STEP GUIDE FOR EVALUATION WORKSHEET

Address each item that is applicable to your article. Do not answer simply as “yes” or “no”, but
include your thoughts on whether quality of the item and how it impacts on the overall usability of
the article.

Your Name:

Provide Article Citation:

1. Framework (Everything prior to the detailed description of the researcher's methods:

Problem Statement, Literature Review, Hypotheses, and Definitions)

Problem Statement

.State the problem of the study either by quoting it or, if no problem statement was given,
in your own words.

.Is the problem stated in such a way as not to imply bias toward the subject studied?

.Is the problem actually solvable - what factors will have to be identified?

Definitions

.Are definitions of important constructs provided? Are they sufficient for the
understanding of this study?

.Are definitions conceptual (based on theory) or operational (made up for this study?

Literature Review

.What is the researcher's orientation? Do they seem to have the expertise to perform the
research?

.Are recent works included?

.Are both sides of issues presented?

.Is a historical basis for the study provided? Are findings explained or just "box-scored"?
 .Is "probable cause" for the expected results provided? This is typically done by
presenting one or more theories that provide a foundation for why the study should have
useful results.

Variables

.What are the independent and dependent variables?

.Are the independent and dependent variables stated or clearly implied?

.Are the variables adequately related to the research question(s)?

.Are other important variables held constant or otherwise accounted for?

Hypothesis

..What is the hypothesis (hypotheses)? Provide the hypothesis (hypotheses) as stated in

the article and/or ones that you inferred.

If stated, is it a null or directional format? If hypotheses are not stated in the null provide
the hypotheses in a null format in addition to those above.

.Are the stated hypotheses, if any, related to the statement of the problem?

2. Methodology

Population

.Describe the population

.Is the population stated or did it have to be inferred? If the latter, how difficult was it to
infer the population?

.Does the derivation of the population make sense based on the literature? Is the
population mostly opportunistic?

Sample

.Describe the sample

.Was the sample drawn at random? If not, does it represent a clearly biased portion
of the population?

.Is the sample size appropriate?

Data Gathering

.Briefly describe the process used to gather data including its clarity of the data gathering
description.
 .What are strengths and weaknesses of the process? Impact on validity?

.Could you replicate the data gathering process with the information provided?

Instrumentation

.Was the instrument developed for this study or was an existing instrument used? Is the
instrument used appropriate to the research questions?

.Was a portion or all of the instrument provided with the report or available online? If not,
how significant an impact is there on your ability to judge the report?

.If accessible, comment on the quality of the survey/questionnaire.

.Was the instrument pilot tested to work out problems?

. Was reliability of the instrument addressed? Was data provided?

(e.g., split-half, test-retest, parallel form, inter coder reliability)

.Was validity of the instrument addressed , i.e., is the instrument really measuring what it
is supposed to? (validity-face, concurrent, construct (factor analysis))

Method

.Describe the method used in the study, e.g., Survey, Quasi Experimental, Experimental,
Historical, or Qualitative.

.Are important details of the method used provided, including important descriptions?
e.g. characteristics of assistants, time of day, room conditions, and other characteristics.

.If experimental design, chart and describe the design.

3. Results

.Provide a brief summary of what occurred in the implementation of the study including
response rates and any problems the researchers experienced.

.Are the results clearly provided in both tabular and narrative form?

.Are all results provided? If not, does the missing data jeopardize the study.

.Are all observations attributed to data collected?

.Is unimportant, confusing detail avoided?

.Do the statistical analyses used make sense given the level of data of the variables and
other factors?
 . Does the results section flow clearly by interweaving statistics (when appropriate) and
narration? Ideally, the narration should make sense when read without the numbers.

4. Discussions/Conclusions

.Does the Discussion/Conclusion section begin with a brief summary of the study's
purpose?

.Is the hypothesis (either one stated or one that you inferred) addressed? Is the hypothesis
addressed in terms of rejection?

If the hypothesis was rejected, did the researcher discuss possible Type I errors (That the
hypothesis was rejected in error and should not have been)?

If the hypothesis was not rejected, did the researcher discuss possible Type II errors (That
the hypothesis should have been rejected and was not)?

.Briefly summarize the conclusions

.Are conclusions warranted by the data?

.Are findings discussed in terms of the literature search? If the results are contrary to the
literature search, does the author address this?

As an example: “This study found that research chemists left their lab and walked across
campus to the library to conduct an online search. This was not expected in light of
Roberts's (1985) finding. However, in this study, the research chemists had been used to
conducting searches on the office computer, but the modem was malfunctioning. In
Roberts study the researchers had no record of online searching.”

.Are alternate explanations for the study’s findings considered and set aside?

5. Recommendations

Briefly summarize the recommendations, if any.

.Are recommendations justified by the findings?

.Check all internal and external validity factors.

6. In about 200-300 words, summarize your thoughts about the quality of the article based on
your input above. Include any observations not covered in 1-5 above. Conclude with a statement
regarding this article that you would make to a researcher pursuing this topic at the time of the
article’s publication.