Professional Documents
Culture Documents
Week002 PaperReview02 OverviewOfSoftwareEngineering
Week002 PaperReview02 OverviewOfSoftwareEngineering
1
Overview of Software Engineering
Look for a recent journal (not more than 5 years ago) that is
related to the 2nd week discussed topics. Use the same uploaded
structure and format guidelines for research paper review stated
below:
This set of guidelines was originally designed for a course, so it includes grading
guidelines, which most readers can ignore.
Notes:
1. Follow the guidelines here on structuring and formatting your
review.
2. Include each category in a separate section or subsection.
Ensure that you discuss each category! (If a category does not apply, then
actually write in that (sub)section that the category does not apply.) We will
be looking for each category when grading.
3. Reference other articles you've read, if appropriate. Just as in
the articles you've read, include a reference marker where you refer to each
in your review, and provide full bibliographic references in a bibliography
section at the end of your report.
4. Your final report should be 4 full pages or more (if measured in
Times New Roman 10pt single spaced), which is equivalent to 7-10 pages in
most Web browsers. This does not include your bibliography, citation
analysis, diagrams or large gaps between parts of your report. There is no
maximum length.
5. Do not copy any part of the article into your review. If you
want to use more than 3-4 of the author's words, then use quotation marks,
and add a page number from the article (if available) when you cite this
quotation.
6. Proof-read your review for grammar and clarity as well as
spelling. Spell checkers do not fix grammar. You should be sure that you have
caught all mistakes and written clearly. Most word processors have a
grammar checker - use it!
7. For the printed copy of your review, turn in just a plain stapled
black & white copy of your review. Do not put your review in a folder. Avoid
Assessments
fancy covers, colors or fonts. We want to judge your content, not visual
presentation.
8. Please reread the note on plagiarism posted in the syllabus and
on the course Web site.
In sections 6-14, utilize at least 3 citations to articles outside those on the class reading
list. Use these articles to back up your analysis, to describe or address issues, implications or
recommendations for further research or development.
Grading Notes
We have included the grading criteria in these guidelines.
Unless otherwise noted, the categories are:
great: fully discussed, explained well and well-justified
ok: partial discussion; not explained well or well-justified
poor: barely discussed
zero: not discussed
Length Grading
We take 5 points off for each full page missing (i.e., less than 4 full pages) and 3 points
off for every half page missing. Note that large gaps between sections will not count towards the
full page length.
Editing Grading
Has the review been prooofread for both spelling and grammar? If not up to 3 points off
for poor grammar and up to 3 points off for poor spelling.
In addition, you can lose points if your review is too short or poorly edited. See the
Grading Notes section above.
Assessments
Paragraph 3: State whether the article is "conceptual" or "empirical", and why you
believe it is conceptual or empirical. Empirical articles and conceptual articles have a similar
objective: to substantiate an argument proposed by the author. While a conceptual article supports
such an argument based on logical and persuasive reasoning, an empirical article offers empirical
evidence to support the argument. Empirical articles offer substantial, detailed evidence which the
authors analyze using statistical methods. Empirical articles must include hypotheses (or
propositions), detailed research results, and (statistical) analyses of this empirical evidence.
Empirical research includes experiments, surveys, questionnaires, field studies, etc, and to limited
degree, case studies. Conceptual articles may refer to such empirical evidence, but do not provide
the detailed analysis of that evidence. Of course, both types of articles can use real life examples to
back up their points. Just because an article provides examples, does not necessarily mean that it is
empirical. (The lesson to take home is not to consider a conceptual article to be an empirical one
just because it provides some summarized or some unanalyzed data.)
Grading: Objectives: great - 3; ok - 2; poor - 1
Grading: Audience/Journal Appropriateness: great - 3; ok - 2; poor - 1
Grading: Conceptual vs. empirical: great - 2; ok/poor - 1
4. Results
Very briefly summarize the important points (observations, conclusions, findings) and
"take home messages" in the article.
Please do not repeat lists of items in the articles - just summarize the essence of these if
you feel they are necessary to include.
Grading: great - 8; ok - 5; poor - 2
5. Class Readings
1. Does this article directly cite any of the class readings, i.e., does any class reading
appear explicitly in its bibliography or reference section? If not, state this explicitly. If so, clearly
describe how the authors use the cited article. How does the article you are reviewing relate to
and/or build upon the class article it cites?
If this article does not cite any class readings then just state this. (If you do not state this
explicitly, you will not receive credit for this section.) Do not discuss any other readings, such as
other readings on the same topic or by the same author. Save any discussions of similar articles for
your synthesis section below.
2. Do any of the class readings cite your article (besides the textbook)? If so, clearly
describe how.
DIT101 – Advanced Software Engineering
5
Overview of Software Engineering
If no class readings cite your article, then write in your review "No class readings cite this
article." (If you do not state this explicitly, you will not receive credit for this section.)
Be sure to add all references you cite to the bibliography.
Grading: great - 4; ok - 2; poor - 1 {If none, then score 4 by default if this has been
stated explicitly.}
6. Contributions
An article makes a "contribution" by adding to the knowledge of researchers in a research
field. An article can make a contribution to the research field in many ways. Does it provide a new
way to look at a problem? Does it bring together or "synthesize" several concepts (or frameworks,
models, etc.) together in an insightful way that has not been done before? Does it provide new
solutions? Does it provide new results? Does it identify new issues? Does it provide a
comprehensive survey or review of a domain? Does it provide new insights?
Also, is it salient (relevant and current) to a particular scientific issue or managerial
problem? Are the issues addressed introduced in a way that their relevance to practice is evident?
Would answers to the questions raised in the article likely to be useful to researchers and
managers?
Note: Do not discuss the contributions of the technologies the article describes, but rather
the contributions of the article itself!
The article's contributions should be original. To the best of your knowledge, are they?
Are the article's take-home messages new?
Describe each contribution clearly in a separate paragraph or bullet point. Discuss why
the contribution is important.
Alternatively, if you believe the article makes no contributions, explain why clearly.
Grading: great - 8; ok - 5; poor - 2
7. Foundation
Good research often is built upon theories and frameworks that other researchers have
developed. Sometimes articles will be substantially based upon this prior work, and refer back to it
in some detail. (Not all research articles will do this.)
Which theoretical foundations does this article and research build on, if any? In what
ways? Include references/citations of the foundation work. (You can determine this in part from
the works the article cites.)
Note, however, that most works cited are not core foundational work, but rather just
support certain aspects of the article. Similarly, do not confuse a general discussion of related
topics as foundational work.
If the article does not build upon key pieces of prior research, then write in your review
"This article does not build upon any foundation research." (If you do not state this explicitly, you
will not receive credit for this section.)
Grading: great - 4; ok - 3; poor -1 {If none, then score 4 by default if this has been stated
explicitly}
Assessments
Discuss the article's research ideas and results in terms of any relevant materials covered
in class or which you have found in the readings. You can also check the concepts in the "to know"
link on the "quick links" portion of the course Web site. Cite these readings explicitly, including
their source in the bibliography and a bibliographic marker in the text (e.g., [Turoff et al., 1999]).
You also could analyze the approach the author took to the article's analysis and
discussion. Discuss the article's approach and results in terms of one or more of the frameworks,
etc., from the text or readings, or any you find elsewhere. For example, if the authors discuss any
type of information system, you could use Alter's WCA analysis to examine how they approached
that information system. Try to do this for all the models and frameworks, etc., which apply to
your article.
As part of this analysis, reference other articles you've read, when appropriate. Compare
the approach, results and contribution with all articles about similar topics or with a similar
approach. For example, if your article develops a new framework, compare it with
Bandyopadyhah's framework criteria (and vice versa - whoever does Bandyopadyhah's article
could test his criteria on frameworks from the other readings). Include any articles you cite in the
bibliography and use bibliographic markers in the text.
For all of these, do your synthesis comparison in as much depth as you can!
Grading: four items up to 20 points total (12 points plus 8 points extra credit) - for each
item: great - 5 ok - 2; poor - 1
Great: discussed deeply and relating the article in detail with the synthesized models and
frameworks.
OK: the synthesized information is only discussed in general
9. Analysis
Note: Many people assume this category is the same as "General Critique". It is not.
General Critique is a different category from this, and follows below.
What has changed since the article was written? How do it's lessons, ideas and theories
still apply? To what extent has its issues been resolved?
Grading: great - 4; ok - 2; poor - 1
Additional Analysis
Optionally, try applying the article's models, frameworks and guidelines, etc. yourself.
Do you find them useful?
In addition, you may optionally add your own additional analysis in a separate
subsection. (Do not repeat the author's analysis in the paper - you could summarize this as part of
the results section.)
Grading: this section is extra credit only: great - 8; ok - 5; poor - 2
For full credit, ask yourself these questions when justifying your critique points:
why/why not?
how?
what distinguishes the differences/different approaches, and
in what ways?
Grading: four items up to 16 points total (10 points plus 6 points extra credit) - for each
item: great - 4; ok - 2; poor - 1
Assessments
11e. Further Critique of an Empirical Article
*** only for empirical articles
{adapted from guidelines from Dr. Dan Robey, Georgia State University}
A critique of an empirical article examines the strength of the empirical evidence
supporting the author's argument. Both strengths and weaknesses should be identified in a critique.
Explain and justify each of your critique points in at least 3-4 sentences.
To the best of your abilities, discuss each of the following categories in a separate
paragraph:
1. CLARITY: Is the article's purpose and argument clear? Do the researchers clearly
develop a major research question, proposition, or hypothesis that is to be evaluated in the
empirical study and discussed in this article? If the study is exploratory (preliminary), is sufficient
justification for an exploratory strategy given?
2. THEORETICAL GROUNDING: Is the researcher's argument grounded in more basic
theory? Is it clear whether the structure of the empirical study (i.e., what they do) was derived from
theory, or just made up? In theory-building articles, is the need for new theory adequately
established?
3. DESIGN OF RESEARCH INVESTIGATION: Is it clear exactly how the empirical
study was carried out? Is the design of the research approach (field study, experiments,
questionnaires, etc. - both contents and how they will be used) adequate to address the common
threats to internal and external validity? Have appropriate controls been established, and is the
selection of research sites justified? Are the hypotheses and experiments, etc., significant?
4. MEASUREMENT: Empirical studies can have quantitative measurements (i.e.,
numeric results) and qualitative or subjective measurements. Are the measures used adequately
described (i.e., what is measured in the study and how)? Are data on the reliability and validity of
these measures reported? Does the article feel anecdotal or solidly supported with evidence? For
example, in case or field studies, are the results well documented? Is it clear who the subjects were,
and with whom interviews were carried out? Were important results cross-checked, i.e.,
determined across a range of subjects or just gotten from one or two subjects?
5. ANALYSIS: Is the analysis of empirical data conducted properly? Do the data
conform to the requirements of any statistical tests used? Are qualitative data adequately described
and presented?
6. DISCUSSION AND CONCLUSIONS: In discussing the results of the empirical study,
do the authors remain true to the actual findings of the study? Are the claims made in the
conclusion of the article actually supported by the empirical data? If the study is exploratory, do
the authors offer research questions or hypotheses for future research?
7. BIASES: Do the biases of the authors affect the design of the research or the
interpretation of the results? Are the authors aware of potential biases and the affect on the study?
Grading: for each: great - 2 ok/poor - 1
Note: If you have any critiques in this section, they most likely belong in the General
Critique section instead.
DIT101 – Advanced Software Engineering
9
Overview of Software Engineering
Grading: 3 items up to 9 points total (6 points plus 3 points extra credit) - for each item:
great - 3; ok - 2; poor - 1
Note: If you have any critiques in this section, they most likely belong in the General
Critique section instead.
Grading: 4 items up to 12 points total (6 points plus 6 points extra credit) - for each
item: great - 3; ok - 2; poor - 1
14. Impact
To determine how much impact this article has had, do a citation analysis. Discuss what
this citation analysis shows, and why; don't just list the citations! (See the Citation Analysis
Guidelines (.doc) and Handout (.pdf) posted on the course Web site.)
If the article has no citations, then write in your review "I found no citations in the
Science Citation Index, the Social Sciences Citation Index or on the Internet." Then clearly explain
why you believe there were no citations at all. If you found citations in some indexes or on the
Internet but not the others, then explain this as well.
Include your citation lists in an appendix to your review (see below for details).
Grading - impact discussion: great - 3; ok - 2; poor - 1
15. Questions
List three insightful questions of your own, arising from this article. Do not ask
definitions, but rather questions that really make one think.
Grading: 3 questions, up to 6 points total - for each question: great/ok - 2; poor - 1
Assessments
Also, be sure that you have included a bibliographic marker to each (such as [Bieber &
Smith, 2001]) in the text of your review.
Grading: -5 if missing references; -3 if you mention the authors explicitly in your text
and put the references in this bibliography section, but forget to explicitly place citation markers
in your text.
If the article has no citations for any of these three, then write in that section "I found no
citations in the [Science Citation Index or the Social Sciences Citation Index or on the Internet]."
Note, if your article has more than 20 citations, you only need to include a selection of
them:
State how many citations each index has and the Web search
found
List 1-2 citations for each year in which the article has been
cited. Try to include citations from several different journals spread over
your selection ? Include a citation analysis to see who has cited it and how.
Grading - citation analysis: 2 points for each of the regular indexes, and 2 points for the
Internet search; if the article is from 1998 or later then: 1 point for each of the regular indexes, 4
points for the Internet search.