How to read and understand a

You will read a lot of papers

paper and its contribution
Carole Goble Most will be irrelevant
Many will be poor
Some will be important
A few will change your life

Strategies for choosing papers Smart Reading

• Salami paper • Understanding the context of the paper
writing • Beginnings and endings
• Survey the structure.
• Using surveys and • Using figures and tables. Or generating them if
summary papers they don’t exist.
• Reading every word. Or not.
• Mining citations
• Summarising the paper.
• Multiple reading passes.
• Reading out loud
• Explaining the paper to your cat

1
 Additional material Where does it fit with my work
• Conference papers means a presentation • Is it relevant? If not why not?
• Web pages • How does it fit with your framework?
• Technical reports • Yes – you will need a framework!
• Can you relate the terminology and
• Other? notation to yours?
• Keeping a record of the contribution.

When will the paper become

Tools for managing reading
relevant?
• Over time • Endnote
• An ongoing framework • BibTex
• Revisiting • Mindmaps
• JabRef

FreeMind http://freemind.sourceforge.net/wiki/index.php/Main_Page

JabRef http://jabref.sourceforge.net/

2
 Make yourself a template
Author housekeeping stuff
Paper genre
Problem statement/motivation
Key ideas
Technical contribution
Technical flaws
Presentation
Comparison
– To authors’ other work
– To third party’s work
– To your work
How would I extend this paper?
What questions does it raise?
Future work of author.
What else?
Author log
Discussion Items
• Strategies for choosing papers
– Salami paper writing
– Using surveys and summary papers
– Mining citations
• Smart reading
– Understanding the context of the
paper
– Beginnings and endings
– Survey the structure.
– Using figures and tables. Or
generating them if they don’t exist.
– Reading every word. Or not.
– Summarising the paper.
– Multiple reading passes.
– Reading out loud
– Explaining the paper to your cat
• Additional material
– Conference paper means a
presentation
– Web pages
• Where does it fit with my work
– Is it relevant? If not why not?
– How does it fit with your
framework?
– Can you relate the terminology and
notation to yours?
– Keeping a record of the
contribution.
• When will the paper become
relevant?
– Over time
– An ongoing framework
– Revisiting
• Tools for managing reading
– Endnote
– Mindmaps, e.g. FreeMind
http://freemind.sourceforge.net/wiki/
index.php/Main_Page
– JabRef http://jabref.sourceforge.net/

Different Kinds of Paper • Problem(s) investigated:

– well-established class of
• Techniques used for their
solution:
problems, e.g., FO theorem – new, i.e., developed by
proving, SAT, TSP, image authors?
retrieval etc.
• Position paper – novel class of problems (is
– existing? Developed by
authors or others?
it really new?)
• Systems paper – single problem or many
– new combination of
existing techniques?
problems
• Theory paper • Learn more about the
– good or better/worse than
X and why
problem
• Vision paper – its computational
complexity, phase
transition, increase
understanding/conceptualis
• Deep and narrow ation, etc.

• Broad and shallow • Evaluation?

– empirical (run tests): test suite and testing must match problem
targeted
– theoretical: correct and understandable/convincing and relevant

3
 Ask yourself…
1. What is the research paradigm that the author is using?
• psychological experiments, formalization and theorem proving, and artefact

•
design and construction.
If the paper is part of a well established field, should describe the field and its
A muddled description of a simple idea
current state.
2. What is the problem area with which the paper is concerned? =
• "Automatic Generation of Compilers from Denotational Semantic Descriptions of

3.
the Source Code" would describe a research paper on compilation.
What is the author's thesis? What are they trying to convince you of?
a clear description of a muddled idea
4. Summarize the author's argument. How does the author go about trying to
convince you of the thesis?
5. Does the author describe other work in the field? If so, how does the
research described in the paper differ from the other work?
6. Does the paper succeed? Are you convinced of the thesis by the time that
you have finished reading the paper?
7. Does the author indicate how the work should be followed up on? Does the
paper generate new ideas?
8. Some papers implicitly or explicitly provide a new way of doing things or of
thinking about problems. If your paper does so, describe the approach.

Different kinds of papers

• Position paper vs Systems paper vs Theory paper vs Vision paper
• Deep and narrow vs Broad and shallow
• Problem(s) investigated:
– well-established class of problems, e.g., FO theorem proving, SAT, TSP, image
retrieval etc.
– novel class of problems (is it really new?)
– single problem or many problems
Understand Reviewing
• Techniques used for their solution:
– new, i.e., developed by authors?
– existing? Developed by authors or others?
– new combination of existing techniques?
• Show that techniques used are good or better/worse than X and understand
why
• Learn more about the problem (e.g., its computational complexity, phase
transition, increase understanding/conceptualisation, etc.)
• Evaluation?
– empirical (run tests): test suite and testing must match problem targeted
– theoretical: correct and understandable/convincing and relevant

4
 Classical review form
Hints for Reviewing Papers • OVERALL EVALUATION: 3 (strong accept) 2 (accept) 1 (weak accept) 0 (borderline paper)
-1 (weak reject) -2 (reject) -3 (strong reject)
• The answer to each question tells you something about the technical content of the paper • REVIEWER's CONFIDENCE 4 (expert) 3 (high) 2 (medium) 1 (low) 0 (null)
• The ease of extracting the answer to each question tells you something about the quality
of the writing. • RELEVANCE TO THE CONFERENCE: 10 (very good) 9 8 7 6 5 (borderline) 4 3 2 1 (very
Questions bad)
• Is this a vision/position/direction paper, or a measurement/implementation paper? • SCHOLARSHIP: 1 =no citations, 3 =huge gaps, 5 =ignorant of a major relevant piece of
work, 8 =good but missing a few references, 10=clear specification of relation to rest of
• If you know the area well, can you mentally slot this paper somewhere in the taxonomy? field
("Differs from X as follows; has the following in common with Y;" etc.) If the paper is
radically brilliant, new, or iconoclastic work, this question may not apply. • CLARITY: 1 =unreadable, 3 =significant problems with exposition, 5 =some problems with
grammar or structure, 8 =basically okay, 10=beautifully clear
• Can you summarize the single most important contribution in one or two sentences?
• NOVELTY: 1=same thing has been done before, 3=very similar to existing work, 5=a direct
Issues extension of existing work, 8=major step in a previously existing direction, 10=surprisingly
• Will this advance the state of the art? new model, approach, or result
• Did you learn anything new? • SOUNDNESS AND TECHNICAL QUALITY: 1 =technically incoherent with no validation, 3
• Does it provide evidence which supports/contradicts hypotheses? =major mistakes, weak experiments/studies, theorems/assumptions false, 5
• Experimental validation? =experiments/studies not well designed, minor theoretical/conceptual problems, 8
=basically okay, 10=beautiful formalization and proof, or exceptionally good experimental
• How readable is the paper? methods/technical design
• Is the paper relevant to a broader community? • SIGNIFICANCE AND IMPACT: 1 =no-one will ever read this paper, 3 =no-one will ever cite
Goals of Review this paper, 5 =this paper will moderately influence researchers close to the area, 8 =this
• Guide the program committee in selection process paper will moderately influence a wide community of researchers, or have a large influence
• Help authors (to revise paper for acceptance, to understand rejection, to improve further on a smaller community, 10=this paper will change computer science
research and future projects)
• DETAILED COMMENTS Please provide a detailed review, including justification for your
scores.
John Ousterhout's Hints for Reviewing Papers