Peer Review in the

Google Age 

Is technology changing the way science is done

and evaluated?

Peggy Dominy & Jay Bhatt

When did peer review start?
Some would say that “Peer
Review” goes back as far
as the 17th century, when it
was known as “The
Inquisition of the Holy
Roman and Catholic
Church”. Scholars’ works
were examined for any hints Galileo
of “heresy”.

Wikipedia
http://en.wikipedia.org/wiki/Peer_review
Peer review in “modern times”
Peer review (known as refereeing
in some academic fields) is
used in:
1. Publication process
2. Awarding of funding for
research
3. Patents
4. Standards

Each of these involve slightly

different practices, but
ultimately colleagues are
evaluating each other.
Process of peer review

Once a paper has been

submitted for consideration of
publication, the editor will
select 1-2 or 3 scholars from
a pool of volunteers to read
and evaluate the paper.

Typically it is a double blind process: the reviewers do not

know who the author is and the author does not know who
the reviewers are. That way only the merits of the paper are
evaluated.
Process of peer review (cont.)
The reviewers (within a reasonable time period)
respond with their comments which are then
forwarded to the author for response to or
compliance with reviewer’s suggestions. In
the days before the “Internet”, this added
weeks (months?) to the publishing process.

Today, moving text back and forth electronically

has dramatically accelerated the process,
though the imposition on an overburden
volunteer researcher has not changed much.
Why do peer review?
 Filter
 More papers submitted than could be “printed”
 Eliminate “bad” science, pseudo-science,
harmful science...
 Aura of “quality” (only the best gets in)
 Collegial stamp of approval
 Professional obligation to the principles of
one’s discipline
 So, what’s the problem?
 Famous papers that were published and did NOT get peer reviewed:
 Watson & Crick’s 1951 paper on the structure of DNA in Nature
 Abdus Salam’s paper “Weak and electromagnetic interactions” (1968). Led to Nobel Prize
 Alan Sokal’s “Transgressing the Boundaries...” in 1996 turned out to be a hoax. Now known
as the Sokal Affair.
 Famous papers that were published and passed peer review that later proved to be fraudulent:
 Jan Hendrik Schon (Bell Labs) submitted and passed peer review 15 papers published in
Science and Nature (1998-2001) found to be fraudulent.
 Igor and Grichka Bogdanov 1999 & 2002 published papers in theoretical physics believed
by many to be jargon-rich nonsense.
 Famous papers that got rejected that later turned out to be seminal works:
 Krebs & Johnson’s 1937 paper on the role of citric acid on metabolism was rejected by
Nature as being of “insufficient importance”, was eventually published in the Dutch journal
Enzymologia. This discovery, now known as the Krebs Cycle, was recognized with a Nobel
prize in 1953.
 Black & Scholes 1973 paper on “the pricing of options and corporate liabilities”, rejected
many times, was eventually published at the intersession of Merton Miller to get it accepted
by the Journal of Political Economy. This work led to the Nobel Prize.
Who’s worried about peer review?
Using Google Scholar search engine, a search using “peer review”
and limiting to subject areas (as defined by Google folks):

7,220 Biology, Life Sciences, and Environmental Science

12,000 Business, Administration, Finance, and Economics
12,500 Chemistry and Materials Science
157,000 Engineering, Computer Science, and Mathematics
15,300 Medicine, Pharmacology, and Veterinary Science
35,900 Physics, Astronomy, and Planetary Science
35,100 Social Sciences, Arts, and Humanities
Two Recent Articles...
 “Is Peer Review Broken?"
by Alison McCook
The Scientist, vol 20 (2), Feb 2006, pg 26.
http://www.the-scientist.com/2006/2/1/26/1/

Submissions are up, reviewers are overtaxed, and authors are lodging
complaint after complaint about the process at top-tier journals. What's
wrong with peer review?

 "Journal lays bare remarks from peer reviewers"

by Emma Marris
Nature, vol. 439, 9 February 2006, page 642
http://www.nature.com/nature/journal/v439/n7077/full/439642b.html

Cloak of anonymity shed by new publication. Editors of a journal launched

this week are out to revolutionize peer review. By publishing signed
reviews alongside papers, they hope to make the process more
transparent and improve the quality of the articles.
So, what’s changed?
 Papers can be “published” on the Web
without the constraints (peer reviewers and
editors) of traditional “journals”.

 Papers can be “published” on the Web with

the constraints (peer reviewers and editors) of
traditional “journals”.

Is this a good thing or a bad thing?

Without “constraints”…
 Wild and Open—no gatekeepers—no
censors
 “Where’s the beef?”—will the good stuff
percolate to the top?
 Discernment—how do you know if it’s good?
 Who’s the authority?
With “constraints”…
 The good stuff is vetted
 Scholarship is monitored and maintained
 Exposure beyond the internet?
 Publishers “sharing” content with the internet (a short list):

 American Physical Society  ACM

 BioMed Central  AIP
 Blackwell  IOP
 BMJ  American Mathematical
 Cambridge University Society
Press  Public Library of Science
 IEEE
 Royal Society of Chemistry
 MIT Press
 Kluwer
 PNAS
 And many more
Peer Review?
 It’s not perfect—grist for a lot of mills
 The Web has made it less of a obstacle to
access
 Different disciplines have different
perspectives—different issues
 Pedagogical yardstick for students
Peer review on the Internet
 Using email
email based online peer review
See Peer Review of Scholarly Publications in Health,
Online Manuscript Peer Review and Tracking Systems and
Physics of Plasmas Online Manuscript Submission and Peer
Review
 Discussion approach
better interaction among authors, reviewers and the editorial
body
JIME – Open Peer Review Process
 Wikis
Immense potential to conduct peer review
 Blogs
post publication comments
See Article Note: On Blogging
as Tool, but Really About Using RSS
Access to Scientific Literature
 Author home pages linking their papers
Google finds them.
 Institutional Repositories
Provide access to faculty authored research
papers. See Publisher Policies that shows
listing of which publishers allow either
publisher or post print version on IRs
 Indexed by Google; increases visibility of
scholarly material
Access to Scientific Literature
 SHERPA: Securing a Hybrid Environment for Resear
ch Preservation and Access
. It is developing open-access institutional
repositories in a number of research universities to
disseminate research findings worldwide
 Preserving EPrints:Scaling the Preservation Mountain
 DSpace at Drexel
 University of Pennsylvania Institutional Repository

 Institutional Repositories are increasing and hence

open access to scholarly literature increasing
Global benefits
 Worldwide increase in access to scientific
literature
 Increased opportunities for collaboration
among experts worldwide
 Increased speed to disseminate scientific
literature with electronic communities
 More informal peer reviews
 Quality needs to be maintained
 See Scholarly Electronic Publishing Weblog
A (very, very) Brief Bibliography
 Chubin, D. E., & Hackett, E. J. (1990). Peerless science: Peer review
and U. S. science policy. Albany: State University of New York Press.
 Davenas, E., Beauvais, F., Amara, J., Oberbaum, M., Robinzon, B., &
Miadonnai, A. et al. (1988). Human basophil degranulation triggered by
very dilute . Nature, 333(6176), 816-818. (published with editorial
reservation on validity)
 Emch, A. (1937). Rejected papers of three famous mathematicians.
National Mathematics Magazine, 11(4), 186-189.
 Garfield, E. (1993). Essays on refereeing and peer review. Retrieved
2/22, 2006 from http://www.garfield.library.upenn.edu/peerreview.html
 Godlee, F., Gale, C. R., & Martyn, C. N. (1998). Effect on the quality of
peer review of blinding reviewers and asking them to sign their reports: A
randomized controlled trial. JAMA, 280(3), 237.
 Harnad, S. (2000). The invisible hand of peer review. Exploit
Interactive(5), February 15, 2006
 Harnad, S. (1996). Implementing peer review on the net: Scientific
quality control in scholarly electronic journals. In R. Peek, & G. Newby
(Eds.), Scholarly publication: The electronic frontier (pp. 103).
Cambridge MA: MIT Press.
A (very, very) Brief Bibliography
 Harnad, S., & Hemus, M. (1997). ALL-OR-NONE: NO STABLE
HYBRID OR HALF-WAY SOLUTIONS FOR LAUNCHING THE
LEARNED PERIODICAL LITERATURE INTO THE
POSTGUTENBERG GALAXY. In I. Butterworth (Ed.), The impact of
electronic publishing on the academic community (pp. 18). London:
Portland Press.
 Horrobin, D. F. (1990). The philosophical basis of peer review and the
suppression of innovation. JAMA, 263(10), 1438-1441.
 Judson, H. F. (1994). Structural transformations of the sciences and
the end of peer review. Second international congress on biomedical
peer review and scientific publication, Chicago, JAMA 272, 92-94.
 Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., &
and the PEER Investigators. (1998). Does masking author identity
improve peer review quality?: A randomized controlled trial. JAMA,
280(3), 240-242.
 Kassirer, J. P., & Campion, E. W. (1994). Peer review: Crude and
understudied, but indispensable. Second international congress on
biomedical peer review and scientific publication, Chicago, JAMA, 272
96-97.
A (very, very) Brief Bibliography
 Mahoney, M. J. E. -. (1977). Publication prejudices: An experimental
study of confirmatory bias in the peer review system. Cognitive Therapy
and Research, 1(2), 161-175.
 McNutt, R. A., Evans, A. T., Fletcher, R. H., & Fletcher, S. W. (1990).
The effects of blinding on the quality of peer review. A randomized trial.
JAMA, 263(10), 1371-1376.
 Moller, A. P., & Jennions, M. D. (2001). Testing and adjusting for
publication bias. Trends in Ecology & Evolution, 16(10), 580.
 Scaria, V. (2003). Peer review of scholarly communication in health:
Perspectives in the internet age. Internet Health, Journal of Research,
Application, Communications & Ethics, 2(6)
 Scaria, V. (2003). Scholarly communication in biomedical sciences,
open access and the developing world. Internet Health, Journal of
Research, Application, Communication & Ethics, 1(1)
 Wenneras, C., & Wold, A. (1997). Nepotism and sexism in peer-review.
Nature, 387(6631), 341-343.
 Wikipedia. (2006). Peer review. Retrieved February 20, 2006, 2006
from http://en.wikipedia.org/wiki/Peer_review
A (very, very) Brief Bibliography
 Wilkinson, S. L. (1998). Electronic publishing takes journals
into a new realm; publications slip off restrictions of print world
and carve out a unique identity Chemical & Engineering News,
76

 JAMA Peer Review Theme Issues

Containing abstracts and articles from the Fourth, Third, and
Second Peer Review Congresses.
June 5, 2002 Issue
July 15, 1998 Issue
July 13, 1994 Issue
 International Congress on Peer Review and Biomedical
Publication
http://www.ama-assn.org/public/peer/peerhome.htm
There have been 5 since 1991