Tobin MJ, "Rigor of Peer Review and the standing of a journal" AM J RESP CRIT CARE 166 (8): 1013-1014 OCT 15 2002

[Eugene Garfield]

To : Members of SIG-Metrics List

 Tobin MJ, "Rigor of Peer Review and the standing of a journal"
 AM J RESP CRIT CARE 166 (8): 1013-1014 OCT 15 2002


The article by M.J. Tobin "Rigor of Peer Review and the standing of a
journal"  AM J RESP CRIT CARE 166 (8): 1013-1014 OCT 15 2002(reproduced
below) introduces the  paper by

Frederic G. Hoppin, Jr. "How I Review an Original Scientific Article" BY
Hoppin FG
AM J RESP CRIT CARE 166 (8): 1019-1023 OCT 15 2002 .

Since the  paper by FG Hoppin Jr. is too long to send in one message, you
will receive it in two parts.

The subject column  will show:

Part #1 - Frederic G. Hoppin, Jr. "How I Review an Original Scientific
Article" BY Hoppin FG
AM J RESP CRIT CARE 166 (8): 1019-1023 OCT 15 2002

Part #2 - Frederic G. Hoppin, Jr. "How I Review an Original Scientific
Article" BY Hoppin FG
AM J RESP CRIT CARE 166 (8): 1019-1023 OCT 15 2002

____________________________________________________________________

Martin J. Tobin, Editor Am. J. Resp. Critical Care  : MTOBIN2 at lumc.edu

AM J RESP CRIT CARE 166 (8): 1013-1014 OCT 15 2002

Editorial

Title   :    Rigor of Peer Review and the Standing of a Journal

Author   :   Martin J. Tobin, Editor

Shortly after launching The Philosophical Transactions of the Royal Society
in 1665, Henry Oldenberg, the first editor, wrote to Robert Boyle that a
submitted manuscript needs to be carefully scrutinized "before we give a
publick testimony of it to ye world, as is desired of us" (1). Right from
the start, peer review was used as an instrument of quality control to
distinguish scientific journals from book publishing. Through the emergence
of reviewers, journals could insist on the highest standards for excellence
before publishing a report of new research findings. This exacting scrutiny
by fellow experts is without parallel in any other field, and the reviewer
has been described as "the lynchpin about which the whole business of
science is pivoted" (2).

For a component of pivotal importance to the progress of science, journals
provide scant guidance to reviewers. The confidential and anonymous nature
of editorial peer review makes it especially difficult for the novice to
learn the skill. In this issue of AJRCCM (pp. 1019-1023), Hoppin (3)
describes how he reviews a manuscript. He considers the task one of the
utmost responsibility, and writes with wisdom garnered from more than 30
years of experience. His essay is unique: I am not aware of another
published description of how to review a manuscript and the cognitive
processes involved. All reviewers will benefit from reading Hoppin's essay,
with resulting enhancement of the quality of their critiques.

The publication of an article in a journal represents not only the work of
the authors, but it also reflects the standards of that journal. The
standing of a journal among other journals is determined by the expectations
and demands of the scientists who serve as reviewers. Because the reviewers
also submit their own research to the journal, they embody the journal's
standards for scientific excellence. When reviewers believe that a journal
publishes work of a low standard, they will be less demanding in their
critiques, the journal will publish manuscripts of lower quality, and its
standing will fall. Authors regard reviewers as hurdles to vault before
their work gets published and find it hard to see them as offering help.
Readers, however, depend on reviewers to scrutinize a manuscript and
guarantee them it will be worth the time invested in reading it. Readers
also benefit from the improvements made to articles through the intellectual
capital added by reviewers; in turn, the effort made by the reviewers
depends on their assessment of the role of that journal in the advancement
of science. And rounding this positive feedback loop, journals that employ
the toughest reviewing procedures are the most attractive to authors who
hold themselves to the highest standards.

Editorial peer review has always had its critics. At its dawn, Isaac Newton
was complaining that addressing criticisms of his submission to The
Philosophical Transactions "had sacrificed my peace (of mind), a matter of
real substance" (4). Research into peer review has only gotten underway in
the last 10 to 15 years and has been slow to uncover major benefits (5-7).
These studies have been interpreted to mean that editorial peer review has
no value (8). Absence of proof, however, is not proof of absence. It is not
possible to do the definitive outcome study: to randomize two continents to
the stricture, or lack thereof, that the results of all experiments be
published in peer-reviewed journals and then to follow the progress of
science in each continent for 300 years. Moreover, research into peer review
has not included rigorous studies of its most critical and ineffable
components: the making of subjective value judgments, and the effect on
manuscript quality and relevance.

Any discussion of the validity of peer review and science must ultimately
grapple with ontology, the branch of philosophy that asks what actually
exists, and epistemology, the branch of philosophy that asks what, if
anything, we can know (9). Most scientists would be satisfied if they could
achieve the epistemologic status of empirical truth revealed by objective
measurements in an experiment. In this context, the term "objective" refers
to information obtained without human intervention. But the design of an
instrument and the interpretation of findings always involve a component of
human judgment. In scientific epistemology, the most rigorous standard of
empirical objectivity we can hope for is "intersubjective agreement" on the
findings (10-13). The objectivity of scientific knowledge resides in its
being a social construct: its communal institutions, norms, and activities,
including the peer-review system, are not just a background for the logic of
the scientific method-they are constitutive of science as we know it (13).
Peer review, of course, can never guarantee validity, and being a human
enterprise, peer review cannot be free of error, any more than can polygraph
tracings or digital readouts. Scientific journals are archives of work done,
not of revealed truth (14). It is time that decides validity.

Hoppin touches on the issue of why scientists donate so much time to the
review of manuscripts submitted by the fellows of other scientists. This
community of scholars recognizes that they belong to an intricate social
system in which one has to contribute if one is to receive (15). Scientists
also derive several benefits from reviewing: they learn about cutting-edge
research long before it is published; they improve their skills as critical
appraisers by comparing their critiques against those of other reviewers and
seeing the decision letters of associate editors; and they learn how to
write more competitive manuscripts by seeing the faults of others. Each
year, AJRCCM publishes a list of individuals who reviewed manuscripts for
the Journal. And appointment to the Editorial Board of AJRCCM is based
solely on the cogency, rigor, and timeliness of the individual's performance
as a reviewer. The greatest reward, however, is the self-awareness that an
individual is serving as one small cog in the wheel of medical progress. In
any social enterprise, the best individuals volunteer their services without
caring about payment. They find sufficient reward in knowing that the work
of the moment is connected to a much greater process: in this case, science
leading to improved care of patients-work of ever-lasting value. You cannot,
however, have a social enterprise of this magnitude without some parasites.
Every journal has a handful of authors who phone or write about their every
submission, demanding expedited treatment, and in turn are delinquent or
provide superficial comments when invited to review the manuscripts of other
authors.

Online peer review has reduced the time between submission and first
decision to 33 days at AJRCCM. More important than speed, the online system
has enhanced the rigor of peer review. Formerly, we assigned a manuscript to
two reviewers, but now commonly assign four or more reviewers. The
descriptor number on a manuscript allows us to instantly identify the
reviewers with the greatest expertise in each of the 172 areas covered by
AJRCCM. We have more than 5,600 reviewers in our database, and associate
editors can assess level of expertise and prior performance by clicking a
mouse when assigning manuscripts. New reviewers are constantly added to the
database, and we also maintain a list of delinquent and superficial
reviewers. The internet is revolutionizing the speed of processing
manuscripts and the dissemination of new findings, but the bedrock of
science has not changed since the 1660s: experiments are converted into
science only after the results have been published in a peer-reviewed
journal. This point is well illustrated by Leonardo da Vinci, perhaps the
finest scientific mind ever. Leonardo's notebooks are records of the most
intense scientific efforts by a single individual. But his research efforts
came to nothing, because the stricture for disciplined publication was not
formulated until more than 150 years after Leonardo's death.

REFERENCES

1. Zuckerman H, Merton RK. Patterns of evaluation in science: institutional
-ization, structure and functions of the referee system. Minerva
1971;9:66-100.

2. Ziman JM. Public knowledge: an essay concerning the social dimension of
science. Cambridge: Cambridge University Press; 1968. p. 111.

3. Hoppin FG. How I review an original scientific article. Am J Respir Crit
Care Med 2002;166:1019-1023.[Free Full Text]

4. Siegelman SS. The genesis of modern science: contribution of scientific
societies and scientific journals. Radiology 1998;208:9-16.[Medline]

5. Lock S. A difficult balance: editorial peer review in medicine. London:
British Medical Journal; 1991.

6. Goodman SN, Berlin J, Fletcher SW, Fletcher RH. Manuscript quality before
and after peer review and editing at Annals of Internal Medicine. Ann Intern
Med 1994;121:11-21.[Medline]

7. Pierie JP, Walvoort HC, Overbeke AJ. Readers' evaluation of effect of
peer review and editing on quality of articles in the Nederlands Tijdschrift
voor Geneeskunde. Lancet 1996;348:1480-1483.[CrossRef][Medline]

8. Smith R. The future of peer review. In: Godlee F, Jefferson T, editors.
Peer review in health sciences. London: BMJ Books; 1999. p. 244-253.

9. Magee B. The story of philosophy. London: Dorling Kindersley; 1998. p. 8.

10. Polanyi M. Science, faith and society. Oxford: Oxford University Press;
1946.

11. Polanyi M. Personal knowledge: towards a post-critical philosophy.
London: Routledge and Kegan; 1958. p. 225-257.

12. Gross AG. The rhetoric of science. Cambridge, MA: Harvard University
Press; 1996. p. 129-143.

13. Ziman J. An introduction to science studies: the philosophical and
social aspects of science and technology. Cambridge: Cambridge University
Press; 1984. p. 34-57.

14. Relman A. Journals. In: Warren KS, editor. Coping with the medical
literature: a primer for the scientist and the clinician. New York: Praeger
Publishers. 1981. p. 67-78.

15. Ingelfinger F. Peer review in biomedical publication. Am J Med
1974;56:686-692.[Medline]

When responding, please attach my original message
_______________________________________________________________________
Eugene Garfield, PhD.  email: garfield at codex.cis.upenn.edu
home page: www.eugenegarfield.org
Tel: 215-243-2205 Fax 215-387-1266
President, The Scientist LLC. www.the-scientist.com
Chairman Emeritus, ISI www.isinet.com
Past President, American Society for Information Science and Technology
(ASIS&T)  www.asis.org
_______________________________________________________________________