Maier G (Maier, Gunther) "Impact factors and peer judgment: The case of regional science journals " SCIENTOMETRICS 69 (3): 651-667 DEC 2006

[Stephen J Bensman]

For whatever they are worth, below are comments I wrote Gunther in a letter
I sent him after reading his article:

Gunther,
I read your ariticle.  In general, I was not surprised by your finding of
no relationship of the impact factor to the importance of journals as rated
by European regional scientists, but I was very surprised by the extreme
lowness and insignificance of the correlations.  That said, I have some
quibbles with your paper and its findings.

First, I think that you have overstated Garfield's view of the impact
factor as a measure of journal significance.  In general, he considered
total citations a better holistic measure of journal significance, and he
used the impact factor to identify to certain facets of journal
significance not identified by total citations due to size.  Gafield never
considered the impact factor a precise measure but thought it one riddled
by error and accurate to only one decimal place instead of the three
published by ISI to avoid too many ties in ranks.  In using impact factor
as a holistic measure of significance, he did not use anything precise as
the Pearson correletation but merely bifurcated his journal sample between
the few journals very high in the impact factor and the rest.  At this
crude level of measurement the impact factor does identify signicant
journals.

Seaond, I am somewhat suspicious of your use of the Pearson r, which
requires a bivarate normal distribution.  Your impact factor data and
perhaps your rating data may have required the logarithmic transformatimn,
and I did not read that you tested your data for the underlying probability
distributions.

Third, since Garfield regarded total citations as the better holistic
measure of journal value, I think that you should have tested this measure
against your regional scientist ratings.  I think that you would have found
higher correlations.  You are making the common mistake of treating the
impact factor as a holistic measure, when in theory and fact it is not.

Having stated this, I would interpret your low correlations as due to the
following reasons.  First, human ratings of importance are positively
influenced by size, and the impact factor is specifically designed to
control for size.  You should have tested your regional scientist ratings
against the total citations in the JCR.  Second, your regional scientist
ratings are European, and ISI citation data are heavily dominated by the US
scientific social stratification system.  Your sets need to be better
defined from this perspective, and you should have used outlier analysis
for this purpose.  And, third, regional science--like most social
sciences--is a very fuzzy discipline, and your data are probably plagued by
exogenous citations, requiring more precise subject set definition.  You
should have also used outlier analysis for this purpose.  You did discuss
the latter two reasons, but you require more analysis, including total
citations analysis to capture the size in your regional scientists ratings.

etc. etc.

SB





Eugene Garfield <garfield at CODEX.CIS.UPENN.EDU>@LISTSERV.UTK.EDU> on
01/12/2007 01:22:18 PM

Please respond to ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>

Sent by:    ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>


To:    SIGMETRICS at LISTSERV.UTK.EDU
cc:     (bcc: Stephen J Bensman/notsjb/LSU)

Subject:    [SIGMETRICS] Maier G (Maier, Gunther) "Impact factors and peer
       judgment: The case of regional science journals " SCIENTOMETRICS 69
       (3): 651-667 DEC 2006


E-mail Addresses: gunther.maier at wu-wien.ac.at


Title: Impact factors and peer judgment: The case of regional science
journals

Author(s): Maier G (Maier, Gunther)

Source: SCIENTOMETRICS 69 (3): 651-667 DEC 2006

Document Type: Article
Language: English
Cited References: 32      Times Cited: 0

Abstract:
this paper discusses the relationship between Journal Impact Factors and
the scientific community's judgment of the quality of journals in regional
science, a discipline closely related to economics and geography. The paper

compares the results of a survey inquiring the quality of journals in the
discipline with the impact factors of these journals for a total of five
years. The comparison shows that no significant positive correlation
between the impact factors and the peer judgments can be found. In many
cases the correlation turns out to be negative - in some cases even
significantly.

Addresses: Maier G (reprint author), Vienna Univ Econ & Business Adm,
Augasse 2-6, Vienna, Austria
Vienna Univ Econ & Business Adm, Vienna, Austria

E-mail Addresses: gunther.maier at wu-wien.ac.at

Publisher: SPRINGER, VAN GODEWIJCKSTRAAT 30, 3311 GZ DORDRECHT, NETHERLANDS


IDS Number: 114NF

ISSN: 0138-9130

CITED REFERENCES :
AKSNES DW
Peer reviews and bibliometric indicators: a comparative study at a
Norwegian university
RESEARCH EVALUATION 13 : 33 2004

 ALTMANN KG
The usefulness of impact factor in serial selection: A rank and mean
analysis using ecology journals
LIBRARY ACQUISITIONS-PRACTICE AND THEORY 22 : 147 1998

 ANDERSON RC
PUBLICATION RATINGS VERSUS PEER RATINGS OF UNIVERSITIES
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 29 : 91 1978

 AZEVEDO CD
Combining revealed and qtated preferences: Consistency tests and their
interpretations
AMERICAN JOURNAL OF AGRICULTURAL ECONOMICS 85 : 525 2003

 BAYLIS M
Sprucing up one's impact factor
NATURE 401 : 322 1999

 BENSMAN SJ
Scientific and technical serials holdings optimization in an inefficient
market: A LSU serials redesign project exercise
LIBRARY RESOURCES & TECHNICAL SERVICES 42 : 147 1998

 BENSMAN SJ
The structure of the library market for scientific journals: The case of
chemistry
LIBRARY RESOURCES & TECHNICAL SERVICES 40 : 145 1996

 BRADFORD SC
DOCUMENTATION : 1948

 BRADFORD SC
ENGINEERING-LONDON 137 : 85 1934

 BRINN T
Measuring research quality: peer review 1, citation indices 0
OMEGA-INTERNATIONAL JOURNAL OF MANAGEMENT SCIENCE 28 : 237 2000

 CAMERON BD
Trends in the usage of ISI bibliometric data: Uses, abuses, and
implications
PORTAL-LIBRARIES AND THE ACADEMY 5 : 105 2005

 DILLMAN DA
RESPONSE RATE MEASUR : 2001

 FLORAX RJGM
Introducing the brightest of dawns: Regional science in "Papers"
PAPERS IN REGIONAL SCIENCE 83 : 5 2004

 GARFIELD E
Journal impact factor: a brief review
CANADIAN MEDICAL ASSOCIATION JOURNAL 161 : 979 1999

GARFIELD E
CITATION INDEXING IT : 1979

 HECHT F
The journal "impact factor": A misnamed, misleading, misused measure
CANCER GENETICS AND CYTOGENETICS 104 : 77 1998

 HOEFFEL C
Journal impact factors
ALLERGY 53 : 1225 1998

 ISSERMAN AM
Intellectual leaders of regional science: A half-century citation study
PAPERS IN REGIONAL SCIENCE 83 : 91 2004

 LAWANI SM
VALIDITY OF CITATION CRITERIA FOR ASSESSING THE INFLUENCE OF SCIENTIFIC
PUBLICATIONS - NEW EVIDENCE WITH PEER ASSESSMENT
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 34 : 59 1983

 LEYDESDORFF L
VISUALIZATION CITATI : 2005

MACROBERTS MH
PROBLEMS OF CITATION ANALYSIS - A CRITICAL-REVIEW
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 40 : 342 1989

MAIER G
20053 SRE I REG UNW : 2005

MAIER G
J QUESTIONNAIRE : 2005

MEHO LI
Citation ranking versus peer evaluation of senior faculty research
performance: A case study of Kurdish scholarship
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 51 : 123 2000

 MONASTERSKY R
CHRONICLE HIGHER OCT 14 : A12 2005

 RINIA EJ
Comparative analysis of a set of bibliometric indicators and central peer
review criteria - Evaluation of condensed matter physics in the Netherlands
RESEARCH POLICY 27 : 95 1998

 SEGLEN PO
BRIT MED J 314 : 497 1997

 SEGLEN PO
CAUSAL RELATIONSHIP BETWEEN ARTICLE CITEDNESS AND JOURNAL IMPACT
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 45 : 1 1994

 VANDIJK J
UNPUB KNOWLEDGE CREA : 2005

 VANRAAN AFJ
Fatal attraction: Conceptual and methodological problems in the ranking of
universities by bibliometric methods
SCIENTOMETRICS 62 : 133 2005

VICKERY BC
BRADFORD LAW OF SCATTERING
JOURNAL OF DOCUMENTATION 4 : 198 1948

 WEST R
What do citation counts count for in the field of addiction? An empirical
evaluation of citation counts and their link with peer ratings of quality
ADDICTION 97 : 501 2002