"Bibliometric Distortion": The Babblarazzi Are At It Again...

[Stephen J Bensman]

The Guardian article is interesting.  It seems that the Brits have
finally realized what the American Council on Education and National
Research Council have known for decades--that to rate research programs
you have to define your disciplinary sets carefully and that you should
use multiple measures--peer ratings, publication rates, citation rates,
grants, awards, etc.  Of these, peer ratings are still probably the most
important, for the human mind can do what quantitative measures cannot
do--encapsulate multiple factors into one number.  Despite everything,
it is still more of an art form than a science.  This particularly true
in defining disciplinary sets not only due to the inherent fuzziness of
these sets but also due to institutional parameters often not matching
subject parameters. 

Stephen J. Bensman
LSU Libraries
Louisiana State University
Baton Rouge, LA   70803
USA
notsjb at lsu.edu

-----Original Message-----
From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stevan Harnad
Sent: Friday, November 09, 2007 5:11 AM
To: SIGMETRICS at listserv.utk.edu
Subject: [SIGMETRICS] "Bibliometric Distortion": The Babblarazzi Are At
It Again...


     Comment on: "Bibliometrics could distort research assessment"
     Guardian Education, Friday  9 November 2007
     http://education.guardian.co.uk/RAE/story/0,,2207678,00.html

Yes, any system (including democracy, health care, welfare, taxation,
market economics, justice, education and the Internet) can be abused.
But
abuses can be detected, exposed and punished, and this is especially
true in the case of scholarly/scientific research, where "peer review"
does not stop with publication, but continues for as long as research
findings are read and used. And it's truer still if it is all online and
openly accessible.

The researcher who thinks his research impact can be spuriously enhanced
by producing many small, "salami-sliced" publications instead of fewer
substantial ones will stand out against peers who publish fewer, more
substantial papers. Paper lengths and numbers are metrics too, hence
they too can be part of the metric equation. And if most or all peers do
salami-slicing, then it becomes a scale factor that can be factored out
(and the metric equation and its payoffs can be adjusted to discourage
it).

Citations inflated by self-citations or co-author group citations can
also be detected and weighted accordingly. Robotically inflated download
metrics are also detectable, nameable and shameable. Plagiarism is
detectable too, when all full-text content is accessible online.

The important thing is to get all these publications as well as their
metrics out in the open for scrutiny by making them Open Access. Then
peer and public scrutiny -- plus the analytic power of the algorithms
and the Internet -- can collaborate to keep them honest.

     Harnad, S. (2007) Open Access Scientometrics and the UK Research
     Assessment Exercise. In Proceedings of 11th Annual Meeting of the
     International Society for Scientometrics and Informetrics 11(1),
pp.
     27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.
     http://eprints.ecs.soton.ac.uk/13804/

Stevan Harnad
AMERICAN SCIENTIST OPEN ACCESS FORUM:
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-For
um.html
     http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/

UNIVERSITIES and RESEARCH FUNDERS:
If you have adopted or plan to adopt an policy of providing Open Access
to your own research article output, please describe your policy at:
     http://www.eprints.org/signup/sign.php
     http://openaccess.eprints.org/index.php?/archives/71-guid.html
     http://openaccess.eprints.org/index.php?/archives/136-guid.html

OPEN-ACCESS-PROVISION POLICY:
     BOAI-1 ("Green"): Publish your article in a suitable toll-access
journal
     http://romeo.eprints.org/
OR
     BOAI-2 ("Gold"): Publish your article in an open-access journal
if/when
     a suitable one exists.
     http://www.doaj.org/
AND
     in BOTH cases self-archive a supplementary version of your article
     in your own institutional repository.
     http://www.eprints.org/self-faq/
     http://archives.eprints.org/
     http://openaccess.eprints.org/