STI conference Leiden--Quality standards for evaluation indicators

Bornmann, Lutz lutz.bornmann at GV.MPG.DE
Thu Aug 28 02:48:46 EDT 2014

Hi Loet,

I think we can share our manuscript which you quoted in your email and which will be published in a similar form in EMBO Reports, since it is relevant in the context of standards in scientometrics.



From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Loet Leydesdorff
Sent: Thursday, August 28, 2014 8:29 AM
Subject: Re: [SIGMETRICS] STI conference Leiden--Quality standards for evaluation indicators

Adminstrative info for SIGMETRICS (for example unsubscribe):
Dear Ismael,

It seems to me that we know from the innovation-science literature that standards are to the interests of incumbent firms. In this context, one can expect process innovation more than product innovations. The further development of the field, in my opinion, needs the fluidity of intellectual exchanges and the space to propose new variants.

Perhaps, as an intellectual community we have increasingly interests different from the professional practices of (quasi)industries on a market of evaluation studies that one may wish to certify ( and thus to shield the market against “amateurs”; our PhD students?).

Let me quote from a recent text (that I coauthored for other reasons):

“There exists a professional community with experts in bibliometrics who develop advanced bibliometric indicators for productivity and citation impact measurements (see an overview in Vinkler, 2010). Only experts from this community should undertake a bibliometric study. These centres of professional expertise can be found, for example, at the Centre for Science and Technology Studies (CWTS, Leiden) or the Centre for Research & Development Monitoring (ECOOM, Leuven).”

Is this the dream to come through? Or do we hear institutional interests? Perhaps, we need smaller dreams ☺


Loet Leydesdorff
University of Amsterdam
Amsterdam School of Communications Research (ASCoR)
loet at <mailto:loet at> ;
Honorary Professor, SPRU, <> University of Sussex;
Guest Professor Zhejiang Univ.<>, Hangzhou; Visiting Professor, ISTIC, <> Beijing;
Visiting Professor, Birkbeck<>, University of London;

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Ismael Rafols
Sent: Thursday, August 28, 2014 2:21 AM
Subject: [SIGMETRICS] STI conference Leiden--Quality standards for evaluation indicators

Adminstrative info for SIGMETRICS (for example unsubscribe):
With apologies for cross-posting)

Dear all,
to warm up forweek in the ST Indicators Conference in Leiden, let us share the topic of a debate:
Quality standards for evaluation indicators: Any chance for the dream to come true?
Special session at the STI-ENID conference in Leiden, 3 September 2014, 16-17.30h
Organisers: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden University), Sarah de Rijcke (CWTS, Leiden University)
Location:  Aalmarkt-hall, Stadsgehoorzaal Leiden

There is a growing realization in the scientometrics community of the need to offer clearer guidance to users and further develop standards for professional use of bibliometrics in research evaluations. Indeed the STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This session continues from the 2013 ISSI and STI conferences in Vienna and Berlin, where full plenary sessions were convened on the need for standards in evaluative bibliometrics, and the ethical and policy implications of individual-level bibliometrics. The need to debate these issues has come to the forefront in light of reports that uses of certain easy-to-use metrics for evaluative purposes have become a routine part of academic life, despite misgivings within the profession itself about its validity. Very recently high-profile movements against certain metric indicators (e.g. the DORA declaration about the Journal Impact Factor) have brought possible misuses of metrics further to the center of attention. There may be a growing need for standards – also to promote for accountability of scientometricians as experts.

Indeed the relationship between scientometricians and end-users has been changing over the years due to factors like: 1. Increasing demands for bibliometric services in research management at various levels of aggregation, 2. New capacities and demands for performance information through the greater availability of new research technologies and their applications, and 3. The emergence of “citizen bibliometrics” (i.e. bibliometrics carried out by non-expert end-users) due to larger availability of data and indicators. Some of these developments may result in new opportunities for research contributions and information-use, and may increase effectiveness of bibliometrics due to more advanced indicators and increased availability of data sets (including web data). Yet some innovations also risk bypassing the quality control mechanisms of fields like scientometrics and the standards they promote. The implications of this increasing scope and intensity of bibliometric practices requires a concerted response from scientometrics to produce more explicit guidelines and expert advice on good scientometric practices for specific evaluative practices such as recruitment, grant awards, institutional or national benchmarking.

This special session will bring together scientometric experts, representatives of funding agencies, policy makers and opinion leaders on the role of metrics in research assessment to discuss the extent to which moving towards clearer, standardised guidelines over usage and consultancy can be achieved, both technically and strategically, and what the guidelines should look like concretely.

Background material:
- Report on International workshop "Guidelines and good practices on quantitative assessments of research" (OST, Paris, 12 May 2014):<>
- Blogposts Paul Wouters on previous debates at the ISSI and STI conferences in 2013, and on the DORA declaration:<><><>
- Information on the Higher Education Funding Council for England (HEFCE) "Independent review of the role of metrics in research assessment" + SPRU response<><><>
- Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To intervene, or not to intervene; is that the question? On the role of scientometrics in research evaluation."<>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: EMBOReports.pdf
Type: application/pdf
Size: 131404 bytes
Desc: EMBOReports.pdf
URL: <>

More information about the SIGMETRICS mailing list