STI conference Leiden--Quality standards for evaluation indicators

[Ismael Rafols]

With apologies for cross-posting)

Dear all,
to warm up forweek in the ST Indicators Conference in Leiden, let us share
the topic of a debate:

 *Quality standards for evaluation indicators: Any chance for the dream to
come true?*
 *Special session at the STI-ENID conference in Leiden, 3 September
2014, 16-17.30h *
 *Organisers*: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden
University), Sarah de Rijcke (CWTS, Leiden University)
 *Location*:  Aalmarkt-hall, Stadsgehoorzaal Leiden

 There is a growing realization in the scientometrics community of the need
to offer clearer guidance to users and further develop standards for
professional use of bibliometrics in research evaluations. Indeed the
STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This
session continues from the 2013 ISSI and STI conferences in Vienna and
Berlin, where full plenary sessions were convened on the need for standards
in evaluative bibliometrics, and the ethical and policy implications of
individual-level bibliometrics. The need to debate these issues has come to
the forefront in light of reports that uses of certain easy-to-use metrics
for evaluative purposes have become a routine part of academic life,
despite misgivings within the profession itself about its validity. Very
recently high-profile movements against certain metric indicators (e.g. the
DORA declaration about the Journal Impact Factor) have brought possible
misuses of metrics further to the center of attention. There may be a
growing need for standards – also to promote for accountability of
scientometricians as experts.

 Indeed the relationship between scientometricians and end-users has been
changing over the years due to factors like: 1. Increasing demands for
bibliometric services in research management at various levels of
aggregation, 2. New capacities and demands for performance information
through the greater availability of new research technologies and their
applications, and 3. The emergence of “citizen bibliometrics” (i.e.
bibliometrics carried out by non-expert end-users) due to larger
availability of data and indicators. Some of these developments may result
in new opportunities for research contributions and information-use, and
may increase effectiveness of bibliometrics due to more advanced indicators
and increased availability of data sets (including web data). Yet some
innovations also risk bypassing the quality control mechanisms of fields
like scientometrics and the standards they promote. The implications of
this increasing scope and intensity of bibliometric practices requires a
concerted response from scientometrics to produce more explicit guidelines
and expert advice on good scientometric practices for specific evaluative
practices such as recruitment, grant awards, institutional or national
benchmarking.

 This special session will bring together scientometric experts,
representatives of funding agencies, policy makers and opinion leaders on
the role of metrics in research assessment to discuss the extent to which
moving towards clearer, standardised guidelines over usage and consultancy
can be achieved, both technically and strategically, and what the
guidelines should look like concretely.

 ---
 *Background material*:
 - Report on International workshop "Guidelines and good practices on
quantitative assessments of research" (OST, Paris, 12 May 2014):
http://www.obs-ost.fr/fractivit%C3%A9s/workshop_international
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_international>
 - Blogposts Paul Wouters on previous debates at the ISSI and STI
conferences in 2013, and on the DORA declaration:

http://citationculture.wordpress.com/2013/07/29/bibliometrics-of-individual-researchers/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibliometrics-of-individual-researchers%2f>

http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individual-researchers-the-debate-in-berlin/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibliometrics-of-individual-researchers-the-debate-in-berlin%2f>

http://citationculture.wordpress.com/2013/05/23/dora-a-stimulus-for-a-new-evaluation-culture-in-science/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-a-stimulus-for-a-new-evaluation-culture-in-science%2f>
 - Information on the Higher Education Funding Council for England (HEFCE)
"Independent review of the role of metrics in research assessment" + SPRU
response

http://citationculture.wordpress.com/2014/05/02/metrics-in-research-assessment-under-review/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetrics-in-research-assessment-under-review%2f>
 http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetrics%2f>

https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.pdf&site=25
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fname%3dspru-response-final.pdf%26site%3d25>
 - Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To
intervene, or not to intervene; is that the question? On the role of
scientometrics in research evaluation."

https://citationculture.files.wordpress.com/2014/08/de-rijcke_rushforth_jasist_preprint2014.pdf
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-rijcke_rushforth_jasist_preprint2014.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140828/78a4c310/attachment.html>