STI conference Leiden--Quality standards for evaluation indicators

Thu Aug 28 11:25:02 EDT 2014

Very interesting is also the following page:

http://www.academicanalytics.com/Public/WhatWeDo 

Many US universities use these services.

Best,

Loet

  _____  

Loet Leydesdorff 

University of Amsterdam
Amsterdam School of Communications Research (ASCoR)

 <mailto:loet at leydesdorff.net> loet at leydesdorff.net ;
<http://www.leydesdorff.net/> http://www.leydesdorff.net/ 
Honorary Professor, SPRU,  <http://www.sussex.ac.uk/spru/> University of
Sussex; 

Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/> , Hangzhou;
Visiting Professor, ISTIC,  <http://www.istic.ac.cn/Eng/brief_en.html>
Beijing;

Visiting Professor, Birkbeck <http://www.bbk.ac.uk/> , University of London;

http://scholar.google.com/citations?user=ych9gNYAAAAJ
<http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en> &hl=en  

From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Yves Gingras
Sent: Thursday, August 28, 2014 4:44 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] STI conference Leiden--Quality standards for
evaluation indicators

Hello all

Here is an importnt document about good and bad indicators to add up to the
discussion on research evaluation:

http://www.scienceadvice.ca/en/assessments/completed/science-performance.asp
x

The whole report is free in pdf.

Best regards

Yves Gingras

Le 27/08/14 20:21, « Ismael Rafols » <ismaelrafols at gmail.com> a écrit :

With apologies for cross-posting)

Dear all,
to warm up forweek in the ST Indicators Conference in Leiden, let us share
the topic of a debate:

Quality standards for evaluation indicators: Any chance for the dream to
come true?
Special session at the STI-ENID conference in Leiden, 3 September 2014,
16-17.30h 
Organisers: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden
University), Sarah de Rijcke (CWTS, Leiden University)
Location:  Aalmarkt-hall, Stadsgehoorzaal Leiden

There is a growing realization in the scientometrics community of the need
to offer clearer guidance to users and further develop standards for
professional use of bibliometrics in research evaluations. Indeed the
STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This
session continues from the 2013 ISSI and STI conferences in Vienna and
Berlin, where full plenary sessions were convened on the need for standards
in evaluative bibliometrics, and the ethical and policy implications of
individual-level bibliometrics. The need to debate these issues has come to
the forefront in light of reports that uses of certain easy-to-use metrics
for evaluative purposes have become a routine part of academic life, despite
misgivings within the profession itself about its validity. Very recently
high-profile movements against certain metric indicators (e.g. the DORA
declaration about the Journal Impact Factor) have brought possible misuses
of metrics further to the center of attention. There may be a growing need
for standards – also to promote for accountability of scientometricians as
experts.

Indeed the relationship between scientometricians and end-users has been
changing over the years due to factors like: 1. Increasing demands for
bibliometric services in research management at various levels of
aggregation, 2. New capacities and demands for performance information
through the greater availability of new research technologies and their
applications, and 3. The emergence of “citizen bibliometrics” (i.e.
bibliometrics carried out by non-expert end-users) due to larger
availability of data and indicators. Some of these developments may result
in new opportunities for research contributions and information-use, and may
increase effectiveness of bibliometrics due to more advanced indicators and
increased availability of data sets (including web data). Yet some
innovations also risk bypassing the quality control mechanisms of fields
like scientometrics and the standards they promote. The implications of this
increasing scope and intensity of bibliometric practices requires a
concerted response from scientometrics to produce more explicit guidelines
and expert advice on good scientometric practices for specific evaluative
practices such as recruitment, grant awards, institutional or national
benchmarking. 

This special session will bring together scientometric experts,
representatives of funding agencies, policy makers and opinion leaders on
the role of metrics in research assessment to discuss the extent to which
moving towards clearer, standardised guidelines over usage and consultancy
can be achieved, both technically and strategically, and what the guidelines
should look like concretely.

---
Background material:
- Report on International workshop "Guidelines and good practices on
quantitative assessments of research" (OST, Paris, 12 May 2014):
http://www.obs-ost.fr/fractivit%C3%A9s/workshop_international
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_internati
onal>
&URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_inter
national> 
- Blogposts Paul Wouters on previous debates at the ISSI and STI conferences
in 2013, and on the DORA declaration:
http://citationculture.wordpress.com/2013/07/29/bibliometrics-of-individual-
researchers/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibliomet
rics-of-individual-researchers%2f>
&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibli
ometrics-of-individual-researchers%2f> 
http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individual-
researchers-the-debate-in-berlin/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibliomet
rics-of-individual-researchers-the-debate-in-berlin%2f>
&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibli
ometrics-of-individual-researchers-the-debate-in-berlin%2f> 
http://citationculture.wordpress.com/2013/05/23/dora-a-stimulus-for-a-new-ev
aluation-culture-in-science/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-a-st
imulus-for-a-new-evaluation-culture-in-science%2f>
&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-
a-stimulus-for-a-new-evaluation-culture-in-science%2f> 
- Information on the Higher Education Funding Council for England (HEFCE)
"Independent review of the role of metrics in research assessment" + SPRU
response
http://citationculture.wordpress.com/2014/05/02/metrics-in-research-assessme
nt-under-review/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetrics-i
n-research-assessment-under-review%2f>
&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetri
cs-in-research-assessment-under-review%2f> 
http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetrics%2f
>
&URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetric
s%2f> 
https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.p
df
<https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.
pdf&site=25> &site=25
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fname%3d
spru-response-final.pdf%26site%3d25>
&URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fnam
e%3dspru-response-final.pdf%26site%3d25> 
- Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To
intervene, or not to intervene; is that the question? On the role of
scientometrics in research evaluation."
https://citationculture.files.wordpress.com/2014/08/de-rijcke_rushforth_jasi
st_preprint2014.pdf
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006
<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&amp
;URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-rijc
ke_rushforth_jasist_preprint2014.pdf>
&URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-
rijcke_rushforth_jasist_preprint2014.pdf> 

Yves Gingras

Professeur 
Département d'histoire
Centre interuniversitaire de recherche
sur la science et la technologie (CIRST) 
Chaire de recherche du Canada en histoire
et sociologie des sciences
Observatoire des sciences et des technologies (OST) 
UQAM
C.P. 8888, Succ. Centre-Ville
Montréal, Québec
Canada, H3C 3P8

Tel: (514)-987-3000-7053
Fax: (514)-987-7726

http://www.chss.uqam.ca
http://www.cirst.uqam.ca
http://www.ost.uqam.ca

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140828/9cb5f394/attachment.html>