STI conference Leiden--Quality standards for evaluation indicators

Thu Aug 28 04:38:17 EDT 2014

Dear Ismael,

Since it is very difficult to formulate standards to which all experts agree, one could formulate as standard that bibliometric evaluations should be done by an expert in bibliometrics. There are a lot of problems with a standardized practice (which should be followed by a non-expert): For example, the advanced bibliometric indicators are so complex now that they cannot be understand (not to mentioned used) by non-experts. Standards change over time and one has to be up-to-date. Furthermore, there are different standards for different contexts necessary.

Best,

Lutz

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Ismael Rafols
Sent: Thursday, August 28, 2014 10:22 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] STI conference Leiden--Quality standards for evaluation indicators

Adminstrative info for SIGMETRICS (for example unsubscribe): http://web.utk.edu/~gwhitney/sigmetrics.html
Thanks Loet,

the role of professional standards is indeed to certify good practices and this may not have neutral effects --notice that I say "professional" standards. The debate is precisely (thanks for stressing it!) WHICH form of standardisation is desirable given that there is discontent among users about the proliferations of metrics (some of dubious reliability), and there are initiatives setting de facto standards (http://www.snowballmetrics.com/) without wider stakeholder engagement.

As Jochen Gläser pointed out last year, one may want to distinguish professional practices -offered to users, which have professional repercussions and can do good or harm-- from from scientometric research. Just the same way that you distinguish between medical practices with patients from experimentation with mice in the lab. Medical standards do not suppress research in the lab --though do pose ethical questions on how research is conducted.

We look forward to the discussion next week!
Ismael

2014-08-28 10:12 GMT+02:00 Paul Wouters <p.f.wouters at cwts.leidenuniv.nl<mailto:p.f.wouters at cwts.leidenuniv.nl>>:
Adminstrative info for SIGMETRICS (for example unsubscribe): http://web.utk.edu/~gwhitney/sigmetrics.html
Dear Loet and Lutz,

Many thanks for this contribution. The motivation for the discussion about standards, as far as I am concerned, is the need to protect research groups and researchers against sloppy or damaging evaluation practices. I agree with Loet that standards are often a powerful competition weapon to protect industry interests. It is certainly not the motivation for this panel, but it may end up like that if the process of standard setting, and the sociological interpretation of those standards, is not taken into account carefully. In my view the STI conference is the best place to have this discussion, because it is a meeting place between metrics experts and policy experts. In my view, this does not lead to the question whether or not one should have some quality control process of evaluation processes, but what kind of quality control we need and what kind of standards with respect to data and indicators can play a role in this.

In other words, you have raised a crucial point for the panel discussion next week.

Regards,

Paul Wouters
Professor of Scientometrics
Director Centre for Science and Technology Studies
Leiden University

PS: I am pleased to announce the release of our completely renewed CWTS website:
cwts.nl<http://cwts.nl> - all information now easily available!

Visiting address:
Willem Einthoven Building
Wassenaarseweg 62A
2333 AL Leiden
Mail address: P.O. Box 905
2300 AX Leiden
T: +31 71 5273909<tel:%2B31%2071%C2%A05273909> (secr.)
F: +31 71 5273911<tel:%2B31%2071%205273911>
E: p.f.wouters at cwts.leidenuniv.nl<mailto:p.f.wouters at cwts.leidenuniv.nl>

CWTS home page: www.cwts.nl<http://www.cwts.nl>
Blog about Citation Cultures: http://citationculture.wordpress.com/
Research Dreams: www.researchdreams.nl<http://www.researchdreams.nl>

On Thu, Aug 28, 2014 at 8:29 AM, Loet Leydesdorff <loet at leydesdorff.net<mailto:loet at leydesdorff.net>> wrote:
Adminstrative info for SIGMETRICS (for example unsubscribe): http://web.utk.edu/~gwhitney/sigmetrics.html
Dear Ismael,

It seems to me that we know from the innovation-science literature that standards are to the interests of incumbent firms. In this context, one can expect process innovation more than product innovations. The further development of the field, in my opinion, needs the fluidity of intellectual exchanges and the space to propose new variants.

Perhaps, as an intellectual community we have increasingly interests different from the professional practices of (quasi)industries on a market of evaluation studies that one may wish to certify ( and thus to shield the market against “amateurs”; our PhD students?).

Let me quote from a recent text (that I coauthored for other reasons):

“There exists a professional community with experts in bibliometrics who develop advanced bibliometric indicators for productivity and citation impact measurements (see an overview in Vinkler, 2010). Only experts from this community should undertake a bibliometric study. These centres of professional expertise can be found, for example, at the Centre for Science and Technology Studies (CWTS, Leiden) or the Centre for Research & Development Monitoring (ECOOM, Leuven).”

Is this the dream to come through? Or do we hear institutional interests? Perhaps, we need smaller dreams ☺

Best,
Loet

________________________________
Loet Leydesdorff
University of Amsterdam
Amsterdam School of Communications Research (ASCoR)
loet at leydesdorff.net <mailto:loet at leydesdorff.net> ; http://www.leydesdorff.net/
Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/> University of Sussex;
Guest Professor Zhejiang Univ.<http://www.zju.edu.cn/english/>, Hangzhou; Visiting Professor, ISTIC, <http://www.istic.ac.cn/Eng/brief_en.html> Beijing;
Visiting Professor, Birkbeck<http://www.bbk.ac.uk/>, University of London;
http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>] On Behalf Of Ismael Rafols
Sent: Thursday, August 28, 2014 2:21 AM
To: SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>
Subject: [SIGMETRICS] STI conference Leiden--Quality standards for evaluation indicators

Adminstrative info for SIGMETRICS (for example unsubscribe): http://web.utk.edu/~gwhitney/sigmetrics.html
With apologies for cross-posting)

Dear all,
to warm up forweek in the ST Indicators Conference in Leiden, let us share the topic of a debate:
Quality standards for evaluation indicators: Any chance for the dream to come true?
Special session at the STI-ENID conference in Leiden, 3 September 2014, 16-17.30h
Organisers: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden University), Sarah de Rijcke (CWTS, Leiden University)
Location:  Aalmarkt-hall, Stadsgehoorzaal Leiden

There is a growing realization in the scientometrics community of the need to offer clearer guidance to users and further develop standards for professional use of bibliometrics in research evaluations. Indeed the STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This session continues from the 2013 ISSI and STI conferences in Vienna and Berlin, where full plenary sessions were convened on the need for standards in evaluative bibliometrics, and the ethical and policy implications of individual-level bibliometrics. The need to debate these issues has come to the forefront in light of reports that uses of certain easy-to-use metrics for evaluative purposes have become a routine part of academic life, despite misgivings within the profession itself about its validity. Very recently high-profile movements against certain metric indicators (e.g. the DORA declaration about the Journal Impact Factor) have brought possible misuses of metrics further to the center of attention. There may be a growing need for standards – also to promote for accountability of scientometricians as experts.

Indeed the relationship between scientometricians and end-users has been changing over the years due to factors like: 1. Increasing demands for bibliometric services in research management at various levels of aggregation, 2. New capacities and demands for performance information through the greater availability of new research technologies and their applications, and 3. The emergence of “citizen bibliometrics” (i.e. bibliometrics carried out by non-expert end-users) due to larger availability of data and indicators. Some of these developments may result in new opportunities for research contributions and information-use, and may increase effectiveness of bibliometrics due to more advanced indicators and increased availability of data sets (including web data). Yet some innovations also risk bypassing the quality control mechanisms of fields like scientometrics and the standards they promote. The implications of this increasing scope and intensity of bibliometric practices requires a concerted response from scientometrics to produce more explicit guidelines and expert advice on good scientometric practices for specific evaluative practices such as recruitment, grant awards, institutional or national benchmarking.

This special session will bring together scientometric experts, representatives of funding agencies, policy makers and opinion leaders on the role of metrics in research assessment to discuss the extent to which moving towards clearer, standardised guidelines over usage and consultancy can be achieved, both technically and strategically, and what the guidelines should look like concretely.

---
Background material:
- Report on International workshop "Guidelines and good practices on quantitative assessments of research" (OST, Paris, 12 May 2014):  http://www.obs-ost.fr/fractivit%C3%A9s/workshop_international<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_international>
- Blogposts Paul Wouters on previous debates at the ISSI and STI conferences in 2013, and on the DORA declaration:
http://citationculture.wordpress.com/2013/07/29/bibliometrics-of-individual-researchers/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibliometrics-of-individual-researchers%2f>
http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individual-researchers-the-debate-in-berlin/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibliometrics-of-individual-researchers-the-debate-in-berlin%2f>
http://citationculture.wordpress.com/2013/05/23/dora-a-stimulus-for-a-new-evaluation-culture-in-science/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-a-stimulus-for-a-new-evaluation-culture-in-science%2f>
- Information on the Higher Education Funding Council for England (HEFCE) "Independent review of the role of metrics in research assessment" + SPRU response
http://citationculture.wordpress.com/2014/05/02/metrics-in-research-assessment-under-review/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetrics-in-research-assessment-under-review%2f>
http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetrics%2f>
https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.pdf&site=25<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fname%3dspru-response-final.pdf%26site%3d25>
- Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To intervene, or not to intervene; is that the question? On the role of scientometrics in research evaluation."
https://citationculture.files.wordpress.com/2014/08/de-rijcke_rushforth_jasist_preprint2014.pdf<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-rijcke_rushforth_jasist_preprint2014.pdf>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140828/7ba6fad3/attachment.html>