STI conference Leiden--Quality standards for evaluation indicators

Thu Aug 28 05:37:57 EDT 2014

Dear All,

I wholeheartedly agree with Jonathan here.  I am a little unnerved at the discussion regarding who is considered amateurish and insinuations about who is to be considered an scientometric expert and therefore has the right to offer advice on standards etc.  The popularity of our field by different stakeholder groups is one that is to be celebrated and promoted perhaps through greater engagement of our community members with these stakeholders rather than ringfencing the advances of our field against unstructured or perhaps perceivingly unauthorised use by outsiders.  Afterall, a greater level of engagement with our growing list of stakeholders is not asking of us anything more than it is asking of other academic fields of research.

Nonetheless, some interesting points to raise next week and a discussion I am very much looking forward to hearing.  I am looking forward to seeing everybody there.

Sincerely,
Gemma

Dr Gemma E. Derrick | ESRC Future Research Leader Fellow | HERG | Brunel University London
Address: Health Economics Research Group, Brunel University, Uxbridge, Middlesex, UB8 3PH, UK
Tel: +44 (0)1895 265454
Fax: +44 (0)1895 269708
E-mail: gemma.derrick at brunel.ac.uk<mailto:gemma.derrick at brunel.ac.uk>
Website: Health Economics Research Group (HERG)<http://www.brunel.ac.uk/about/acad/herg>

________________________________
From: ASIS&T Special Interest Group on Metrics [SIGMETRICS at LISTSERV.UTK.EDU] on behalf of Jonathan Adams [j.adams at DIGITAL-SCIENCE.COM]
Sent: Thursday, August 28, 2014 10:11 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] STI conference Leiden--Quality standards for evaluation indicators

I agree with the thrust of Loet's argument.  It is dangerous for any self-appointed group to think that it can set a particular set of standards. This is surely more risky in an area with a healthy research and innovation culture that has advanced scientometric thinking and practice significantly over recent decades.

However, Isidro Aguillo recently reminded us that there is still surprisingly widespread ignorance of what good practice in scientometrics looks like, hence recourse by too many research managers to weak indicators of performance like the Journal Impact factor.  For this reason, the development of reference points for good (and bad) practice may be valuable and therefore worthwhile.  These should not be constraining, except where they point out what should not be done.

At the same time, we are few and they (research managers and bureaucrats) are many. If we are to be effective then what we say needs to be short and simple: not a handbook (or a cook-book).  And it has to reach the right audience, not just other scientometricians.

On 28 August 2014 09:33, Loet Leydesdorff <loet at leydesdorff.net<mailto:loet at leydesdorff.net>> wrote:
Dear Paul,

It seems to me that one can distinguish different stakeholders. Professional centers may wish to exploit their competitive advantages and produce “revised SNIP indicators” and “new Leiden Rankings” from time to time. (For example, the Leiden Ranking 2014 has a base for the normalization different from the Leiden Ranking 2013.) The critique and deconstruction/reconstruction of these indicators and their use is very legitimate in academia.

PhD students, for example, in a remote university are not able to normalize citation rates using sophisticated standards that may be company property. Should their efforts (e.g., using Publish or Perish) be considered as amateurish a priori (and thus be rejected)? The criteria, in my opinion, have to be intellectual: do we gain new (theoretical) insights from the critique? Of course, we need also up-to-date methods.

Best,
Loet

________________________________
Loet Leydesdorff
University of Amsterdam
Amsterdam School of Communications Research (ASCoR)
loet at leydesdorff.net <mailto:loet at leydesdorff.net> ; http://www.leydesdorff.net/
Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/> University of Sussex;
Guest Professor Zhejiang Univ.<http://www.zju.edu.cn/english/>, Hangzhou; Visiting Professor, ISTIC, <http://www.istic.ac.cn/Eng/brief_en.html> Beijing;
Visiting Professor, Birkbeck<http://www.bbk.ac.uk/>, University of London;
http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>] On Behalf Of Paul Wouters
Sent: Thursday, August 28, 2014 10:12 AM
To: SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>
Subject: Re: [SIGMETRICS] STI conference Leiden--Quality standards for evaluation indicators

Dear Loet and Lutz,

Many thanks for this contribution. The motivation for the discussion about standards, as far as I am concerned, is the need to protect research groups and researchers against sloppy or damaging evaluation practices. I agree with Loet that standards are often a powerful competition weapon to protect industry interests. It is certainly not the motivation for this panel, but it may end up like that if the process of standard setting, and the sociological interpretation of those standards, is not taken into account carefully. In my view the STI conference is the best place to have this discussion, because it is a meeting place between metrics experts and policy experts. In my view, this does not lead to the question whether or not one should have some quality control process of evaluation processes, but what kind of quality control we need and what kind of standards with respect to data and indicators can play a role in this.

In other words, you have raised a crucial point for the panel discussion next week.

Regards,

Paul Wouters
Professor of Scientometrics
Director Centre for Science and Technology Studies
Leiden University

PS: I am pleased to announce the release of our completely renewed CWTS website:
cwts.nl<http://cwts.nl> - all information now easily available!

Visiting address:
Willem Einthoven Building
Wassenaarseweg 62A
2333 AL Leiden
Mail address: P.O. Box 905
2300 AX Leiden
T: +31 71 5273909<tel:%2B31%2071%C2%A05273909> (secr.)
F: +31 71 5273911<tel:%2B31%2071%205273911>
E: p.f.wouters at cwts.leidenuniv.nl<mailto:p.f.wouters at cwts.leidenuniv.nl>

CWTS home page: www.cwts.nl<http://www.cwts.nl>
Blog about Citation Cultures: http://citationculture.wordpress.com/
Research Dreams: www.researchdreams.nl<http://www.researchdreams.nl>

On Thu, Aug 28, 2014 at 8:29 AM, Loet Leydesdorff <loet at leydesdorff.net<mailto:loet at leydesdorff.net>> wrote:
Dear Ismael,

It seems to me that we know from the innovation-science literature that standards are to the interests of incumbent firms. In this context, one can expect process innovation more than product innovations. The further development of the field, in my opinion, needs the fluidity of intellectual exchanges and the space to propose new variants.

Perhaps, as an intellectual community we have increasingly interests different from the professional practices of (quasi)industries on a market of evaluation studies that one may wish to certify ( and thus to shield the market against “amateurs”; our PhD students?).

Let me quote from a recent text (that I coauthored for other reasons):

“There exists a professional community with experts in bibliometrics who develop advanced bibliometric indicators for productivity and citation impact measurements (see an overview in Vinkler, 2010). Only experts from this community should undertake a bibliometric study. These centres of professional expertise can be found, for example, at the Centre for Science and Technology Studies (CWTS, Leiden) or the Centre for Research & Development Monitoring (ECOOM, Leuven).”

Is this the dream to come through? Or do we hear institutional interests? Perhaps, we need smaller dreams :)

Best,
Loet

________________________________
Loet Leydesdorff
University of Amsterdam
Amsterdam School of Communications Research (ASCoR)
loet at leydesdorff.net <mailto:loet at leydesdorff.net> ; http://www.leydesdorff.net/
Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/> University of Sussex;
Guest Professor Zhejiang Univ.<http://www.zju.edu.cn/english/>, Hangzhou; Visiting Professor, ISTIC, <http://www.istic.ac.cn/Eng/brief_en.html> Beijing;
Visiting Professor, Birkbeck<http://www.bbk.ac.uk/>, University of London;
http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>] On Behalf Of Ismael Rafols
Sent: Thursday, August 28, 2014 2:21 AM
To: SIGMETRICS at LISTSERV.UTK.EDU<mailto:SIGMETRICS at LISTSERV.UTK.EDU>
Subject: [SIGMETRICS] STI conference Leiden--Quality standards for evaluation indicators

With apologies for cross-posting)

Dear all,
to warm up forweek in the ST Indicators Conference in Leiden, let us share the topic of a debate:
Quality standards for evaluation indicators: Any chance for the dream to come true?
Special session at the STI-ENID conference in Leiden, 3 September 2014, 16-17.30h
Organisers: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden University), Sarah de Rijcke (CWTS, Leiden University)
Location:  Aalmarkt-hall, Stadsgehoorzaal Leiden

There is a growing realization in the scientometrics community of the need to offer clearer guidance to users and further develop standards for professional use of bibliometrics in research evaluations. Indeed the STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This session continues from the 2013 ISSI and STI conferences in Vienna and Berlin, where full plenary sessions were convened on the need for standards in evaluative bibliometrics, and the ethical and policy implications of individual-level bibliometrics. The need to debate these issues has come to the forefront in light of reports that uses of certain easy-to-use metrics for evaluative purposes have become a routine part of academic life, despite misgivings within the profession itself about its validity. Very recently high-profile movements against certain metric indicators (e.g. the DORA declaration about the Journal Impact Factor) have brought possible misuses of metrics further to the center of attention. There may be a growing need for standards – also to promote for accountability of scientometricians as experts.

Indeed the relationship between scientometricians and end-users has been changing over the years due to factors like: 1. Increasing demands for bibliometric services in research management at various levels of aggregation, 2. New capacities and demands for performance information through the greater availability of new research technologies and their applications, and 3. The emergence of “citizen bibliometrics” (i.e. bibliometrics carried out by non-expert end-users) due to larger availability of data and indicators. Some of these developments may result in new opportunities for research contributions and information-use, and may increase effectiveness of bibliometrics due to more advanced indicators and increased availability of data sets (including web data). Yet some innovations also risk bypassing the quality control mechanisms of fields like scientometrics and the standards they promote. The implications of this increasing scope and intensity of bibliometric practices requires a concerted response from scientometrics to produce more explicit guidelines and expert advice on good scientometric practices for specific evaluative practices such as recruitment, grant awards, institutional or national benchmarking.

This special session will bring together scientometric experts, representatives of funding agencies, policy makers and opinion leaders on the role of metrics in research assessment to discuss the extent to which moving towards clearer, standardised guidelines over usage and consultancy can be achieved, both technically and strategically, and what the guidelines should look like concretely.

---
Background material:
- Report on International workshop "Guidelines and good practices on quantitative assessments of research" (OST, Paris, 12 May 2014):  MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be http://www.obs-ost.fr/fractivit%C3%A9s/workshop_international<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_international>
- Blogposts Paul Wouters on previous debates at the ISSI and STI conferences in 2013, and on the DORA declaration:
MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be http://citationculture.wordpress.com/2013/07/29/bibliometrics-of-individual-researchers/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibliometrics-of-individual-researchers%2f>
MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individual-researchers-the-debate-in-berlin/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibliometrics-of-individual-researchers-the-debate-in-berlin%2f>
MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be http://citationculture.wordpress.com/2013/05/23/dora-a-stimulus-for-a-new-evaluation-culture-in-science/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-a-stimulus-for-a-new-evaluation-culture-in-science%2f>
- Information on the Higher Education Funding Council for England (HEFCE) "Independent review of the role of metrics in research assessment" + SPRU response
MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be http://citationculture.wordpress.com/2014/05/02/metrics-in-research-assessment-under-review/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetrics-in-research-assessment-under-review%2f>
MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetrics%2f>
MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.pdf&site=25<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fname%3dspru-response-final.pdf%26site%3d25>
- Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To intervene, or not to intervene; is that the question? On the role of scientometrics in research evaluation."
MailScanner has detected a possible fraud attempt from "agenda.upv.es" claiming to be https://citationculture.files.wordpress.com/2014/08/de-rijcke_rushforth_jasist_preprint2014.pdf<https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-rijcke_rushforth_jasist_preprint2014.pdf>

--
Sincere regards,

Dr Jonathan Adams
Chief Scientist, Digital Science
Visiting Professor, King's College London

M/ +44 7964 908449
E/ j.adams at digital-science.com<mailto:j.adams at digital-science.com>

Macmillan Publishers Ltd
4 Crinan Street
London N1 9XW, UK

http://www.timeshighereducation.co.uk/news/research/research-intelligence-proof-is-in-the-numbers/411118.article

********************************************************************************
DISCLAIMER: This e-mail is confidential and should not be used by anyone who is
not the original intended recipient. If you have received this e-mail in error
please inform the sender and delete it from your mailbox or any other storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail or
its attachments and it is your responsibility to scan the e-mail and
attachments (if any). No contracts may be concluded on behalf of Macmillan
Publishers Limited or its agents by means of e-mail communication. Macmillan
Publishers Limited Registered in England and Wales with registered number 785998
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS
********************************************************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140828/902c6e47/attachment.html>