STI conference Leiden--Quality standards for evaluation indicators

Thu Aug 28 05:11:07 EDT 2014

I agree with the thrust of Loet's argument.  It is dangerous for any
self-appointed group to think that it can set a particular set of
standards. This is surely more risky in an area with a healthy research and
innovation culture that has advanced scientometric thinking and practice
significantly over recent decades.

However, Isidro Aguillo recently reminded us that there is still
surprisingly widespread ignorance of what good practice in scientometrics
looks like, hence recourse by too many research managers to weak indicators
of performance like the Journal Impact factor.  For this reason, the
development of reference points for good (and bad) practice may be valuable
and therefore worthwhile.  These should not be constraining, except where
they point out what should not be done.

At the same time, we are few and they (research managers and bureaucrats)
are many. If we are to be effective then what we say needs to be short and
simple: not a handbook (or a cook-book).  And it has to reach the right
audience, not just other scientometricians.

On 28 August 2014 09:33, Loet Leydesdorff <loet at leydesdorff.net> wrote:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Paul,
>
>
>
> It seems to me that one can distinguish different stakeholders.
> Professional centers may wish to exploit their competitive advantages and
> produce “revised SNIP indicators” and “new Leiden Rankings” from time to
> time. (For example, the Leiden Ranking 2014 has a base for the
> normalization different from the Leiden Ranking 2013.) The critique and
> deconstruction/reconstruction of these indicators and their use is very
> legitimate in academia.
>
>
>
> PhD students, for example, in a remote university are not able to
> normalize citation rates using sophisticated standards that may be company
> property. Should their efforts (e.g., using Publish or Perish) be
> considered as amateurish a priori (and thus be rejected)? The criteria, in
> my opinion, have to be intellectual: do we gain new (theoretical) insights
> from the critique? Of course, we need also up-to-date methods.
>
>
>
> Best,
>
> Loet
>
>
> ------------------------------
>
> Loet Leydesdorff
>
> University of Amsterdam
> Amsterdam School of Communications Research (ASCoR)
>
> loet at leydesdorff.net ; http://www.leydesdorff.net/
> Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/>University of
> Sussex;
>
> Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/>,
> Hangzhou; Visiting Professor, ISTIC,
> <http://www.istic.ac.cn/Eng/brief_en.html>Beijing;
>
> Visiting Professor, Birkbeck <http://www.bbk.ac.uk/>, University of
> London;
>
> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>
>
>
> *From:* ASIS&T Special Interest Group on Metrics [mailto:
> SIGMETRICS at LISTSERV.UTK.EDU] *On Behalf Of *Paul Wouters
> *Sent:* Thursday, August 28, 2014 10:12 AM
> *To:* SIGMETRICS at LISTSERV.UTK.EDU
> *Subject:* Re: [SIGMETRICS] STI conference Leiden--Quality standards for
> evaluation indicators
>
>
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Loet and Lutz,
>
>
>
> Many thanks for this contribution. The motivation for the discussion about
> standards, as far as I am concerned, is the need to protect research groups
> and researchers against sloppy or damaging evaluation practices. I agree
> with Loet that standards are often a powerful competition weapon to protect
> industry interests. It is certainly not the motivation for this panel, but
> it may end up like that if the process of standard setting, and the
> sociological interpretation of those standards, is not taken into account
> carefully. In my view the STI conference is the best place to have this
> discussion, because it is a meeting place between metrics experts and
> policy experts. In my view, this does not lead to the question whether or
> not one should have some quality control process of evaluation processes,
> but what kind of quality control we need and what kind of standards with
> respect to data and indicators can play a role in this.
>
>
>
> In other words, you have raised a crucial point for the panel discussion
> next week.
>
>
>
> Regards,
>
>
>
>
> Paul Wouters
>
> Professor of Scientometrics
> Director Centre for Science and Technology Studies
> Leiden University
>
>
>
> PS: I am pleased to announce the release of our completely renewed CWTS
> website:
>
> cwts.nl - all information now easily available!
>
>
>
> Visiting address:
> Willem Einthoven Building
> Wassenaarseweg 62A
> 2333 AL Leiden
> Mail address: P.O. Box 905
> 2300 AX Leiden
> T: +31 71 5273909 (secr.)
> F: +31 71 5273911
> E: p.f.wouters at cwts.leidenuniv.nl
>
> CWTS home page: www.cwts.nl
> Blog about Citation Cultures: http://citationculture.wordpress.com/
> Research Dreams: www.researchdreams.nl
>
>
>
> On Thu, Aug 28, 2014 at 8:29 AM, Loet Leydesdorff <loet at leydesdorff.net>
> wrote:
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Ismael,
>
>
>
> It seems to me that we know from the innovation-science literature that
> standards are to the interests of incumbent firms. In this context, one can
> expect process innovation more than product innovations. The further
> development of the field, in my opinion, needs the fluidity of intellectual
> exchanges and the space to propose new variants.
>
>
>
> Perhaps, as an intellectual community we have increasingly interests
> different from the professional practices of (quasi)industries on a market
> of evaluation studies that one may wish to certify ( and thus to shield the
> market against “amateurs”; our PhD students?).
>
>
>
> Let me quote from a recent text (that I coauthored for other reasons):
>
>
>
> “There exists a professional community with experts in bibliometrics who
> develop advanced bibliometric indicators for productivity and citation
> impact measurements (see an overview in Vinkler, 2010). Only experts from
> this community should undertake a bibliometric study. These centres of
> professional expertise can be found, for example, at the Centre for Science
> and Technology Studies (CWTS, Leiden) or the Centre for Research &
> Development Monitoring (ECOOM, Leuven).”
>
>
>
> Is this the dream to come through? Or do we hear institutional interests?
> Perhaps, we need smaller dreams J
>
>
>
> Best,
>
> Loet
>
>
> ------------------------------
>
> Loet Leydesdorff
>
> University of Amsterdam
> Amsterdam School of Communications Research (ASCoR)
>
> loet at leydesdorff.net ; http://www.leydesdorff.net/
> Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/>University of
> Sussex;
>
> Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/>,
> Hangzhou; Visiting Professor, ISTIC,
> <http://www.istic.ac.cn/Eng/brief_en.html>Beijing;
>
> Visiting Professor, Birkbeck <http://www.bbk.ac.uk/>, University of
> London;
>
> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>
>
>
> *From:* ASIS&T Special Interest Group on Metrics [mailto:
> SIGMETRICS at LISTSERV.UTK.EDU] *On Behalf Of *Ismael Rafols
> *Sent:* Thursday, August 28, 2014 2:21 AM
> *To:* SIGMETRICS at LISTSERV.UTK.EDU
> *Subject:* [SIGMETRICS] STI conference Leiden--Quality standards for
> evaluation indicators
>
>
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> With apologies for cross-posting)
>
> Dear all,
> to warm up forweek in the ST Indicators Conference in Leiden, let us share
> the topic of a debate:
>
> *Quality standards for evaluation indicators: Any chance for the dream to
> come true?*
>
> *Special session at the STI-ENID conference in Leiden, 3 September
> 2014, 16-17.30h *
>
> *Organisers*: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden
> University), Sarah de Rijcke (CWTS, Leiden University)
>
> *Location*:  Aalmarkt-hall, Stadsgehoorzaal Leiden
>
>
>
> There is a growing realization in the scientometrics community of the need
> to offer clearer guidance to users and further develop standards for
> professional use of bibliometrics in research evaluations. Indeed the
> STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This
> session continues from the 2013 ISSI and STI conferences in Vienna and
> Berlin, where full plenary sessions were convened on the need for standards
> in evaluative bibliometrics, and the ethical and policy implications of
> individual-level bibliometrics. The need to debate these issues has come to
> the forefront in light of reports that uses of certain easy-to-use metrics
> for evaluative purposes have become a routine part of academic life,
> despite misgivings within the profession itself about its validity. Very
> recently high-profile movements against certain metric indicators (e.g. the
> DORA declaration about the Journal Impact Factor) have brought possible
> misuses of metrics further to the center of attention. There may be a
> growing need for standards – also to promote for accountability of
> scientometricians as experts.
>
>
>
> Indeed the relationship between scientometricians and end-users has been
> changing over the years due to factors like: 1. Increasing demands for
> bibliometric services in research management at various levels of
> aggregation, 2. New capacities and demands for performance information
> through the greater availability of new research technologies and their
> applications, and 3. The emergence of “citizen bibliometrics” (i.e.
> bibliometrics carried out by non-expert end-users) due to larger
> availability of data and indicators. Some of these developments may result
> in new opportunities for research contributions and information-use, and
> may increase effectiveness of bibliometrics due to more advanced indicators
> and increased availability of data sets (including web data). Yet some
> innovations also risk bypassing the quality control mechanisms of fields
> like scientometrics and the standards they promote. The implications of
> this increasing scope and intensity of bibliometric practices requires a
> concerted response from scientometrics to produce more explicit guidelines
> and expert advice on good scientometric practices for specific evaluative
> practices such as recruitment, grant awards, institutional or national
> benchmarking.
>
>
>
> This special session will bring together scientometric experts,
> representatives of funding agencies, policy makers and opinion leaders on
> the role of metrics in research assessment to discuss the extent to which
> moving towards clearer, standardised guidelines over usage and consultancy
> can be achieved, both technically and strategically, and what the
> guidelines should look like concretely.
>
>
>
> ---
>
> *Background material*:
>
> - Report on International workshop "Guidelines and good practices on
> quantitative assessments of research" (OST, Paris, 12 May 2014):
> http://www.obs-ost.fr/fractivit%C3%A9s/workshop_international
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_international>
>
> - Blogposts Paul Wouters on previous debates at the ISSI and STI
> conferences in 2013, and on the DORA declaration:
>
>
> http://citationculture.wordpress.com/2013/07/29/bibliometrics-of-individual-researchers/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibliometrics-of-individual-researchers%2f>
>
>
> http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individual-researchers-the-debate-in-berlin/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibliometrics-of-individual-researchers-the-debate-in-berlin%2f>
>
>
> http://citationculture.wordpress.com/2013/05/23/dora-a-stimulus-for-a-new-evaluation-culture-in-science/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-a-stimulus-for-a-new-evaluation-culture-in-science%2f>
>
> - Information on the Higher Education Funding Council for England (HEFCE)
> "Independent review of the role of metrics in research assessment" + SPRU
> response
>
>
> http://citationculture.wordpress.com/2014/05/02/metrics-in-research-assessment-under-review/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetrics-in-research-assessment-under-review%2f>
>
> http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetrics%2f>
>
>
> https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.pdf&site=25
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fname%3dspru-response-final.pdf%26site%3d25>
>
> - Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To
> intervene, or not to intervene; is that the question? On the role of
> scientometrics in research evaluation."
>
>
> https://citationculture.files.wordpress.com/2014/08/de-rijcke_rushforth_jasist_preprint2014.pdf
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-rijcke_rushforth_jasist_preprint2014.pdf>
>
>
>
>
>

-- 
Sincere regards,

Dr Jonathan Adams
Chief Scientist, Digital Science
Visiting Professor, King's College London

M/ +44 7964 908449
E/ j.adams at digital-science.com

Macmillan Publishers Ltd
4 Crinan Street
London N1 9XW, UK

http://www.timeshighereducation.co.uk/news/research/research-intelligence-proof-is-in-the-numbers/411118.article

-- 
******************************************************************************** 

DISCLAIMER: This e-mail is confidential and should not be used by anyone 
who is
not the original intended recipient. If you have received this e-mail in 
error
please inform the sender and delete it from your mailbox or any other 
storage
mechanism. Neither Macmillan Publishers Limited nor any of its agents accept
liability for any statements made which are clearly the sender's own and not
expressly made on behalf of Macmillan Publishers Limited or one of its 
agents.
Please note that neither Macmillan Publishers Limited nor any of its agents
accept any responsibility for viruses that may be contained in this e-mail 
or
its attachments and it is your responsibility to scan the e-mail and 
attachments (if any). No contracts may be concluded on behalf of Macmillan 
Publishers Limited or its agents by means of e-mail communication. 
Macmillan 
Publishers Limited Registered in England and Wales with registered number 
785998 
Registered Office Brunel Road, Houndmills, Basingstoke RG21 6XS   
********************************************************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140828/cddb7a65/attachment.html>