STI conference Leiden--Quality standards for evaluation indicators

[Paul Wouters]

Dear Loet, we clearly agree on the existence of this tension. I think, this
discussion thread is an excellent starting point to further explore next
week!

With regards,



Paul Wouters
Professor of Scientometrics
Director Centre for Science and Technology Studies
Leiden University

PS: I am pleased to announce the release of our completely renewed CWTS
website:
cwts.nl - all information now easily available!

Visiting address:
Willem Einthoven Building
Wassenaarseweg 62A
2333 AL Leiden
Mail address: P.O. Box 905
2300 AX Leiden
T: +31 71 5273909 (secr.)
F: +31 71 5273911
E: p.f.wouters at cwts.leidenuniv.nl

CWTS home page: www.cwts.nl
Blog about Citation Cultures: http://citationculture.wordpress.com/
Research Dreams: www.researchdreams.nl


On Thu, Aug 28, 2014 at 2:18 PM, Loet Leydesdorff <loet at leydesdorff.net>
wrote:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Paul,
>
>
>
> I did not want to hurt anybody; I continuously study the papers from your
> excellent center and use the tools in my own research. I look forward to
> the meeting next week.
>
>
>
> The contribution was meant analytically: we are in the middle of a tension
> between professional practices that need standards (and standards to be
> further developed), and an academic environment in which novelty is
> crucial. One can analyze this in terms of stakeholders, value systems, or –
> more our approach ?– differently codified (yet blurring) discourses.
>
>
>
> Best,
>
> Loet
>
>
> ------------------------------
>
> Loet Leydesdorff
>
> University of Amsterdam
> Amsterdam School of Communications Research (ASCoR)
>
> loet at leydesdorff.net ; http://www.leydesdorff.net/
> Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/>University of
> Sussex;
>
> Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/>,
> Hangzhou; Visiting Professor, ISTIC,
> <http://www.istic.ac.cn/Eng/brief_en.html>Beijing;
>
> Visiting Professor, Birkbeck <http://www.bbk.ac.uk/>, University of
> London;
>
> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>
>
>
> *From:* ASIS&T Special Interest Group on Metrics [mailto:
> SIGMETRICS at LISTSERV.UTK.EDU] *On Behalf Of *Paul Wouters
> *Sent:* Thursday, August 28, 2014 11:14 AM
>
> *To:* SIGMETRICS at LISTSERV.UTK.EDU
> *Subject:* Re: [SIGMETRICS] STI conference Leiden--Quality standards for
> evaluation indicators
>
>
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> My dear Loet, I have the feeling that you are opening doors that are
> already wide open. Of course work of PhD students on the basis of the data
> they have available should not be rejected as amateurish! Indeed, only
> intellectual criteria should be used.
>
>
>
> I think professional centers are regularly improving their work not only
> for competitive reasons but also for intellectual reasons. For example,
> CWTS is (or if one wishes has become) first and foremost an academic
> research institute that also provides commercial services. But the
> intellectual agenda is dominant (I am not denying that there are of course
> tensions between the two missions). The change in the Leiden Ranking re
> normalization is motivated primarily by the problems with the WoS subject
> categories that you, as we ourselves and others, have identified. I find it
> ironic that you see this only as a commercial move! Ludo Waltman wil give a
> paper on our new classification system at the STI conference next week.
>
>
>
> Last, I agree with Jonathan's argument about the main direction the
> initiative re quality checks on evaluation should take.
>
>
>
>
> Paul Wouters
>
> Professor of Scientometrics
> Director Centre for Science and Technology Studies
> Leiden University
>
>
>
> PS: I am pleased to announce the release of our completely renewed CWTS
> website:
>
> cwts.nl - all information now easily available!
>
>
>
> Visiting address:
> Willem Einthoven Building
> Wassenaarseweg 62A
> 2333 AL Leiden
> Mail address: P.O. Box 905
> 2300 AX Leiden
> T: +31 71 5273909 (secr.)
> F: +31 71 5273911
> E: p.f.wouters at cwts.leidenuniv.nl
>
> CWTS home page: www.cwts.nl
> Blog about Citation Cultures: http://citationculture.wordpress.com/
> Research Dreams: www.researchdreams.nl
>
>
>
> On Thu, Aug 28, 2014 at 10:33 AM, Loet Leydesdorff <loet at leydesdorff.net>
> wrote:
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Paul,
>
>
>
> It seems to me that one can distinguish different stakeholders.
> Professional centers may wish to exploit their competitive advantages and
> produce “revised SNIP indicators” and “new Leiden Rankings” from time to
> time. (For example, the Leiden Ranking 2014 has a base for the
> normalization different from the Leiden Ranking 2013.) The critique and
> deconstruction/reconstruction of these indicators and their use is very
> legitimate in academia.
>
>
>
> PhD students, for example, in a remote university are not able to
> normalize citation rates using sophisticated standards that may be company
> property. Should their efforts (e.g., using Publish or Perish) be
> considered as amateurish a priori (and thus be rejected)? The criteria, in
> my opinion, have to be intellectual: do we gain new (theoretical) insights
> from the critique? Of course, we need also up-to-date methods.
>
>
>
> Best,
>
> Loet
>
>
> ------------------------------
>
> Loet Leydesdorff
>
> University of Amsterdam
> Amsterdam School of Communications Research (ASCoR)
>
> loet at leydesdorff.net ; http://www.leydesdorff.net/
> Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/>University of
> Sussex;
>
> Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/>,
> Hangzhou; Visiting Professor, ISTIC,
> <http://www.istic.ac.cn/Eng/brief_en.html>Beijing;
>
> Visiting Professor, Birkbeck <http://www.bbk.ac.uk/>, University of
> London;
>
> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>
>
>
> *From:* ASIS&T Special Interest Group on Metrics [mailto:
> SIGMETRICS at LISTSERV.UTK.EDU] *On Behalf Of *Paul Wouters
> *Sent:* Thursday, August 28, 2014 10:12 AM
> *To:* SIGMETRICS at LISTSERV.UTK.EDU
> *Subject:* Re: [SIGMETRICS] STI conference Leiden--Quality standards for
> evaluation indicators
>
>
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Loet and Lutz,
>
>
>
> Many thanks for this contribution. The motivation for the discussion about
> standards, as far as I am concerned, is the need to protect research groups
> and researchers against sloppy or damaging evaluation practices. I agree
> with Loet that standards are often a powerful competition weapon to protect
> industry interests. It is certainly not the motivation for this panel, but
> it may end up like that if the process of standard setting, and the
> sociological interpretation of those standards, is not taken into account
> carefully. In my view the STI conference is the best place to have this
> discussion, because it is a meeting place between metrics experts and
> policy experts. In my view, this does not lead to the question whether or
> not one should have some quality control process of evaluation processes,
> but what kind of quality control we need and what kind of standards with
> respect to data and indicators can play a role in this.
>
>
>
> In other words, you have raised a crucial point for the panel discussion
> next week.
>
>
>
> Regards,
>
>
>
>
> Paul Wouters
>
> Professor of Scientometrics
> Director Centre for Science and Technology Studies
> Leiden University
>
>
>
> PS: I am pleased to announce the release of our completely renewed CWTS
> website:
>
> cwts.nl - all information now easily available!
>
>
>
> Visiting address:
> Willem Einthoven Building
> Wassenaarseweg 62A
> 2333 AL Leiden
> Mail address: P.O. Box 905
> 2300 AX Leiden
> T: +31 71 5273909 (secr.)
> F: +31 71 5273911
> E: p.f.wouters at cwts.leidenuniv.nl
>
> CWTS home page: www.cwts.nl
> Blog about Citation Cultures: http://citationculture.wordpress.com/
> Research Dreams: www.researchdreams.nl
>
>
>
> On Thu, Aug 28, 2014 at 8:29 AM, Loet Leydesdorff <loet at leydesdorff.net>
> wrote:
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> Dear Ismael,
>
>
>
> It seems to me that we know from the innovation-science literature that
> standards are to the interests of incumbent firms. In this context, one can
> expect process innovation more than product innovations. The further
> development of the field, in my opinion, needs the fluidity of intellectual
> exchanges and the space to propose new variants.
>
>
>
> Perhaps, as an intellectual community we have increasingly interests
> different from the professional practices of (quasi)industries on a market
> of evaluation studies that one may wish to certify ( and thus to shield the
> market against “amateurs”; our PhD students?).
>
>
>
> Let me quote from a recent text (that I coauthored for other reasons):
>
>
>
> “There exists a professional community with experts in bibliometrics who
> develop advanced bibliometric indicators for productivity and citation
> impact measurements (see an overview in Vinkler, 2010). Only experts from
> this community should undertake a bibliometric study. These centres of
> professional expertise can be found, for example, at the Centre for Science
> and Technology Studies (CWTS, Leiden) or the Centre for Research &
> Development Monitoring (ECOOM, Leuven).”
>
>
>
> Is this the dream to come through? Or do we hear institutional interests?
> Perhaps, we need smaller dreams J
>
>
>
> Best,
>
> Loet
>
>
> ------------------------------
>
> Loet Leydesdorff
>
> University of Amsterdam
> Amsterdam School of Communications Research (ASCoR)
>
> loet at leydesdorff.net ; http://www.leydesdorff.net/
> Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/>University of
> Sussex;
>
> Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/>,
> Hangzhou; Visiting Professor, ISTIC,
> <http://www.istic.ac.cn/Eng/brief_en.html>Beijing;
>
> Visiting Professor, Birkbeck <http://www.bbk.ac.uk/>, University of
> London;
>
> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>
>
>
> *From:* ASIS&T Special Interest Group on Metrics [mailto:
> SIGMETRICS at LISTSERV.UTK.EDU] *On Behalf Of *Ismael Rafols
> *Sent:* Thursday, August 28, 2014 2:21 AM
> *To:* SIGMETRICS at LISTSERV.UTK.EDU
> *Subject:* [SIGMETRICS] STI conference Leiden--Quality standards for
> evaluation indicators
>
>
>
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> With apologies for cross-posting)
>
> Dear all,
> to warm up forweek in the ST Indicators Conference in Leiden, let us share
> the topic of a debate:
>
> *Quality standards for evaluation indicators: Any chance for the dream to
> come true?*
>
> *Special session at the STI-ENID conference in Leiden, 3 September
> 2014, 16-17.30h *
>
> *Organisers*: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden
> University), Sarah de Rijcke (CWTS, Leiden University)
>
> *Location*:  Aalmarkt-hall, Stadsgehoorzaal Leiden
>
>
>
> There is a growing realization in the scientometrics community of the need
> to offer clearer guidance to users and further develop standards for
> professional use of bibliometrics in research evaluations. Indeed the
> STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This
> session continues from the 2013 ISSI and STI conferences in Vienna and
> Berlin, where full plenary sessions were convened on the need for standards
> in evaluative bibliometrics, and the ethical and policy implications of
> individual-level bibliometrics. The need to debate these issues has come to
> the forefront in light of reports that uses of certain easy-to-use metrics
> for evaluative purposes have become a routine part of academic life,
> despite misgivings within the profession itself about its validity. Very
> recently high-profile movements against certain metric indicators (e.g. the
> DORA declaration about the Journal Impact Factor) have brought possible
> misuses of metrics further to the center of attention. There may be a
> growing need for standards – also to promote for accountability of
> scientometricians as experts.
>
>
>
> Indeed the relationship between scientometricians and end-users has been
> changing over the years due to factors like: 1. Increasing demands for
> bibliometric services in research management at various levels of
> aggregation, 2. New capacities and demands for performance information
> through the greater availability of new research technologies and their
> applications, and 3. The emergence of “citizen bibliometrics” (i.e.
> bibliometrics carried out by non-expert end-users) due to larger
> availability of data and indicators. Some of these developments may result
> in new opportunities for research contributions and information-use, and
> may increase effectiveness of bibliometrics due to more advanced indicators
> and increased availability of data sets (including web data). Yet some
> innovations also risk bypassing the quality control mechanisms of fields
> like scientometrics and the standards they promote. The implications of
> this increasing scope and intensity of bibliometric practices requires a
> concerted response from scientometrics to produce more explicit guidelines
> and expert advice on good scientometric practices for specific evaluative
> practices such as recruitment, grant awards, institutional or national
> benchmarking.
>
>
>
> This special session will bring together scientometric experts,
> representatives of funding agencies, policy makers and opinion leaders on
> the role of metrics in research assessment to discuss the extent to which
> moving towards clearer, standardised guidelines over usage and consultancy
> can be achieved, both technically and strategically, and what the
> guidelines should look like concretely.
>
>
>
> ---
>
> *Background material*:
>
> - Report on International workshop "Guidelines and good practices on
> quantitative assessments of research" (OST, Paris, 12 May 2014):
> http://www.obs-ost.fr/fractivit%C3%A9s/workshop_international
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_international>
>
> - Blogposts Paul Wouters on previous debates at the ISSI and STI
> conferences in 2013, and on the DORA declaration:
>
>
> http://citationculture.wordpress.com/2013/07/29/bibliometrics-of-individual-researchers/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibliometrics-of-individual-researchers%2f>
>
>
> http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individual-researchers-the-debate-in-berlin/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibliometrics-of-individual-researchers-the-debate-in-berlin%2f>
>
>
> http://citationculture.wordpress.com/2013/05/23/dora-a-stimulus-for-a-new-evaluation-culture-in-science/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-a-stimulus-for-a-new-evaluation-culture-in-science%2f>
>
> - Information on the Higher Education Funding Council for England (HEFCE)
> "Independent review of the role of metrics in research assessment" + SPRU
> response
>
>
> http://citationculture.wordpress.com/2014/05/02/metrics-in-research-assessment-under-review/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetrics-in-research-assessment-under-review%2f>
>
> http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetrics%2f>
>
>
> https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.pdf&site=25
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fname%3dspru-response-final.pdf%26site%3d25>
>
> - Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To
> intervene, or not to intervene; is that the question? On the role of
> scientometrics in research evaluation."
>
>
> https://citationculture.files.wordpress.com/2014/08/de-rijcke_rushforth_jasist_preprint2014.pdf
> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-rijcke_rushforth_jasist_preprint2014.pdf>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140828/b1b25fdd/attachment.html>