STI conference Leiden--Quality standards for evaluation indicators

Thu Aug 28 04:21:34 EDT 2014

Thanks Loet,

the role of professional standards is indeed to certify good practices and
this may not have neutral effects --notice that I say "professional"
standards. The debate is precisely (thanks for stressing it!) WHICH form of
standardisation is desirable given that there is discontent among users
about the proliferations of metrics (some of dubious reliability), and
there are initiatives setting de facto standards (
http://www.snowballmetrics.com/) without wider stakeholder engagement.

As Jochen Gläser pointed out last year, one may want to distinguish
professional practices -offered to users, which have professional
repercussions and can do good or harm-- from from scientometric research.
Just the same way that you distinguish between medical practices with
patients from experimentation with mice in the lab. Medical standards do
not suppress research in the lab --though do pose ethical questions on how
research is conducted.

We look forward to the discussion next week!

Ismael

2014-08-28 10:12 GMT+02:00 Paul Wouters <p.f.wouters at cwts.leidenuniv.nl>:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
> Dear Loet and Lutz,
>
> Many thanks for this contribution. The motivation for the discussion about
> standards, as far as I am concerned, is the need to protect research groups
> and researchers against sloppy or damaging evaluation practices. I agree
> with Loet that standards are often a powerful competition weapon to protect
> industry interests. It is certainly not the motivation for this panel, but
> it may end up like that if the process of standard setting, and the
> sociological interpretation of those standards, is not taken into account
> carefully. In my view the STI conference is the best place to have this
> discussion, because it is a meeting place between metrics experts and
> policy experts. In my view, this does not lead to the question whether or
> not one should have some quality control process of evaluation processes,
> but what kind of quality control we need and what kind of standards with
> respect to data and indicators can play a role in this.
>
> In other words, you have raised a crucial point for the panel discussion
> next week.
>
> Regards,
>
>
> Paul Wouters
> Professor of Scientometrics
> Director Centre for Science and Technology Studies
> Leiden University
>
> PS: I am pleased to announce the release of our completely renewed CWTS
> website:
> cwts.nl - all information now easily available!
>
> Visiting address:
> Willem Einthoven Building
> Wassenaarseweg 62A
> 2333 AL Leiden
> Mail address: P.O. Box 905
> 2300 AX Leiden
> T: +31 71 5273909 (secr.)
> F: +31 71 5273911
> E: p.f.wouters at cwts.leidenuniv.nl
>
> CWTS home page: www.cwts.nl
> Blog about Citation Cultures: http://citationculture.wordpress.com/
> Research Dreams: www.researchdreams.nl
>
>
> On Thu, Aug 28, 2014 at 8:29 AM, Loet Leydesdorff <loet at leydesdorff.net>
> wrote:
>
>> Adminstrative info for SIGMETRICS (for example unsubscribe):
>> http://web.utk.edu/~gwhitney/sigmetrics.html
>>
>> Dear Ismael,
>>
>>
>>
>> It seems to me that we know from the innovation-science literature that
>> standards are to the interests of incumbent firms. In this context, one can
>> expect process innovation more than product innovations. The further
>> development of the field, in my opinion, needs the fluidity of intellectual
>> exchanges and the space to propose new variants.
>>
>>
>>
>> Perhaps, as an intellectual community we have increasingly interests
>> different from the professional practices of (quasi)industries on a market
>> of evaluation studies that one may wish to certify ( and thus to shield the
>> market against “amateurs”; our PhD students?).
>>
>>
>>
>> Let me quote from a recent text (that I coauthored for other reasons):
>>
>>
>>
>> “There exists a professional community with experts in bibliometrics who
>> develop advanced bibliometric indicators for productivity and citation
>> impact measurements (see an overview in Vinkler, 2010). Only experts
>> from this community should undertake a bibliometric study. These centres of
>> professional expertise can be found, for example, at the Centre for Science
>> and Technology Studies (CWTS, Leiden) or the Centre for Research &
>> Development Monitoring (ECOOM, Leuven).”
>>
>>
>>
>> Is this the dream to come through? Or do we hear institutional interests?
>> Perhaps, we need smaller dreams J
>>
>>
>>
>> Best,
>>
>> Loet
>>
>>
>> ------------------------------
>>
>> Loet Leydesdorff
>>
>> University of Amsterdam
>> Amsterdam School of Communications Research (ASCoR)
>>
>> loet at leydesdorff.net ; http://www.leydesdorff.net/
>> Honorary Professor, SPRU, <http://www.sussex.ac.uk/spru/>University of
>> Sussex;
>>
>> Guest Professor Zhejiang Univ. <http://www.zju.edu.cn/english/>,
>> Hangzhou; Visiting Professor, ISTIC,
>> <http://www.istic.ac.cn/Eng/brief_en.html>Beijing;
>>
>> Visiting Professor, Birkbeck <http://www.bbk.ac.uk/>, University of
>> London;
>>
>> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en
>>
>>
>>
>> *From:* ASIS&T Special Interest Group on Metrics [mailto:
>> SIGMETRICS at LISTSERV.UTK.EDU] *On Behalf Of *Ismael Rafols
>> *Sent:* Thursday, August 28, 2014 2:21 AM
>> *To:* SIGMETRICS at LISTSERV.UTK.EDU
>> *Subject:* [SIGMETRICS] STI conference Leiden--Quality standards for
>> evaluation indicators
>>
>>
>>
>> Adminstrative info for SIGMETRICS (for example unsubscribe):
>> http://web.utk.edu/~gwhitney/sigmetrics.html
>>
>> With apologies for cross-posting)
>>
>> Dear all,
>> to warm up forweek in the ST Indicators Conference in Leiden, let us
>> share the topic of a debate:
>>
>> *Quality standards for evaluation indicators: Any chance for the dream to
>> come true?*
>>
>> *Special session at the STI-ENID conference in Leiden, 3 September
>> 2014, 16-17.30h *
>>
>> *Organisers*: Ismael Rafols (INGENIO & SPRU), Paul Wouters (CWTS, Leiden
>> University), Sarah de Rijcke (CWTS, Leiden University)
>>
>> *Location*:  Aalmarkt-hall, Stadsgehoorzaal Leiden
>>
>>
>>
>> There is a growing realization in the scientometrics community of the
>> need to offer clearer guidance to users and further develop standards for
>> professional use of bibliometrics in research evaluations. Indeed the
>> STI-ENID Conference 2014 has the telling sub-title ‘Context Matters’. This
>> session continues from the 2013 ISSI and STI conferences in Vienna and
>> Berlin, where full plenary sessions were convened on the need for standards
>> in evaluative bibliometrics, and the ethical and policy implications of
>> individual-level bibliometrics. The need to debate these issues has come to
>> the forefront in light of reports that uses of certain easy-to-use metrics
>> for evaluative purposes have become a routine part of academic life,
>> despite misgivings within the profession itself about its validity. Very
>> recently high-profile movements against certain metric indicators (e.g. the
>> DORA declaration about the Journal Impact Factor) have brought possible
>> misuses of metrics further to the center of attention. There may be a
>> growing need for standards – also to promote for accountability of
>> scientometricians as experts.
>>
>>
>>
>> Indeed the relationship between scientometricians and end-users has been
>> changing over the years due to factors like: 1. Increasing demands for
>> bibliometric services in research management at various levels of
>> aggregation, 2. New capacities and demands for performance information
>> through the greater availability of new research technologies and their
>> applications, and 3. The emergence of “citizen bibliometrics” (i.e.
>> bibliometrics carried out by non-expert end-users) due to larger
>> availability of data and indicators. Some of these developments may result
>> in new opportunities for research contributions and information-use, and
>> may increase effectiveness of bibliometrics due to more advanced indicators
>> and increased availability of data sets (including web data). Yet some
>> innovations also risk bypassing the quality control mechanisms of fields
>> like scientometrics and the standards they promote. The implications of
>> this increasing scope and intensity of bibliometric practices requires a
>> concerted response from scientometrics to produce more explicit guidelines
>> and expert advice on good scientometric practices for specific evaluative
>> practices such as recruitment, grant awards, institutional or national
>> benchmarking.
>>
>>
>>
>> This special session will bring together scientometric experts,
>> representatives of funding agencies, policy makers and opinion leaders on
>> the role of metrics in research assessment to discuss the extent to which
>> moving towards clearer, standardised guidelines over usage and consultancy
>> can be achieved, both technically and strategically, and what the
>> guidelines should look like concretely.
>>
>>
>>
>> ---
>>
>> *Background material*:
>>
>> - Report on International workshop "Guidelines and good practices on
>> quantitative assessments of research" (OST, Paris, 12 May 2014):
>> http://www.obs-ost.fr/fractivit%C3%A9s/workshop_international
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.obs-ost.fr%2ffractivit%25C3%25A9s%2fworkshop_international>
>>
>> - Blogposts Paul Wouters on previous debates at the ISSI and STI
>> conferences in 2013, and on the DORA declaration:
>>
>>
>> http://citationculture.wordpress.com/2013/07/29/bibliometrics-of-individual-researchers/
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f07%2f29%2fbibliometrics-of-individual-researchers%2f>
>>
>>
>> http://citationculture.wordpress.com/2013/10/03/bibliometrics-of-individual-researchers-the-debate-in-berlin/
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f10%2f03%2fbibliometrics-of-individual-researchers-the-debate-in-berlin%2f>
>>
>>
>> http://citationculture.wordpress.com/2013/05/23/dora-a-stimulus-for-a-new-evaluation-culture-in-science/
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2013%2f05%2f23%2fdora-a-stimulus-for-a-new-evaluation-culture-in-science%2f>
>>
>> - Information on the Higher Education Funding Council for England (HEFCE)
>> "Independent review of the role of metrics in research assessment" + SPRU
>> response
>>
>>
>> http://citationculture.wordpress.com/2014/05/02/metrics-in-research-assessment-under-review/
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fcitationculture.wordpress.com%2f2014%2f05%2f02%2fmetrics-in-research-assessment-under-review%2f>
>>
>> http://www.hefce.ac.uk/whatwedo/rsrch/howfundr/metrics/
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=http%3a%2f%2fwww.hefce.ac.uk%2fwhatwedo%2frsrch%2fhowfundr%2fmetrics%2f>
>>
>>
>> https://www.sussex.ac.uk/webteam/gateway/file.php?name=spru-response-final.pdf&site=25
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fwww.sussex.ac.uk%2fwebteam%2fgateway%2ffile.php%3fname%3dspru-response-final.pdf%26site%3d25>
>>
>> - Opinion article for JASIST by Sarah de Rijcke and Alex Rushforth "To
>> intervene, or not to intervene; is that the question? On the role of
>> scientometrics in research evaluation."
>>
>>
>> https://citationculture.files.wordpress.com/2014/08/de-rijcke_rushforth_jasist_preprint2014.pdf
>> <https://agenda.upv.es/owa/redir.aspx?C=55006d6295ec47ed88d25cb18366d006&URL=https%3a%2f%2fcitationculture.files.wordpress.com%2f2014%2f08%2fde-rijcke_rushforth_jasist_preprint2014.pdf>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20140828/a4e5adeb/attachment.html>