Open Access Metrics: Use REF2014 to Validate Metrics for REF2020

Wed Dec 17 15:01:12 EST 2014

Dear Stevan, 

I apologize for misspelling your name. 

Best,

Loet

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stevan Harnad
Sent: Wednesday, December 17, 2014 8:43 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Open Access Metrics: Use REF2014 to Validate Metrics for REF2020

On Wed, Dec 17, 2014 at 1:58 PM, Loet Leydesdorff <loet at leydesdorff.net <mailto:loet at leydesdorff.net> > wrote:

Dear Steven, 

Since you have said this so many times and for so many years, I assumed that you would have done this for REF2014. Did you? 

Are you calling for this to be done, or is this your own research program for the next few years?

Dear Loet,

I did say that I had suggested it before, unheeded!

Yes, when the REF2014 rankings are in, we will use them to start initializing the weights on a REF2020 vector.

The methodology is similar to what we used recently to test and then update the weights on the MELIBEA vector <http://arxiv.org/abs/1410.2926> . 

With the MELIBEA vector weights we were trying to predict Open Access Mandate effectiveness, whereas with the REF2020 vector will try to predict REF2014 rank.

Best wishes,

Stevan

Best,

Loet

  _____  

Loet Leydesdorff 

Emeritus University of Amsterdam
Amsterdam School of Communications Research (ASCoR)

 <mailto:loet at leydesdorff.net> loet at leydesdorff.net ;  <http://www.leydesdorff.net/> http://www.leydesdorff.net/ 
Honorary Professor,  <http://www.sussex.ac.uk/spru/> SPRU, University of Sussex; 

Guest Professor  <http://www.zju.edu.cn/english/> Zhejiang Univ., Hangzhou; Visiting Professor,  <http://www.istic.ac.cn/Eng/brief_en.html> ISTIC, Beijing;

Visiting Professor,  <http://www.bbk.ac.uk/> Birkbeck, University of London; 

 <http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en> http://scholar.google.com/citations?user=ych9gNYAAAAJ&hl=en

From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Stevan Harnad
Sent: Wednesday, December 17, 2014 6:35 PM
To: SIGMETRICS at LISTSERV.UTK.EDU <mailto:SIGMETRICS at LISTSERV.UTK.EDU> 
Subject: Re: [SIGMETRICS] Open Access Metrics: Use REF2014 to Validate Metrics for REF2020

On Dec 17, 2014, at 12:15 PM, Taylor, Nicholas K <N.K.Taylor at hw.ac.uk <mailto:N.K.Taylor at hw.ac.uk> > wrote:

Why should the REF 2014 panellists become the Gold Standard? No disrespect but I would venture to suggest that even they would not see themselves as the optimal arbiters for all time.

REF 2014 is certainly not the gold standard for all, or for all time — but it’s clearly going to be the standard for doling out the 2014 REF gold!

And as such, it is just fine for also initializing the weights on a first approximation to a REF 2020 metric equation:

REF2020Rank = 

w1(pubcount) + w2(JIF) + w3(cites) +w4(art-age) + w5(art-growth) + w6(hits) + w7(cite-peak-latency) + w8(hit-peak-latency) + w9(citedecay) + w10(hitdecay) + w11(hub-score) + w12(authority+score) + w13(h-index) + w14(prior-funding) +w15(bookcites) + w16(student-counts) + w17(co-cites + w18(co-hits) + w19(co-authors) + w20(endogamy) + w21(exogamy) + w22(co-text) + w23(tweets) + w24(tags), + w25(comments) + w26(acad-likes) etc. etc. 

There’s plenty of time between now and 2020 to keep optimizing the equation, discipline by discipline, updating the initial weights as well as adding more metrics to the battery.

Stevan 

From: Council of Professors and Heads of Computing in UK universities [ <mailto:cphc-members at JISCMAIL.AC.UK> mailto:cphc-members at JISCMAIL.AC.UK] On Behalf Of Stevan Harnad
Sent: 17 December 2014 14:27
To:  <mailto:cphc-members at JISCMAIL.AC.UK> cphc-members at JISCMAIL.AC.UK
Subject: Open Access Metrics: Use REF2014 to Validate Metrics for REF2020

Steven Hill of HEFCE has posted “an overview of the work HEFCE are currently commissioning which they are hoping will build a robust evidence base for research assessment” in LSE Impact Blog 12(17) 2014 entitled  <http://blogs.lse.ac.uk/impactofsocialsciences/2014/12/17/time-for-reflection/> Time for REFlection: HEFCE look ahead to provide rounded evaluation of the REF

Let me add a suggestion, updated for REF2014, that I have made before (unheeded):

Scientometric predictors of research performance need to be validated by showing that they have a high correlation with the external criterion they are trying to predict. The UK Research Excellence Framework (REF) -- together with the growing movement toward making the full-texts of research articles freely available on the web -- offer a unique opportunity to test and validate a wealth of old and new scientometric predictors, through multiple regression analysis: Publications, journal impact factors, citations, co-citations, citation chronometrics (age, growth, latency to peak, decay rate), hub/authority scores, h-index, prior funding, student counts, co-authorship scores, endogamy/exogamy, textual proximity, download/co-downloads and their chronometrics, tweets, tags, etc.) can all be tested and validated jointly, discipline by discipline, against their REF panel rankings in REF2014. The weights of each predictor can be calibrated to maximize the joint correlation with the rankings. Open Access Scientometrics will provide powerful new means of navigating, evaluating, predicting and analyzing the growing Open Access database, as well as powerful incentives for making it grow faster.

Harnad, S. (2009)  <http://eprints.ecs.soton.ac.uk/17142/> Open Access Scientometrics and the UK Research Assessment Exercise. Scientometrics 79 (1) Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.  (2007) 

See also:
 <http://openaccess.eprints.org/index.php?/archives/1136-The-Only-Substitute-for-Metrics-is-Better-Metrics.html> The Only Substitute for Metrics is Better Metrics (2014)
and
 <http://openaccess.eprints.org/index.php?/archives/479-On-Metrics-and-Metaphysics.html> On Metrics and Metaphysics (2008)

We invite research leaders and ambitious early career researchers to join us in leading and driving research in key inter-disciplinary themes. Please see  <http://www.hw.ac.uk/researchleaders> www.hw.ac.uk/researchleaders for further information and how to apply. 

Heriot-Watt University is a Scottish charity registered under charity number SC000278.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20141217/9c940774/attachment.html>