Open Access Metrics: Use REF2014 to Validate Metrics for REF2020

[Stevan Harnad]

> On Dec 17, 2014, at 9:35 AM, Jon Crowcroft <jon.crowcroft at cl.cam.ac.uk> wrote:
> 
> if you wanted to do this properly, you should have to take a lot of outputs that were NOT submitted and run any metric scheme on them as well as those submitted. 
> too late:)

Not too late. REF2014 gives the 2014 ranking based on the 4 outputs submitted. That is then the criterion against which the many the other metrics I list below can be jointly validated, through multiple regression, to initialize their weights for REF2020, as well as for other assessments. In fact, open access metrics can be — and will be — continuously assessed, as open access grows. And as the metric equation (per discipline) is optimized for predictive power, it can replace the peer rankings (except for periodic cross-checks and updates).


> On Wed, Dec 17, 2014 at 2:26 PM, Stevan Harnad <harnad at ecs.soton.ac.uk <mailto:harnad at ecs.soton.ac.uk>> wrote:
> Steven Hill of HEFCE has posted “an overview of the work HEFCE are currently commissioning which they are hoping will build a robust evidence base for research assessment” in LSE Impact Blog 12(17) 2014 entitled Time for REFlection: HEFCE look ahead to provide rounded evaluation of the REF <http://blogs.lse.ac.uk/impactofsocialsciences/2014/12/17/time-for-reflection/>
> 
> Let me add a suggestion, updated for REF2014, that I have made before (unheeded):
> 
> Scientometric predictors of research performance need to be validated by showing that they have a high correlation with the external criterion they are trying to predict. The UK Research Excellence Framework (REF) -- together with the growing movement toward making the full-texts of research articles freely available on the web -- offer a unique opportunity to test and validate a wealth of old and new scientometric predictors, through multiple regression analysis: Publications, journal impact factors, citations, co-citations, citation chronometrics (age, growth, latency to peak, decay rate), hub/authority scores, h-index, prior funding, student counts, co-authorship scores, endogamy/exogamy, textual proximity, download/co-downloads and their chronometrics, tweets, tags, etc.) can all be tested and validated jointly, discipline by discipline, against their REF panel rankings in REF2014. The weights of each predictor can be calibrated to maximize the joint correlation with the rankings. Open Access Scientometrics will provide powerful new means of navigating, evaluating, predicting and analyzing the growing Open Access database, as well as powerful incentives for making it grow faster.
> 
> Harnad, S. (2009) Open Access Scientometrics and the UK Research Assessment Exercise <http://eprints.ecs.soton.ac.uk/17142/>. Scientometrics 79 (1) Also in Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27-33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds.  (2007) 
> 
> See also:
> The Only Substitute for Metrics is Better Metrics <http://openaccess.eprints.org/index.php?/archives/1136-The-Only-Substitute-for-Metrics-is-Better-Metrics.html> (2014)
> and
> On Metrics and Metaphysics <http://openaccess.eprints.org/index.php?/archives/479-On-Metrics-and-Metaphysics.html> (2008)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20141217/cb76e3a7/attachment.html>