Future UK RAEs to be Metrics-Based

[Ian Rowlands]

This is certainly an interesting technical argument, but I think we (the metrics
/ science policy community) ought to be trying to lead this debate to a more
broadly socially useful conclusion.

We know that targets are a very strong motivator and that they will have a
profound effect on human behaviour.  The argument so far (which I agree with)
is that the RAE is a waste of money in that simple metrics could provide a
solution which pretty well fits the rankings obtained - expensively - by
committees.

But the bigger question is what are we trying to achieve by the RAE or its
successors?  If it's simply research concentration, why not just let market
forces rip?  That would be both efficient and transparent.  If the RAE is
intended as a redistributive exercise, to address market failure, then we will
need metrics (or fudges) that address that issue.  Like how to deal with
scholarship, editorial contributions, etc.

But, fundamentally, what are we trying to achieve?  Would not the RAE gain a
modicum of respect if it justified itself in terms of changing academic
behaviour towards some agreed community goals.  I think this proposal is the
worst of all worlds, it hardens the edges around the decisions that are made
about a very complex set of scholarly communication behaviours and most
definitely will not gain community respect, it does not address social value,
the role of universities and where research should go (rather than where it's
been).

No good manager needs a load of trailing edge indicators to tell him or her who
their best staff are.  But they can see the value of setting aspirational
targets that get the best from them by inspiring them to change their behaviour
for the common good.  We need new indicators that will enable policy makers to
set socially desirable targets: should keep us going for a few years!
Otherwise what is the point?

Ian Rowlands




Quoting Stevan Harnad <harnad at ECS.SOTON.AC.UK>:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> On Tue, 28 Mar 2006, Loet Leydesdorff wrote:
>
> > I agree with this need for calibration, but it seems unlikely that this
> will
> > be achieved within one RAE round. Thereafter, you have to wait four years
> > for a second measurement point.
>
> No, there are retro data available for the past RAEs as well. And the
> calibration is to improve the predictive power of the RAE. To merely
> duplicate its prior predictive power, the strongly correlated metrics are
> already enough.
>
> > Furthermore, there is the problem of how
> > to combine the multivariate regression with the time series perspective.
> The
> > former will provide you with comparative statics, but not with insights in
> > the dynamics of the system (autocorrelation problems!). It seems that you
> > are formulating a research program rather than a policy measure.
>
> There's definitely an emerging research programme there (Open Access Web
> Scientometrics) but there's no need for the results of the research to
> be in in order to conclude that the wasteful, inefficient non-metric
> components of the RAE should be replaced by the cheap, fast, efficient
> metrics that are already highly correlated with the outcome. The
> research programme is for *improving* on the prior outcomes.
>
> Bref: Scrapping the non-metric white-elephant in favour of existing
> metrics is the policy part; improving on existing metrics is the
> research part.
>
> > Perhaps, I lost you. I downloaded the report of the UK government which you
> > cited, but it is more on technology and innovation than on science. After a
> > lot of the usual rhetoric, they advise to develop "a metrics" without
> > specifying anything.
>
> The government's is just a policy document. Have a look at
>
>     http://openaccess.eprints.org/index.php?/archives/75-guid.html
>
> for a summary and at the references cited in
>
>      Harnad, S., Carr, L., Brody, T. & Oppenheim, C. (2003) Mandated
>      online RAE CVs Linked to University Eprint Archives: Improving the
>      UK Research Assessment Exercise whilst making it cheaper and easier.
>      Ariadne 35.  http://www.ariadne.ac.uk/issue35/harnad/
>
> for some of the correlational evidence.
>
> Stevan Harnad
> American Scientist Open Access Forum
>
http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html
>
> Chaire de recherche du Canada           Professor of Cognitive Science
> Ctr. de neuroscience de la cognition    Dpt. Electronics & Computer Science
> Université du Québec à Montréal         University of Southampton
> Montréal, Québec                        Highfield, Southampton
> Canada  H3C 3P8                         SO17 1BJ United Kingdom
> http://www.crsc.uqam.ca/                http://www.ecs.soton.ac.uk/~harnad/
>




----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.