RAE Questions

[Stephen J Bensman]

I would like to use the commentary of David Watkins below to raise some
questions I have concerning the British RAE.  I have some insight on this
matter, for I worked closely with the National Research Council (NRC) on
its last evaluation of US research-doctorate programs.  I was chosen as one
of three persons to test the database which the NRC developed during this
evalution.  The NRC data is some of the best data in world with which to
analyze variables involved in academic evaluations, and it is freely
available at the following web site for those interested:

http://www.nap.edu/readingroom/books/researchdoc/

Unfortunately it is not the full data set but only that published in the
book, and you have to buy the book to understand it.

My questions concern the effect of the types of distributions with which
the RAEs are dealing.  It is well known that informetric distributions are
not only highly skewed but highly stable over time.  Using the NRC data I
found that peer rating, publication rates, citation rates, etc. correlated
from one period to another about at 0.9.  Moreover, the same programs
maintained their dominant positions despite the addition of numerous new
programs.  This is a natural consequence of the cumulative advantage or
success-breeds-success process underlying this stratification system.
Translating this into UK terms, it means that Oxford and Cambridge have
been dominant for about the last millenium and will maintain their
dominance for the next millenium, barring their conversion into madrassas.
The RAES should not only be finding this but reinforcing it.

Second, such a hierarchical and stable social system is probably
functionally necessary for the advance of science.  Then comes the question
about what is the mobility of individual scientists within this system.
Here I think may be the primary fault of the RAEs.  The distributions are
not only skewed between programs but within programs.  Analysis of the
elite programs reveals a surprising amount of dead wood  among their
scientists.  Some of it results of extinct volcanos that have ceased
producing new work decades ago.  The way the RAE allocates research
resources seems to have the potential of feather-bedding these drones at
the expense of the productive scientists at lesser institutions, making the
system not only hierarchical but closed.

And, third, evaluating research on a departmental basis seems to run the
risk of introducing variables that extraneous to the quality of the
research.  It is well known that large departments are automatically more
highly rated than small ones just on that basis alone.  Moreover, such a
broad method of evaluating research reduces the flexibility in defining
proper sets for comparative purposes.  Thus, there is the danger of losing
sight of new and developing fields.

Taking the above points all together, I think that the US utilization of
such evaluations is more correct than that of the UK.  In the US such
evaluations are regarded as what they actually are--beauty contests in
which academic programs strut down the runway in their swim suits,
flaunting their particulars.  ("And, Miss Harvard Physics, how do you think
the world can be immediately saved?")  The data can be used to determine
how the system is actually functioning and thus explore ways in which the
system can be proved.  For example, I was able to use the NRC data to
determine that there is no anti-Southern bias in the evaluation of
research-doctorate programs, thus breaking numerous faculty hearts here at
LSU.  However, such evaluations should in no way be utilized for allocating
research funds, which is best done on a project-by-project basis.
Fortunately, the UK way is politically impossible in the US, which is a
Federal state.  The South may have lost the Civil War, but its soldiers
inflicted 9 casualties for every one suffered, putting strict limits on the
central Leviathan due to the huge butcher's bill.  Any attempt to implement
the UK system would be immediately killed in Congress and probably by the
Southern delegations.

These musings are only speculative, and I may be entirely wrong about the
dangers of the UK system.

SB




David Watkins <David.Watkins at SOLENT.AC.UK>@LISTSERV.UTK.EDU> on 04/02/2006
08:32:03 AM

Please respond to ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>

Sent by:    ASIS&T Special Interest Group on Metrics
       <SIGMETRICS at LISTSERV.UTK.EDU>


To:    SIGMETRICS at LISTSERV.UTK.EDU
cc:     (bcc: Stephen J Bensman/notsjb/LSU)

Subject:    Re: [SIGMETRICS] SIGMETRICS Digest - 30 Mar 2006 to 31 Mar 2006
       (#2006-44)


RAE et al.....


For me the basic problem with the RAE was an unintended consequence. The
government's problem was that of 'big science'. HEFCE's desire was to
concentrate funding for expensive, scientific / medical / engineering
research in a few institutions. Probably a good idea. However, it swept up
all the other disciplines, too, into a one size fits all system. Thus the
lone researcher who can do good work in the history stacks and local
archives - if given sufficient time - was treated like the guy who needs
CERN time and a cast of hundreds to check out his/her theory. That is just
bizarre. In the humanities and social sciences the results have been poor
for academe since the basic premise was wrong.

Switching the whole system to a metrics based one merely continues this and
distorts scholarly good practice outside the sciences; we know that
publication practices, citation behaviour etc are quite different in
non-science disciplines in the absence of a distortion like RAE. What is
needed is a funding regime which gives scholarly space-time and base-line
funding to all academics - possibly working on the basis of a notionally
equal 4 way split between teaching/teaching
preparation/administration/research. Those who needed no special facilities
could do good work on this basis as they always did traditionally.The rest
of HEFCE's research funds would go to the Research Councils. Other
academics who needed large scale funding / facilities would use their
'quartile' to work up proposals for substantial funding from the Research
Councils and others. This could - but needn't  - involve buying out more of
their own admin - or even teaching - time. Teaching everywhere would be
enhanced since few established academics would have / need non-teaching
roles (evaluation of the whole scholarly role internally, with a level
playing field) and the costs of external evaluation would be restricted to
the project level, which is the most appropriate one. There would be no
institutional 'halo' to support underperformers, but good scholars working
in isolation or in small / peripheral institutions would not be
discriminated against, and recruitment could return to the assessment of
academic institutional need rather than just looking for 'four alpha
publications' and to hell with the other academic skills and interests we
would all favour seeing in a new colleague.


************************************************
Professor David Watkins
Postgraduate Research Centre
Southampton Business School
East Park Terrace
Southampton SO14 0RH

David.Watkins at solent.ac.uk
 023 80 319610 (Tel)
+44 23 80 31 96 10 (Tel)

02380 33 26 27 (fax)
+44 23 80 33 26 27 (fax)