RAE Questions

[Stevan Harnad]

May I point out that the UK's is a *dual* funding system: (1) direct,
peer-reviewed, competitive research proposals, exactly as in the US,
submitted to the UK funding councils (RCUK) plus (2) a much smaller but
significant portion of top-sliced funding, based on the RAE. Stephen
Bensman, below, seems to think it's all RAE. It's not! Not even
mostly. But the RAE top-slice is important too. Is it just a Matthew
effect? Surely not just, but no doubt it is in part, and perhaps it's not
bad to have some longer-term reinforcement of prior productivity:
the length of its tail (its time constant) can be calibrated, but it
is quite possible that the dual system is both more stable, robust and
equitable than an exclusive bid-by-bid horse-race.

Very little of this has been objectively measures and monitored, let
alone calibrated and optimized as it went all along (the excellent NRC
data notwithstanding). Now that the RAE is going metric, all sorts of
new possibilities for enriching, diversifying, monitoring and maximizing
predictivity and validity open up, particularly with a webwide, digital
open-access research database to base it all on.

Keep your mind open open and hold onto your hats because things
should soon be picking up for PostGutenberg Scientometrics!

Stevan Harnad

On Mon, 3 Apr 2006, Stephen J Bensman wrote:

> I would like to use the commentary of David Watkins below to raise some
> questions I have concerning the British RAE.  I have some insight on this
> matter, for I worked closely with the National Research Council (NRC) on
> its last evaluation of US research-doctorate programs.  I was chosen as one
> of three persons to test the database which the NRC developed during this
> evaluation.  The NRC data is some of the best data in world with which to
> analyze variables involved in academic evaluations, and it is freely
> available at the following web site for those interested:
>
> http://www.nap.edu/readingroom/books/researchdoc/
>
> Unfortunately it is not the full data set but only that published in the
> book, and you have to buy the book to understand it.
>
> My questions concern the effect of the types of distributions with which
> the RAEs are dealing.  It is well known that informetric distributions are
> not only highly skewed but highly stable over time.  Using the NRC data I
> found that peer rating, publication rates, citation rates, etc. correlated
> from one period to another about at 0.9.  Moreover, the same programs
> maintained their dominant positions despite the addition of numerous new
> programs.  This is a natural consequence of the cumulative advantage or
> success-breeds-success process underlying this stratification system.
> Translating this into UK terms, it means that Oxford and Cambridge have
> been dominant for about the last millennium and will maintain their
> dominance for the next millennium, barring their conversion into madrassas.
> The RAES should not only be finding this but reinforcing it.
>
> Second, such a hierarchical and stable social system is probably
> functionally necessary for the advance of science.  Then comes the question
> about what is the mobility of individual scientists within this system.
> Here I think may be the primary fault of the RAEs.  The distributions are
> not only skewed between programs but within programs.  Analysis of the
> elite programs reveals a surprising amount of dead wood  among their
> scientists.  Some of it results of extinct volcanoes that have ceased
> producing new work decades ago.  The way the RAE allocates research
> resources seems to have the potential of feather-bedding these drones at
> the expense of the productive scientists at lesser institutions, making the
> system not only hierarchical but closed.
>
> And, third, evaluating research on a departmental basis seems to run the
> risk of introducing variables that extraneous to the quality of the
> research.  It is well known that large departments are automatically more
> highly rated than small ones just on that basis alone.  Moreover, such a
> broad method of evaluating research reduces the flexibility in defining
> proper sets for comparative purposes.  Thus, there is the danger of losing
> sight of new and developing fields.
>
> Taking the above points all together, I think that the US utilization of
> such evaluations is more correct than that of the UK.  In the US such
> evaluations are regarded as what they actually are--beauty contests in
> which academic programs strut down the runway in their swim suits,
> flaunting their particulars.  ("And, Miss Harvard Physics, how do you think
> the world can be immediately saved?")  The data can be used to determine
> how the system is actually functioning and thus explore ways in which the
> system can be proved.  For example, I was able to use the NRC data to
> determine that there is no anti-Southern bias in the evaluation of
> research-doctorate programs, thus breaking numerous faculty hearts here at
> LSU.  However, such evaluations should in no way be utilized for allocating
> research funds, which is best done on a project-by-project basis.
> Fortunately, the UK way is politically impossible in the US, which is a
> Federal state.  The South may have lost the Civil War, but its soldiers
> inflicted 9 casualties for every one suffered, putting strict limits on the
> central Leviathan due to the huge butcher's bill.  Any attempt to implement
> the UK system would be immediately killed in Congress and probably by the
> Southern delegations.
>
> These musings are only speculative, and I may be entirely wrong about the
> dangers of the UK system.
>
> SB
>
> David Watkins <David.Watkins at SOLENT.AC.UK>@LISTSERV.UTK.EDU> on 04/02/2006
> 08:32:03 AM
>
> RAE et al.....
>
> For me the basic problem with the RAE was an unintended consequence. The
> government's problem was that of 'big science'. HEFCE's desire was to
> concentrate funding for expensive, scientific / medical / engineering
> research in a few institutions. Probably a good idea. However, it swept up
> all the other disciplines, too, into a one size fits all system. Thus the
> lone researcher who can do good work in the history stacks and local
> archives - if given sufficient time - was treated like the guy who needs
> CERN time and a cast of hundreds to check out his/her theory. That is just
> bizarre. In the humanities and social sciences the results have been poor
> for academe since the basic premise was wrong.
>
> Switching the whole system to a metrics based one merely continues this and
> distorts scholarly good practice outside the sciences; we know that
> publication practices, citation behaviour etc are quite different in
> non-science disciplines in the absence of a distortion like RAE. What is
> needed is a funding regime which gives scholarly space-time and base-line
> funding to all academics - possibly working on the basis of a notionally
> equal 4 way split between teaching/teaching
> preparation/administration/research. Those who needed no special facilities
> could do good work on this basis as they always did traditionally.The rest
> of HEFCE's research funds would go to the Research Councils. Other
> academics who needed large scale funding / facilities would use their
> 'quartile' to work up proposals for substantial funding from the Research
> Councils and others. This could - but needn't  - involve buying out more of
> their own admin - or even teaching - time. Teaching everywhere would be
> enhanced since few established academics would have / need non-teaching
> roles (evaluation of the whole scholarly role internally, with a level
> playing field) and the costs of external evaluation would be restricted to
> the project level, which is the most appropriate one. There would be no
> institutional 'halo' to support underperformers, but good scholars working
> in isolation or in small / peripheral institutions would not be
> discriminated against, and recruitment could return to the assessment of
> academic institutional need rather than just looking for 'four alpha
> publications' and to hell with the other academic skills and interests we
> would all favour seeing in a new colleague.
>
>
> ************************************************
> Professor David Watkins
> Postgraduate Research Centre
> Southampton Business School
> East Park Terrace
> Southampton SO14 0RH
>
> David.Watkins at solent.ac.uk
>  023 80 319610 (Tel)
> +44 23 80 31 96 10 (Tel)
>
> 02380 33 26 27 (fax)
> +44 23 80 33 26 27 (fax)
>