RAE Questions

[Loet Leydesdorff]

Dear Steven,

Now that we have amply discussed the political side of the RAE, let us turn
to your research program of replacing the RAE with a metrics. Two problems
have been mentioned which cannot easily be solved:

1. the skewness of the distributions
2. the heterogeneity of department as units of analysis

The first problem can be solved by using non-parametric regression analysis
(probit or logit) instead of multi-variate regression analysis of the LISREL
type. However, will this provide you with a ranking? I cannot oversee it
because I never did it myself. Stephen Bensman also mentioned the
instability of these skewed curves over time. I would anyhow be worried
about the comparisons over time because of auto-correlation
(auto-covariance) effects.

I have run into these problems before, and therefore I am a big fan of
entropy statistics. But policy makers tend not to understand the results if
one can teach them something about "reduction of the uncertainty". They will
wish firm numbers to legitimate decisions.

The second problem is generated because you will have institutional units of
analysis which may be composed of different disciplinary affiliations and to
a variable extent. For example, I am myself misplaced in a unit of
communication studies. In other cases, universities will have set up
"interdisciplinary units" on purpose while individual scholars continue to
affiliate themselves with their original disciplines. We know that
publication and citation practices vary among disciplines. Thus, one should
not compare apples with oranges.

I would be inclined to disadvise to embark on this research project before
one has an idea of how to handle these two problems. Fortunately, I was not
the reviewer :-).

With best wishes,


Loet

________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681;
loet at leydesdorff.net ; http://www.leydesdorff.net/