British Classification Soc post-RAE talk/discussion - 6 July (fwd)

[Loet Leydesdorff]

> (1) It's not explaining, it's predicting -- like weather-forecasting.

Multi-variate regression is based on a static model. (Otherwise, there is
auto-correlation in the data and also in the error terms.) You may wish to
develop a more dynamic conceptualization. 

In the case of a system, the Markov assumption is often a good one: the best
prediction of a system at t+1 is its state at t. The research system is more
stable than the weather. (I don't know how stable the RAE has been, but it
seems to me that the British system is rather stable.)

> (2) It's not a model. (Multiple regression is just the bog-standard
> general linear model for statistics.)
> 
> (3) The idea is first to find metrics that are closely enough 
> correlated
> with the RAE panel rankings to be confidently substituted for them.
> 
> (4) Then the idea is to make them better, more powerful, more
> predictive, field by field, adjusting the regression weights on each
> metric as needed.

Yes: this is why I suggested in a previous email to develop a structural
equation model (LISREL). LISREL is also static. If you wish to extend the
multi-variate regression with a dynamic analysis, entropy statistics would
be my prime candidate ("The Challenge of Scientometrics", Leiden:
DSWO/Leiden University Press, 1995).

> (5) Not slavishly predictive of the RAE panel rankings any more (that
> would be circular), but predictive of future research 
> performance, other
> (validated) metrics, other human rankings.

It would be more interesting to take "research performance" as the
explanandum. But I had understood that you wished to use it as a predictor
for the ranking in the RAE ("first things first!"). For example, one would
expect research performance to be confounded with other factors (e.g.,
sexism and nepotism; Wenneras, C., & Wold, A. (1997). Sexism and Nepotism in
Peer-Review. Nature, 387, 341-343) in a peer-based ranking exercise like the
RAE. 

> Stevan Harnad
> 

With best wishes,  Loet