"Academics strike back at spurious rankings" (Nature, 31 May)

[Loet Leydesdorff]

PS. 

1. Explaining the RAE ratings does not inform us about their validity! 
It only informs us about how they are constructed and can perhaps be
automated. 

2. If it works with multiple regression analysis after proper normalization
and validation, the validation of the parameters would not be expected to
hold for a next round because there is a feedback loop ("learning") involved
(in addition to the problem of auto-correlations between the rankings).

Best,  Loet



OK: that is a clear answer. The multiple regression serves to explain the
RAE ratings. 
Eventually, you may wish to build an expert system which makes it possible
to generate the RAE ratings without the panelists. 

Thank you for the clarification. (I had not clearly understood this from the
previous mailings.)


Best wishes,  Loet 

> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics 
> [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stevan Harnad
> Sent: Sunday, June 03, 2007 7:43 PM
> To: SIGMETRICS at listserv.utk.edu
> Subject: Re: [SIGMETRICS] "Academics strike back at spurious 
> rankings" (Nature, 31 May)
> 
> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
> 
> On Sun, 3 Jun 2007, Loet Leydesdorff wrote:
> 
> > Yes, I agree that multiple regression is a classical 
> technique. But one
> > needs a dependent variable in that case which can be 
> operationalized. Unlike
> > the case of barometric pressure, we don't have an objective 
> measure, but the
> > standard has to be constructed. 
> 
> Loet, we are beginning to repeat ourselves. I said that in the case
> of weather forecasting, the barometric pressure is the independent
> (predictor) variable and rain is the dependent (predicted) 
> variable. We
> first validate pressure as a predictor of rain, against rain itself,
> and then once pressure is shown to correlate highly enough with rain,
> we plan our picnics based on pressure, without having to wait for them
> to be rained on.
> 
> The same is true with scientometrics. We take our battery of 
> independent
> variables -- the many candidate metrics -- and we do a 
> multiple regression on a
> criterion, the dependent variable, first to validate them. In 
> the example I
> gave, the dependent variable is the RAE panel rankings. Once 
> we validate our
> predictor metrics (by field), we can then give top-sliced 
> research funding (in
> the UK dual-funding system) without having to waste the time 
> and energies of
> the panelists.
> 
> > All the validated measures seem predictors (independent 
> variables) to me
> > when one thinks within the model of multiple regression. 
> What do you propose
> > as the predicted variable? 
> 
> See above.
> 
> Stevan Harnad
>