Suggested Reading

[Stephen J Bensman]

I meant to post this before but then rejected doing so.  However, I have
just received another inquiry about academic evaluations using citations
and other measures.  The US National Research Council (NRC) is again
evaluating of the quality of US research-doctorate programs.  It is in the
process of doing opinion surveys and collecting data.  The last such NRC
evaluation was in 1993.  Those of you doing this sort of thing should go to
the following Web site and carefully read what is on it.  It has the NRC
survey documents and assessments of the mistakes made during previous
evaluations.  Of particular interest is the discussions of the taxonomy of
such evaluations.

http://www7.nationalacademies.org/resdoc/index.html

One of the main mistakes made during the last evaluation was failure to
allow the program faculty to verify and correct the citation counts to
their work.  There were a lot of mistakes in this during the last
assessment.  Therefore, instructions are being issued to the faculty on how
to do this.  One such set of instructions is posted at the Web site.

http://www.library.miami.edu/search/eresources/nrc_citations.html

I am sending this out, because I would like it to be understood that
"metric" evaluations are not so simple and accurate as certain people seem
to think.  Before you do one of these things, you should know the
difficulties involved in these things.  The NRC is probably the most
experienced body in the world in making such evaluations, and these web
sites give you a good idea of all the possible screw ups.  The
possibilities of such screw ups are endless.

SB