HistCite and Program Evaluation

[Stephen J Bensman]

I once recommended the software HistCite developed by Sasha Pudovkin and
Gene Garfield for the evaluation of scientific programs.  Sasha Pudovikin
has kindly supplied me with a description of the way this software works,
and I am pasting it below, so that you can better understand its operation.
As you can see, the software fulfils the first basic requirement for such
evaluations by being very flexible in defining a subject set of papers,
upon which to base the evaluation.  Once this subject set of papers has
been defined, it allows the ranking of institutions strongest in the given
field.  I have seen it work and was mightily impressed with the results.  I
would only issue a number of caveats.  First, the ISI data, on which the
software works, has a lot of biases in it particularly in respect to non-US
institutions.  Therefore, any rankings derived from it should be carefully
checked against the opinions of the pertinent experts in the given subject
field.  Second, there are subject areas which are not amenable to
evaluation by publication and citation counts.  For a good overall view of
this, one should look at the criteria developed by the US National Research
Council to evaluate programs in different academic fields.  The HistCite
program is much better to use than something like the impact factor.  Here
you are working with a defined subject set of actual papers, whereas the
impact factor is nothing but a crude estimate of the arithmetic mean of the
citations to the papers of a given journal.  While this has the advantage
of identifying journals that have the propensity to publish highly cited
articles, the arithmetic mean is in no way a viable estimate of the central
tendency of the papers published by these journals, given the type of
distributions, with which one is dealing.

Sasha's description is below.

SB


The program HistCite is capable of identifying papers, which are most
important for a delimited research area. To find such topically important
papers (hence, authors, journals, institutions) one should do the
following:

1) perform a topical search within Science Citation Index (using Web of
Science portal) and export the generated bibliography (containing cited
references for each retrieved paper);
2) enter the bibliography into HistCite software;
3) sort the papers by Local Citation Score.

The papers which are at the top of the sorted list will be the most cited
papers within the bibliography, that is most cited by the authors working
within the research area; thus these papers will be of the most impact for
this research area.

Besides these high impact topically relevant papers the HistCite software
will identify the most important (for this research area) prior literature
which lie in the foundation (or background) of the topical research. These
will be listed as "Outer References", that is the papers, which are not
included in the generated topical bibliography, but are rather inferred
from
the lists of cited references within each paper in the bibliography.

The efficiency of the procedure (of finding the most topically important
and
relevant papers) strongly depends on the quality of the topical search
within the SCI: the better the search query (the better it fits to the
topic
of research), the better the result. The searches necessary for the
generation of topical bibliographies may be key-word searches, or citation
searches, or author searches, or a combination of them.

>From the identified set of topically relevant papers of high impact the
HistCite software is capable to generate lists of authors, institutions or
journals, most productive in the research area.