On Metrics and Metaphysics

[Stevan Harnad]

On Tue, Oct 21, 2008 at 1:23 PM, J.F.Rowland <J.F.Rowland at lboro.ac.uk> wrote:

>>> HM: Literature - authors.  There are many researchers studying
>>> Shakespeare.  A lesser-known local author will be lucky
>>> to receive the attention of even one researcher.  In a
>>> metrics-based system, it seems reasonable to hypothesize
>>> that this bias will increase, and the odds of studying local
>>> culture decrease.
>>
>> SH: What bias? If a lesser-known researcher does good work,
>> it will be used, and this will be reflected in the metrics.
>
> Stevan - You misunderstood Heather's point.  She didn't say the researcher -
> the author of the current research article in question - was little-known.
> She said the literary author that (s)he was studying was little-known.
> Therefore, not many researchers will be interested in that literary author,
> so not many people will cite the article, however good it is.
>
> There is a real and valid point in Heather's message, and simply saying 'use
> other metrics' is vague, to say the least.  Please specify what metrics
> might be used to provide a valid quality measure to the work of researchers
> who study minority subjects which will excite interest, and therefore usage,
> and citations, from only a few people worldwide.

Fair question. Here is a quick and dirty solution, by way of illustration.
(Many more are possible along analogous lines.)

(1) Compare like with like. So we are only looking at literary
studies. The subject matter can be narrowed down by Boolean means,
including taxonomic descriptors. No comparing Shakespeare scholarship
works with molecular biology.

(2) Now, for illustrative purposes, let us suppose there are only two
authors studied by literary scholars: William Shakespeare and Joe
Bloggs.

(3) Do an OA space search (remember, we are assuming universal OA) to find
out how many litcrit papers there are on Shakespeare and on Bloggs:
Let's say there are S and B papers, respectively.

(4) In comparing the citation counts of Shakespeare scholars with Bloggs
scholars, divide the Shakespeare citations by S and the Bloggs citations
by B.

Notice that I used two metrics, jointly: citation counts and a semiometric
content-word count.

This is just a simplified illustration. With universal OA there will
be a vast palate of potential metrics to use to normalize comparisons
of like with like. Call it metric "profiling."

Separate this question from the question of how you would do it today,
with just ISI web of science metrics and no universal OA. The answer is
that you can't (though Google Scholar and Google Books might still allow
you to do a quick and dirty approximation).

Stevan Harnad

PS the absolute number of researchers working on a topic is itself a
metric, which may for some fields and some purposes be a valid and
informative one, and for other fields not. Compare like with like and
validate, initialize and weight metrics fields by field (or, for some
purposes, even subfield by subfield).