On Metrics and Metaphysics

[Stevan Harnad]

On 22-Oct-08, at 10:10 AM, David E. Wojick wrote:

> Speaking metaphysically, I find the following problematic -- "The  
> point is  that metrics need to be plural, diverse, and validated and  
> weighted by  field and
> subfield."
>
> I would argue that the concepts of field and subfield are too vague  
> and unscientific to be the basis for objective metrics. Science is  
> an ever changing body of inquiry, with personal, methodological and  
> subject matter clusters that change with scale and over time,  
> sometimes very rapidly. No two papers are about exactly the same  
> thing. Each paper is related to its neighbors in multiple ways. Each  
> object of study can be looked at in multiple ways.
>
> Thus field and subfield are always artificial divisions of  
> convenience imposed on an unbroken web of belief and inquiry.  
> Moreover, these divisions can often be usefully made in multiple  
> ways. If the science of science depends on these artificial  
> divisions as a basis for measurement then we are in serious trouble.

Yes, field and subfield are artificial divisions of convenience. So  
are institutions, laboratories, research funders, research projects,  
research assessment exercises, etc. But we do make these artificial  
divisions, so we need metrics to provide objective ways to navigate  
within them. If the artificial divisions (such as disciplines) prove  
inconvenient, or have untoward consequences, we can artificially  
redivide them.

The point is that we should not demand more of our metrics than we  
demand of the data to which we apply them. And that multiple metrics  
are much more promising than sticking to just one.

The kind of radical relativism and subjectivism that is implied by:

"No two papers are about exactly the same thing. Each paper is related  
to its neighbors in multiple ways. Each object of study can be looked  
at in multiple ways."

would rule out not only metrics, but any sort of evaluation or  
comparison. By that token, we may as well toss a coin (or conduct an  
opinion poll) in deciding whom or what to fund, credit, or otherwise  
reward.

'the man who is ready to prove that metaphysics is wholly  
impossible... is a brother metaphysician with a rival theory.'

Stevan Harnad


> David Wojick
> DOE OSTI
>
>>
>> On 22-Oct-08, at 7:33 AM, Loet Leydesdorff wrote:
>>
>>> It seems to me that the expectation of the citation frequency is   
>>> among other
>>> things a function of the local density of the citation network. A   
>>> problem,
>>> however, remains how to define the locale: a journal, a theme, a   
>>> patent
>>> class? "Quick and dirty" skips these problems, in my opinion. I   
>>> agree that
>>> it may be pragmatical and shows that a solution is possible in   
>>> principle.
>>
>> With open access, it is no longer univariate (i.e., not just  
>> citation  counts) and
>> it is definitely no longer journal-centric (author and article   
>> metrics, not journal
>> JIFs, though journal JIFs can be among the metrics used). The point  
>> is  that
>> metrics need to be plural, diverse, and validated and weighted by   
>> field and
>> subfield.
>>
>> To repeat: The "quick and dirty" example I gave was not meant to  
>> be  used,
>> but to show that solutions (many solutions) are possible in  
>> principle,  and
>> that their main features are that they are (1) multivariate, (2)  
>> field  or even
>> subfield-based, (3) require prior (joint) validation, field by  
>> field,  against an
>> already validated or face-valid criterion (such as peer  
>> evaluation),  and, most
>> important, they are (4) conditional on the provision of a full  
>> Open  Access
>> database on which to base them -- a condition that does not yet  
>> exist,  but
>> one for which we are now fighting (using the potential of multiple   
>> Open Access
>> metrics as an incentive).
>>
>>> The problem seems to me in the inference from aggregated citing   
>>> behavior to
>>> an expectation of being cited. The analyst transposed the
>>> citation-transaction matrix (Wouters, 1999).
>>
>> The matrix I have in mind is not a citation matrix, but a matrix   
>> consisting of a
>> rich and diverse set of metrics, including downloads, chronometrics
>> (growth/decay of citations, downloads), co-citation metrics, co-  
>> authorship
>> metrics, funding metrics, student metrics. patent metrics, link  
>> metrics,
>> hub/authority metrics, endogamy/exogamy metrics, years of  
>> publication,
>> total publications, tag metrics, comment metrics, semiometrics,  
>> and  more --
>> all these harvested from the Open Access Research Web, once all  
>> articles
>> are OA, citation-linked, and download-metered.
>>
>> The difference between this plurimetric world and the world of   
>> univariate
>> citations will be like the difference between night and day. But  
>> we  are still in
>> the night...
>>
>> Stevan Harnad
>
> -- 
>
> "David E. Wojick, PhD" <WojickD at osti.gov>
> Senior Consultant for Innovation
> Office of Scientific and Technical Information
> US Department of Energy
> http://www.osti.gov/innovation/
> 391 Flickertail Lane, Star Tannery, VA 22654 USA
> 540-858-3136
>
> http://www.bydesign.com/powervision/resume.html provides my bio and  
> past client list.
> http://www.bydesign.com/powervision/Mathematics_Philosophy_Science/  
> presents some of my own research on information structure and  
> dynamics.