On Metrics and Metaphysics

[David E. Wojick]

On the contrary Steve, I believe there are good objective measures to be had. It is just that field and subfield are not among them. The measures need to reflect the actual topological and metrical properties of the thing under study. In this case we are moslty talking about networks and diffusion processes. Neither is properly divisible into bounded lumps called fields. If there are sets they are fuzzy at best. In fact I was surprised to learn that scientometrics requires fields. Citation and co-authorship do not intrinsically require field attributes.

As for multiple measures, it depends on what you are trying to measure. As I have said before there seems to be a lot of data driven measurement with no clear idea what is being measured. Science has objective properties that we are trying to understand.

David Wojick

>On 22-Oct-08, at 10:10 AM, David E. Wojick wrote:
>
>> Speaking metaphysically, I find the following problematic -- "The  point is  that metrics need to be plural, diverse, and validated and  weighted by  field and
>> subfield."
>>
>> I would argue that the concepts of field and subfield are too vague  and unscientific to be the basis for objective metrics. Science is  an ever changing body of inquiry, with personal, methodological and  subject matter clusters that change with scale and over time,  sometimes very rapidly. No two papers are about exactly the same  thing. Each paper is related to its neighbors in multiple ways. Each  object of study can be looked at in multiple ways.
>>
>> Thus field and subfield are always artificial divisions of  convenience imposed on an unbroken web of belief and inquiry.  Moreover, these divisions can often be usefully made in multiple  ways. If the science of science depends on these artificial  divisions as a basis for measurement then we are in serious trouble.
>
>Yes, field and subfield are artificial divisions of convenience. So  are institutions, laboratories, research funders, research projects,  research assessment exercises, etc. But we do make these artificial  divisions, so we need metrics to provide objective ways to navigate  within them. If the artificial divisions (such as disciplines) prove  inconvenient, or have untoward consequences, we can artificially  redivide them.
>
>The point is that we should not demand more of our metrics than we  demand of the data to which we apply them. And that multiple metrics  are much more promising than sticking to just one.
>
>The kind of radical relativism and subjectivism that is implied by:
>
>"No two papers are about exactly the same thing. Each paper is related  to its neighbors in multiple ways. Each object of study can be looked  at in multiple ways."
>
>would rule out not only metrics, but any sort of evaluation or  comparison. By that token, we may as well toss a coin (or conduct an  opinion poll) in deciding whom or what to fund, credit, or otherwise  reward.
>
>'the man who is ready to prove that metaphysics is wholly  impossible... is a brother metaphysician with a rival theory.'
>
>Stevan Harnad
>
>
>> David Wojick
>> DOE OSTI
>>
>>>
>>> On 22-Oct-08, at 7:33 AM, Loet Leydesdorff wrote:
>>>
>>>> It seems to me that the expectation of the citation frequency is   
>>>> among other
>>>> things a function of the local density of the citation network. A   
>>>> problem,
>>>> however, remains how to define the locale: a journal, a theme, a   
>>>> patent
>>>> class? "Quick and dirty" skips these problems, in my opinion. I   
>>>> agree that
>>>> it may be pragmatical and shows that a solution is possible in   
>>>> principle.
>>>
>>> With open access, it is no longer univariate (i.e., not just  citation  counts) and
>>> it is definitely no longer journal-centric (author and article   
>>> metrics, not journal
>>> JIFs, though journal JIFs can be among the metrics used). The point  is  that
>>> metrics need to be plural, diverse, and validated and weighted by   
>>> field and
>>> subfield.
>>>
>>> To repeat: The "quick and dirty" example I gave was not meant to  be  used,
>>> but to show that solutions (many solutions) are possible in  principle,  and
>>> that their main features are that they are (1) multivariate, (2)  field  or even
>>> subfield-based, (3) require prior (joint) validation, field by  field,  against an
>>> already validated or face-valid criterion (such as peer  evaluation),  and, most
>>> important, they are (4) conditional on the provision of a full  Open  Access
>>> database on which to base them -- a condition that does not yet  exist,  but
>>> one for which we are now fighting (using the potential of multiple   
>>> Open Access
>>> metrics as an incentive).
>>>
>>>> The problem seems to me in the inference from aggregated citing   
>>>> behavior to
>>>> an expectation of being cited. The analyst transposed the
>>>> citation-transaction matrix (Wouters, 1999).
>>>
>>> The matrix I have in mind is not a citation matrix, but a matrix   
>>> consisting of a
>>> rich and diverse set of metrics, including downloads, chronometrics
>>> (growth/decay of citations, downloads), co-citation metrics, co-  authorship
>>> metrics, funding metrics, student metrics. patent metrics, link  metrics,
>>> hub/authority metrics, endogamy/exogamy metrics, years of  publication,
>>> total publications, tag metrics, comment metrics, semiometrics,  and  more --
>>> all these harvested from the Open Access Research Web, once all  articles
>>> are OA, citation-linked, and download-metered.
>>>
>>> The difference between this plurimetric world and the world of   
>>> univariate
>>> citations will be like the difference between night and day. But  we  are still in
>>> the night...
>>>
>>> Stevan Harnad
>>
>> -- 
>> "David E. Wojick, PhD" <WojickD at osti.gov>
>> Senior Consultant for Innovation
>> Office of Scientific and Technical Information
>> US Department of Energy
>> http://www.osti.gov/innovation/
>> 391 Flickertail Lane, Star Tannery, VA 22654 USA
>> 540-858-3136
>>
>> http://www.bydesign.com/powervision/resume.html provides my bio and  past client list.
>> http://www.bydesign.com/powervision/Mathematics_Philosophy_Science/  presents some of my own research on information structure and  dynamics.