Is multidisciplinary research more highly cited? A macro-level study

[Jonathan Levitt]

Dear Loet and Jonathan,

Thank you very much for your interesting responses to our posting.  I am 
replying without first consulting Mike, as he is currently away.

In our investigation of science categories (for articles in Scopus 
published in 1995) we found that the citation levels of Multi-disciplinary 
articles were on average roughly half those of Mono-disciplinary articles.  
This finding (Table 8 of our paper), contrasts sharply with that of 
Evidence Ltd: “If articles are indexed on their normalised citation impact, 
there is no reason to suppose that those which appear to be more 
interdisciplinary will be in any way systematically disadvantaged” 
(Bibliometric analysis of interdisciplinary research, 2007, p. 57-58, 
http://www.hefce.ac.uk/pubs/rdreports/2007/rd19_07/rd19_07.doc).

One possible reason for the contrasting findings is that Levitt & Thelwall 
use the subject designations of Scopus, whereas Evidence Ltd. uses the 
designations of Current Contents.  In order to illustrate how these 
designations differ, let us identify the categories 
containing ‘engineering’.  Scopus has only two categories containing the 
word (Chemical Engineering; Engineering), whereas the Current Contents’ 
subject area of ‘Engineering, Computing and Technology’ has ten categories 
(Chemical Engineering; Civil Engineering; Computer Science & Engineering; 
Electrical and Electronics Engineering; Engineering Management; Engineering 
Mathematics; Environmental Engineering & Energy; Geological, Petroleum & 
Mining Engineering; Materials Science & Engineering; Mechanical 
Engineering).  

Could the prevalence of closely allied subject categories in Current 
Contents at least in part account for the differences in our findings?  A 
simple way of addressing this question is for Evidence Ltd. to clarify the 
extent to which their findings vary between subject areas (I think they use 
the subject areas of Agriculture, Biology and Environmental Sciences; 
Clinical Medicine; Engineering, Computing and Technology; Life Sciences; 
Physical, Chemical and Earth Sciences).  This data would help support or 
refute the conjecture that their findings could have been shaped by subject 
areas that have many closely allied subject categories.

Evidence Ltd. wrote “We would conclude that there is no reason to single 
out interdisciplinary research for differential treatment under a metrics-
based system of assessment” (Bibliometric analysis of interdisciplinary 
research, 2007, p. 58).  Given that the findings of Levitt & Thelwall and 
Evidence Ltd. differ strongly and the urgency of the matter in the UK, I 
agree with both Loet and Jonathan on the importance of further 
investigation.  Please excuse any typing errors; I am visually challenged 
and at this late hour cannot ask a sighted person to proof-read my posting.

Best wishes,
Jonathan.

   
 
  
Date:         Mon, 13 Oct 2008 12:20:46 +0200
Reply-To:     ASIS&T Special Interest Group on Metrics
              <SIGMETRICS at LISTSERV.UTK.EDU>
Sender:       ASIS&T Special Interest Group on Metrics
              <SIGMETRICS at LISTSERV.UTK.EDU>
From:         Loet Leydesdorff <loet at LEYDESDORFF.NET>
Subject:      Re: Is multidisciplinary research more highly cited? A
              macro-level study
In-Reply-To:  
<592818C3D92EAE4791031E3E622927B3014703 at evidence1.Evidence.local>
Content-Type: multipart/alternative;


Dear Jonathan,

I mean something simple using your data. You could for example plot the
citation scores of your "interdisciplinary" papers in decreasing order of
being-citedness (one expects a Lotka-type distribution), and then add as a
second curve the impact factors of the journals. One could then get an
impression whether the noted effect could maily be caused by the upper end
of the curve (interdisciplinary articles cited more than average in
disciplinary journals) or the long tail of "interdisciplinary papers" in the
"carbage can".

But I see immediately that one may have to differentiate this for fields of
science and then it easily becomes more complex. Nevertheless, I would be
interested to know in which journals the top-100 (in terms of citation
rates) of your "interdisciplinary" papers were published. Is that in the
report?

Best, Loet

On Mon, Oct 13, 2008 at 11:39 AM, Jonathan Adams <
jonathan.adams at evidence.co.uk> wrote:

> Adminstrative info for SIGMETRICS (for example unsubscribe):
> http://web.utk.edu/~gwhitney/sigmetrics.html
>
> I think Michel Zitt drew attention to this when he pointed out that the
> 'zoom' with which you look at an item makes a difference to its citation
> calibration.  Context is all!
> And I agree: there are further tests to elaborate the findings and the
> initial hypotheses that emerge.  So, we need a sponsor!
> Jonathan Adams
>
> Evidence Ltd
> 103 Clarendon Road, Leeds LS2 9DF, UK
> t/ +44 (0) 113 384 5680
>
>
> -----Original Message-----
> From: ASIS&T Special Interest Group on Metrics
>  [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff
> Sent: 13 October 2008 10:05
>
> Dear Jonathan,
>
> If both of you are right -- which I have no reason to doubt -- than the
> aggregation at the journal level would make the difference. In that
> case,
> this would be a nice example of the so-called ecological fallacy: what
> one
> can see at the level of the wood, one cannot see at the level of
> individual
> trees, and vice versa (Robertson, 1950).
>
> Given the urgency of the matter in the UK, it may be worth testing this
> hypothesis: if you would reorganize your sample in terms of the journals
> involved, would then ...? Or, in other words, would interdisciplinary
> articles in disciplinary journals do better than average?
>
> Best wishes,
>
>
> Loet
>
> ________________________________
>
> Loet Leydesdorff
> Amsterdam School of Communications Research (ASCoR),
> Kloveniersburgwal 48, 1012 CX Amsterdam.
> Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
> loet at leydesdorff.net ; http://www.leydesdorff.net/
>
>
>
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Jonathan Adams
> > Sent: Monday, October 13, 2008 10:40 AM
> > To: SIGMETRICS at listserv.utk.edu
> > Subject: [SIGMETRICS] Is multidisciplinary research more
> > highly cited? A macro-level study
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > People may also be interested in Evidence Ltd's earlier study on
> > interdisciplinary research for the Higher Education Funding
> > Council for
> > England, as part of the background development of the Research
> > Excellence Framework.
> > http://www.hefce.ac.uk/pubs/rdreports/2007/rd19_07/
> > This report is a background document to HEFCE's consultation on
> > proposals for the Research Excellence Framework ('Research Excellence
> > Framework: consultation on the assessment and funding of higher
> > education research post-2008', HEFCE 2007/34).
> > We set out to develop an acceptable indicator of 'interdisciplinarity'
> > at the article level, rather than looking at articles in
> > multidisciplinary journals.  The report therefore considers how
> > interdisciplinary research could be defined for the purposes of
> > bibliometric analysis, and whether such research is
> > systematically cited
> > less often and could potentially be disadvantaged in a
> > bibliometrics-based system of research assessment.
> > The Evidence study found that there is no strong case for research
> > outputs to be treated differently for the purposes of research
> > assessment on the grounds of interdisciplinarity, but advises that
> > bibliometric analysis of such outputs should be carried out
> > carefully to
> > ensure they are treated appropriately.
> > Of course, the way in which we categorise more or less
> > interdisciplinary
> > research is important.  We exposed the methodology to senior
> > researchers
> > who had worked on previous RAE panels.  They agreed that our working
> > definition seemed to make intuitive sense in the context of their
> > (various) fields - which include social science.
> > Regards
> > Jonathan Adams
> >
> > Evidence Ltd
> > 103 Clarendon Road, Leeds LS2 9DF, UK
> > t/ +44 (0) 113 384 5680
> >
> > Registered in England No. 4036650
> > VAT Registration: GB 758 4671 85
> >
> > Please note that Evidence Ltd does not enter into any form of contract
> > via this medium, nor is our staff authorised to do so on our behalf.
> > -----Original Message-----
> > From: ASIS&T Special Interest Group on Metrics
> > [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Jonathan Levitt and
> > Mike Thelwall
> > Sent: 13 October 2008 00:23
> > To: SIGMETRICS at LISTSERV.UTK.EDU
> > Subject: [SIGMETRICS] Is multidisciplinary research more
> > highly cited? A
> > macro-level study
> >
> > Adminstrative info for SIGMETRICS (for example unsubscribe):
> > http://web.utk.edu/~gwhitney/sigmetrics.html
> >
> > Levitt, J.M. and Thelwall, M. (2008). Is multidisciplinary
> > research more
> >
> > highly cited? A macro-level study. Journal of the American
> > Society for
> > Information Science and Technology, 59(12), 1973-1984.
> >
> > Inter-disciplinary collaboration is a major goal in research policy.
> > This
> > study uses citation analysis to examine diverse subjects in
> > the Web of
> > Science and Scopus to ascertain whether, in general, research
> > published
> > in
> > journals classified in more than one subject is more highly
> > cited than
> > research published in journals classified in a single
> > subject. For each
> > subject the study divides the journals into two disjoint sets called
> > Multi
> > and Mono: Multi consists of all journals in the subject and
> > at least one
> >
> > other subject, whereas Mono consists of all journals in the
> > subject and
> > in
> > no other subject. The main findings are: (a) For social
> > science subject
> > categories in both the Web of Science and Scopus, the average
> > citation
> > levels of articles in Mono and Multi are very similar, and (b) For
> > Scopus
> > subject categories within Life Sciences, Health Sciences, and
> > Physical
> > Sciences, the average citation level of Mono articles is roughly twice
> > that
> > of Multi articles. Hence one cannot assume that, in general, multi-
> > disciplinary research will be more highly cited, and the converse is
> > probably true for many areas of science. A policy implication is that,
> > at
> > least in the sciences, multi-disciplinary researchers should not be
> > evaluated by citations on the same basis as mono-disciplinary
> > researchers.
> >
> > Reported in the Times Higher Education (REF could penalise
> > those working
> >
> > across disciplines,
> > http://www.timeshighereducation.co.uk/story.asp?
> > sectioncode=26&storycode=403796&c=2)
> >
> > Jonathan Levitt and Mike Thelwall