Perils of Press-Release Journalism: NSF and Chronicle of Higher Education

Sat Mar 7 09:39:37 EST 2009

Dear David, 

The non-ISI-sources for citations are available in the JCR. This covers the
second of your arrows. The first can be traced because major newspapers have
excellent search engines on their archives. Alternatively, one can use Lexis
Nexis. I do this regularly with students from communication studies. We then
distinguish among three types of communication: scientific communication,
public communication (newspapers), and political communication
(parliamentary proceedings). The causality is very different in different
domains. 

For example, in the case of obesitas scientific communications feed into the
other domains. In the case of issues like violence in computer games, public
concern is leading both scientific interest and political debate. However, I
would not overestimate the feedback arrow in specific cases. I don't
remember having read anything about citation analysis in a newspaper which I
did not already know from the scientific communication. 

Best wishes,

Loet

PS. Below is the citation impact environment of the Wall Street Journal in
2006 using JCR data:

  _____  

Loet Leydesdorff 
Amsterdam School of Communications Research (ASCoR), 
Kloveniersburgwal 48, 1012 CX Amsterdam. 
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 
 <mailto:loet at leydesdorff.net> loet at leydesdorff.net ;
<http://www.leydesdorff.net/> http://www.leydesdorff.net/ 

  _____  

From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
Sent: Saturday, March 07, 2009 1:07 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and
Chronicle of Higher Education

Thank you Gene, this is indeed relevant and a good example of looking at the
role of news in the diffusion of science within science. 

However, I do question their use of the term "disproportionate" in their
conclusion that "Articles in the Journal that were covered by the Times
received a disproportionate number of scientific citations in each of the 10
years after the Journal articles appeared." The term "significantly larger"
is better because disproportionate sound like a criticism. 

In any case my point is that channels of communication other than reading
journal articles play a major role in the diffusion of ideas within science.
Not just news outlets, but channels like this listserv are very important.
My conjecture is that when it comes to first awareness of a new result
reading the journal is probably a relative small part of the system of
information transactions. That is, most scientists learn about new results
in other ways. The structure of the magazine/journal Science is very
revealing in this context. The number of news-like articles about research
results is much higher than the number of actual articles, perhaps 20 times
higher.

What we need is to look at this non-journal diffusion as systematically as
we look at the systems of citation and co-authorships. First awareness and
citation or co-authorship are opposite extremes in the system of
transactions. The approach will have to be different because the tracking of
individual transactions is impossible. A reading a story about B for
example. This is more like true diffusion analysis, in groundwater for
example, than like network analysis. I think the flow of new language, which
always characterizes the flow of new knowledge, will be an important
approach. It can also be in real time.

So far as I know this kind of diffusion research is not being done, so the
dynamics of new scientific thinking is not being seen, in an important
respect. That was my original point. 

My best regards,

David

David Wojick, Ph.D.
http://www.osti.gov/innovation/

Mar 6, 2009 04:31:17 PM, SIGMETRICS at LISTSERV.UTK.EDU wrote:

Dear David and Christina: I don't know whether it is relevant to your
discussion, but the following reference from the 1991 NEJM sounds like it
should be.

IMPORTANCE OF        THE LAY PRESS        IN THE TRANSMISSION OF MEDICAL
KNOWLEDGE TO THE SCIENTIFIC COMMUNITY

 <http://javascript:void> Order full text <http://javascript:void> Context
Sensitive LinksGo to NCBI for additional information Holdings a UIUC Catalog
b ILLINET Catalog c UIC CatalogGo to Holdings

Format this record for printingE-mail this recordAdd this record to your
Marked List.Save to EndNote Web librarySave to EndNote, Reference Manager,
or similar bibliographic management tool
<http://apps.isiknowledge.com/full_record.do?product=WOS&search_mode=General
Search&qid=1&SID=1A2NKmLJc29M9 at G5LK2&page=1&doc=2#output_options#output_opti
ons> more options

 <http://javascript:void> Context Sensitive Links

Author(s):
<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode
=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9 at G5LK2&name=PHILLIPS%20DP&ut=A1
991GK53800030&pos=1> PHILLIPS DP, KANTER
<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode
=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9 at G5LK2&name=KANTER%20EJ&ut=A199
1GK53800030&pos=2>        EJ, BEDNARCZYK
<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode
=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9 at G5LK2&name=BEDNARCZYK%20B&ut=A
1991GK53800030&pos=3>        B, TASTAD
<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode
=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9 at G5LK2&name=TASTAD%20PL&ut=A199
1GK53800030&pos=4>        PL 

Source: NEW ENGLAND JOURNAL OF MEDICINE    Volume: 325    Issue: 16
Pages:        1180-1183    Published:        OCT 17 1991   

Times Cited: 105
<http://apps.isiknowledge.com/CitingArticles.do?product=WOS&search_mode=Citi
ngArticles&SID=1A2NKmLJc29M9 at G5LK2&db_id=WOS&parentQid=1&parentDoc=2&recid=7
7483183>      References: 15
<http://apps.isiknowledge.com/CitedRefList.do?product=WOS&search_mode=CitedR
efList&SID=1A2NKmLJc29M9 at G5LK2&db_id=WOS&parentQid=1&parentDoc=2&recid=77483
183>       <http://javascript:void();> Citation MapCitation Map beta     

Abstract:        Background. Efficient, undistorted communication of the
results of        medical research is important to physicians, the
scientific community,        and the public. Information that first appears
in the scientific        literature is frequently retransmitted in the
popular press. Does        popular coverage of medical research in turn
amplify the effects of        that research on the scientific community? 

Methods. To test the hypothesis that researchers are more likely        to
cite papers that have been publicized in the popular press, we
compared the number of references in the Science Citation Index to
articles in the New England Journal of Medicine that were covered by
The New York Times with the number of references to similar articles
that were not covered by the Times. We also performed the comparison
during a three-month period when the Times was on strike but continued
to prepare an "edition of record" that was not distributed;        doing so
enabled us to address the possibility that coverage in the        Times was
simply a marker of the most important articles, which would        therefore
be cited more frequently, even without coverage in the        popular press.

Results. Articles in the Journal that were covered by the Times
received a disproportionate number of scientific citations in each of
the 10 years after the Journal articles appeared. The effect was
strongest in the first year after publication, when Journal articles
publicized by the Times received 72.8 percent more scientific citations
than control articles. This effect was not present for articles
published during the strike; articles covered by the Times during this
period were no more likely to be cited than those not covered.
Conclusions. Coverage of medical research in the popular press
amplifies the transmission of medical information from the scientific
literature to the research community.

Document Type: Note        

Language:        English 

KeyWords Plus:        CITATION ANALYSIS 

Reprint Address:        PHILLIPS, DP (reprint author), UNIV CALIF SAN DIEGO,
DEPT SOCIOL, LA JOLLA, CA         92093 USA 

Publisher: MASS        MEDICAL SOC, 10 SHATTUCK, BOSTON,         MA 02115

Subject Category:        Medicine, General & Internal 

IDS Number:        GK538 

ISSN: 

The PDF for this article was sent to me by Barbara Gastel. If you don't have
access to the full text let me know. I presume you can access the list of
100 plus citing papers. If not let me know. Best wishes. Gene Garfield

  _____  

From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
Sent: Friday, February 27, 2009 6:28 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and
Chronicle of Higher Education

Thanks Christina, I will follow up on these leads.

However, what I am looking for is something more global, analogous to
citation analysis but in two steps. First, on the supply side, which
research is being reported and spreading, throughout the news system? This
is akin to publication. Second, who in the scientific community is reading
about which research? This is analogous to citation. This writing-reading
transaction system is much harder to track than citations, but it might be
just as important, if not more so. In fact how to track it is the biggest
research challenge. The write-read news system is bigger, faster, more
turbulent and much less tangible, especially the reading part. 

Plus there is the significant difference that in the news system many
publications are about topics rather than specific results. The negative
impact of biofuel production for example. Yet this too is the spread of
scientific ideas among scientists.

My conjecture is that the news system is far more important than scholarly
publication when it comes to generating first awareness of research results
and new ideas within the scientific community. Scholarly publication
probably plays an intermediate role, that is it is something one reads after
becoming interested as a result of news, and before one contacts the author.
If so then the role of scholarly publication may be misunderstood, but this
is presently just a guess.

My best regards,

David

David Wojick, Ph.D.

Senoir consultant for innovation
http://www.osti.gov

Feb 26, 2009 09:52:56 AM, SIGMETRICS at listserv.utk.edu wrote:

Certainly there has been research on how press releases figure into the
diffusion of scientific information.  For example, there have been
bibliometric studies that included press release coverage in a regression
equations regarding citedness.  There have also been STS and public
understanding of science (I refuse to use the unpleasant abbreviation)
papers about this in general as well as the particular case surrounding cold
fusion.  There are also studies in scholarly communication that discuss the
Ingelfinger rule and the like.

Actually, an editorial in today's Nature is about this issue with blogs,
pre-prints, and press embargos:
http://www.nature.com/nature/journal/v457/n7233/full/4571058a.html

Based on conversations with PLOS and Nature editors, it seems likely that
they will both add more information to article pages regarding web
commentary on blogs and other social computing technologies.  Their goal is
to provide a more 360 view of article/author impact than journal article
citations do alone.

As far as how to study, I think there have even been some relevant questions
on the GSS as well as smaller surveys, qualitative/ethnographic studies,
critical/historical studies, etc.  I'm not saying it's a done deal, but it
certainly has been addressed.

Christina K. Pikas, MLS 
R.E. Gibson Library & Information Center
The Johns Hopkins University Applied Physics Laboratory 
Voice  240.228.4812 (Washington), 443.778.4812 (Baltimore) 
Fax 443.778.5353 

From: ASIS&T Special Interest Group on Metrics
[mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of David Wojick
Sent: Thursday, February 26, 2009 6:53 AM
To: SIGMETRICS at listserv.utk.edu
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and
Chronicle of Higher Education

Steve raises an important scientometric issue, quite apart from the issue of
what Evans did or found. This is the role of press releases, and the news
articles they engender, in the diffusion of scientific infromation. The
question is how to observe and measure such diffusion? The number of
information transactions, or A reading about B's results, via news is
several orders of magnitude greater than via journal articles. I don't think
we even know how many orders of magnitude. Yet this is in some respects the
most important mode of scientific knowledge diffusion.

How this news based diffusion affects the dynamics of science is likewise
unknown. Is anyone studying this formally? I am doing so informally. The web
is providing some new approaches, such a blog tracking and the occurrence of
embedded URLs. The spread of characteristic language is also a likely
avenue. This is much more like true diffusion analysis than is citation and
co-author network analysis, in that it goes beyond tracking large, discrete
transactions to looking at a vague spreading cloud of information.

Steve also raises the issue of the spread of misinformation via diffusion of
news. This has been studied in the context of general social thought,
especially rumors. It is certainly significant in the realm of science and
public policy, where the Evans case lies. I study this phenomenon in the
climate change debate and in energy policy. Whether it is important in
science per se I do not know. It is not even clear how one would approach
it, but it seems like an important research topic. Perhaps it should be
approached as the diffusion and dynamics of controversy or disagreement.

Cheers,

David

David Wojick, Ph.D.

http://www.osti.gov

Feb 25, 2009 06:40:34 PM, SIGMETRICS at listserv.utk.edu wrote:

<http://openaccess.eprints.org/index.php?/archives/535-guid.html> critique
of his
<http://chronicle.com/news/article/6026/fee-based-journals-get-better-result
s-study-in-fee-based-journal-reports?commented=1#c033840> Chronicle of
Higher Education posting on Evans and Reimer's (2009)
<http://www.sciencemag.org/cgi/content/full/323/5917/1025> Science article
(which I likewise
<http://openaccess.eprints.org/index.php?/archives/533-guid.html> critiqued,
though much more mildly), I got an email from Paul Basken asking me to
explain what, if anything he had got wrong, since his posting was based
entirely on a  <http://www.nsf.gov/news/news_summ.jsp?cntn_id=114225> press
release from NSF. Sure enough, the silly spin originated from the NSF Press
release (though the buck stops with E & R's vague and somewhat tendentious
description and interpretation of some of their findings). Here is the NSF
Press Release, enhanced with my comments, for your delectation and verdict: 

  _____  

If you offer something of value to people for free while someone else
charges a hefty sum of money for the same type of product, one would
logically assume that most people would choose the free option. According to
new research in today's edition of the journal Science, if the product in
question is access to scholarly papers and research, that logic might just
be wrong. These findings provide new insight into the nature of scholarly
discourse and the future of the open source publication movement[sic,
emphasis added].

(1) If you offer something valuable for free, people will choose the free
option unless they've already paid for the paid option (especially if they
needed -- and could afford -- it earlier).

(2) Free access after an embargo of a year is not the same "something" as
immediate free access. Its "value" for a potential user is lower. (That's
one of the reasons institutions keep paying for subscription/license access
to journals.)

(3) Hence it is not in the least surprising that immediate print-on-paper
access + (paid) online access (IP + IO) generates more citations than
immediate (paid) print-on-paper access (IP) alone.

(4) Nor is it surprising that immediate (paid) print-on-paper access +
online access + delayed free online access (IP +IO + DF) generates more
citations than just immediate (paid) print-on-paper + online access (IO +
IP) alone -- even if the free access is provided a year later than the paid
access.

(5) Why on earth would anyone conclude that the fact that the increase in
citations from IP to IP + IO is 12% and the increase in citations from IP +
IO to IP + IO + DF is a further 8% implies anything whatsoever about
people's preference for paid access over free access? Especially when the
free access is not even immediate (IF) but delayed (DF)? 

Most research is published in scientific journals and reviews, and
subscriptions to these outlets have traditionally cost money--in some cases
a great deal of money. Publishers must cover the costs of producing
peer-reviewed publications and in most cases also try to turn a profit. To
access these publications, other scholars and researchers must either be
able to afford subscriptions or work at institutions that can provide
access.

In recent years, as the Internet has helped lower the cost of publishing,
more and more scientists have begun publishing their research in open source
outlets online. Since these publications are free to anyone with an Internet
connection, the belief has been that more interested readers will find them
and potentially cite them. Earlier studies had postulated that being in an
open source format could more than double the number of times a journal
article is used by other researchers.

What on earth is an "open source outlet"? ("Open source" is a software
matter.) Let's assume what's meant is "open access"; but then is this
referring to (i) publishing in an open access journal, to (ii) publishing in
a subscription journal but also self-archiving the published article to make
it open access, or to (iii) self-archiving an unpublished paper?

What (many)  <http://opcit.eprints.org/oacitation-biblio.html> previous
studies had measured (not "postulated") was that (ii) publishing in a
subscription journal (IP + IO) and also self-archiving the published article
to make it Open Access (IP + IO + OA) could more than double the citations,
compared to IP + IO alone. 

To test this theory, James A. Evans, an assistant professor of sociology at
the University of Chicago, and Jacob Reimer, a student of neurobiology also
at the University of Chicago, analyzed millions of articles available
online, including those from open source publications and those that
required payment to access.

No, they did nothing of the sort; and no "theory" was tested.

Evans & Reimer (E & R) only analyzed articles from subscription access
journals before and after they became accessible online (to paid subscribers
only) (i.e., IP vs IP + IO) as well as before and after the online version
was made accessible free for all (after a paid-access-only embargo of up to
a year or more: i.e., IP +IO vs IP + IO + DF). Their methodology was based
on comparing citation counts for articles within the same journals before
and after being made free online at various intervals. 

The results were surprising. On average, when a given publication was made
available online after being in print for a year, being published in an open
source format increased the use of that article by about 8 percent. When
articles are made available online in a commercial format a year after
publication, however, usage increases by about 12 percent.

In other words, the citation count increase from just (paid) IP to (paid) IP
+ IO was 12% and the citation count increase from just (paid) IP + IO to
(paid) IP + IO + DF was 8%. Not in the least surprising: Making paid-access
articles accessible online increases their citations, and making them free
online (even if only after a delay of a year) increases them still more.

What is surprising is the rather absurd spin that this press release appears
to be trying to put on this unsurprising finding. 

"Across the scientific community," Evans said in an interview, "it turns out
that open access does have a positive impact on the attention that's given
to the journal articles, but it's a small impact."

We already knew that OA increased citations, as the
<http://opcit.eprints.org/oacitation-biblio.html> many prior published
studies have shown.  Most of those studies, however, were based on immediate
OA (i.e., IF), not embargoed OA. What E & R do show, interestingly, is that
even delaying OA for a year still increases citations, though not nearly as
much as immediate OA (IF). 

Yet Evans and Reimer's research also points to one very positive impact of
the open source movement that is sometimes overlooked in the debate about
scholarly publications. Researchers in the developing world, where research
funding and libraries are not as robust as they are in wealthier countries,
were far more likely to read and cite open source articles.

A large portion of the citation increase from (delayed) OA turns out to come
from Developing Countries (refuting
<http://www.hprints.org/hprints-00328270/en/> Frandsen's recent report to
the contrary). (A similar comparison, within the US, of citations from the
Have-Not Universities (with the smaller journal subscription budgets)
compared to the Harvards may well reveal the same effect closer to home,
though probably at a smaller scale.) 

The University of Chicago team concludes that outside the developed world,
the open source movement "widens the global circle of those who can
participate in science and benefit from it."

And it will be interesting to test for the same effect comparing the
Harvards and the Have-Nots in the US -- but a more realistic estimate might
come from looking at immediate OA (IF) rather than just embargoed OA (DF). 

So while some scientists and scholars may chose to pay for scientific
publications even when free publications are available, their colleagues in
other parts of the world may find that going with open source works is the
only choice they have.

It would be interesting to hear the authors of this NSF press release -- or
E & R, for that matter -- explain how this paradoxical "preference" for paid
access over free access was tested during the access embargo period...

 <http://www.eprints.org/openaccess/> Stevan Harnad

<http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.
html> American Scientist Open Access Forum

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20090307/714c25e1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: WallStreetJ.JPG
Type: image/jpeg
Size: 70066 bytes
Desc: not available
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20090307/714c25e1/attachment.jpg>