Perils of Press-Release Journalism: NSF and Chronicle of Higher Education
Pikas, Christina K.
Christina.Pikas at JHUAPL.EDU
Mon Mar 9 09:40:21 EDT 2009
Oh, but there definitely has been research on informal scholarly communication. My qualifying exam or experience or whatever they're calling it now was based on a review I did on the topic. It's available online at: http://terpconnect.umd.edu/~cpikas/878/Pikas_The_Impact_of_ICTs_on_ISSC_0506.pdf
In particular, studies looked at listservs just like this one and things like pre-print servers (which Bohlin 2004 also covers), and of course my research is centered on social computing technologies (and blogs in particular).
Examples include:
Matzat, U. (2004). Academic communication and internet discussion groups: Transfer of information or creation of social contacts? Social Networks, 26(3), 221-255.
Rojo, A., & Ragsdale, R. G. (1997). A process perspective on participation in scholarly electronic forums. Science Communication, 18(4), 320-341.
Talja, S., Savolainen, R., & Maula, H. (2004). Field differences in the use and perceived usefulness of scholarly mailing lists. Information Research-an International Electronic Journal, 10(1), 200.
Various things by J.P. Walsh with Maloney, Bayma, and other co-authors.
Also D.J. de Solla Price's work and others of that era described how insiders in the invisible college received the information first through informal channels before the journal articles are ever published.
Christina K. Pikas, MLS
R.E. Gibson Library & Information Center
The Johns Hopkins University Applied Physics Laboratory
Voice 240.228.4812 (Washington), 443.778.4812 (Baltimore)
Fax 443.778.5353
From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of David E. Wojick
Sent: Sunday, March 08, 2009 4:10 PM
To: SIGMETRICS at listserv.utk.edu
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and Chronicle of Higher Education
Dear Loet,
I agree about the arrows, although I think being reported is more like citation than publication. Also it is important to note that the news system includes many channels within the scientific community itself, about which we have little data. There are thousands of listservs like Sigmetrics, which I suspect are far more important than the public newspapers. Whether they are or not is the kind of question I am referring to.
To wander a bit, citation itself is often a form of news. As I have said before, the beginning part of a typical journal article is devoted to explaining the problem addressed by the research. It is therefore about (hence reporting) the work of others, which is why most of the citations occur in this section. I often find myself reading these overviews and citations even though the specific research result in the paper does not interest me.
Your point about discourse is very important. How the reasoning of a community is progressing is arguably far more important than who learns what from whom. (Discourse analysis is my basic field. I even claim to have discovered how sentences fit together to create reasoned discourse, such as science.) In fact the concept of knowledge diffusion is somewhat misleading here. It suggests that knowledge circulates without changing, but almost everyone adds some thinking along the way. To me the dynamics of science is the dynamics of that reasoning.
Thanks for your interest,
David
Dear David,
I meant with the two arrows:
First, on the supply side, which research is being reported and spreading, throughout the news system? This is akin to publication. Second, who in the scientific community is reading about which research?
In my opinion, it is helpful to distinguish between individuals and how they collect and disseminate information versus how discourses evolve (scientific discourses, political discourses, and the common discourse in the media). The three are interconnected by people and organizations.
Best, Loet
________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
loet at leydesdorff.net<mailto:loet at leydesdorff.net> ; http://www.leydesdorff.net/
________________________________
From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David E. Wojick
Sent: Saturday, March 07, 2009 5:08 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and Chronicle of Higher Education
Thank you Loet, but I am not talking about newspapers. That was merely the example that Gene supplied. I am talking about all significant forms of scientific communication, in the sense of all the forms whereby scientists learn about new science.
For example, I get most of my initial information about scientometrics papers from this listserv. In many cases I only read the abstract. I get a great deal of knowledge about other topics that I am involved in from the magazine part of Science, which I read carefully each week. For particular topics I get a lot from Google, Google Scholar, and the various OSTI products like Science.gov and WorldWideScience.gov. In one case (climate change) I rely mostly on blogs.
I am often led to journal articles but I read no specific journals regularly, so journals are never my original awareness source. The journal article typically shows up somewhere between steps 2 and 10, although I may not get to an article in many cases.
Taken all together this is the diffusion system of science. Journals play an essential role but from a transaction point of view it is quite small. Yet this small subsystem, the journals, gets 99% of our attention. We have very little idea what is going on in the rest of the diffusion system, probably because we have very little data on it. But it is there and it is important.
(Sorry but I did not use or mention arrows so I do not know what you are referring to by the second of my arrows.)
All my best,
David
Dear David,
The non-ISI-sources for citations are available in the JCR. This covers the second of your arrows. The first can be traced because major newspapers have excellent search engines on their archives. Alternatively, one can use Lexis Nexis. I do this regularly with students from communication studies. We then distinguish among three types of communication: scientific communication, public communication (newspapers), and political communication (parliamentary proceedings). The causality is very different in different domains.
For example, in the case of obesitas scientific communications feed into the other domains. In the case of issues like violence in computer games, public concern is leading both scientific interest and political debate. However, I would not overestimate the feedback arrow in specific cases. I don't remember having read anything about citation analysis in a newspaper which I did not already know from the scientific communication.
Best wishes,
Loet
PS. Below is the citation impact environment of the Wall Street Journal in 2006 using JCR data:
Picture deleted.
________________________________
Loet Leydesdorff
Amsterdam School of Communications Research (ASCoR),
Kloveniersburgwal 48, 1012 CX Amsterdam.
Tel.: +31-20- 525 6598; fax: +31-20- 525 3681
loet at leydesdorff.net<mailto:loet at leydesdorff.net> ; http://www.leydesdorff.net/
________________________________
From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
Sent: Saturday, March 07, 2009 1:07 PM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and Chronicle of Higher Education
Thank you Gene, this is indeed relevant and a good example of looking at the role of news in the diffusion of science within science.
However, I do question their use of the term "disproportionate" in their conclusion that "Articles in the Journal that were covered by the Times received a disproportionate number of scientific citations in each of the 10 years after the Journal articles appeared." The term "significantly larger" is better because disproportionate sound like a criticism.
In any case my point is that channels of communication other than reading journal articles play a major role in the diffusion of ideas within science. Not just news outlets, but channels like this listserv are very important. My conjecture is that when it comes to first awareness of a new result reading the journal is probably a relative small part of the system of information transactions. That is, most scientists learn about new results in other ways. The structure of the magazine/journal Science is very revealing in this context. The number of news-like articles about research results is much higher than the number of actual articles, perhaps 20 times higher.
What we need is to look at this non-journal diffusion as systematically as we look at the systems of citation and co-authorships. First awareness and citation or co-authorship are opposite extremes in the system of transactions. The approach will have to be different because the tracking of individual transactions is impossible. A reading a story about B for example. This is more like true diffusion analysis, in groundwater for example, than like network analysis. I think the flow of new language, which always characterizes the flow of new knowledge, will be an important approach. It can also be in real time.
So far as I know this kind of diffusion research is not being done, so the dynamics of new scientific thinking is not being seen, in an important respect. That was my original point.
My best regards,
David
David Wojick, Ph.D.
http://www.osti.gov/innovation/
Mar 6, 2009 04:31:17 PM, SIGMETRICS at LISTSERV.UTK.EDU wrote:
Dear David and Christina: I don't know whether it is relevant to your discussion, but the following reference from the 1991 NEJM sounds like it should be.
IMPORTANCE OF THE LAY PRESS IN THE TRANSMISSION OF MEDICAL KNOWLEDGE TO THE SCIENTIFIC COMMUNITY
Author(s): PHILLIPS DP<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9@G5LK2&name=PHILLIPS%20DP&ut=A1991GK53800030&pos=1>, KANTER EJ<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9@G5LK2&name=KANTER%20EJ&ut=A1991GK53800030&pos=2>, BEDNARCZYK B<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9@G5LK2&name=BEDNARCZYK%20B&ut=A1991GK53800030&pos=3>, TASTAD PL<http://apps.isiknowledge.com/DaisyOneClickSearch.do?product=WOS&search_mode=DaisyOneClickSearch&db_id=&SID=1A2NKmLJc29M9@G5LK2&name=TASTAD%20PL&ut=A1991GK53800030&pos=4>
Source: NEW ENGLAND JOURNAL OF MEDICINE Volume: 325 Issue: 16 Pages: 1180-1183 Published: OCT 17 1991
Times Cited: 105<http://apps.isiknowledge.com/CitingArticles.do?product=WOS&search_mode=CitingArticles&SID=1A2NKmLJc29M9@G5LK2&db_id=WOS&parentQid=1&parentDoc=2&recid=77483183> References: 15<http://apps.isiknowledge.com/CitedRefList.do?product=WOS&search_mode=CitedRefList&SID=1A2NKmLJc29M9@G5LK2&db_id=WOS&parentQid=1&parentDoc=2&recid=77483183> [driver?nimlet=download&fn=INBOX&mid=16301&partIndex=6&disp=inline] Citation Map[driver?nimlet=download&fn=INBOX&mid=16301&partIndex=7&disp=inline] <http://javascript:void();>
Abstract: Background. Efficient, undistorted communication of the results of medical research is important to physicians, the scientific community, and the public. Information that first appears in the scientific literature is frequently retransmitted in the popular press. Does popular coverage of medical research in turn amplify the effects of that research on the scientific community?
Methods. To test the hypothesis that researchers are more likely to cite papers that have been publicized in the popular press, we compared the number of references in the Science Citation Index to articles in the New England Journal of Medicine that were covered by The New York Times with the number of references to similar articles that were not covered by the Times. We also performed the comparison during a three-month period when the Times was on strike but continued to prepare an "edition of record" that was not distributed; doing so enabled us to address the possibility that coverage in the Times was simply a marker of the most important articles, which would therefore be cited more frequently, even without coverage in the popular press.
Results. Articles in the Journal that were covered by the Times received a disproportionate number of scientific citations in each of the 10 years after the Journal articles appeared. The effect was strongest in the first year after publication, when Journal articles publicized by the Times received 72.8 percent more scientific citations than control articles. This effect was not present for articles published during the strike; articles covered by the Times during this period were no more likely to be cited than those not covered. Conclusions. Coverage of medical research in the popular press amplifies the transmission of medical information from the scientific literature to the research community.
The PDF for this article was sent to me by Barbara Gastel. If you don't have access to the full text let me know. I presume you can access the list of 100 plus citing papers. If not let me know. Best wishes. Gene Garfield
________________________________
From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of David Wojick
Sent: Friday, February 27, 2009 6:28 AM
To: SIGMETRICS at LISTSERV.UTK.EDU
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and Chronicle of Higher Education
Thanks Christina, I will follow up on these leads.
However, what I am looking for is something more global, analogous to citation analysis but in two steps. First, on the supply side, which research is being reported and spreading, throughout the news system? This is akin to publication. Second, who in the scientific community is reading about which research? This is analogous to citation. This writing-reading transaction system is much harder to track than citations, but it might be just as important, if not more so. In fact how to track it is the biggest research challenge. The write-read news system is bigger, faster, more turbulent and much less tangible, especially the reading part.
Plus there is the significant difference that in the news system many publications are about topics rather than specific results. The negative impact of biofuel production for example. Yet this too is the spread of scientific ideas among scientists.
My conjecture is that the news system is far more important than scholarly publication when it comes to generating first awareness of research results and new ideas within the scientific community. Scholarly publication probably plays an intermediate role, that is it is something one reads after becoming interested as a result of news, and before one contacts the author. If so then the role of scholarly publication may be misunderstood, but this is presently just a guess.
My best regards,
David
David Wojick, Ph.D.
Senoir consultant for innovation
http://www.osti.gov
Feb 26, 2009 09:52:56 AM, SIGMETRICS at listserv.utk.edu wrote:
Certainly there has been research on how press releases figure into the diffusion of scientific information. For example, there have been bibliometric studies that included press release coverage in a regression equations regarding citedness. There have also been STS and public understanding of science (I refuse to use the unpleasant abbreviation) papers about this in general as well as the particular case surrounding cold fusion. There are also studies in scholarly communication that discuss the Ingelfinger rule and the like.
Actually, an editorial in today's Nature is about this issue with blogs, pre-prints, and press embargos: http://www.nature.com/nature/journal/v457/n7233/full/4571058a.html
Based on conversations with PLOS and Nature editors, it seems likely that they will both add more information to article pages regarding web commentary on blogs and other social computing technologies. Their goal is to provide a more 360 view of article/author impact than journal article citations do alone.
As far as how to study, I think there have even been some relevant questions on the GSS as well as smaller surveys, qualitative/ethnographic studies, critical/historical studies, etc. I'm not saying it's a done deal, but it certainly has been addressed.
Christina K. Pikas, MLS
R.E. Gibson Library & Information Center
The Johns Hopkins University Applied Physics Laboratory
Voice 240.228.4812 (Washington), 443.778.4812 (Baltimore)
Fax 443.778.5353
From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of David Wojick
Sent: Thursday, February 26, 2009 6:53 AM
To: SIGMETRICS at listserv.utk.edu
Subject: Re: [SIGMETRICS] Perils of Press-Release Journalism: NSF and Chronicle of Higher Education
Steve raises an important scientometric issue, quite apart from the issue of what Evans did or found. This is the role of press releases, and the news articles they engender, in the diffusion of scientific infromation. The question is how to observe and measure such diffusion? The number of information transactions, or A reading about B's results, via news is several orders of magnitude greater than via journal articles. I don't think we even know how many orders of magnitude. Yet this is in some respects the most important mode of scientific knowledge diffusion.
How this news based diffusion affects the dynamics of science is likewise unknown. Is anyone studying this formally? I am doing so informally. The web is providing some new approaches, such a blog tracking and the occurrence of embedded URLs. The spread of characteristic language is also a likely avenue. This is much more like true diffusion analysis than is citation and co-author network analysis, in that it goes beyond tracking large, discrete transactions to looking at a vague spreading cloud of information.
Steve also raises the issue of the spread of misinformation via diffusion of news. This has been studied in the context of general social thought, especially rumors. It is certainly significant in the realm of science and public policy, where the Evans case lies. I study this phenomenon in the climate change debate and in energy policy. Whether it is important in science per se I do not know. It is not even clear how one would approach it, but it seems like an important research topic. Perhaps it should be approached as the diffusion and dynamics of controversy or disagreement.
Cheers,
David
David Wojick, Ph.D.
http://www.osti.gov
Feb 25, 2009 06:40:34 PM, SIGMETRICS at listserv.utk.edu wrote:
.eprints.org/index.php?/archives/535-guid.html> of his Chronicle of Higher Education posting<http://chronicle.com/news/article/6026/fee-based-journals-get-better-results-study-in-fee-based-journal-reports?commented=1#c033840> on Evans and Reimer's (2009) Science article<http://www.sciencemag.org/cgi/content/full/323/5917/1025> (which I likewise critiqued<http://openaccess.eprints.org/index.php?/archives/533-guid.html>, though much more mildly), I got an email from Paul Basken asking me to explain what, if anything he had got wrong, since his posting was based entirely on a press release from NSF<http://www.nsf.gov/news/news_summ.jsp?cntn_id=114225>. Sure enough, the silly spin originated from the NSF Press release (though the buck stops with E & R's vague and somewhat tendentious description and interpretation of some of their findings). Here is the NSF Press Release, enhanced with my comments, for your delectation and verdict:
________________________________
If you offer something of value to people for free while someone else charges a hefty sum of money for the same type of product, one would logically assume that most people would choose the free option. According to new research in today's edition of the journal Science, if the product in question is access to scholarly papers and research, that logic might just be wrong. These findings provide new insight into the nature of scholarly discourse and the future of the open source publication movement[sic, emphasis added].
(1) If you offer something valuable for free, people will choose the free option unless they've already paid for the paid option (especially if they needed -- and could afford -- it earlier).
(2) Free access after an embargo of a year is not the same "something" as immediate free access. Its "value" for a potential user is lower. (That's one of the reasons institutions keep paying for subscription/license access to journals.)
(3) Hence it is not in the least surprising that immediate print-on-paper access + (paid) online access (IP + IO) generates more citations than immediate (paid) print-on-paper access (IP) alone.
(4) Nor is it surprising that immediate (paid) print-on-paper access + online access + delayed free online access (IP +IO + DF) generates more citations than just immediate (paid) print-on-paper + online access (IO + IP) alone -- even if the free access is provided a year later than the paid access.
(5) Why on earth would anyone conclude that the fact that the increase in citations from IP to IP + IO is 12% and the increase in citations from IP + IO to IP + IO + DF is a further 8% implies anything whatsoever about people's preference for paid access over free access? Especially when the free access is not even immediate (IF) but delayed (DF)?
Most research is published in scientific journals and reviews, and subscriptions to these outlets have traditionally cost money--in some cases a great deal of money. Publishers must cover the costs of producing peer-reviewed publications and in most cases also try to turn a profit. To access these publications, other scholars and researchers must either be able to afford subscriptions or work at institutions that can provide access.
In recent years, as the Internet has helped lower the cost of publishing, more and more scientists have begun publishing their research in open source outlets online. Since these publications are free to anyone with an Internet connection, the belief has been that more interested readers will find them and potentially cite them. Earlier studies had postulated that being in an open source format could more than double the number of times a journal article is used by other researchers.
What on earth is an "open source outlet"? ("Open source" is a software matter.) Let's assume what's meant is "open access"; but then is this referring to (i) publishing in an open access journal, to (ii) publishing in a subscription journal but also self-archiving the published article to make it open access, or to (iii) self-archiving an unpublished paper?
What (many) previous studies<http://opcit.eprints.org/oacitation-biblio.html> had measured (not "postulated") was that (ii) publishing in a subscription journal (IP + IO) and also self-archiving the published article to make it Open Access (IP + IO + OA) could more than double the citations, compared to IP + IO alone.
To test this theory, James A. Evans, an assistant professor of sociology at the University of Chicago, and Jacob Reimer, a student of neurobiology also at the University of Chicago, analyzed millions of articles available online, including those from open source publications and those that required payment to access.
No, they did nothing of the sort; and no "theory" was tested.
Evans & Reimer (E & R) only analyzed articles from subscription access journals before and after they became accessible online (to paid subscribers only) (i.e., IP vs IP + IO) as well as before and after the online version was made accessible free for all (after a paid-access-only embargo of up to a year or more: i.e., IP +IO vs IP + IO + DF). Their methodology was based on comparing citation counts for articles within the same journals before and after being made free online at various intervals.
The results were surprising. On average, when a given publication was made available online after being in print for a year, being published in an open source format increased the use of that article by about 8 percent. When articles are made available online in a commercial format a year after publication, however, usage increases by about 12 percent.
In other words, the citation count increase from just (paid) IP to (paid) IP + IO was 12% and the citation count increase from just (paid) IP + IO to (paid) IP + IO + DF was 8%. Not in the least surprising: Making paid-access articles accessible online increases their citations, and making them free online (even if only after a delay of a year) increases them still more.
What is surprising is the rather absurd spin that this press release appears to be trying to put on this unsurprising finding.
"Across the scientific community," Evans said in an interview, "it turns out that open access does have a positive impact on the attention that's given to the journal articles, but it's a small impact."
We already knew that OA increased citations, as the many prior published studies<http://opcit.eprints.org/oacitation-biblio.html> have shown. Most of those studies, however, were based on immediate OA (i.e., IF), not embargoed OA. What E & R do show, interestingly, is that even delaying OA for a year still increases citations, though not nearly as much as immediate OA (IF).
Yet Evans and Reimer's research also points to one very positive impact of the open source movement that is sometimes overlooked in the debate about scholarly publications. Researchers in the developing world, where research funding and libraries are not as robust as they are in wealthier countries, were far more likely to read and cite open source articles.
A large portion of the citation increase from (delayed) OA turns out to come from Developing Countries (refuting Frandsen<http://www.hprints.org/hprints-00328270/en/>'s recent report to the contrary). (A similar comparison, within the US, of citations from the Have-Not Universities (with the smaller journal subscription budgets) compared to the Harvards may well reveal the same effect closer to home, though probably at a smaller scale.)
The University of Chicago team concludes that outside the developed world, the open source movement "widens the global circle of those who can participate in science and benefit from it."
And it will be interesting to test for the same effect comparing the Harvards and the Have-Nots in the US -- but a more realistic estimate might come from looking at immediate OA (IF) rather than just embargoed OA (DF).
So while some scientists and scholars may chose to pay for scientific publications even when free publications are available, their colleagues in other parts of the world may find that going with open source works is the only choice they have.
It would be interesting to hear the authors of this NSF press release -- or E & R, for that matter -- explain how this paradoxical "preference" for paid access over free access was tested during the access embargo period...
Stevan Harnad<http://www.eprints.org/openaccess/>
American Scientist Open Access Forum<http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html>
--
"David E. Wojick, Ph.D., PE" <WojickD at osti.gov>
Senior Consultant for Innovation
Office of Scientific and Technical Information
US Department of Energy
http://www.osti.gov/innovation/
391 Flickertail Lane, Star Tannery, VA 22654 USA
http://www.bydesign.com/powervision/resume.html provides my bio and past client list.
http://www.bydesign.com/powervision/Mathematics_Philosophy_Science/ presents some of my own research on information structure and dynamics.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.asis.org/pipermail/sigmetrics/attachments/20090309/ef94dde1/attachment.html>
More information about the SIGMETRICS
mailing list