From loet at LEYDESDORFF.NET Fri Sep 1 08:00:58 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Fri, 1 Sep 2006 14:00:58 +0200 Subject: Nanotechnology: Its Delineation in Terms of Journals and Patents -- preprint version available Message-ID: Nanotechnology as a Field of Science: Its Delineation in terms of Journals and Patents Loet Leydesdorff and Ping Zhou The Journal Citation Reports of the Science Citation Index 2004 were used to delineate a core set of nanotechnology journals and a nanotechnology-relevant set. In comparison with 2003, the core set has grown and the relevant set has decreased. This suggests a higher degree of codification in the field of nanotechnology: the field has become more focused in terms of citation practices. Using the citing patterns among journals at the aggregate level, a core group of ten nanotechnology journals in the vector space can be delineated on the criterion of betweenness centrality. National contributions to this core group of journals are evaluated for the years 2003, 2004, and 2005. Additionally, the specific class of nanotechnology patents in the database of the U.S. Patent and Trade Office (USPTO) is analyzed to determine if non-patent literature references can be used as a source for the delineation of the knowledge base in terms of scientific journals. The references are primarily to general science journals and letters, and therefore not specific enough for the purpose of delineating a journal set. ** apologies for cross-postings _____ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ NEW: The Knowledge-Based Economy: Modeled, Measured, Simulated. The Self-Organization of the Knowledge-Based Society; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: att62f04.gif Type: image/gif Size: 1101 bytes Desc: not available URL: From kboyack at SANDIA.GOV Fri Sep 1 11:41:16 2006 From: kboyack at SANDIA.GOV (Boyack, Kevin W) Date: Fri, 1 Sep 2006 09:41:16 -0600 Subject: Nanotechnology: Its Delineation in Terms of Journals and Patents -- preprint version available Message-ID: Dear Loet, Thanks for continuing to post your work as it becomes available. I very much enjoy keeping up to date on your work. In scanning through this paper, I thought I might make a point that could be of interest to those on this list that work with patent data. Your paper shows the lead position of "The Regents of the University of California" in the USPTO class 977 patents (nano). You correctly state that this includes all of the various branches of the University of California (Berkeley, SF, SD, SB, LA, Davis, Riverside, Irvine, Santa Cruz, Merced). But what many people may not know is that there are three other institutions that also patent under the "UC" umbrella, and in some cases these institutions dominate the university component. They are: Los Alamos National Laboratory Lawrence Livermore National Laboratory Lawrence Berkeley Laboratory All three institutions, although US Department of Energy labs, have the UC as their GOCO (government-owned contractor-operated) prime contractor, and patent under the UC umbrella (in most, but not all, cases). The best way to differentiate these patents from the true university patents is to look for the government contract number in the "government information" field, which, unfortunately, is no longer included in the weekly XML front page files on the USPTO weekly update site. The prime contract for Los Alamos was recently awarded to a Bechtel consortium (with some UC involvement). Whether or not the Los Alamos patents will appear under Bechtel or LANL or ??? in the future remains to be seen. Regarding the splitting out of UC patents to the various university locations - good luck. I have done it for one small project, and the best way I found was to use Google Scholar to find a recent paper by the inventor(s) to get a university address. Use of inventor city and state is not definitive in this case, because there are five institutions in the SF Bay area (UCSF, UCB, UCSC, LLNL, LBL) and three in the greater Los Angeles area (UCLA, UCI, UCSB). I hope this information is useful to someone. Best wishes, Kevin ________________________________ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff Sent: Friday, September 01, 2006 6:01 AM To: SIGMETRICS at listserv.utk.edu Subject: [SIGMETRICS] Nanotechnology: Its Delineation in Terms of Journals and Patents -- preprint version available Nanotechnology as a Field of Science: Its Delineation in terms of Journals and Patents Loet Leydesdorff and Ping Zhou The Journal Citation Reports of the Science Citation Index 2004 were used to delineate a core set of nanotechnology journals and a nanotechnology-relevant set. In comparison with 2003, the core set has grown and the relevant set has decreased. This suggests a higher degree of codification in the field of nanotechnology: the field has become more focused in terms of citation practices. Using the citing patterns among journals at the aggregate level, a core group of ten nanotechnology journals in the vector space can be delineated on the criterion of betweenness centrality. National contributions to this core group of journals are evaluated for the years 2003, 2004, and 2005. Additionally, the specific class of nanotechnology patents in the database of the U.S. Patent and Trade Office (USPTO) is analyzed to determine if non-patent literature references can be used as a source for the delineation of the knowledge base in terms of scientific journals. The references are primarily to general science journals and letters, and therefore not specific enough for the purpose of delineating a journal set. ** apologies for cross-postings ________________________________ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ NEW: The Knowledge-Based Economy: Modeled, Measured, Simulated . The Self-Organization of the Knowledge-Based Society ; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: att62f04.gif Type: image/gif Size: 1101 bytes Desc: att62f04.gif URL: From quentinburrell at MANX.NET Fri Sep 1 11:39:04 2006 From: quentinburrell at MANX.NET (Quentin L. Burrell) Date: Fri, 1 Sep 2006 16:39:04 +0100 Subject: A model for the Hirsch index Message-ID: Hirsch's h-index: a preliminary stochastic model Quentin L Burrell Paper to be presented at the 9th International Conference on Science and Technology Indicators: "New Challenges in Quantitative Science and Technology Research", Leuven, Belgium, 7-9 September, 2006. The fully developed version of this presentation will appear as a paper in the new Journal of Informetrics in due course. A preprint version will be available following the conference. Preprint requests should be addressed to the author at q.burrell at ibs.ac.im Abstract We propose a simple stochastic model for an author's production/citation process in order to investigate the recently proposed h-index for measuring an author's research output and its impact. The parametric model distinguishes between an author's publication process and the subsequent citation processes of the published papers. This allows us to investigate different scenarios such as varying the production/publication rates and citation rates as well as the researcher's career length. We are able to draw tentative results regarding the dependence of Hirsch's h-index on each of these fundamental parameters. We conjecture that the h-index is, according to this model, (approximately) linear in career length, log publication rate and log citation rate, at least for moderate citation rates. ****************************************************************************************************** Dr Quentin L Burrell Isle of Man International Business School The Nunnery, Old Castletown Road, Douglas, Isle of Man IM2 1QB, via United Kingdom Tel. +44(0)1624 693706 Website www.ibs.ac.im -------------- next part -------------- An HTML attachment was scrubbed... URL: From loet at LEYDESDORFF.NET Fri Sep 1 12:50:53 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Fri, 1 Sep 2006 18:50:53 +0200 Subject: Nanotechnology: Its Delineation in Terms of Journals and Patents -- preprint version available In-Reply-To: <46EAC19F3066C14BB20DF799A649C5F4020DB92A@ES23SNLNT.srn.sandia.gov> Message-ID: Thanks, Kevin. It is interesting and I 'll add a footnote about this in the next version! Best wishes, Loet _____ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at LISTSERV.UTK.EDU] On Behalf Of Boyack, Kevin W Sent: Friday, September 01, 2006 5:41 PM To: SIGMETRICS at LISTSERV.UTK.EDU Subject: Re: [SIGMETRICS] Nanotechnology: Its Delineation in Terms of Journals and Patents -- preprint version available Dear Loet, Thanks for continuing to post your work as it becomes available. I very much enjoy keeping up to date on your work. In scanning through this paper, I thought I might make a point that could be of interest to those on this list that work with patent data. Your paper shows the lead position of "The Regents of the University of California" in the USPTO class 977 patents (nano). You correctly state that this includes all of the various branches of the University of California (Berkeley, SF, SD, SB, LA, Davis, Riverside, Irvine, Santa Cruz, Merced). But what many people may not know is that there are three other institutions that also patent under the "UC" umbrella, and in some cases these institutions dominate the university component. They are: Los Alamos National Laboratory Lawrence Livermore National Laboratory Lawrence Berkeley Laboratory All three institutions, although US Department of Energy labs, have the UC as their GOCO (government-owned contractor-operated) prime contractor, and patent under the UC umbrella (in most, but not all, cases). The best way to differentiate these patents from the true university patents is to look for the government contract number in the "government information" field, which, unfortunately, is no longer included in the weekly XML front page files on the USPTO weekly update site. The prime contract for Los Alamos was recently awarded to a Bechtel consortium (with some UC involvement). Whether or not the Los Alamos patents will appear under Bechtel or LANL or ??? in the future remains to be seen. Regarding the splitting out of UC patents to the various university locations - good luck. I have done it for one small project, and the best way I found was to use Google Scholar to find a recent paper by the inventor(s) to get a university address. Use of inventor city and state is not definitive in this case, because there are five institutions in the SF Bay area (UCSF, UCB, UCSC, LLNL, LBL) and three in the greater Los Angeles area (UCLA, UCI, UCSB). I hope this information is useful to someone. Best wishes, Kevin _____ From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Loet Leydesdorff Sent: Friday, September 01, 2006 6:01 AM To: SIGMETRICS at listserv.utk.edu Subject: [SIGMETRICS] Nanotechnology: Its Delineation in Terms of Journals and Patents -- preprint version available Nanotechnology as a Field of Science: Its Delineation in terms of Journals and Patents Loet Leydesdorff and Ping Zhou The Journal Citation Reports of the Science Citation Index 2004 were used to delineate a core set of nanotechnology journals and a nanotechnology-relevant set. In comparison with 2003, the core set has grown and the relevant set has decreased. This suggests a higher degree of codification in the field of nanotechnology: the field has become more focused in terms of citation practices. Using the citing patterns among journals at the aggregate level, a core group of ten nanotechnology journals in the vector space can be delineated on the criterion of betweenness centrality. National contributions to this core group of journals are evaluated for the years 2003, 2004, and 2005. Additionally, the specific class of nanotechnology patents in the database of the U.S. Patent and Trade Office (USPTO) is analyzed to determine if non-patent literature references can be used as a source for the delineation of the knowledge base in terms of scientific journals. The references are primarily to general science journals and letters, and therefore not specific enough for the purpose of delineating a journal set. ** apologies for cross-postings _____ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ NEW: The Knowledge-Based Economy: Modeled, Measured, Simulated. The Self-Organization of the Knowledge-Based Society; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: att62f04.gif Type: image/gif Size: 1101 bytes Desc: not available URL: From j.s.katz at SUSSEX.AC.UK Fri Sep 1 15:45:53 2006 From: j.s.katz at SUSSEX.AC.UK (Sylvan Katz) Date: Fri, 1 Sep 2006 13:45:53 -0600 Subject: Indicators for complex innovation systems Message-ID: Katz, J. S., 2006: Indicators for complex innovation systems. Research Policy, 35, 893-909. Preprint ABSTRACT Performance indicators such as national wealth (GDP per capita), R&D intensity (GERD/GDP) and scientific impact (citations/ paper) are used to compare innovation systems. These indicators are derived from the ratio of primary measures such as population, GDP, GERD and papers. Frequently they are used to rank members of an innovation system and to inform decision makers. This is illustrated by the European Research Area S&T indicators scoreboard used to compare the performance of member states. A formal study of complex systems has evolved over the past few decades from common observations made by researchers from many fields. Complex systems are dynamic and many of their properties emerge from the interactions among the entities in them. They also have a propensity to exhibit power law or scaling correlations between primary measures used to characterize them. Katz [Katz, J.S., 2000. Scale independent indicators and research assessment. Science and Public Policy 27, 23?36] showed that scientific impact (citations/paper) scales with the size of the group (papers). In this paper it will be shown that two other common measures, R&D intensity and national wealth, scale with the sizes of European countries and Canadian provinces. Some of these scaling correlations are predictable. These findings illustrate that a performance indicator derived from the ratio of two measures may not be properly normalized for size. This paper argues that innovation systems are complex systems. Hence scaling correlations are expected to exist between the primary measures used to characterize them. These scaling correlations can be used to construct scale-independent (scale-adjusted) indicators and models that are truly normalized for size. Scale-independent indicators can more accurately inform decision makers how groups of different sizes contribute to an innovation system. The ranks of member groups of an innovation system by scale-independent indicators can be subtly and profoundly different than the ranks given by conventional indicators. The differences can result in a shift in perspective about the performance of members of an innovation system that has public policy implications. Keywords: Complex system; Emergent; Indicator; Power law; Innovation system Dr. J. Sylvan Katz, Visiting Fellow SPRU, University of Sussex http://www.sussex.ac.uk/Users/sylvank Adjunct Professor Mathematics & Statistics, University of Saskatchewan Associate Researcher Institut national de la recherche scientifique, University of Quebec From Michel.Menou at WANADOO.FR Thu Sep 7 11:03:19 2006 From: Michel.Menou at WANADOO.FR (Michel J. Menou) Date: Thu, 7 Sep 2006 17:03:19 +0200 Subject: [Sigkm-l] International Calendar of Information Science Conferences Message-ID: ** Please excuse cross-postings and forward as appropriate, especially to LIS societies and conference organizers ** A periodic reminder to post your event and look for conferences around the world in the: INTERNATIONAL CALENDAR OF INFORMATION SCIENCE CONFERENCES (http://icisc.neasist.org) Over 500 conferences listed in the past year! Over 50 events posted in the past 3 weeks! Over 75 countries hosting events on all continents! Over 80 countries represented by visitors to the Calendar! Over 9,000 visits a month! If you work in the information sciences and related disciplines (libraries, archives, museums, information and communication technology, telecommunications, etc.), be sure to stay up to date on the latest opportunities to learn and share your work throughout the world by regularly checking (RSS available) and posting your events in the Calendar. Help colleagues by registering the conferences you are aware of in case they are not listed. If your favourite conference or the one you organise is not listed, be sure to let us know of your event. There are three ways to submit events. See the FAQ here: http://icisc.neasist.org/about.html. ACCESSIBILITY The "Quick Calendar" is a static version of the calendar grouped by date and by region: http://icisc.neasist.org/quickcalendar.html ... great for a quick glance, slower connection speeds or printing. CONFERENCE ORGANIZERS AND PROFESSIONAL ASSOCIATIONS Use the Calendar to coordinate events with other groups. Feel free to post "PROPOSED" events. We encourage you to enter your events directly: http://icisc.neasist.org/instructions.html VOLUNTEERS NEEDED If you would like to play a more active role in facilitating the international exchange of ideas through this Calendar, please contact Caryn Anderson at icisc at asist.org. Volunteers with all skills are welcome, and those with knowledge of calendar software are especially appreciated. The Calendar is a nonprofit collaboration between the International Information Issues Special Interest Group and the European and New England chapters of the American Society for Information Science and Technology (http://www.asist.org), with the additional support of Haworth Press (http://haworthpress.com/library/). Thank you for bookmarking and/or subscribing to the International Calendar of Information Science Conferences (http://icisc.neasist.org/) and sharing with your colleagues! -- Caryn Anderson Program Coordinator PhD/Managerial Leadership in the Information Professions GSLIS, Simmons College 300 The Fenway, P-204E Boston, MA 02115 caryn.anderson at simmons.edu 617.521.2829 http://www.simmons.edu/gslis/phdmlip -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.1/440 - Release Date: 06/09/2006 From kretschmer.h at T-ONLINE.DE Sun Sep 10 13:20:37 2006 From: kretschmer.h at T-ONLINE.DE (kretschmer.h@t-online.de) Date: Sun, 10 Sep 2006 19:20:37 +0200 Subject: Call for Papers: 3rd Intern. Conf. on Webometrics, Informetrics & Scientometrics In-Reply-To: Message-ID: An HTML attachment was scrubbed... URL: From loet at LEYDESDORFF.NET Sun Sep 10 17:04:48 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Sun, 10 Sep 2006 23:04:48 +0200 Subject: Percentage of World Shares and Numbers of Publications, including 2005 Message-ID: _____ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ NEW: The Knowledge-Based Economy: Modeled, Measured, Simulated. The Self-Organization of the Knowledge-Based Society; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook.jpg Type: image/jpeg Size: 71695 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Outlook.jpg Type: image/jpeg Size: 69180 bytes Desc: not available URL: From eugene.garfield at THOMSON.COM Mon Sep 11 13:15:02 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Mon, 11 Sep 2006 13:15:02 -0400 Subject: Citation analysis in history of Google book search Message-ID: Readers of Sigmetrics may be interested in this historical note about citation analysis. >From Google's Angle Google also was excited about adding the new partner and, one assumes, by the vote of confidence it represented for the ongoing and ultimate success of the controversial project. Susan Wojcicki, vice president of product management at Google, stated: "We're thrilled to begin working with the University of California libraries to include their incredible collection in Google Book Search." Progress continues at the five initial research libraries participating in the project-the University of Michigan, Harvard University, Stanford University, Oxford University, and the New York Public Library. Google also has a pilot project with the Library of Congress' World Digital Library, a project to digitize rare, important material held in U.S. and Western libraries but focusing on non-Western countries and cultures. If you're interested in the Google Book Search project, you might want to read its history (http://books.google.com/googlebooks/newsviews/history.html). Oddly enough, the project appears to be older than Google itself. Two Stanford graduate students, working on the Stanford Digital Library Technologies Project back in 1996, built a specialized crawler for book content called BackRub. BackRub's citation analysis technique later evolved into the PageRank algorithms underlying Google. Of course, the names of the two students were Sergey Brin and Larry Page. After a brief detour to solve the problem of indexing the World Wide Web, they finally got back to the world of books in 2002. The full text of the story from this is quoted is at: http://www.infotoday.com/newsbreaks/nb060814-2.shtml When responding, please attach my original message __________________________________________________ Eugene Garfield, PhD. email: garfield at codex.cis.upenn.edu home page: www.eugenegarfield.org Tel: 215-243-2205 Fax 215-387-1266 Chairman Emeritus, ISI www.isinet.com 3501 Market Street, Philadelphia, PA 19104-3302 President, The Scientist LLC. www.the-scientist.com 400 Market Street, Suite 1250, Philadelphia, PA 19106-2501 Past President, American Society for Information Science and Technology (ASIS&T) www.asist.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugene.garfield at THOMSON.COM Mon Sep 11 15:58:38 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Mon, 11 Sep 2006 15:58:38 -0400 Subject: FW: Usage Statistics Message-ID: Eugene Garfield, PhD. email: garfield at codex.cis.upenn.edu -----Original Message----- From: NFAIS Listserv [mailto:NFAIS-L at LISTSERV.SILVERPLATTER.COM] On Behalf Of Jill O'Neill Sent: Monday, September 11, 2006 3:41 PM To: NFAIS-L at LISTSERV.SILVERPLATTER.COM Subject: Usage Statistics NFAIS Forum on the Current Status and Future Trends of Usage Statistics. Registration Discount Available Through September 29th. The NFAIS Committee on Best Practices/Usage Statistics is organizing a one-day meeting, Online Usage Statistics: Current Status and Future Directions, to be held on Friday, October 27, 2006 at PALINET Headquarters, in Philadelphia, PA, from 9:00am to 4:30pm. Usage statistics are a key metric. They are used by publishers and librarians alike to determine the value of electronic products and services, as a tool for collection development, and in many decision-making processes related to marketing, sales and the development of new products and library services. This one-day program will provide a brief overview of usage statistics - the standards used in their development, the major initiatives that have emerged to improve data collection and usage, and the pros and cons of the current status of usage statistics development. It will highlight the two major initiatives related to usage statistics, Project COUNTER and SUSHI, focusing on the objectives of each initiative, their current status and planned enhancements, and the relationship between the two projects. In addition, publishers and librarians will present case studies on their implementation of COUNTER and SUSHI, with a focus on the challenges and opportunities that both of these initiatives provide. The remainder of the program will discuss the collection of usage statistics, how they can be leveraged in the decision-making process with regard to the development of new products and product enhancements, and how they are applied in library collection development. The meeting will close with a futuristic discussion on what developments are required in order to improve and fine-tune the creation, collection and analysis of usage statistics and what changes can be expected to occur in the next few years. Whether you are new to this field and want to quickly come up-to-speed or an experienced information professional who wants to remain up-to-date on the rapidly evolving field of usage statistics, this meeting is for you. Come and join your peers for an enjoyable and fact-filled day that will provide information essential to your organization, and provide you with an opportunity for networking within the Information Community. The preliminary program, registration form, directions to the meeting location, list of nearby hotels and general information on Philadelphia is available at: http://www.nfais.org/events/event_details.cfm?id=39 Register early, as seating is limited. On or before September 29, 2006, NFAIS members pay $215 and non-members pay $265 (registration fee includes continental breakfast, a box lunch and an afternoon refreshment break). After September 29, 2006 NFAIS members pay $245 and non-members pay $295. For more information contact:? Jill O'Neill, NFAIS Director, Communication and Planning, 215-893-1561 (phone); 215-893-1564 (fax); mail to: mailto:jilloneill at nfais.org .?? Founded in 1958, NFAIS is a premier membership organization of more than 50 of the world's leading producers of databases, information services, and information technology in the sciences, engineering, social sciences, business, and the arts and humanities. -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.2/443 - Release Date: 9/11/2006 From eugene.garfield at THOMSON.COM Tue Sep 12 12:15:35 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Tue, 12 Sep 2006 12:15:35 -0400 Subject: FW: [Asis-l] New book about Paul Otlet in French Message-ID: Hopefully this book will appear one day in an English translation. All scientometricians should be aware of the pioneering work of Paul Otlet and his "Traite de Documentation" published in 1934. See also Michael Buckland's page at http://www.ischool.berkeley.edu/~buckland/otlet.html -----Original Message----- From: asis-l-bounces at asis.org [mailto:asis-l-bounces at asis.org] On Behalf Of Michel J. Menou Sent: Tuesday, September 12, 2006 10:23 AM To: asis-l at asis.org; sighfis-l at asis.org; eurchap; Euro_Student_ASIST at yahoogroups.com Subject: [Asis-l] New book about Paul Otlet in French A Belgian publisher, Les impressions nouvelles, announces the publication next month of a book by Fran?oise Levie "L'homme qui voulait classer le monde: Paul Otlet et le Mondaneum. (The man who wanted to systematize the world: Paul Otlet and the Mondaneum -- I'm not to keen of translation for "classer") Reduced subscription price till Sept. 30. See http://www.lesimpressionsnouvelles.com/OTLET.pdf - this not a commercial ad on my part - Mondaneum may be less attractive to techies than the semantic web, but ... Nil novi sub sole. Michel -- ===================================================================== Dr. Michel J. Menou Consultant in ICT policies and Knowledge & Information Management Adviser of Somos at Telecentros board http://www.tele-centros.org Member of the founding steering committee of Telecenters of the Americas Partnership http://www.tele-centers.net/ B.P. 15 F-49350 Les Rosiers sur Loire, France Email: micheljmenou [at] gmail.com Michel.Menou [at]wanadoo.fr Phone: +33 (0)2 41511043 http://www.ucl.ac.uk/ciber/peoplemenou.php ===================================================================== -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.3/445 - Release Date: 11/09/2006 ____ ________________________________________ Asis-l mailing list Asis-l at asis.org http://mail.asis.org/mailman/listinfo/asis-l -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.3/445 - Release Date: 9/11/2006 From garfield at CODEX.CIS.UPENN.EDU Tue Sep 12 17:07:53 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Tue, 12 Sep 2006 17:07:53 -0400 Subject: Herubel JPVM "Tradition and protean nature-journals and scholarly communication: A review essay " Libraries & Culture 41(2): 233-257 Spr 2006 Message-ID: JPVM Herubel : jpvmh at purdue.edu Title: Tradition and protean nature-journals and scholarly communication: A review essay Author(s): Herubel JPVM Source: LIBRARIES & CULTURE 41 (2): 233-257 SPR 2006 Document Type: Review Language: English Cited References: 162 Times Cited: 0 Addresses: Herubel JPVM (reprint author), Purdue Univ, W Lafayette, IN 47907 USA Purdue Univ, W Lafayette, IN 47907 USA Publisher: UNIV TEXAS PRESS, BOX 7819, AUSTIN, TX 78713-7819 USA Subject Category: HISTORY & PHILOSOPHY OF SCIENCE IDS Number: 066DR ISSN: 0894-8631 -------------- next part -------------- A non-text attachment was scrubbed... Name: D:\MMistry\Desktop\herubel.pdf Type: application/pdf Size: 145262 bytes Desc: not available URL: From antoine.blanchard at GMAIL.COM Fri Sep 15 03:40:53 2006 From: antoine.blanchard at GMAIL.COM (Antoine Blanchard) Date: Fri, 15 Sep 2006 09:40:53 +0200 Subject: FW: [Asis-l] New book about Paul Otlet in French In-Reply-To: <311174B69873F148881A743FCF1EE53702398345@TSHUSPAPHIMBX02.ERF.THOMSON.COM> Message-ID: Before that day comes, you can have a look at Fran?oise Levie's article in English about her related movie "L'homme qui voulait classer le monde", shown at the 2002 conference on "The History and Heritage of Scientific and Technological Information Systems" : http://www.chemheritage.org/events/asist2002/33-levie-film.pdf Regards, Antoine 2006/9/12, Eugene Garfield : > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Hopefully this book will appear one day in an English translation. All scientometricians should be aware of the pioneering work of Paul Otlet and his "Traite de Documentation" published in 1934. > See also Michael Buckland's page at http://www.ischool.berkeley.edu/~buckland/otlet.html > > > -----Original Message----- > From: asis-l-bounces at asis.org [mailto:asis-l-bounces at asis.org] On Behalf Of Michel J. Menou > Sent: Tuesday, September 12, 2006 10:23 AM > To: asis-l at asis.org; sighfis-l at asis.org; eurchap; Euro_Student_ASIST at yahoogroups.com > Subject: [Asis-l] New book about Paul Otlet in French > > A Belgian publisher, Les impressions nouvelles, announces the > publication next month of a book by Fran?oise Levie "L'homme qui voulait > classer le monde: Paul Otlet et le Mondaneum. (The man who wanted to > systematize the world: Paul Otlet and the Mondaneum -- I'm not to keen > of translation for "classer") > Reduced subscription price till Sept. 30. > See http://www.lesimpressionsnouvelles.com/OTLET.pdf > - this not a commercial ad on my part - > Mondaneum may be less attractive to techies than the semantic web, but ... > Nil novi sub sole. > Michel > > -- > ===================================================================== > Dr. Michel J. Menou > Consultant in ICT policies and Knowledge & Information Management > Adviser of Somos at Telecentros board http://www.tele-centros.org > Member of the founding steering committee of > Telecenters of the Americas Partnership http://www.tele-centers.net/ > B.P. 15 > F-49350 Les Rosiers sur Loire, France > Email: micheljmenou [at] gmail.com > Michel.Menou [at]wanadoo.fr > Phone: +33 (0)2 41511043 > http://www.ucl.ac.uk/ciber/peoplemenou.php > ===================================================================== > > > > -- > No virus found in this outgoing message. > Checked by AVG Free Edition. > Version: 7.1.405 / Virus Database: 268.12.3/445 - Release Date: 11/09/2006 > > ____ > ________________________________________ > Asis-l mailing list > Asis-l at asis.org > http://mail.asis.org/mailman/listinfo/asis-l > > -- > No virus found in this incoming message. > Checked by AVG Free Edition. > Version: 7.1.405 / Virus Database: 268.12.3/445 - Release Date: 9/11/2006 > > From Michel.Menou at WANADOO.FR Fri Sep 15 09:42:05 2006 From: Michel.Menou at WANADOO.FR (Michel J. Menou) Date: Fri, 15 Sep 2006 15:42:05 +0200 Subject: FW: [Asis-l] New book about Paul Otlet in French In-Reply-To: Message-ID: Thank you Antoine for this useful addition. BTW - again no ad - our colleague Prof. T. Aparac told me that this movie can be purchased from Mrs. Levie' company and is of great educational value. Mrs. levie can be reacged at flevie {at} yucom {dot} be _MM _ Antoine Blanchard a ?crit : > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > Before that day comes, you can have a look at Fran?oise Levie's > article in English about her related movie "L'homme qui voulait > classer le monde", shown at the 2002 conference on "The History and > Heritage of Scientific and Technological Information Systems" : > > http://www.chemheritage.org/events/asist2002/33-levie-film.pdf > > Regards, > > Antoine -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.4/448 - Release Date: 14/09/2006 From quentinburrell at MANX.NET Sat Sep 16 17:32:05 2006 From: quentinburrell at MANX.NET (Quentin L. Burrell) Date: Sat, 16 Sep 2006 22:32:05 +0100 Subject: Fw: Journal's turnaround time for first decision Message-ID: A colleague has requested that I post this to the list. If anyone has useful input, please send it direct to Pelle. Thanks for any help. Quentin ****************************** Dr Quentin L Burrell Isle of Man International Business School The Nunnery Old Castletown Road Douglas Isle of Man IM2 1QB via United Kingdom ----- Original Message ----- From: "P?r Sj?lander" To: Sent: Saturday, September 16, 2006 1:23 PM Subject: Journal's turnaround time for first decision >I am looking for a list over the average (or median) turnaround time to > a first decision for different journals. This is relevant since most of > us do not want to wait for 2 years before a first decision about our > submitted paper is taken. Is there such a list available? > > I am mainly interested in statistics, economics, and econometrics > journals. > > (Rejection frequency, number of issues per year is also interesting... > but this can usually be found on the webpage of every journal... a list > of all these statistics would be handy to have when we choose which > journal to submit our papers to). > > Regards, > Pelle From notsjb at LSU.EDU Sun Sep 17 12:04:24 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Sun, 17 Sep 2006 11:04:24 -0500 Subject: Fw: Journal's turnaround time for first decision Message-ID: The only thing I can think of is that he should take a random sample of issues of journals he is interested in publishing in. There is usually in the articles some sort of information on when paper was submitted and when accepted. There is is also the issue date. He should be able to figure out from this where best to publish according to his standards. I am afraid, though, that there may be some sort of inverse relationship between the quality of the journals and the rapidity of publication. Quality journals have many more submissions and tend to have greater backlogs of articles to publish than journals of lesser repute. JASIST is an example of this. SB "Quentin L. Burrell" @listserv.utk.edu> on 09/16/2006 04:32:05 PM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at listserv.utk.edu cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: [SIGMETRICS] Fw: Journal's turnaround time for first decision A colleague has requested that I post this to the list. If anyone has useful input, please send it direct to Pelle. Thanks for any help. Quentin ****************************** Dr Quentin L Burrell Isle of Man International Business School The Nunnery Old Castletown Road Douglas Isle of Man IM2 1QB via United Kingdom ----- Original Message ----- From: "P?r Sj?lander" To: Sent: Saturday, September 16, 2006 1:23 PM Subject: Journal's turnaround time for first decision >I am looking for a list over the average (or median) turnaround time to > a first decision for different journals. This is relevant sinae most of > us do not want to wait for 2 years before a first decision about our > submitted paper is taken. Is there such a list available? > > I am mainly interested in statistics, economics, and econometrics > journals. > > (Rejection frequency, number of issues per year is also interesting... > but this can usually be found on the webpage of every journal... a list > of all these statistics would be handy to have when we choose which > journal to submit our papers to). > > Regards, > Pelle From paul.wouters at VKS.KNAW.NL Mon Sep 18 06:32:12 2006 From: paul.wouters at VKS.KNAW.NL (Paul Wouters) Date: Mon, 18 Sep 2006 12:32:12 +0200 Subject: VKS LAUNCH In-Reply-To: Message-ID: Apologies for cross-postings Invitation Dear colleagues, The Virtual Knowledge Studio requests the pleasure of your company at the Launch of our research programme on October 11, 2006 in the Trippenhuis, Royal Netherlands Academy of Arts and Sciences, Kloveniersburgwal 29, 1011 JV Amsterdam. The Launch will be the occasion to take stock of the state of the art with respect to e-research in the humanities and social sciences and new research practices. You will be able to meet with scholars from a variety of disciplines, as well as specialists from the creative industry. You will have the occasion to get acquainted with their work in a series of special events in the Research Gallery which we will host for the first time during this launch. The programme: 13.00 Arrival and coffee 13.30 - 13.40 Welcome by artist and television host Bart Peeters 13.40 - 14.00 Introduction of the Studio by dr. Paul Wouters 14.00- 15.00 Panel discussion hosted by Bart Peeters, with prof. dr. Frits van Oostrom (president of the Royal Netherlands Academy of Arts and Sciences), prof. dr. Eep Talstra (chair of the Scientific Advisory Board of the VKS), prof. dr. Nina Wakeford (INCITE, University of Surrey, UK), Marie-Jos? Klaver (NRC Handelsblad), and prof. dr. Seamus Ross (HATII, University of Glasgow, UK) 15.15 - 16. 30 Research Gallery: * Web, Digital Media and Virtual Ethnography * Institutions, Governance and Culture * Web Archiving, Communities and Collaboration * Simulation, Annotation and Metaphors * Arts, Languages, and Sound 16. 30 Opening of the VKS by prof. dr. Frits van Oostrom 17.00- 18.30 The Benjamin Herman Trio and dr. Henkjan Honing (Music Cognition Group, University of Amsterdam) 18.30 Reception We hope that you will be able to join us in this event. My colleagues and I are looking forward to meeting you at the Launch, Yours sincerely, Paul Wouters Programme Leader VKS RSVP: Could you please confirm your participation by email to jeannette.haagsma at vks.knaw.nl and indicate in which gallery event you are especially interested? -- Paul Wouters Virtual Knowledge Studio for the Humanities and Social Sciences Royal Netherlands Academy of Arts and Sciences Cruquiusweg 31 1019 AT Amsterdam, NL T: +3120 850270 F: +3120 8500271 www.virtualknowledgestudio.nl -------------- next part -------------- An HTML attachment was scrubbed... URL: From Christina.Pikas at JHUAPL.EDU Mon Sep 18 09:27:34 2006 From: Christina.Pikas at JHUAPL.EDU (Pikas, Christina K.) Date: Mon, 18 Sep 2006 09:27:34 -0400 Subject: Fw: Journal's turnaround time for first decision Message-ID: I don't know that field, but I would assume there are "letters" type journals -- or sections of journals. These should be publishing much more quickly and will skew the results. The example I've given in the past is in the JCR, Optics for 2005 you have (sorted by impact factor) 1 ADV ATOM MOL OPT PHY 0065-2199 813 5.833 2.400 5 7.7 2 OPT EXPRESS 1094-4087 7564 3.764 0.604 1231 2.2 3 OPT LETT 0146-9592 25963 3.599 0.621 985 6.3 4 J BIOMED OPT 1083-3668 2266 3.557 0.274 226 3.6 5 PHYS REV A 1050-2947 61693 2.997 0.696 2039 8.3 So #2 and #3 are letters and #1 is a monographic series. IOW, I agree that it's probably a good idea to sample the same types of journals, then for each platform work out a way to extract the submitted and published dates (and if there's an "available online" date separate from published date as for JASIST). Of course, as our contributors here would note, there are e-print servers that could fill the gap while the article is in press. In economics, the author should be certain that his institutional repository is part of RePEc for greatest visibility. Christina K. Pikas, MLS R.E. Gibson Library & Information Center The Johns Hopkins University Applied Physics Laboratory Voice 240.228.4812 (Washington), 443.778.4812 (Baltimore) Fax 443.778.5353 -----Original Message----- From: ASIS&T Special Interest Group on Metrics [mailto:SIGMETRICS at listserv.utk.edu] On Behalf Of Stephen J Bensman Sent: Sunday, September 17, 2006 12:04 PM To: SIGMETRICS at listserv.utk.edu Subject: Re: [SIGMETRICS] Fw: Journal's turnaround time for first decision The only thing I can think of is that he should take a random sample of issues of journals he is interested in publishing in. There is usually in the articles some sort of information on when paper was submitted and when accepted. There is is also the issue date. He should be able to figure out from this where best to publish according to his standards. I am afraid, though, that there may be some sort of inverse relationship between the quality of the journals and the rapidity of publication. Quality journals have many more submissions and tend to have greater backlogs of articles to publish than journals of lesser repute. JASIST is an example of this. SB "Quentin L. Burrell" @listserv.utk.edu> on 09/16/2006 04:32:05 PM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at listserv.utk.edu cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: [SIGMETRICS] Fw: Journal's turnaround time for first decision A colleague has requested that I post this to the list. If anyone has useful input, please send it direct to Pelle. Thanks for any help. Quentin ****************************** Dr Quentin L Burrell Isle of Man International Business School The Nunnery Old Castletown Road Douglas Isle of Man IM2 1QB via United Kingdom ----- Original Message ----- From: "P?r Sj?lander" To: Sent: Saturday, September 16, 2006 1:23 PM Subject: Journal's turnaround time for first decision >I am looking for a list over the average (or median) turnaround time to >a first decision for different journals. This is relevant sinae most of >us do not want to wait for 2 years before a first decision about our >submitted paper is taken. Is there such a list available? > > I am mainly interested in statistics, economics, and econometrics > journals. > > (Rejection frequency, number of issues per year is also interesting... > but this can usually be found on the webpage of every journal... a > list of all these statistics would be handy to have when we choose > which journal to submit our papers to). > > Regards, > Pelle From eugene.garfield at THOMSON.COM Mon Sep 18 10:57:55 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Mon, 18 Sep 2006 10:57:55 -0400 Subject: FW: Research on Peer Review Message-ID: -----Original Message----- From: Finn Hansson [mailto:fh.lpf at cbs.dk] Sent: Monday, September 18, 2006 8:03 AM Subject: SV: Research on Peer Review Finn Hansson, Ass. professor, Copenhagen Business School _____ Fra: Lutz Bornmann [mailto:bornmann at gess.ethz.ch] Sendt: 18. september 2006 13:48 Emne: Research on Peer Review Dear colleagues Please find attached a recently published paper (pdf-file) that might be of interest to you. The paper contains new results of our research on peer review: We investigated committee peer review for awarding long-term fellowships to post-doctoral researchers as practiced by the Boehringer Ingelheim Fonds (B.I.F.) ? a foundation for the promotion of basic research in biomedicine. Assessing the validity of selection decisions requires a generally accepted criterion for research impact. A widely used approach is to use citation counts as a proxy for the impact of scientific research. Therefore, a citation analysis for articles published previous to the applicants? approval or rejection for a B.I.F. fellowship was conducted. Based on our model estimation (negative binomial regression model), journal articles that had been published by applicants approved for a fellowship award (n = 64) prior to applying for the B.I.F. fellowship award can be expected to have 37% (straight counts of citations) and 49% (complete counts of citations) more citations than articles that had been published by rejected applicants (n = 333). Furthermore, comparison with international scientific reference values revealed (a) that articles published by successful and non-successful applicants are cited considerably more often than the ?average? publication and (b) that excellent research performance can be expected more of successful than non-successful applicants. The findings confirm that the foundation is not only achieving its goal of selecting the best junior scientists for fellowship awards, but also successfully attracting highly talented young scientists to apply for B.I.F. fellowships. We would greatly appreciate receiving your comments on our paper. Sincerely yours Lutz Bornmann PS: If you are active in peer review research, we would be deeply grateful, if you would send us reprints of related papers of your new research on peer review. ----------------------------------------------------------------------------- Dr. Lutz Bornmann Download of publications: HYPERLINK "http://www.lutz-bornmann.de/Publications.htm" \nwww.lutz-bornmann.de/Publications.htm ETH Zurich, D-GESS Professorship for Social Psychology and Research on Higher Education Zaehringerstr. 24 / ZAE CH-8092 Zurich Phone: 0041 44 632 48 25 Fax: 0041 44 632 12 83 HYPERLINK "https://mail.student.ethz.ch/exchweb/bin/redir.asp?URL=http://www.psh.ethz.ch" \nwww.psh.ethz.ch HYPERLINK "mailto:bornmann at gess.ethz.ch" \nbornmann at gess.ethz.ch -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.4/449 - Release Date: 9/15/2006 -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.4/449 - Release Date: 9/15/2006 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Hansson evaluation learning.pdf Type: application/pdf Size: 161254 bytes Desc: Hansson evaluation learning.pdf URL: From eugene.garfield at THOMSON.COM Mon Sep 18 10:58:42 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Mon, 18 Sep 2006 10:58:42 -0400 Subject: FW: Research on Peer Review Message-ID: Dear colleagues Please find attached a recently published paper (pdf-file) that might be of interest to you. The paper contains new results of our research on peer review: We investigated committee peer review for awarding long-term fellowships to post-doctoral researchers as practiced by the Boehringer Ingelheim Fonds (B.I.F.) ? a foundation for the promotion of basic research in biomedicine. Assessing the validity of selection decisions requires a generally accepted criterion for research impact. A widely used approach is to use citation counts as a proxy for the impact of scientific research. Therefore, a citation analysis for articles published previous to the applicants? approval or rejection for a B.I.F. fellowship was conducted. Based on our model estimation (negative binomial regression model), journal articles that had been published by applicants approved for a fellowship award (n = 64) prior to applying for the B.I.F. fellowship award can be expected to have 37% (straight counts of citations) and 49% (complete counts of citations) more citations than articles that had been published by rejected applicants (n = 333). Furthermore, comparison with international scientific reference values revealed (a) that articles published by successful and non-successful applicants are cited considerably more often than the ?average? publication and (b) that excellent research performance can be expected more of successful than non-successful applicants. The findings confirm that the foundation is not only achieving its goal of selecting the best junior scientists for fellowship awards, but also successfully attracting highly talented young scientists to apply for B.I.F. fellowships. We would greatly appreciate receiving your comments on our paper. Sincerely yours Lutz Bornmann PS: If you are active in peer review research, we would be deeply grateful, if you would send us reprints of related papers of your new research on peer review. ----------------------------------------------------------------------------- Dr. Lutz Bornmann Download of publications: HYPERLINK "http://www.lutz-bornmann.de/Publications.htm" \nwww.lutz-bornmann.de/Publications.htm ETH Zurich, D-GESS Professorship for Social Psychology and Research on Higher Education Zaehringerstr. 24 / ZAE CH-8092 Zurich Phone: 0041 44 632 48 25 Fax: 0041 44 632 12 83 HYPERLINK "https://mail.student.ethz.ch/exchweb/bin/redir.asp?URL=http://www.psh.ethz.ch" \nwww.psh.ethz.ch HYPERLINK "mailto:bornmann at gess.ethz.ch" \nbornmann at gess.ethz.ch -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.4/449 - Release Date: 9/15/2006 -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.12.4/449 - Release Date: 9/15/2006 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Hansson evaluation learning.pdf Type: application/pdf Size: 161254 bytes Desc: Hansson evaluation learning.pdf URL: From harnad at ECS.SOTON.AC.UK Mon Sep 18 12:16:40 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Mon, 18 Sep 2006 17:16:40 +0100 Subject: Future UK RAEs to be Metrics-Based (fwd) Message-ID: ---------- Forwarded message ---------- Date: Mon, 18 Sep 2006 17:14:21 +0100 (BST) From: Stevan Harnad To: AmSci Forum Subject: Re: Future UK RAEs to be Metrics-Based Charles Oppenheim has authorised me to post this on his behalf: "Research I have done indicates that the same correlations between RAE scores and citation counts already noted in the sciences and social sciences apply just as strongly (sometimes more strongly) in the humanities! But you are right, Richard, that metrics are PERCEIVED to be inappropriate for the humanities and a lot of educating is needed on this topic." Professor Charles Oppenheim Head Department of Information Science Loughborough University From harnad at ECS.SOTON.AC.UK Mon Sep 18 12:07:14 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Mon, 18 Sep 2006 17:07:14 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: On Mon, 18 Sep 2006, Richard Poynder wrote: > >CO: > >However, thank heavens for (4) because, as you rightly point out, a > >metrics based system will sweep all this nonsense aside. > > Does not a metrics based system pose some problems for the > humanities? I haven't followed the discussion closely, but I get the > feeling that this is primarily intended as a fix for the sciences > isn't it? How is it envisaged working in the humanities? I hope Charles Oppenheim will respond directly about the studies he has done on metrics in the RAE humanities disciplines. Meanwhile: (1) Wherever anyone has checked the correlation between journal citation counts and RAE outcome, the correlation has always been significant and sizeable, hence predictive. Humanities disciplines have not been exceptions. (2) In book-based fields, what has likewise not been looked at is supplementing the journal-article citation metric with a book-citation metric. (That's still metrics!) (3) And then there are all the other candidate metrics, most still untested: downloads, co-citations, hubs/authorities, recursive CiteRank, download/citation growth parameters (latency, slope, peak, longevity), semantic metrics, etc. An a-priori declaration, free of any supporting evidence -- by any discipline today -- that its work is an exception, not assessable by metrics, makes about as much sense as an a-priori declaration, without any supporting evidence, that a discipline's work is not assessable by any form of comparative performance evaluation at all. (Who's to say whether subjective evaluations have any validity either?) Stevan Harnad From dgoodman at PRINCETON.EDU Mon Sep 18 17:05:37 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Mon, 18 Sep 2006 17:05:37 -0400 Subject: Future UK RAEs to be Metrics-Based (fwd) In-Reply-To: Message-ID: Charles, this would seem to imply that research in the humanities is now mainly dependent upon journal articles, as it is in the sciences. -- and if its from the RAE, this might be applicable only in the UK? (Might there be more diffeence between countries in the structure of humanities research than there is in the sciences.) According to what I've seen in Chronicle of Higher Education and elsewhere, in US research universities the typical publication requirement for tenure is two monograps, and only a few such universities have been experimenting with accepting journal articles as a partial substitute.) David Goodman dgoodman at princeton.edu ----- Original Message ----- From: Stevan Harnad Date: Monday, September 18, 2006 1:03 pm Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based (fwd) To: SIGMETRICS at LISTSERV.UTK.EDU > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > ---------- Forwarded message ---------- > Date: Mon, 18 Sep 2006 17:14:21 +0100 (BST) > From: Stevan Harnad > To: AmSci Forum > Subject: Re: Future UK RAEs to be Metrics-Based > > Charles Oppenheim has authorised me to post this on his behalf: > > "Research I have done indicates that the same correlations between > RAE scores and citation counts already noted in the sciences and > social sciences apply just as strongly (sometimes more strongly) > in the humanities! But you are right, Richard, that metrics are > PERCEIVED to be inappropriate for the humanities and a lot of > educating is needed on this topic." > > Professor Charles Oppenheim > Head > Department of Information Science > Loughborough University > From harnad at ECS.SOTON.AC.UK Mon Sep 18 19:05:31 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 00:05:31 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <20060918203748.xkicyfnp9ckgogws@www.staffmail.ed.ac.uk> Message-ID: On Mon, 18 Sep 2006 l.hurtado at ED.AC.UK wrote: > Well, I'm all for empirically-based views in these matters. So, if > Oppenheim or others have actually soundly based studies showing what > Stevan and Oppenheim claim, then that's to be noted. I'll have to see > the stuff when it's published. In the meanwhile, a couple of further > questions: Many studies are already published. In fact many are cited in: Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated online RAE CVs Linked to University Eprint Archives. Ariadne 35. http://eprints.ecs.soton.ac.uk/7725/ > --Pardon me for being out of touch, perhaps, but more precisely what is > being measured? What does journal "citation counts" refer to? The total number of citations to the articles by submitted authors (and not just those for their 4 submitted articles!) > Citation of journal articles? Or citation of various things in journal > articles (and why privilege this medium?)? Or . . . what? Citation of the articles, but that usually means citing things in the articles! Journal articles are privileged in many disciplines because they are the main means of reporting research. In book-based disciplines the balance is otherwise, but the interesting thing is that even in book-based disciplines there is a journal article citation correlation with the RAE rankings. One would expect it to be somewhat weaker than in article-based disciplines, but more data are needed to be exact about this. > --What does "correlation" between RAE results and "citation counts" > actually comprise? The RAE ranks the departments of the c. 73 UK research universities, with ranks from 1 to 5*. Correlation is the measure of the degree to which values on one variable co-vary with, hence predict, values on another variable (e.g. height is correlated with weight, the higher on one, the higher on the other, and vice versa). When two variables are correlated, you can predict one from the other. How accurately you can predict is reflected by the square of the correlation coefficient: If there is a correlation of 0.8, then the predictivity (the percentage of the variation in one of the variables that you can already predict from the other) is 64%. For a correlation of 0.9 it's 81% etc. Well, as you will see in the reference list of the above-cited article, Smith & Eysenck found a correlation of about 0.9 between the RAE ranks and the total citation counts for the submitted researchers in Psychology. Looking at Charles Oppenheim's studies, you will see that the correlations varied from about 0.6 to 0.9, depending on field and year, which is all quite high, but *especially* give that the RAE does not count citations! The correlation is even higher with another metric, in science and biology: prior research funding. There it can be as high as 0.99, but that is not so good, because (1) prior funding *is* directly seen and counted by the panel, so that high correlation could be an effect of direct influence. Worse, using prior funding as a criterion generates a Matthew Effect, with the already-highly-funded getting richer and richer, and the less-funded getting poorer and poorer. That is why a multiple regression equation is best, with many predictor metrics, each one weighted according to the desiderata and particulars of each discipline, and validated against further criteria, to make sure they are measuring what we want to measure. There will be many candidate metrics in the OA era. > Let me lay out further reasons for some skepticism. In my own field > (biblical studies/theology), I'd say most senior-level scholars > actually publish very infrequently in refereed journals. We do perhaps > more in earlier years, but as we get to senior levels we tend (a) to > get requests for papers for multi-author volumes, and (b) we devote > ourselves to projects that best issue in book-length publications. That happens in other fields too, and as metric equations are calibrated and optimised, factors like seniority will enter into the weightings too. (Book chapter citations can and will of course be cited too -- and are, to a limited degree, already being counted by ISI and others, because journal articles cite books and book chapters too, and those citations are caught by ISI.) > So if my own productivity and impact were assessed by how many journal > articles I've published in the last five years, I'd look poor (even > though . . . well, let's say that I rather suspect that wouldn't be the > way I'm perceived by peers in the field). The RAE ranks departments via individuals, and a department needs a blend of junior and senior people, with their different style of publication. And remember that RAE is comparing like with like. So you might be interested in checking how your own journal article and book chapter citation counts compare with those of your peers (or juniors). You might be (pleasantly) surprised! And of course in the (soon-to-hand) OA era, other metrics will be available too, such as download counts ("hits"), which happen much earlier, yet are correlated with later citations -- and are of course maximized by self-archiving your papers in your institutional IR to make them OA. Odd new metrics will also include endogamy/exogamy scores (their preferred polarity depending on field!), depending on the degree of self-citing, co-author citing, co-citation circle citing, within/outside specialty citing, intra/interdisciplinary citing, both for the citing article/author and the cited article/author. Then there's text-proximity scores (of which an extreme would be plagiarism), latency/longevity metrics, co-citation to/from, CiteRank (where the weight of each citation is recursively ranked, google style, by the degree of citedness of the citer), etc. etc. > Or is the metric to comprise how many times I'm *cited* in journals? It's how many times you're cited, which means how many times your articles are cited -- in journals, but in principle also in book chapters, conferences and books. And whether what is *being* cited is articles, chapters or books. > If so, is there some proven correlation between a scholar's impact or > significance of publications in the field and how many times he happens > to be cited in this one genre of publication? I'm just a bit > suspicious of the assumptions, which I still suspect are drawn (all > quite innocently, but naively) from disciplines in which journal > publication is much more the main and significant venue for scholarly > publication. I don't know of systematic genre comparisons (journals vs book chapters, even empirical vs theoretical journals, reviews, etc.) but they no doubt exist. I will branch this to the sigmetrics list where the experts are! I am just an amateur... > And, as we all know, "empirical" studies depend entirely on the > assumptions that lie at their base. So their value is heavily framed > by the validity and adequacy of the governing assumptions. No > accusations, just concerns. Interpretations may be influenced by assumptions, but the empirical fact that atmospheric pressure predicts RAE ranking would be an empirical datum (and, if it predicted it with a correlation of, say, 0.9) that would be a reason for scrapping RAE panels for barometers theory-independently.... Stevan Harnad > Quoting Stevan Harnad : > > > On Mon, 18 Sep 2006, Larry Hurtado wrote: > > > >> Stevan and I have exchanged views on the *feasibility* of a metrics > >> approach to assessing research strength in the Humanities, and he's > >> impressed me that something such *might well* be feasible *when/if* > >> certain as-yet untested and undeveloped things fall into place. I note, > >> e.g., in Stevan's addendum to Oppenheim's comment that a way of handling > >> book-based disciplines "has not yet been looked at", and that a number > >> of other matters are as yet "untested". > > > > Larry is quite right that the (rather obvious and straightforward) > > procedure of self-archiving books' metadata and cited references in > > order to derive a comprehensive book-citation index (which would > > of course include journal articles citing books, books citing books, > > and books citing journal articles) had not yet been implemented or > > tested. > > > > However, the way to go about it is quite clear, and awaits only OA > > self-archiving mandates (to which a mandate to self-archive one's book > > metadata and reference list should be added as a matter of course). > > > > But please recall that I am an evangelist for OA self-archiving, because > > I *know* it can be done, that it works, and that it confers substantial > > benefits in terms of research access, usage and impact. > > > > Insofar as metrics are concerned, I am not an evangelist, but merely an > > enthusiast: The evidence is there, almost as clearly as it is with the > > OA impact-advantage, that citation counts are strongly correlated with > > RAE rankings in every discipline so far tested. Larry seems to pass over > > evidence in his remark about the as yet incomplete book citation data > > (ISI has some, but they are only partial). But what does he have to say > > about the correlation between RAE rankings and *journal article citation > > counts* in the humanities (i.e., in the "book-based" disciplines)? > > Charles will, for example, soon be reporting strong correlations in > > Music. Even without having to wait for a book-impact index, it seems > > clear that there are as yet no reported empirical exceptions to the > > correlation between journal article citation metrics and RAE outcomes. > > > > (I hope Charles will reply directly, posting some references to his and > > others' studies.) > > > >> This being the case, it is certainly not so a priori to say that a > >> metrics approach is not now really feasible for some disciplines. > > > > Nothing a priori about it: A posteriori, every discipline so far tested > > has shown positive correlations between its journal citation counts and its > > RAE rankings, including several Humanities disciplines. > > > > The advantage of having one last profligate panel-based RAE in parallel > > with the metric one in 2008 is that not a stone will be left unturned. > > If there prove to be any disciplines having small or non-existent > > correlations with metrics, they can and should be evaluated otherwise. > > But let us not assume, a priori, that there will be any such > > disciplines. > > > >> I emphasize that my point is not a philosophical one, but strictly > >> whether as yet a worked out scheme for handling all Humanities > >> disciplines rightly is in place, or capable of being mounted without > >> some significant further developments, or even thought out adequately. > > > > It depends entirely on the size of the metric correlations with the > > present RAE rankings. Some disciplines may need some supplementary forms > > of (non-metric) evaluation if their correlations are too weak. That is an > > empirical question. Meanwhile, the metrics will also be growing in power > > and diversity. > > > >> That's not an antagonistic question, simply someone asking for the > >> basis for the evangelistic stance of Stevan and some others. > > > > I evangelize for OA self-archiving of research and merely advocate > > further development, testing and use of metrics in research performance > > assessment, in all disciplines, until/unless evidence appears that there > > are exceptions. So far, the objections I know of are all only in the > > form of a priori preconceptions and habits, not objective data. > > > > Stevan Harnad > > > >> > Charles Oppenheim has authorised me to post this on his behalf: > >> > > >> > "Research I have done indicates that the same correlations between > >> > RAE scores and citation counts already noted in the sciences and > >> > social sciences apply just as strongly (sometimes more strongly) > >> > in the humanities! But you are right, Richard, that metrics are > >> > PERCEIVED to be inappropriate for the humanities and a lot of > >> > educating is needed on this topic." > > > > > > L. W. Hurtado, Professor of New Testament Language, Literature & Theology > Director of Postgraduate Studies > School of Divinity, New College > University of Edinburgh > Mound Place > Edinburgh, UK. EH1 2LX > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > From harnad at ECS.SOTON.AC.UK Mon Sep 18 20:07:56 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 01:07:56 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <1158620418.450f25025a727@staff-webmail.lboro.ac.uk> Message-ID: Fytton Rowland is absolutely right. It would be *extremely* bad and extremely foolish if the current RAE-reform juggernaut continued to veer mindlessly toward prior-funding as the mother-of-all-metrics. I hope clearer heads will prevail, and that the RAE will use its wise decision to scrap panel review in favour of metrics in order to develop a rich set of candidate metrics, adapted to each discipline, rather than to abandon measurement and prediction altogether for and opt instead for self-fulfilling prophecy and the Matthew Effect that would result from relying solely or mainly on the prior-funding metric. Stevan Harnad On Tue, 19 Sep 2006, Fytton Rowland wrote: > All the exchanges in this thread over the last day or so seem to be assuming > that the 'metrics' to be used in future RAEs in the UK will measure > citations or some other similar measure of use of published papers. If > that were so, I would not be too unhappy - Charles Oppenheim's earlier > research has shown that such measures correlate quite well with past RAE > results achieved by much more labour-intnsive and expensive methods. > > But if we look at the actual proposals from the funding councils that are > currently out for consultation, we find that their idea of 'metrics' is > something quite different. Essentially their proposals give primacy to a > measure of money earned from outside research grants. This is bad in two > important ways. First, it measures resaerch inputs not outputs, and > potentially rewards inefficiency - surely those who achieve good research > results on a shoestring should be admired, not penalised? Second, it > removes diversity of research funding - essentially, the only way to build > a research-active departmenbt is to get money from research councils, etc. > The HEFCE money would simply apply a multiplier to that from research > funders. The current RAE system, or a hypothetical one based on citation, > offers an alternative route ro research success for those who do good work > without outside funding; if their outputs (published papers and books) are > adjudged good and influential, they receive QR funding independently of any > outside research grants they may have. The proposed money-based metrics > offer no alternative means of support, but simply apply 'to him that hath, > it shall be given'. > > So, for me, metrics in principle are not necessarily bad but the actual > metrics being proposed are very bad indeed. > > Fytton Rowland, Department of Information Science, Loughborough University, > UK. > From harnad at ECS.SOTON.AC.UK Mon Sep 18 20:55:36 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 01:55:36 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: David Goodman seems to have misunderstood the RAE, the metric RAE, and my point about the metric RAE: (1) The RAE assesses peer-reviewed, published journal articles. (2) For years now UK authors and their institutions have been wasting their time re-submitting -- and the RAE panels have been wasting their time re-reviewing -- *already peer-reviewed, published articles*. (3) They have been re-reviewing them in order to generate the RAE competitive rankings, in proportion to which the authors' departments get top-sliced funding and prestige. (4) The rankings turn out to be strongly correlated with citation counts (even though citations are not actually counted by the panels) in all fields tested so far. (5) Hence most of the time and effort of re-submission and re-review by the panels has been a complete waste, since much the same rankings could have been generated by just counting citations (and other metrics). (6) Hence it has at last been wisely decided to abandon the re-review and just use metrics. (7) In its one last incarnation in 2008, there will be a parallel exercise: both the panel re-review and the metrics. (8) The panels never did, and never could, perform peer review rigorously for each paper submitted, because there was neither the time nor the expertise: and besides, the papers had already been peer-reviewed by the journals in which they were published. (9) Hence it was always absurd to insist that the submissions should be the page images of the journal: Most only got skimmed anyway, and the journal's quality level most likely exerts more influence on the outcome than the peer expertise of the panel members to which each paper happened to be assigned: this too was probably reflected in the correlations with citations. David Goodman seems to think that the RAE panel review was actually performing the primary peer review -- refereeing for originality, primacy correctness, etc. But that was already done, once (and once is enough) by the journal's peer reviewers. That's not what the RAE panelists were doing. (It's actually not clear *what* the RAE panelists were doing: probably just assigning an RAE rank to each paper, based on one does not quite know what.) On Mon, 18 Sep 2006, David Goodman wrote: > > SH: > > (b) So what if authors correct or improve their publications after > > publication? That's *good*, not bad! > > Stevan, you have previously argued consistently on this list and > elsewhere that the formally published version is the version of record, > not a version--improved or otherwise, that the author posted, previously > or subsequently. Yes. But what is your point? We are not talking here about the RAE (or the author's IR) being a Deposit Library, archiving the version of record. We are talking about a national research assessment exercise in which a system that wasted everyone's time and money needlessly re-doing peer-review (inexpertly) is to be replaced by metrics. There was never any earthly reason why the RAE should have insisted on seeing the publisher's PDF for its panel review -- and even less so now, when the panel review is at long last fading out. The author's postprint is and always was sufficient. > If an author publishes something, and another person does the work > better or differently, surely the original author cannot update his work > after the fact and pretend he got it right the first time. He can, of > course, publish an additional paper based on the corrected understanding. Priority is examined and assigned by the original journal's peer-reviewers, not by post-hoc RAE review panels assessing research performance based on already published articles. It is not clear what incentive there would be in fraudulently claiming priority by doctoring a published article, but the risk -- for both the author and his institutions -- of having the fraud discovered and made public surely outweighs whatever imaginary incentive it might have had to do it in the first place. Far more likely in an author's postprint is a lapse in incorporating all the recommendations of the referees -- a lapse that puts the author at some slight risk of looking a bit worse in a spot-check by the panelists. Correcting an error that slipped into the published version, on the other hand, gives the author a slight (and legitimate) chance of looking a bit better. All trivial either way, since the panel review is being phased out for metrics and was never rigorous enough to detect much of anything in the first place. > If there is to be an RAE, should it not assess what the author > published? Views like yours will give well-justified concern to > administrators that the OA archives can not be trusted. I very much > regret you have thought this way, let alone publicised it. It brings > disrepute on us all. David, I think you're getting a bit carried away... The author's peer-reviewed final draft is fine for the RAE, and it's also fine for an OA supplement in the author's IR, for users who cannot afford to access the publisher's version of record. It is not a rival publishing system, nor does it provide a rival version of record. > > Charles Oppenheim: > > "Research I have done indicates that the same correlations between > > RAE scores and citation counts already noted in the sciences and > > social sciences apply just as strongly (sometimes more strongly) > > in the humanities! But you are right, Richard, that metrics are > > PERCEIVED to be inappropriate for the humanities and a lot of > > educating is needed on this topic." > > this would seem to imply that research in the humanities is now mainly > dependent upon journal articles, as it is in the sciences. No, it just implies that the RAE outcomes are correlated with article citation counts, even in the humanities. > -- and if its from the RAE, this might be applicable only in the UK? Since only the UK does the RAE (Australia will soon have one too), and since the correlation is between citations and RAE ranks, it clearly applies only to the UK. (But one would expect similar effects for any similar assessment exercise.) > (Might there be more difference between countries in the structure of > humanities research than there is in the sciences.) There might; or there might not. This is (yet another) empirical question that cannot be answered from the speculative armchair. > According to what I've seen in Chronicle > of Higher Education and elsewhere, in US research universities the > typical publication requirement for tenure is two monographs, and only > a few such universities have been experimenting with accepting journal > articles as a partial substitute.) And what is your point here? The RAE is not evaluating individuals for tenure, it is evaluating research performance of departments, for top-sliced research funding. Stevan Harnad From willieezi at YAHOO.COM Tue Sep 19 08:52:51 2006 From: willieezi at YAHOO.COM (Williams Nwagwu) Date: Tue, 19 Sep 2006 05:52:51 -0700 Subject: RESEARCH ON AFRICA Message-ID: Does anyone perchance have answers to these questions: i. statistucs of serials in africa ii.proportion of mainstream journals that come from africa iii. proportion of mainstream journals that actually circulate in africa this will help me complete a paper Dr Willie __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From harnad at ECS.SOTON.AC.UK Tue Sep 19 09:07:32 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 14:07:32 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <20060919095610.6xi6gpzg2sgko0kg@www.staffmail.ed.ac.uk> Message-ID: On Tue, 19 Sep 2006 l.hurtado at ED.AC.UK wrote: > --Humanities scholarly publishing is more diverse in venue/genre than > in some other fields. Indeed, journals are not particularly regarded > as quite so central, but only one among several respected and > frequented genres, which include multi-author books, and (perhaps > particularly) monographs. Citation counts can in principle -- and up to a point already do -- count citations to all these genres: (1) citations *from* articles *to* preprints, articles, chapters, and books (already being partially indexed, e.g., by ISI) (2) citations *from* preprints, articles, chapters, books *to* preprints, articles, chapters, and books (indexable in principle, already partly indexed by citebase, citeseer, google scholar and scopus, and will flourish dramatically once Open Access prevails) Hence whatever statistically significant RAE/citation correlations and effect sizes Charles Oppenheim manages to find *despite* the weak and partial citation coverage to date is actually evidence of the robustness of the RAE/citation correlation in the fields that are less article-based. > QUESTION: Are the studies that supposedly show such meaningful > correlations actually drawing upon the full spread of publication > genres appropriate to the fields in view? Not yet, and that's the point: Charles's findings are all the more remarkable for being so robust, despite the weak signal! > (I'd be surprised but > delighted were the answer yes, because I'm not aware of any mechanism > in place, such as ISI in journal monitoring, for surveying and counting > in such a vast body of material. The point is that citation coverage right now is most definitely incomplete and insufficient. But that can (and will) only improve (especially under pressure from the RAE, and OA!). Meanwhile, though, the successful demonstration of strong correlations even based on the partial coverage is very promising evidence. > I'm not pushing at all for the labour-intensive RAE of the past. Bravo. That means 80% agreement already! > Indeed, if the question is not how do individual scholars stack up in > comparison to others in their field (which the RAE actually wasn't > designed to determine), but instead how can we identify depts into > which a disproportionate amount of govt funding should be pumped, then > I think in almost any field a group of informed scholars could readily > determine the top 5-10 places within 30 minutes, and with time left > over for coffee. For all the UK departments, and in fair proportion to their merits and needs? Or just for a familiar few? And would several such informed-scholar circles agree on their rankings (with one another, or with the current RAE rankings)? > I'm just asking for more transparency and evidence behind the > enthusiasm for replacing RAE with "metrics". A group of informed scholars over coffee does not strike me as the height of transparency and evidence... However, in validating the new weighted metric equation, and adjusting it for the needs of each discipline, one of the criteria against which it will be validated is of course informed peer judgments: The metric equation should not be at odds with informed peer judgment (nor should there be marked discrepancies among metrics themselves, at least among those we assume to be measuring the same sort of thing, such as downloads and citations). In general, with multiple regression equations (which, by the way, capture only linear effects, unless orthogonal polynomials guessing at nonlinear relations are used), one wants the measures that are meant to measure the same sort of thing to be correlated with one another, but one does not want the correlation to be *too* high, otherwise the measures are redundant: Optimally, they should be cross-checks on one another, but also each should be making its own unique contribution to the prediction, over and above corroborating the rest. And of course the weight of each should be adjustable in accordance with the specific profile of the discipline and its needs and values. For example, exogamy might be more of a virtue in some fields than others. Some fields may be more authority-based or co-citation authority-based than others. For some fields, steep early uptake may be predictive, for others, longevity. etc. This will all be brought into focus by the metric validation and calibration and customization phase that will have to precede the use of the scientometric equation for evaluation -- exactly as validation, standardization and the creation of norms and benchmarks must be done in biometrics and psychometrics before using the metrics for clinical or evaluative purposes. We are talking about a rich new OA world of online performance indicators and predictors sitting on top of an even richer primary database: the research itself. Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects, chapter 20. Chandos. http://eprints.ecs.soton.ac.uk/12453/ Stevan Harnad > Larry > > Quoting "C.Oppenheim" : > > > The correlation is between number of citations in total (and average number > > of citations per member of staff) received by a Department over the RAE > > period (1996-2001) and the RAE score received by the Department following > > expert peer review. Correlation analyses are done using Pearson or Spearman > > correlation coefficients. The fact that so few humanities scholars publish > > journal articles does not affect this result. > > > > A paper on the topic is in preparation at the moment. > > > > What intrigues me is why there is so much scepticism about the notion. RAE > > is done by peer review experts. Citations are also done by (presumably) > > experts who choose to cite a particular work. So one would expect a > > correlation between the two, wouldn't one? What it tells us is that high > > quality research leads to both high RAE scores AND high citation counts. > > > > I do these calculations (and I've covered many subject areas over the > > years, but not biblical studies - something for the future!) in a totally > > open-minded manner. If I get a non-significant or zero correlation in such > > a study in the future, I will faithfully report it. But so far, that hasn't > > happened. > > > > Charles > > > > Professor Charles Oppenheim > > Head > > Department of Information Science > > Loughborough University > > Loughborough > > Leics LE11 3TU > > > > Tel 01509-223065 > > Fax 01509-223053 > > e mail C.Oppenheim at lboro.ac.uk > > ----- Original Message ----- > > From: > > To: > > Sent: Monday, September 18, 2006 8:37 PM > > Subject: Re: Future UK RAEs to be Metrics-Based > > > > > School of Divinity, New College > University of Edinburgh > Mound Place > Edinburgh, UK. EH1 2LX > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > From notsjb at LSU.EDU Tue Sep 19 09:46:41 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Tue, 19 Sep 2006 08:46:41 -0500 Subject: Future UK RAEs to be Metrics-Based Message-ID: I should like to point out that quantitative bibliometric measures have usually not been found to be applicable in the humanities. First, despite plans to do so, ISI never developed a JCR for the A&HCI. Second, both the 1981 assessment of US research-doctorate programs by the American Council on Education, etc., and the 1993 assessment of these programs by National Research Council rejected using publication and citation counts for the humanities. The 1993 assessment substituted faculty awards for these measures. In general, the humanities do not conform closely to typical biliometric distributions, being more random and scattered. This has generally been found to be the case in library use studies. My own observation has been that, whereas variance in the sciences is due to accepted "paradigms," such variance as there is in the humanities is due to intellectual "fads." I am afraid that one is reduced in the humanities to either subjective evaluations or the acceptance of the subjective evaluations of others. There are thankfully some things in this world not reducible to quantitative laws. SB Stevan Harnad @LISTSERV.UTK.EDU> on 09/19/2006 08:07:32 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based On Tue, 19 Sep 2006 l.hurtado at ED.AC.UK wrote: > --Humanities scholarly publishing is more diverse in venue/genre than > in some other fields. Indeed, journals are not particularly regarded > as quite so central, but only one among several respected and > frequented genres, which include multi-author books, and (perhaps > particularly) monographs. Citation counts can in principle -- and up to a point already do -- count citations to all these genres: (1) citations *from* articles *to* preprints, articles, chapters, and books (already being partially indexed, e.g., by ISI) (2) citations *from* preprints, articles, chapters, books *to* preprints, articles, chapters, and books (indexable in principle, already partly indexed by citebase, citeseer, google scholar and scopus, and will flourish dramatically once Open Access prevails) Hence whatever statistically significant RAE/citation correlations and effect sizes Charles Oppenheim manages to find *despite* the weak and partial citation coverage to date is actually evidence of the robustness of the RAE/citation correlation in the fields that are less article-based. > QUESTION: Are the studies that supposedly show such meaningful > correlations actually drawing upon the full spread of publication > genres appropriate to the fields in view? Not yet, and that's the point: Charles's findings are all the more remarkable for being so robust, despite the weak signal! > (I'd be surprised but > delighted were the answer yes, because I'm not aware of any mechanism > in place, such as ISI in journal monitoring, for surveying and counting > in such a vast body of material. The point is that citation coverage right now is most definitely incomplete and insufficient. But that can (and will) only improve (especially under pressure from the RAE, and OA!). Meanwhile, though, the successful demonstration of strong correlations even based on the partial coverage is very promising evidence. > I'm not pushing at all for the labour-intensive RAE of the past. Bravo. That means 80% agreement already! > Indeed, if the question is not how do individual scholars stack up in > comparison to others in their field (which the RAE actually wasn't > designed to determine), but instead how can we identify depts into > which a disproportionate amount of govt funding should be pumped, then > I think in almost any field a group of informed scholars could readily > determine the top 5-10 places within 30 minutes, and with time left > over for coffee. For all the UK departments, and in fair proportion to their merits and needs? Or just for a familiar few? And would several such informed-scholar circles agree on their rankings (with one another, or with the current RAE rankings)? > I'm just asking for more transparency and evidence behind the > enthusiasm for replacing RAE with "metrics". A group of informed scholars over coffee does not strike me as the height of transparency and evidence... However, in validating the new weighted metric equation, and adjusting it for the needs of each discipline, one of the criteria against which it will be validated is of course informed peer judgments: The metric equation should not be at odds with informed peer judgment (nor should there be marked discrepancies among metrics themselves, at least among those we assume to be measuring the same sort of thing, such as downloads and citations). In general, with multiple regression equations (which, by the way, capture only linear effects, unless orthogonal polynomials guessing at nonlinear relations are used), one wants the measures that are meant to measure the same sort of thing to be correlated with one another, but one does not want the correlation to be *too* high, otherwise the measures are redundant: Optimally, they should be cross-checks on one another, but also each should be making its own unique contribution to the prediction, over and above corroborating the rest. And of course the weight of each should be adjustable in accordance with the specific profile of the discipline and its needs and values. For example, exogamy might be more of a virtue in some fields than others. Some fields may be more authority-based or co-citation authority-based than others. For some fields, steep early uptake may be predictive, for others, longevity. etc. This will all be brought into focus by the metric validation and calibration and customization phase that will have to precede the use of the scientometric equation for evaluation -- exactly as validation, standardization and the creation of norms and benchmarks must be done in biometrics and psychometrics before using the metrics for clinical or evaluative purposes. We are talking about a rich new OA world of online performance indicators and predictors sitting on top of an even richer primary database: the research itself. Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, N., Eds. Open Access: Key Strategic, Technical and Economic Aspects, chapter 20. Chandos. http://eprints.ecs.soton.ac.uk/12453/ Stevan Harnad > Larry > > Quoting "C.Oppenheim" : > > > The correlation is between number of citations in total (and average number > > of citations per member of staff) received by a Department over the RAE > > period (1996-2001) and the RAE score received by the Department following > > expert peer review. Correlation analyses are done using Pearson or Spearman > > correlation coefficients. The fact that so few humanities scholars publish > > journal articles does not affect this result. > > > > A paper on the topic is in preparation at the moment. > > > > What intrigues me is why there is so much scepticism about the notion. RAE > > is done by peer review experts. Citations are also done by (presumably) > > experts who choose to cite a particular work. So one would expect a > > correlation between the two, wouldn't one? What it tells us is that high > > quality research leads to both high RAE scores AND high citation counts. > > > > I do these calculations (and I've covered many subject areas over the > > years, but not biblical studies - something for the future!) in a totally > > open-minded manner. If I get a non-significant or zero correlation in such > > a study in the future, I will faithfully report it. But so far, that hasn't > > happened. > > > > Charles > > > > Professor Charles Oppenheim > > Head > > Department of Information Science > > Loughborough University > > Loughborough > > Leics LE11 3TU > > > > Tel 01509-223065 > > Fax 01509-223053 > > e mail C.Oppenheim at lboro.ac.uk > > ----- Original Message ----- > > From: > > To: > > Sent: Monday, September 18, 2006 8:37 PM > > Subject: Re: Future UK RAEs to be Metrics-Based > > > > > School of Divinity, New College > University of Edinburgh > Mound Place > Edinburgh, UK. EH1 2LX > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > From harnad at ECS.SOTON.AC.UK Tue Sep 19 10:33:30 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 15:33:30 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: On Tue, 19 Sep 2006, Stephen J Bensman wrote: > I should like to point out that quantitative bibliometric measures have > usually not been found to be applicable in the humanities. The question at hand is a specific and purely empirical one: Is there a significant and sizeable correlation between RAE ranking, as currently determined by the RAE panels, and citation counts? So far, the answer, for every discipline tested to date (including several among the humanities) has been: yes. > First, despite plans to do so, ISI never developed a JCR for the A&HCI. Regrettable, but OA self-archiving mandates will remedy that. > Second, both the > 1981 assessment of US research-doctorate programs by the American Council > on Education, etc., and the 1993 assessment of these programs by National > Research Council rejected using publication and citation counts for the > humanities. Then the question is: Did they reject it for good reasons? Did they try, and fail to get numbers that were proportional to what they were trying to evaluate? Or did they just not try? > The 1993 assessment substituted faculty awards for these > measures. In general, the humanities do not conform closely to typical > bibliometric distributions, being more random and scattered. The empiricial question is: Are citation counts correlated with other measures of research quality/importance/impact in the humanities (including peer evaluation)? > This has generally been found to be the case in library use studies. This is not about library use, but about research use, by researchers, and, in particular these days: online use. > My own > observation has been that, whereas variance in the sciences is due to > accepted "paradigms," such variance as there is in the humanities is due to > intellectual "fads." No matter: Is there a correlation between peer judgments of value and citation counts (and other metrics)? > I am afraid that one is reduced in the humanities to > either subjective evaluations or the acceptance of the subjective > evaluations of others. There are thankfully some things in this world not > reducible to quantitative laws. Deciding whether or not to use and cite a piece of work is also a "subjective judgment" (although the outcome of the use may not be!). The empirical question remains one about correlation: Is there a correlation between citation counts and subjective evaluations. Stevan Harnad > Stevan Harnad @LISTSERV.UTK.EDU> on 09/19/2006 > 08:07:32 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > On Tue, 19 Sep 2006 l.hurtado at ED.AC.UK wrote: > > > --Humanities scholarly publishing is more diverse in venue/genre than > > in some other fields. Indeed, journals are not particularly regarded > > as quite so central, but only one among several respected and > > frequented genres, which include multi-author books, and (perhaps > > particularly) monographs. > > Citation counts can in principle -- and up to a point already do -- > count citations to all these genres: > > (1) citations *from* articles *to* preprints, articles, chapters, and > books (already being partially indexed, e.g., by ISI) > > (2) citations *from* preprints, articles, chapters, books *to* > preprints, articles, chapters, and books (indexable in principle, > already partly indexed by citebase, citeseer, google scholar and > scopus, and will flourish dramatically once Open Access prevails) > > Hence whatever statistically significant RAE/citation correlations > and effect sizes Charles Oppenheim manages to find *despite* the weak and > partial citation coverage to date is actually evidence of the robustness > of the RAE/citation correlation in the fields that are less article-based. > > > QUESTION: Are the studies that supposedly show such meaningful > > correlations actually drawing upon the full spread of publication > > genres appropriate to the fields in view? > > Not yet, and that's the point: Charles's findings are all the more > remarkable for being so robust, despite the weak signal! > > > (I'd be surprised but > > delighted were the answer yes, because I'm not aware of any mechanism > > in place, such as ISI in journal monitoring, for surveying and counting > > in such a vast body of material. > > The point is that citation coverage right now is most definitely > incomplete and insufficient. But that can (and will) only improve > (especially under pressure from the RAE, and OA!). Meanwhile, though, > the successful demonstration of strong correlations even based on the > partial coverage is very promising evidence. > > > I'm not pushing at all for the labour-intensive RAE of the past. > > Bravo. That means 80% agreement already! > > > Indeed, if the question is not how do individual scholars stack up in > > comparison to others in their field (which the RAE actually wasn't > > designed to determine), but instead how can we identify depts into > > which a disproportionate amount of govt funding should be pumped, then > > I think in almost any field a group of informed scholars could readily > > determine the top 5-10 places within 30 minutes, and with time left > > over for coffee. > > For all the UK departments, and in fair proportion to their merits and > needs? > Or just for a familiar few? > > And would several such informed-scholar circles agree on their rankings > (with one another, or with the current RAE rankings)? > > > I'm just asking for more transparency and evidence behind the > > enthusiasm for replacing RAE with "metrics". > > A group of informed scholars over coffee does not strike me as the height > of > transparency and evidence... > > However, in validating the new weighted metric equation, and adjusting > it for the needs of each discipline, one of the criteria against which > it will be validated is of course informed peer judgments: The metric > equation should not be at odds with informed peer judgment (nor should > there be marked discrepancies among metrics themselves, at least among > those we assume to be measuring the same sort of thing, such as downloads > and citations). > > In general, with multiple regression equations (which, by the way, > capture only linear effects, unless orthogonal polynomials guessing at > nonlinear relations are used), one wants the measures that are meant to > measure the same sort of thing to be correlated with one another, but one > does not want the correlation to be *too* high, otherwise the measures > are redundant: Optimally, they should be cross-checks on one another, but > also each should be making its own unique contribution to the prediction, > over and above corroborating the rest. And of course the weight of each > should be adjustable in accordance with the specific profile of the > discipline and its needs and values. > > For example, exogamy might be more of a virtue in some fields than > others. Some fields may be more authority-based or co-citation > authority-based than others. For some fields, steep early uptake may be > predictive, for others, longevity. etc. This will all be brought into > focus by the metric validation and calibration and customization > phase that will have to precede the use of the scientometric equation > for evaluation -- exactly as validation, standardization and the > creation of norms and benchmarks must be done in biometrics and > psychometrics before using the metrics for clinical or evaluative > purposes. > > We are talking about a rich new OA world of online performance > indicators and predictors sitting on top of an even richer primary > database: the research itself. > > Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open > Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, > N., Eds. Open Access: Key Strategic, Technical and Economic Aspects, > chapter 20. Chandos. http://eprints.ecs.soton.ac.uk/12453/ > > Stevan Harnad > > > Larry > > > > Quoting "C.Oppenheim" : > > > > > The correlation is between number of citations in total (and average > number > > > of citations per member of staff) received by a Department over the RAE > > > period (1996-2001) and the RAE score received by the Department > following > > > expert peer review. Correlation analyses are done using Pearson or > Spearman > > > correlation coefficients. The fact that so few humanities scholars > publish > > > journal articles does not affect this result. > > > > > > A paper on the topic is in preparation at the moment. > > > > > > What intrigues me is why there is so much scepticism about the notion. > RAE > > > is done by peer review experts. Citations are also done by > (presumably) > > > experts who choose to cite a particular work. So one would expect a > > > correlation between the two, wouldn't one? What it tells us is that > high > > > quality research leads to both high RAE scores AND high citation > counts. > > > > > > I do these calculations (and I've covered many subject areas over the > > > years, but not biblical studies - something for the future!) in a > totally > > > open-minded manner. If I get a non-significant or zero correlation in > such > > > a study in the future, I will faithfully report it. But so far, that > hasn't > > > happened. > > > > > > Charles > > > > > > Professor Charles Oppenheim > > > Head > > > Department of Information Science > > > Loughborough University > > > Loughborough > > > Leics LE11 3TU > > > > > > Tel 01509-223065 > > > Fax 01509-223053 > > > e mail C.Oppenheim at lboro.ac.uk > > > ----- Original Message ----- > > > From: > > > To: > > > Sent: Monday, September 18, 2006 8:37 PM > > > Subject: Re: Future UK RAEs to be Metrics-Based > > > > > > > > School of Divinity, New College > > University of Edinburgh > > Mound Place > > Edinburgh, UK. EH1 2LX > > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > > > From harnad at ECS.SOTON.AC.UK Tue Sep 19 10:51:21 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 15:51:21 +0100 Subject: Future UK RAEs to be Metrics-Based (fwd) Message-ID: ---------- Forwarded message ---------- Date: Tue, 19 Sep 2006 15:40:59 +0100 From: C.Oppenheim Reply-To: American Scientist Open Access Forum To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM at LISTSERVER.SIGMAXI.ORG Subject: Re: Future UK RAEs to be Metrics-Based The statement "I should like to point out that quantitative bibliometric measures have usually not been found to be applicable in the humanities." would be convincing if supported by references to studies in the literature. I am not aware of any. Perhaps Mr bensmann could give us chapter and verse on this? Charles Professor Charles Oppenheim Head Department of Information Science Loughborough University Loughborough Leics LE11 3TU Tel 01509-223065 Fax 01509-223053 e mail C.Oppenheim at lboro.ac.uk ----- Original Message ----- From: "Stevan Harnad" To: Sent: Tuesday, September 19, 2006 3:33 PM Subject: Re: Future UK RAEs to be Metrics-Based > On Tue, 19 Sep 2006, Stephen J Bensman wrote: > >> I should like to point out that quantitative bibliometric measures have >> usually not been found to be applicable in the humanities. > > The question at hand is a specific and purely empirical one: Is there a > significant and sizeable correlation between RAE ranking, as currently > determined by the RAE panels, and citation counts? > > So far, the answer, for every discipline tested to date (including several > among > the humanities) has been: yes. > >> First, despite plans to do so, ISI never developed a JCR for the A&HCI. > > Regrettable, but OA self-archiving mandates will remedy that. > >> Second, both the >> 1981 assessment of US research-doctorate programs by the American Council >> on Education, etc., and the 1993 assessment of these programs by National >> Research Council rejected using publication and citation counts for the >> humanities. > > Then the question is: Did they reject it for good reasons? Did they try, > and fail > to get numbers that were proportional to what they were trying to > evaluate? Or > did they just not try? > >> The 1993 assessment substituted faculty awards for these >> measures. In general, the humanities do not conform closely to typical >> bibliometric distributions, being more random and scattered. > > The empiricial question is: Are citation counts correlated with other > measures of > research quality/importance/impact in the humanities (including peer > evaluation)? > >> This has generally been found to be the case in library use studies. > > This is not about library use, but about research use, by researchers, > and, in > particular these days: online use. > >> My own >> observation has been that, whereas variance in the sciences is due to >> accepted "paradigms," such variance as there is in the humanities is due >> to >> intellectual "fads." > > No matter: Is there a correlation between peer judgments of value and > citation > counts (and other metrics)? > >> I am afraid that one is reduced in the humanities to >> either subjective evaluations or the acceptance of the subjective >> evaluations of others. There are thankfully some things in this world >> not >> reducible to quantitative laws. > > Deciding whether or not to use and cite a piece of work is also a > "subjective > judgment" (although the outcome of the use may not be!). The empirical > question > remains one about correlation: Is there a correlation between citation > counts and > subjective evaluations. > > Stevan Harnad > > >> Stevan Harnad @LISTSERV.UTK.EDU> on 09/19/2006 >> 08:07:32 AM >> >> Please respond to ASIS&T Special Interest Group on Metrics >> >> >> Sent by: ASIS&T Special Interest Group on Metrics >> >> >> >> To: SIGMETRICS at LISTSERV.UTK.EDU >> cc: (bcc: Stephen J Bensman/notsjb/LSU) >> >> Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based >> >> Adminstrative info for SIGMETRICS (for example unsubscribe): >> http://web.utk.edu/~gwhitney/sigmetrics.html >> >> On Tue, 19 Sep 2006 l.hurtado at ED.AC.UK wrote: >> >> > --Humanities scholarly publishing is more diverse in venue/genre than >> > in some other fields. Indeed, journals are not particularly regarded >> > as quite so central, but only one among several respected and >> > frequented genres, which include multi-author books, and (perhaps >> > particularly) monographs. >> >> Citation counts can in principle -- and up to a point already do -- >> count citations to all these genres: >> >> (1) citations *from* articles *to* preprints, articles, chapters, and >> books (already being partially indexed, e.g., by ISI) >> >> (2) citations *from* preprints, articles, chapters, books *to* >> preprints, articles, chapters, and books (indexable in principle, >> already partly indexed by citebase, citeseer, google scholar and >> scopus, and will flourish dramatically once Open Access prevails) >> >> Hence whatever statistically significant RAE/citation correlations >> and effect sizes Charles Oppenheim manages to find *despite* the weak and >> partial citation coverage to date is actually evidence of the robustness >> of the RAE/citation correlation in the fields that are less >> article-based. >> >> > QUESTION: Are the studies that supposedly show such meaningful >> > correlations actually drawing upon the full spread of publication >> > genres appropriate to the fields in view? >> >> Not yet, and that's the point: Charles's findings are all the more >> remarkable for being so robust, despite the weak signal! >> >> > (I'd be surprised but >> > delighted were the answer yes, because I'm not aware of any mechanism >> > in place, such as ISI in journal monitoring, for surveying and counting >> > in such a vast body of material. >> >> The point is that citation coverage right now is most definitely >> incomplete and insufficient. But that can (and will) only improve >> (especially under pressure from the RAE, and OA!). Meanwhile, though, >> the successful demonstration of strong correlations even based on the >> partial coverage is very promising evidence. >> >> > I'm not pushing at all for the labour-intensive RAE of the past. >> >> Bravo. That means 80% agreement already! >> >> > Indeed, if the question is not how do individual scholars stack up in >> > comparison to others in their field (which the RAE actually wasn't >> > designed to determine), but instead how can we identify depts into >> > which a disproportionate amount of govt funding should be pumped, then >> > I think in almost any field a group of informed scholars could readily >> > determine the top 5-10 places within 30 minutes, and with time left >> > over for coffee. >> >> For all the UK departments, and in fair proportion to their merits and >> needs? >> Or just for a familiar few? >> >> And would several such informed-scholar circles agree on their rankings >> (with one another, or with the current RAE rankings)? >> >> > I'm just asking for more transparency and evidence behind the >> > enthusiasm for replacing RAE with "metrics". >> >> A group of informed scholars over coffee does not strike me as the height >> of >> transparency and evidence... >> >> However, in validating the new weighted metric equation, and adjusting >> it for the needs of each discipline, one of the criteria against which >> it will be validated is of course informed peer judgments: The metric >> equation should not be at odds with informed peer judgment (nor should >> there be marked discrepancies among metrics themselves, at least among >> those we assume to be measuring the same sort of thing, such as downloads >> and citations). >> >> In general, with multiple regression equations (which, by the way, >> capture only linear effects, unless orthogonal polynomials guessing at >> nonlinear relations are used), one wants the measures that are meant to >> measure the same sort of thing to be correlated with one another, but one >> does not want the correlation to be *too* high, otherwise the measures >> are redundant: Optimally, they should be cross-checks on one another, but >> also each should be making its own unique contribution to the prediction, >> over and above corroborating the rest. And of course the weight of each >> should be adjustable in accordance with the specific profile of the >> discipline and its needs and values. >> >> For example, exogamy might be more of a virtue in some fields than >> others. Some fields may be more authority-based or co-citation >> authority-based than others. For some fields, steep early uptake may be >> predictive, for others, longevity. etc. This will all be brought into >> focus by the metric validation and calibration and customization >> phase that will have to precede the use of the scientometric equation >> for evaluation -- exactly as validation, standardization and the >> creation of norms and benchmarks must be done in biometrics and >> psychometrics before using the metrics for clinical or evaluative >> purposes. >> >> We are talking about a rich new OA world of online performance >> indicators and predictors sitting on top of an even richer primary >> database: the research itself. >> >> Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open >> Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, >> N., Eds. Open Access: Key Strategic, Technical and Economic Aspects, >> chapter 20. Chandos. http://eprints.ecs.soton.ac.uk/12453/ >> >> Stevan Harnad >> >> > Larry >> > >> > Quoting "C.Oppenheim" : >> > >> > > The correlation is between number of citations in total (and average >> number >> > > of citations per member of staff) received by a Department over the >> > > RAE >> > > period (1996-2001) and the RAE score received by the Department >> following >> > > expert peer review. Correlation analyses are done using Pearson or >> Spearman >> > > correlation coefficients. The fact that so few humanities scholars >> publish >> > > journal articles does not affect this result. >> > > >> > > A paper on the topic is in preparation at the moment. >> > > >> > > What intrigues me is why there is so much scepticism about the >> > > notion. >> RAE >> > > is done by peer review experts. Citations are also done by >> (presumably) >> > > experts who choose to cite a particular work. So one would expect a >> > > correlation between the two, wouldn't one? What it tells us is that >> high >> > > quality research leads to both high RAE scores AND high citation >> counts. >> > > >> > > I do these calculations (and I've covered many subject areas over >> > > the >> > > years, but not biblical studies - something for the future!) in a >> totally >> > > open-minded manner. If I get a non-significant or zero correlation >> > > in >> such >> > > a study in the future, I will faithfully report it. But so far, that >> hasn't >> > > happened. >> > > >> > > Charles >> > > >> > > Professor Charles Oppenheim >> > > Head >> > > Department of Information Science >> > > Loughborough University >> > > Loughborough >> > > Leics LE11 3TU >> > > >> > > Tel 01509-223065 >> > > Fax 01509-223053 >> > > e mail C.Oppenheim at lboro.ac.uk >> > > ----- Original Message ----- >> > > From: >> > > To: >> > > Sent: Monday, September 18, 2006 8:37 PM >> > > Subject: Re: Future UK RAEs to be Metrics-Based >> > > >> > > >> > School of Divinity, New College >> > University of Edinburgh >> > Mound Place >> > Edinburgh, UK. EH1 2LX >> > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 >> > >> > From j.s.katz at SUSSEX.AC.UK Tue Sep 19 10:58:23 2006 From: j.s.katz at SUSSEX.AC.UK (Sylvan Katz) Date: Tue, 19 Sep 2006 08:58:23 -0600 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: > The question at hand is a specific and purely empirical one: Is there a > significant and sizeable correlation between RAE ranking, as currently > determined by the RAE panels, and citation counts? Linear correlations can be misleading see Katz, J. S., 2006: Indicators for complex innovation systems. Research Policy, 35, 893-909. Dr. J. Sylvan Katz, Visiting Fellow SPRU, University of Sussex http://www.sussex.ac.uk/Users/sylvank Adjunct Professor Mathematics & Statistics, University of Saskatchewan Associate Researcher Institut national de la recherche scientifique, University of Quebec From harnad at ECS.SOTON.AC.UK Tue Sep 19 12:17:00 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 17:17:00 +0100 Subject: Future UK RAEs to be Metrics-Based (fwd) Message-ID: ---------- Forwarded message ---------- Date: Tue, 19 Sep 2006 16:27:38 +0100 From: C.Oppenheim To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM at LISTSERVER.SIGMAXI.ORG Subject: Re: Future UK RAEs to be Metrics-Based My answer is that it is statistically significantly correlated with the RAE results, based upon long, intensive peer group assessment by a group of experts. As Stevan Harnad has frequently commented, what is remarkable is that despite the fact that journals are relatively unimportant in the humanities, the correlation still works. I stress that my approach is purely pragmatic. I'm not suggesting a cause and effect, simply that there is a strong correlation. Charles Professor Charles Oppenheim Head Department of Information Science Loughborough University Loughborough Leics LE11 3TU Tel 01509-223065 Fax 01509-223053 e mail C.Oppenheim at lboro.ac.uk ----- Original Message ----- From: To: Sent: Tuesday, September 19, 2006 3:22 PM Subject: Re: Future UK RAEs to be Metrics-Based Sorry, Charles (if I may), but your response betrays a misunderstanding of my concern and question. Please try to hear me before giving an answer: You're assuming that picking up items that happen to be cited in a selection of journals is somehow adequate, and it's THIS that I'm concerned about. What is your BASIS for your assumption about my field? Yes, of course, you can collect everything cited in a given set of journals, in principle. But is that the same thing as a representative picture of how scholarship is being treated in a field such as mine in which journals are not necessarily the principle medium in which scholarship is established or exhibited? This readily illustrates my concern about what assumptions go into experiments or "empirical" studies before they are run. So, QUESTION: What is your empirical basis for the assumption that simply monitoring a given set of journals is sufficient for any/all fields. You haven't addressed this yet. Larry Quoting "C.Oppenheim" : > The question betrays a misunderstanding of how citation indexes work. > Citation indexes scan the journal literature for citations to all media, > not > just other journals. So it makes no difference what vehicle the > humanities > scholar disseminated his/her output in, the item will get picked up by the > citation index. > > The notion that a group of informed scholars could come up with a ranking > list over 30 minutes is an appealing one, but the fact remains that the > UK's > RAE takes about a year to collect and analyse the data, together with many > meetings of the group of scholars, before decisions are made. one reason > for this tedious approach is to avoid legal challenges that the results > were > not robustly reached. > > Charles > > Professor Charles Oppenheim > Head > Department of Information Science > Loughborough University > Loughborough > Leics LE11 3TU > > Tel 01509-223065 > Fax 01509-223053 > e mail C.Oppenheim at lboro.ac.uk > ----- Original Message ----- > From: > To: > Sent: Tuesday, September 19, 2006 12:34 PM > Subject: Re: Future UK RAEs to be Metrics-Based > > > My scepticism (which is capable of being satisfied) is not toward the > *idea* that there may well be a correlation between the frequency with > which scholarly work is cited and the wider estimate of that > scholar/dept. I am dubious that it has been demonstrated that > conducting such an analysis *can be done* for all disciplines, > particularly at least some Humanities fields. My scepticism rests upon > the bases I've iterated before. I apologize if I seem to be replaying a > record, but it's not yet clear to me that my concerns are effectively > engaged. > --Humanities scholarly publishing is more diverse in venue/genre than > in some other fields. Indeed, journals are not particularly regarded > as quite so central, but only one among several respected and > frequented genres, which include multi-author books, and (perhaps > particularly) monographs. > > QUESTION: Are the studies that supposedly show such meaningful > correlations actually drawing upon the full spread of publication > genres appropriate to the fields in view? (I'd be surprised but > delighted were the answer yes, because I'm not aware of any mechanism > in place, such as ISI in journal monitoring, for surveying and counting > in such a vast body of material. > > I'm not pushing at all for the labor-intensive RAE of the past. > Indeed, if the question is not how do individual scholars stack up in > comparison to others in their field (which the RAE actually wasn't > designed to determine), but instead how can we identify depts into > which a disproportionate amount of govt funding should be pumped, then > I think in almost any field a group of informed scholars could readily > determine the top 5-10 places within 30 minutes, and with time left > over for coffee. > > I'm just asking for more transparency and evidence behind the > enthusiasm for replacing RAE with "metrics". > > Larry > > Quoting "C.Oppenheim" : > >> The correlation is between number of citations in total (and average >> number >> of citations per member of staff) received by a Department over the RAE >> period (1996-2001) and the RAE score received by the Department following >> expert peer review. Correlation analyses are done using Pearson or >> Spearman >> correlation coefficients. The fact that so few humanities scholars >> publish >> journal articles does not affect this result. >> >> A paper on the topic is in preparation at the moment. >> >> What intrigues me is why there is so much scepticism about the notion. >> RAE >> is done by peer review experts. Citations are also done by (presumably) >> experts who choose to cite a particular work. So one would expect a >> correlation between the two, wouldn't one? What it tells us is that high >> quality research leads to both high RAE scores AND high citation counts. >> >> I do these calculations (and I've covered many subject areas over the >> years, but not biblical studies - something for the future!) in a totally >> open-minded manner. If I get a non-significant or zero correlation in >> such >> a study in the future, I will faithfully report it. But so far, that >> hasn't >> happened. >> >> Charles >> >> Professor Charles Oppenheim >> Head >> Department of Information Science >> Loughborough University >> Loughborough >> Leics LE11 3TU >> >> Tel 01509-223065 >> Fax 01509-223053 >> e mail C.Oppenheim at lboro.ac.uk >> ----- Original Message ----- >> From: >> To: >> Sent: Monday, September 18, 2006 8:37 PM >> Subject: Re: Future UK RAEs to be Metrics-Based >> >> >> Well, I'm all for empirically-based views in these matters. So, if >> Openheim or others have actually soundly based studies showing what >> Stevan and Openheim claim, then that's to be noted. I'll have to see >> the stuff when it's published. In the meanwhile, a couple of further >> questions: >> --Pardon me for being out of touch, perhaps, but more precisely what is >> being measured? What does journal "citation counts" refer to? >> Citation of journal articles? Or citation of various things in journal >> articles (and why privilege this medium?)? Or . . . what? >> --What does "correlation" between RAE results and "citation counts" >> actually comprise? >> >> Let me lay out further reasons for some skepticism. In my own field >> (biblical studies/theology), I'd say most senior-level scholars >> actually publish very infrequently in refereed journals. We do perhaps >> more in earlier years, but as we get to senior levels we tend (a) to >> get requests for papers for multi-author volumes, and (b) we devote >> ourselves to projects that best issue in book-length publications. So, >> if my own productivity and impact were assessed by how many journal >> articles I've published in the last five years, I'd look poor (even >> though . . . well, let's say that I rather suspect that wouldn't be the >> way I'm perceived by peers in the field). >> Or is the metric to comprise how many times I'm *cited* in journals? >> If so, is there some proven correlation between a scholar's impact or >> significance of publications in the field and how many times he happens >> to be cited in this one genre of publication? I'm just a bit >> suspicious of the assumptions, which I still suspect are drawn (all >> quite innocently, but naively) from disciplines in which journal >> publication is much more the main and significant venue for scholarly >> publication. >> And, as we all know, "empirical" studies depend entirely on the >> assumptions that lie at their base. So their value is heavily framed >> by the validity and adequacy of the governing assumptions. No >> accusations, just concerns. >> Larry Hurtado >> >> Quoting Stevan Harnad : >> >>> On Mon, 18 Sep 2006, Larry Hurtado wrote: >>> >>>> Stevan and I have exchanged views on the *feasibility* of a metrics >>>> approach to assessing research strength in the Humanities, and he's >>>> impressed me that something such *might well* be feasible *when/if* >>>> certain as-yet untested and undeveloped things fall into place. I note, >>>> e.g., in Stevan's addendum to Oppenheim's comment that a way of >>>> handling >>>> book-based disciplines "has not yet been looked at", and that a number >>>> of other matters are as yet "untested". >>> >>> Larry is quite right that the (rather obvious and straightforward) >>> procedure of self-archiving books' metadata and cited references in >>> order to derive a comprehensive book-citation index (which would >>> of course include journal articles citing books, books citing books, >>> and books citing journal articles) had not yet been implemented or >>> tested. >>> >>> However, the way to go about it is quite clear, and awaits only OA >>> self-archiving mandates (to which a mandate to self-archive one's book >>> metadata and reference list should be added as a matter of course). >>> >>> But please recall that I am an evangelist for OA self-archiving, because >>> I *know* it can be done, that it works, and that it confers substantial >>> benefits in terms of research access, usage and impact. >>> >>> Insofar as metrics are concerned, I am not an evangelist, but merely an >>> enthusiast: The evidence is there, almost as clearly as it is with the >>> OA impact-advantage, that citation counts are strongly correlated with >>> RAE rankings in every discipline so far tested. Larry seems to pass over >>> evidence in his remark about the as yet incomplete book citation data >>> (ISI has some, but they are only partial). But what does he have to say >>> about the correlation between RAE rankings and *journal article >>> citation >>> counts* in the humanities (i.e., in the "book-based" disciplines)? >>> Charles will, for example, soon be reporting strong correlations in >>> Music. Even without having to wait for a book-impact index, it seems >>> clear that there are as yet no reported empirical exceptions to the >>> correlation between journal article citation metrics and RAE outcomes. >>> >>> (I hope Charles will reply directly, posting some references to his and >>> others' studies.) >>> >>>> This being the case, it is certainly not so a priori to say that a >>>> metrics approach is not now really feasible for some disciplines. >>> >>> Nothing a priori about it: A posteriori, every discipline so far tested >>> has shown positive correlations between its journal citation counts and >>> its >>> RAE rankings, including several Humanities disciplines. >>> >>> The advantage of having one last profligate panel-based RAE in parallel >>> with the metric one in 2008 is that not a stone will be left unturned. >>> If there prove to be any disciplines having small or non-existent >>> correlations with metrics, they can and should be evaluated otherwise. >>> But let us not assume, a priori, that there will be any such >>> disciplines. >>> >>>> I emphasize that my point is not a philosophical one, but strictly >>>> whether as yet a worked out scheme for handling all Humanities >>>> disciplines rightly is in place, or capable of being mounted without >>>> some significant further developments, or even thought out adequately. >>> >>> It depends entirely on the size of the metric correlations with the >>> present RAE rankings. Some disciplines may need some supplementary forms >>> of (non-metric) evaluation if their correlations are too weak. That is >>> an >>> empirical question. Meanwhile, the metrics will also be growing in power >>> and diversity. >>> >>>> That's not an antagonistic question, simply someone asking for the >>>> basis for the evangelistic stance of Stevan and some others. >>> >>> I evangelize for OA self-archiving of research and merely advocate >>> further development, testing and use of metrics in research performance >>> assessment, in all disciplines, until/unless evidence appears that there >>> are exceptions. So far, the objections I know of are all only in the >>> form of a priori preconceptions and habits, not objective data. >>> >>> Stevan Harnad >>> >>>> > Charles Oppenheim has authorised me to post this on his behalf: >>>> > >>>> > "Research I have done indicates that the same correlations >>>> > between >>>> > RAE scores and citation counts already noted in the sciences and >>>> > social sciences apply just as strongly (sometimes more strongly) >>>> > in the humanities! But you are right, Richard, that metrics are >>>> > PERCEIVED to be inappropriate for the humanities and a lot of >>>> > educating is needed on this topic." >>> >> >> >> >> L. W. Hurtado, Professor of New Testament Language, Literature & Theology >> Director of Postgraduate Studies >> School of Divinity, New College >> University of Edinburgh >> Mound Place >> Edinburgh, UK. EH1 2LX >> Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 >> > > > > L. W. Hurtado, Professor of New Testament Language, Literature & Theology > Director of Postgraduate Studies > School of Divinity, New College > University of Edinburgh > Mound Place > Edinburgh, UK. EH1 2LX > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > L. W. Hurtado, Professor of New Testament Language, Literature & Theology Director of Postgraduate Studies School of Divinity, New College University of Edinburgh Mound Place Edinburgh, UK. EH1 2LX Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 From harnad at ECS.SOTON.AC.UK Tue Sep 19 13:09:58 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 18:09:58 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <9D7C8A51DA3B2FDF3824261E@[192.168.0.103]> Message-ID: On Tue, 19 Sep 2006, Sylvan Katz wrote: > > The question at hand is a specific and purely empirical one: Is there a > > significant and sizeable correlation between RAE ranking, as currently > > determined by the RAE panels, and citation counts? > > Linear correlations can be misleading see > Katz, J. S., 2006: Indicators for complex innovation systems. Research > Policy, 35, 893-909. You are quite right that linear correlations can be misleading, either because of an underlying nonlinear relation or because of violation of distribution assumptions. But that does not mean that they *are* misleading in the case of the RAE/citation correlations. (Someone should visualise the relationships in each case, to see if there is any indication of nonlinear components, and of course the distribution assumptions should be checked.) Stevan Harnad From notsjb at LSU.EDU Tue Sep 19 11:23:03 2006 From: notsjb at LSU.EDU (Stephen J Bensman) Date: Tue, 19 Sep 2006 10:23:03 -0500 Subject: Future UK RAEs to be Metrics-Based Message-ID: I have only pointed out certain decisions that have been made by persons I consider rather expert in these matters--the dean of the Harvard Graduate School, the provost of Columbia University, etc. etc. I will leave it to you to find out the reasons why these decisions have been made by such persons. For myself, since books are the key publication in the humanities, I have always considered one possible valid measure the number times books have been reviewed by TLC, New York Review of Books, etc. I would be very hesitant to apply this to music though. As for the library use of humanities materials, this has been validated time again. You should be able to find numerous studies of this in the literature. As for the nature of citations in the humanities, I would suggest that you read Gene Garfield's article at the URL below and--what else?--track the material he cited. http://www.garfield.library.upenn.edu/essays/v5p761y1981-82.pdf SB Stevan Harnad @LISTSERV.UTK.EDU> on 09/19/2006 09:33:30 AM Please respond to ASIS&T Special Interest Group on Metrics Sent by: ASIS&T Special Interest Group on Metrics To: SIGMETRICS at LISTSERV.UTK.EDU cc: (bcc: Stephen J Bensman/notsjb/LSU) Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based On Tue, 19 Sep 2006, Stephen J Bensman wrote: > I should like to point out that quantitative bibliometric measures have > usually not been found to be applicable in the humanities. The question at hand is a specific and purely empirical one: Is there a significant and sizeable correlation between RAE ranking, as currently determined by the RAE panels, and citation counts? So far, the answer, for every discipline tested to date (including several among the humanities) has been: yes. > First, despite plans to do so, ISI never developed a JCR for the A&HCI. Regrettable, but OA self-archiving mandates will remedy that. > Second, both the > 1981 assessment of US research-doctorate programs by the American Council > on Education, etc., and the 1993 assessment of these programs by National > Research Council rejected using publication and citation counts for the > humanities. Then the question is: Did they reject it for good reasons? Did they try, and fail to get numbers that were proportional to what they were trying to evaluate? Or did they just not try? > The 1993 assessment substituted faculty awards for these > measures. In general, the humanities do not conform closely to typical > bibliometric distributions, being more random and scattered. The empiricial question is: Are citation counts correlated with other measures of research quality/importance/impact in the humanities (including peer evaluation)? > This has generally been found to be the case in library use studies. This is not about library use, but about research use, by researchers, and, in particular these days: online use. > My own > observation has been that, whereas variance in the sciences is due to > accepted "paradigms," such variance as there is in the humanities is due to > intellectual "fads." No matter: Is there a correlation between peer judgments of value and citation counts (and other metrics)? > I am afraid that one is reduced in the humanities to > either subjective evaluations or the acceptance of the subjective > evaluations of others. There are thankfully some things in this world not > reducible to quantitative laws. Deciding whether or not to use and cite a piece of work is also a "subjective judgment" (although the outcome of the use may not be!). The empirical question remains one about correlation: Is there a correlation between citation counts and subjective evaluations. Stevan Harnad > Stevan Harnad @LISTSERV.UTK.EDU> on 09/19/2006 > 08:07:32 AM > > Please respond to ASIS&T Special Interest Group on Metrics > > > Sent by: ASIS&T Special Interest Group on Metrics > > > > To: SIGMETRICS at LISTSERV.UTK.EDU > cc: (bcc: Stephen J Bensman/notsjb/LSU) > > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > On Tue, 19 Sep 2006 l.hurtado at ED.AC.UK wrote: > > > --Humanities scholarly publishing is more diverse in venue/genre than > > in some other fields. Indeed, journals are not particularly regarded > > as quite so central, but only one among several respected and > > frequented genres, which include multi-author books, and (perhaps > > particularly) monographs. > > Citation counts can in principle -- and up to a point already do -- > count citations to all these genres: > > (1) citations *from* articles *to* preprints, articles, chapters, and > books (already being partially indexed, e.g., by ISI) > > (2) citations *from* preprints, articles, chapters, books *to* > preprints, articles, chapters, and books (indexable in principle, > already partly indexed by citebase, citeseer, google scholar and > scopus, and will flourish dramatically once Open Access prevails) > > Hence whatever statistically significant RAE/citation correlations > and effect sizes Charles Oppenheim manages to find *despite* the weak and > partial citation coverage to date is actually evidence of the robustness > of the RAE/citation correlation in the fields that are less article-based. > > > QUESTION: Are the studies that supposedly show such meaningful > > correlations actually drawing upon the full spread of publication > > genres appropriate to the fields in view? > > Not yet, and that's the point: Charles's findings are all the more > remarkable for being so robust, despite the weak signal! > > > (I'd be surprised but > > delighted were the answer yes, because I'm not aware of any mechanism > > in place, such as ISI in journal monitoring, for surveying and counting > > in such a vast body of material. > > The point is that citation coverage right now is most definitely > incomplete and insufficient. But that can (and will) only improve > (especially under pressure from the RAE, and OA!). Meanwhile, though, > the successful demonstration of strong correlations even based on the > partial coverage is very promising evidence. > > > I'm not pushing at all for the labour-intensive RAE of the past. > > Bravo. That means 80% agreement already! > > > Indeed, if the question is not how do individual scholars stack up in > > comparison to others in their field (which the RAE actually wasn't > > designed to determine), but instead how can we identify depts into > > which a disproportionate amount of govt funding should be pumped, then > > I think in almost any field a group of informed scholars could readily > > determine the top 5-10 places within 30 minutes, and with time left > > over for coffee. > > For all the UK departments, and in fair proportion to their merits and > needs? > Or just for a familiar few? > > And would several such informed-scholar circles agree on their rankings > (with one another, or with the current RAE rankings)? > > > I'm just asking for more transparency and evidence behind the > > enthusiasm for replacing RAE with "metrics". > > A group of informed scholars over coffee does not strike me as the height > of > transparency and evidence... > > However, in validating the new weighted metric equation, and adjusting > it for the needs of each discipline, one of the criteria against which > it will be validated is of course informed peer judgments: The metric > equation should not be at odds with informed peer judgment (nor should > there be marked discrepancies among metrics themselves, at least among > those we assume to be measuring the same sort of thing, such as downloads > and citations). > > In general, with multiple regression equations (which, by the way, > capture only linear effects, unless orthogonal polynomials guessing at > nonlinear relations are used), one wants the measures that are meant to > measure the same sort of thing to be correlated with one another, but one > does not want the correlation to be *too* high, otherwise the measures > are redundant: Optimally, they should be cross-checks on one another, but > also each should be making its own unique contribution to the prediction, > over and above corroborating the rest. And of course the weight of each > should be adjustable in accordance with the specific profile of the > discipline and its needs and values. > > For example, exogamy might be more of a virtue in some fields than > others. Some fields may be more authority-based or co-citation > authority-based than others. For some fields, steep early uptake may be > predictive, for others, longevity. etc. This will all be brought into > focus by the metric validation and calibration and customization > phase that will have to precede the use of the scientometric equation > for evaluation -- exactly as validation, standardization and the > creation of norms and benchmarks must be done in biometrics and > psychometrics before using the metrics for clinical or evaluative > purposes. > > We are talking about a rich new OA world of online performance > indicators and predictors sitting on top of an even richer primary > database: the research itself. > > Shadbolt, N., Brody, T., Carr, L. and Harnad, S. (2006) The Open > Research Web: A Preview of the Optimal and the Inevitable, in Jacobs, > N., Eds. Open Access: Key Strategic, Technical and Economic Aspects, > chapter 20. Chandos. http://eprints.ecs.soton.ac.uk/12453/ > > Stevan Harnad > > > Larry > > > > Quoting "C.Oppenheim" : > > > > > The correlation is between number of citations in total (and average > number > > > of citations per member of staff) received by a Department over the RAE > > > period (1996-2001) and the RAE score received by the Department > following > > > expert peer review. Correlation analyses are done using Pearson or > Spearman > > > correlation coefficients. The fact that so few humanities scholars > publish > > > journal articles does not affect this result. > > > > > > A paper on the topic is in preparation at the moment. > > > > > > What intrigues me is why there is so much scepticism about the notion. > RAE > > > is done by peer review experts. Citations are also done by > (presumably) > > > experts who choose to cite a particular work. So one would expect a > > > correlation between the two, wouldn't one? What it tells us is that > high > > > quality research leads to both high RAE scores AND high citation > counts. > > > > > > I do these calculations (and I've covered many subject areas over the > > > years, but not biblical studies - something for the future!) in a > totally > > > open-minded manner. If I get a non-significant or zero correlation in > such > > > a study in the future, I will faithfully report it. But so far, that > hasn't > > > happened. > > > > > > Charles > > > > > > Professor Charles Oppenheim > > > Head > > > Department of Information Science > > > Loughborough University > > > Loughborough > > > Leics LE11 3TU > > > > > > Tel 01509-223065 > > > Fax 01509-223053 > > > e mail C.Oppenheim at lboro.ac.uk > > > ----- Original Message ----- > > > From: > > > To: > > > Sent: Monday, September 18, 2006 8:37 PM > > > Subject: Re: Future UK RAEs to be Metrics-Based > > > > > > > > School of Divinity, New College > > University of Edinburgh > > Mound Place > > Edinburgh, UK. EH1 2LX > > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > > > From harnad at ECS.SOTON.AC.UK Tue Sep 19 13:48:36 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 18:48:36 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <20060919173459.kbbtmoi20ooggc8o@www.staffmail.ed.ac.uk> Message-ID: On Tue, 19 Sep 2006 l.hurtado at ED.AC.UK wrote: > --Charles Oppenheim insists that he's able to prove a significant > statistical correlation of some sort between RAE results in a variety > of fields, including Humanities subjects. It would be good to verify > this. I presume that online publication(s) are . . . available > somewhere or will be? Several are (all *should* be! Charles?). Look in OAIster and Google Scholar. > --It will be interesting to see the specifics behind the claims. I'm > not clear what is meant by this "correlation". Does it mean that those > depts given a 5* also show up with . . . what, a much higher number of > their publications getting cited in the selected venues, or the > individuals in the dept being cited more frequently, or . . . whatever? Correlation (positive) means high goes with high and low goes with low. So the higher ranked departments have a higher total number of citations of the papers published by the submitted researchers and the lower ranked departments have a lower number. In a word, the papers of higher RAE-ranked departments are cited more. > And what does "significant" correlation mean? (And please, I hope > that any publications don't give me that "regression to the mean" > technospeak, as I'm not a statistician, but I can follow logic. Give > me in plain English what is being counted and compared and how.) The statistical significance of the correlation (which is *not* the same as its size or importance) is the probability that the correlation just happened by chance. If this probability is less the 0.001 or 0.001 it is fairly safe to assume that it is not just a chance accident. And since the same correlation keeps being found, across fields, the likelihood that these are all happening by accident is negligible. The size of the effect is another question: Correlation coefficients vary from -1 to +1. A correlation of 0 is no correlation at all. A simple way to think of the size of the correlation is to square the correlation coefficient. That tells you what percentage of the variation in the values of one variable is predictable from the variation of the values in the other variable. For example, if height and weight have a (statistically significant) correlation of 0.5 (and the average height is 5 feet, standard deviation 6 inches, and the average weight is 150 pounds, standard deviation 50 pounds) then if you tell me that someone is 6 feet tall, then I can predict that he weights 250 pounds, and I will be about 25% right (because 0.5 squared is 0.25 or 25%). If the correlation was instead 0.9, then I could make the same prediction and be 81% right. Another example is the correlation between barometric pressure and rain. If the correlation is 0.5, then tell me the barometric pressure and I can predict how much it will rain, with 25% accuracy; if the correlation is 0.9, I can predict with 81% accuracy. (Another way to express the accuracy is by putting a +/- take range around the predicted value: that range is broader with lower correlations and narrower with higher correlations.) And that's how it is with citations as predictors of RAE ranks. > --I also note a somewhat different tone in Stevan's comments, which > seem to me to admit more forthrightly that "we ain't there yet" when it > comes to the wherewithall actually to conduct an across-the-board > analysis of the kind being mooted. Charles seems to suggest that he's > able to do this now. Or do I misundertand things? I think Charles and I are in 100% agreement: (1) All evidence so far is that RAE ranks can be predicted from citation counts for all disciplines tested so far. (2) There are still disciplines to be tested. (3) Citation counts are not the only possible metrics. (4) RAE panels should definitely be scrapped in favour of metrics in all fields except those where no metric can be shown to correlate sufficiently closely with the RAE rankings. > In any case, I hope that all parties understand the importance of > making sure that all of us affected by any metrics approach understand > it and can see its superiority and full feasibility. I certainly ain't > there yet on any of these matters, but let's see what rolls out. If > Stevan and Charles can put it together and show the rest of us how it > works well, I'll go for it. I just want to be shown (I originated from > Missouri). Y'all hold onto yir hats; ya ain't seed nothin' yet! Stevan Harnad > Quoting "C.Oppenheim" : > > > My answer is that it is statistically significantly correlated with the RAE > > results, based upon long, intensive peer group assessment by a group of > > experts. As Stevan Harnad has frequently commented, what is remarkable is > > that despite the fact that journals are relatively unimportant in the > > humanities, the correlation still works. > > > > I stress that my approach is purely pragmatic. I'm not suggesting a cause > > and effect, simply that there is a strong correlation. > > > > Charles > > > > Professor Charles Oppenheim > > Head > > Department of Information Science > > Loughborough University > > Loughborough > > Leics LE11 3TU > > > > Tel 01509-223065 > > Fax 01509-223053 > > e mail C.Oppenheim at lboro.ac.uk > > ----- Original Message ----- > > From: > > To: > > Sent: Tuesday, September 19, 2006 3:22 PM > > Subject: Re: Future UK RAEs to be Metrics-Based > > > > > > Sorry, Charles (if I may), but your response betrays a misunderstanding > > of my concern and question. Please try to hear me before giving an > > answer: > > You're assuming that picking up items that happen to be cited in a > > selection of journals is somehow adequate, and it's THIS that I'm > > concerned about. What is your BASIS for your assumption about my field? > > Yes, of course, you can collect everything cited in a given set of > > journals, in principle. But is that the same thing as a representative > > picture of how scholarship is being treated in a field such as mine in > > which journals are not necessarily the principle medium in which > > scholarship is established or exhibited? This readily illustrates my > > concern about what assumptions go into experiments or "empirical" > > studies before they are run. > > > > So, QUESTION: What is your empirical basis for the assumption that > > simply monitoring a given set of journals is sufficient for any/all > > fields. You haven't addressed this yet. > > Larry > > > > Quoting "C.Oppenheim" : > > > >> The question betrays a misunderstanding of how citation indexes work. > >> Citation indexes scan the journal literature for citations to all media, > >> not > >> just other journals. So it makes no difference what vehicle the > >> humanities > >> scholar disseminated his/her output in, the item will get picked up by the > >> citation index. > >> > >> The notion that a group of informed scholars could come up with a ranking > >> list over 30 minutes is an appealing one, but the fact remains that the > >> UK's > >> RAE takes about a year to collect and analyse the data, together with many > >> meetings of the group of scholars, before decisions are made. one reason > >> for this tedious approach is to avoid legal challenges that the results > >> were > >> not robustly reached. > >> > >> Charles > >> > >> Professor Charles Oppenheim > >> Head > >> Department of Information Science > >> Loughborough University > >> Loughborough > >> Leics LE11 3TU > >> > >> Tel 01509-223065 > >> Fax 01509-223053 > >> e mail C.Oppenheim at lboro.ac.uk > >> ----- Original Message ----- > >> From: > >> To: > >> Sent: Tuesday, September 19, 2006 12:34 PM > >> Subject: Re: Future UK RAEs to be Metrics-Based > >> > >> > >> My scepticism (which is capable of being satisfied) is not toward the > >> *idea* that there may well be a correlation between the frequency with > >> which scholarly work is cited and the wider estimate of that > >> scholar/dept. I am dubious that it has been demonstrated that > >> conducting such an analysis *can be done* for all disciplines, > >> particularly at least some Humanities fields. My scepticism rests upon > >> the bases I've iterated before. I apologize if I seem to be replaying a > >> record, but it's not yet clear to me that my concerns are effectively > >> engaged. > >> --Humanities scholarly publishing is more diverse in venue/genre than > >> in some other fields. Indeed, journals are not particularly regarded > >> as quite so central, but only one among several respected and > >> frequented genres, which include multi-author books, and (perhaps > >> particularly) monographs. > >> > >> QUESTION: Are the studies that supposedly show such meaningful > >> correlations actually drawing upon the full spread of publication > >> genres appropriate to the fields in view? (I'd be surprised but > >> delighted were the answer yes, because I'm not aware of any mechanism > >> in place, such as ISI in journal monitoring, for surveying and counting > >> in such a vast body of material. > >> > >> I'm not pushing at all for the labor-intensive RAE of the past. > >> Indeed, if the question is not how do individual scholars stack up in > >> comparison to others in their field (which the RAE actually wasn't > >> designed to determine), but instead how can we identify depts into > >> which a disproportionate amount of govt funding should be pumped, then > >> I think in almost any field a group of informed scholars could readily > >> determine the top 5-10 places within 30 minutes, and with time left > >> over for coffee. > >> > >> I'm just asking for more transparency and evidence behind the > >> enthusiasm for replacing RAE with "metrics". > >> > >> Larry > >> > >> Quoting "C.Oppenheim" : > >> > >>> The correlation is between number of citations in total (and average > >>> number > >>> of citations per member of staff) received by a Department over the RAE > >>> period (1996-2001) and the RAE score received by the Department following > >>> expert peer review. Correlation analyses are done using Pearson or > >>> Spearman > >>> correlation coefficients. The fact that so few humanities scholars > >>> publish > >>> journal articles does not affect this result. > >>> > >>> A paper on the topic is in preparation at the moment. > >>> > >>> What intrigues me is why there is so much scepticism about the notion. > >>> RAE > >>> is done by peer review experts. Citations are also done by (presumably) > >>> experts who choose to cite a particular work. So one would expect a > >>> correlation between the two, wouldn't one? What it tells us is that high > >>> quality research leads to both high RAE scores AND high citation counts. > >>> > >>> I do these calculations (and I've covered many subject areas over the > >>> years, but not biblical studies - something for the future!) in a totally > >>> open-minded manner. If I get a non-significant or zero correlation in > >>> such > >>> a study in the future, I will faithfully report it. But so far, that > >>> hasn't > >>> happened. > >>> > >>> Charles > >>> > >>> Professor Charles Oppenheim > >>> Head > >>> Department of Information Science > >>> Loughborough University > >>> Loughborough > >>> Leics LE11 3TU > >>> > >>> Tel 01509-223065 > >>> Fax 01509-223053 > >>> e mail C.Oppenheim at lboro.ac.uk > >>> ----- Original Message ----- > >>> From: > >>> To: > >>> Sent: Monday, September 18, 2006 8:37 PM > >>> Subject: Re: Future UK RAEs to be Metrics-Based > >>> > >>> > >>> Well, I'm all for empirically-based views in these matters. So, if > >>> Oppenheim or others have actually soundly based studies showing what > >>> Stevan and Oppenheim claim, then that's to be noted. I'll have to see > >>> the stuff when it's published. In the meanwhile, a couple of further > >>> questions: > >>> --Pardon me for being out of touch, perhaps, but more precisely what is > >>> being measured? What does journal "citation counts" refer to? > >>> Citation of journal articles? Or citation of various things in journal > >>> articles (and why privilege this medium?)? Or . . . what? > >>> --What does "correlation" between RAE results and "citation counts" > >>> actually comprise? > >>> > >>> Let me lay out further reasons for some skepticism. In my own field > >>> (biblical studies/theology), I'd say most senior-level scholars > >>> actually publish very infrequently in refereed journals. We do perhaps > >>> more in earlier years, but as we get to senior levels we tend (a) to > >>> get requests for papers for multi-author volumes, and (b) we devote > >>> ourselves to projects that best issue in book-length publications. So, > >>> if my own productivity and impact were assessed by how many journal > >>> articles I've published in the last five years, I'd look poor (even > >>> though . . . well, let's say that I rather suspect that wouldn't be the > >>> way I'm perceived by peers in the field). > >>> Or is the metric to comprise how many times I'm *cited* in journals? > >>> If so, is there some proven correlation between a scholar's impact or > >>> significance of publications in the field and how many times he happens > >>> to be cited in this one genre of publication? I'm just a bit > >>> suspicious of the assumptions, which I still suspect are drawn (all > >>> quite innocently, but naively) from disciplines in which journal > >>> publication is much more the main and significant venue for scholarly > >>> publication. > >>> And, as we all know, "empirical" studies depend entirely on the > >>> assumptions that lie at their base. So their value is heavily framed > >>> by the validity and adequacy of the governing assumptions. No > >>> accusations, just concerns. > >>> Larry Hurtado > >>> > >>> Quoting Stevan Harnad : > >>> > >>>> On Mon, 18 Sep 2006, Larry Hurtado wrote: > >>>> > >>>>> Stevan and I have exchanged views on the *feasibility* of a metrics > >>>>> approach to assessing research strength in the Humanities, and he's > >>>>> impressed me that something such *might well* be feasible *when/if* > >>>>> certain as-yet untested and undeveloped things fall into place. I note, > >>>>> e.g., in Stevan's addendum to Oppenheim's comment that a way of > >>>>> handling > >>>>> book-based disciplines "has not yet been looked at", and that a number > >>>>> of other matters are as yet "untested". > >>>> > >>>> Larry is quite right that the (rather obvious and straightforward) > >>>> procedure of self-archiving books' metadata and cited references in > >>>> order to derive a comprehensive book-citation index (which would > >>>> of course include journal articles citing books, books citing books, > >>>> and books citing journal articles) had not yet been implemented or > >>>> tested. > >>>> > >>>> However, the way to go about it is quite clear, and awaits only OA > >>>> self-archiving mandates (to which a mandate to self-archive one's book > >>>> metadata and reference list should be added as a matter of course). > >>>> > >>>> But please recall that I am an evangelist for OA self-archiving, because > >>>> I *know* it can be done, that it works, and that it confers substantial > >>>> benefits in terms of research access, usage and impact. > >>>> > >>>> Insofar as metrics are concerned, I am not an evangelist, but merely an > >>>> enthusiast: The evidence is there, almost as clearly as it is with the > >>>> OA impact-advantage, that citation counts are strongly correlated with > >>>> RAE rankings in every discipline so far tested. Larry seems to pass over > >>>> evidence in his remark about the as yet incomplete book citation data > >>>> (ISI has some, but they are only partial). But what does he have to say > >>>> about the correlation between RAE rankings and *journal article > >>>> citation > >>>> counts* in the humanities (i.e., in the "book-based" disciplines)? > >>>> Charles will, for example, soon be reporting strong correlations in > >>>> Music. Even without having to wait for a book-impact index, it seems > >>>> clear that there are as yet no reported empirical exceptions to the > >>>> correlation between journal article citation metrics and RAE outcomes. > >>>> > >>>> (I hope Charles will reply directly, posting some references to his and > >>>> others' studies.) > >>>> > >>>>> This being the case, it is certainly not so a priori to say that a > >>>>> metrics approach is not now really feasible for some disciplines. > >>>> > >>>> Nothing a priori about it: A posteriori, every discipline so far tested > >>>> has shown positive correlations between its journal citation counts and > >>>> its > >>>> RAE rankings, including several Humanities disciplines. > >>>> > >>>> The advantage of having one last profligate panel-based RAE in parallel > >>>> with the metric one in 2008 is that not a stone will be left unturned. > >>>> If there prove to be any disciplines having small or non-existent > >>>> correlations with metrics, they can and should be evaluated otherwise. > >>>> But let us not assume, a priori, that there will be any such > >>>> disciplines. > >>>> > >>>>> I emphasize that my point is not a philosophical one, but strictly > >>>>> whether as yet a worked out scheme for handling all Humanities > >>>>> disciplines rightly is in place, or capable of being mounted without > >>>>> some significant further developments, or even thought out adequately. > >>>> > >>>> It depends entirely on the size of the metric correlations with the > >>>> present RAE rankings. Some disciplines may need some supplementary forms > >>>> of (non-metric) evaluation if their correlations are too weak. That is > >>>> an > >>>> empirical question. Meanwhile, the metrics will also be growing in power > >>>> and diversity. > >>>> > >>>>> That's not an antagonistic question, simply someone asking for the > >>>>> basis for the evangelistic stance of Stevan and some others. > >>>> > >>>> I evangelize for OA self-archiving of research and merely advocate > >>>> further development, testing and use of metrics in research performance > >>>> assessment, in all disciplines, until/unless evidence appears that there > >>>> are exceptions. So far, the objections I know of are all only in the > >>>> form of a priori preconceptions and habits, not objective data. > >>>> > >>>> Stevan Harnad > >>>> > >>>>> > Charles Oppenheim has authorised me to post this on his behalf: > >>>>> > > >>>>> > "Research I have done indicates that the same correlations > >>>>> > between > >>>>> > RAE scores and citation counts already noted in the sciences and > >>>>> > social sciences apply just as strongly (sometimes more strongly) > >>>>> > in the humanities! But you are right, Richard, that metrics are > >>>>> > PERCEIVED to be inappropriate for the humanities and a lot of > >>>>> > educating is needed on this topic." > >>>> > >>> > >>> > >>> > >>> L. W. Hurtado, Professor of New Testament Language, Literature & Theology > >>> Director of Postgraduate Studies > >>> School of Divinity, New College > >>> University of Edinburgh > >>> Mound Place > >>> Edinburgh, UK. EH1 2LX > >>> Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > >>> > >> > >> > >> > >> L. W. Hurtado, Professor of New Testament Language, Literature & Theology > >> Director of Postgraduate Studies > >> School of Divinity, New College > >> University of Edinburgh > >> Mound Place > >> Edinburgh, UK. EH1 2LX > >> Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > >> > > > > > > > > L. W. Hurtado, Professor of New Testament Language, Literature & Theology > > Director of Postgraduate Studies > > School of Divinity, New College > > University of Edinburgh > > Mound Place > > Edinburgh, UK. EH1 2LX > > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > > > > > > L. W. Hurtado, Professor of New Testament Language, Literature & Theology > Director of Postgraduate Studies > School of Divinity, New College > University of Edinburgh > Mound Place > Edinburgh, UK. EH1 2LX > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 > From harnad at ECS.SOTON.AC.UK Tue Sep 19 15:28:34 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Tue, 19 Sep 2006 20:28:34 +0100 Subject: OA's Impact on the Humanities vs. Science (fwd) Message-ID: ---------- Forwarded message ---------- Date: Tue, 19 Sep 2006 13:17:36 -0400 From: Charles Finley To: AMERICAN-SCIENTIST-OPEN-ACCESS-FORUM at LISTSERVER.SIGMAXI.ORG Subject: OA's Impact on the Humanities vs. Science To follow up on the citation debate and its references to the humanities I am posting this to let you know that at the Knowledge Media Design Institute we have just published an article on OA and the Humanities on our Project OS |OA website by University of Toronto Professor Linda Hutcheon. This article is not on citation specifically, but it is a good exploration of why OA's impact on the humanities has been different than that of the sciences. http://open.utoronto.ca/index.php?option=com_content&task=view&id=389&Itemid=66 /*Open Access offers new opportunities for publishing, research and promotion in the humanities says U of T University Professor Linda Hutcheon. Emphasis on monographs, less time-sensitive research reasons for lack of OA debate and awareness compared to the sciences.*/ Linda Hutcheon predicts the rise of electronic books and a move towards an emphasis on journal articles as a basis for promotion and tenure will contribute to increased demands from humanities scholars for open access in our academic institutions. * *Read more: http://open.utoronto.ca/index.php?option=com_content&task=view&id=389&Itemid=66 -- Charles Finley, Executive Director Project Open Source | Open Access Knowledge Media Design Institute University of Toronto charles.finley at utoronto.ca Phone: 416.978.3778 http://open.utoronto.ca C.Oppenheim wrote: > My latest study is in the field of music, where journal articles are most > certainly not the primary means of dissemination. I also understand > that in > arhcaeology, reports and monographs are as important, if not more > important, > than journal articles. > > However, the comment misses the point that citation counting counts > citations to all media, not just to journal articles. > > Charles > > Professor Charles Oppenheim > Head > Department of Information Science > Loughborough University > Loughborough > Leics LE11 3TU > > Tel 01509-223065 > Fax 01509-223053 > e mail C.Oppenheim at lboro.ac.uk > ----- Original Message ----- > From: > To: > Sent: Tuesday, September 19, 2006 9:56 AM > Subject: Re: Future UK RAEs to be Metrics-Based > > > I've done a quick check of the publications by Openheim supposedly > showing a strong correlation of RAE standings and citations in > journals, and it seems to me that all I can find are studies to do with > psychology, anatomy, archaeology, etc., ALL OF WHICH use > journal-articles as the prized mode of research > productivity/publication. > Can Openheim or STevan point me to studies of, e.g., English Lit, > History, Religion, with similar results?? > I know that this list is not about this issue primarily, but it's the > (over?)confidence of Stevan on this that puzzles me . . . in the > apparent absence of the empirical proof that he so values. Or please > correct me by pointing to the publications I request (preferably > online, of course!). > Larry > > Quoting Stevan Harnad : > >> On Mon, 18 Sep 2006 l.hurtado at ED.AC.UK wrote: >> >>> Well, I'm all for empirically-based views in these matters. So, if >>> Oppenheim or others have actually soundly based studies showing what >>> Stevan and Oppenheim claim, then that's to be noted. I'll have to see >>> the stuff when it's published. In the meanwhile, a couple of further >>> questions: >> >> Many studies are already published. In fact many are cited in: >> >> Harnad, S., Carr, L., Brody, T. and Oppenheim, C. (2003) Mandated >> online RAE CVs Linked to University Eprint Archives. Ariadne 35. >> http://eprints.ecs.soton.ac.uk/7725/ >> >>> --Pardon me for being out of touch, perhaps, but more precisely what is >>> being measured? What does journal "citation counts" refer to? >> >> The total number of citations to the articles by submitted authors >> (and not just those >> for their 4 submitted articles!) >> >>> Citation of journal articles? Or citation of various things in journal >>> articles (and why privilege this medium?)? Or . . . what? >> >> Citation of the articles, but that usually means citing things in the >> articles! >> >> Journal articles are privileged in many disciplines because they are the >> main means of reporting research. In book-based disciplines the balance >> is otherwise, but the interesting thing is that even in book-based >> disciplines there is a journal article citation correlation with the RAE >> rankings. One would expect it to be somewhat weaker than in >> article-based >> disciplines, but more data are needed to be exact about this. >> >>> --What does "correlation" between RAE results and "citation counts" >>> actually comprise? >> >> The RAE ranks the departments of the c. 73 UK research universities, >> with ranks from 1 to 5*. Correlation is the measure of the degree to >> which values on one variable co-vary with, hence predict, values on >> another variable (e.g. height is correlated with weight, the higher on >> one, the higher on the other, and vice versa). >> >> When two variables are correlated, you can predict one from the >> other. How >> accurately you can predict is reflected by the square of the correlation >> coefficient: If there is a correlation of 0.8, then the predictivity >> (the >> percentage of the variation in one of the variables that you can already >> predict from the other) is 64%. For a correlation of 0.9 it's 81% etc. >> >> Well, as you will see in the reference list of the above-cited article, >> Smith & Eysenck found a correlation of about 0.9 between the RAE ranks >> and the total citation counts for the submitted researchers in >> Psychology. >> >> Looking at Charles Oppenheim's studies, you will see that the >> correlations >> varied from about 0.6 to 0.9, depending on field and year, which is all >> quite high, but *especially* give that the RAE does not count citations! >> >> The correlation is even higher with another metric, in science and >> biology: prior research funding. There it can be as high as 0.99, but >> that is not so good, because (1) prior funding *is* directly seen and >> counted by the panel, so that high correlation could be an effect of >> direct influence. Worse, using prior funding as a criterion generates a >> Matthew Effect, with the already-highly-funded getting richer and >> richer, >> and the less-funded getting poorer and poorer. >> >> That is why a multiple regression equation is best, with many predictor >> metrics, each one weighted according to the desiderata and particulars >> of each discipline, and validated against further criteria, to make sure >> they are measuring what we want to measure. There will be many candidate >> metrics in the OA era. >> >>> Let me lay out further reasons for some skepticism. In my own field >>> (biblical studies/theology), I'd say most senior-level scholars >>> actually publish very infrequently in refereed journals. We do perhaps >>> more in earlier years, but as we get to senior levels we tend (a) to >>> get requests for papers for multi-author volumes, and (b) we devote >>> ourselves to projects that best issue in book-length publications. >> >> That happens in other fields too, and as metric equations are calibrated >> and optimised, factors like seniority will enter into the weightings >> too. (Book chapter citations can and will of course be cited too -- >> and are, to a limited degree, already being counted by ISI and others, >> because journal articles cite books and book chapters too, and those >> citations are caught by ISI.) >> >>> So if my own productivity and impact were assessed by how many journal >>> articles I've published in the last five years, I'd look poor (even >>> though . . . well, let's say that I rather suspect that wouldn't be the >>> way I'm perceived by peers in the field). >> >> The RAE ranks departments via individuals, and a department needs >> a blend of junior and senior people, with their different style of >> publication. And remember that RAE is comparing like with like. So >> you might be interested in checking how your own journal article >> and book chapter citation counts compare with those of your peers (or >> juniors). You might be (pleasantly) surprised! >> >> And of course in the (soon-to-hand) OA era, other metrics will be >> available too, such as download counts ("hits"), which happen much >> earlier, yet are correlated with later citations -- and are of course >> maximized by self-archiving your papers in your institutional IR to make >> them OA. >> >> Odd new metrics will also include endogamy/exogamy scores (their >> preferred >> polarity depending on field!), depending on the degree of self-citing, >> co-author citing, co-citation circle citing, within/outside specialty >> citing, intra/interdisciplinary citing, both for the citing >> article/author >> and the cited article/author. Then there's text-proximity scores (of >> which >> an extreme would be plagiarism), latency/longevity metrics, co-citation >> to/from, CiteRank (where the weight of each citation is recursively >> ranked, google style, by the degree of citedness of the citer), etc. >> etc. >> >>> Or is the metric to comprise how many times I'm *cited* in journals? >> >> It's how many times you're cited, which means how many times your >> articles are >> cited -- in journals, but in principle also in book chapters, >> conferences and books. >> And whether what is *being* cited is articles, chapters or books. >> >>> If so, is there some proven correlation between a scholar's impact or >>> significance of publications in the field and how many times he happens >>> to be cited in this one genre of publication? I'm just a bit >>> suspicious of the assumptions, which I still suspect are drawn (all >>> quite innocently, but naively) from disciplines in which journal >>> publication is much more the main and significant venue for scholarly >>> publication. >> >> I don't know of systematic genre comparisons (journals vs book chapters, >> even empirical vs theoretical journals, reviews, etc.) but they no doubt >> exist. I will branch this to the sigmetrics list where the experts >> are! I >> am just an amateur... >> >>> And, as we all know, "empirical" studies depend entirely on the >>> assumptions that lie at their base. So their value is heavily framed >>> by the validity and adequacy of the governing assumptions. No >>> accusations, just concerns. >> >> Interpretations may be influenced by assumptions, but the empirical fact >> that atmospheric pressure predicts RAE ranking would be an empirical >> datum >> (and, if it predicted it with a correlation of, say, 0.9) that would be >> a reason for scrapping RAE panels for barometers >> theory-independently.... >> >> Stevan Harnad >> >>> Quoting Stevan Harnad : >>> >>> > On Mon, 18 Sep 2006, Larry Hurtado wrote: >>> > >>> >> Stevan and I have exchanged views on the *feasibility* of a metrics >>> >> approach to assessing research strength in the Humanities, and he's >>> >> impressed me that something such *might well* be feasible *when/if* >>> >> certain as-yet untested and undeveloped things fall into place. I >>> >> note, >>> >> e.g., in Stevan's addendum to Oppenheim's comment that a way of >>> >> handling >>> >> book-based disciplines "has not yet been looked at", and that a >>> number >>> >> of other matters are as yet "untested". >>> > >>> > Larry is quite right that the (rather obvious and straightforward) >>> > procedure of self-archiving books' metadata and cited references in >>> > order to derive a comprehensive book-citation index (which would >>> > of course include journal articles citing books, books citing books, >>> > and books citing journal articles) had not yet been implemented or >>> > tested. >>> > >>> > However, the way to go about it is quite clear, and awaits only OA >>> > self-archiving mandates (to which a mandate to self-archive one's >>> book >>> > metadata and reference list should be added as a matter of course). >>> > >>> > But please recall that I am an evangelist for OA self-archiving, >>> > because >>> > I *know* it can be done, that it works, and that it confers >>> substantial >>> > benefits in terms of research access, usage and impact. >>> > >>> > Insofar as metrics are concerned, I am not an evangelist, but >>> merely an >>> > enthusiast: The evidence is there, almost as clearly as it is with >>> the >>> > OA impact-advantage, that citation counts are strongly correlated >>> with >>> > RAE rankings in every discipline so far tested. Larry seems to pass >>> > over >>> > evidence in his remark about the as yet incomplete book citation data >>> > (ISI has some, but they are only partial). But what does he have >>> to say >>> > about the correlation between RAE rankings and *journal article >>> > citation >>> > counts* in the humanities (i.e., in the "book-based" disciplines)? >>> > Charles will, for example, soon be reporting strong correlations in >>> > Music. Even without having to wait for a book-impact index, it seems >>> > clear that there are as yet no reported empirical exceptions to the >>> > correlation between journal article citation metrics and RAE >>> outcomes. >>> > >>> > (I hope Charles will reply directly, posting some references to >>> his and >>> > others' studies.) >>> > >>> >> This being the case, it is certainly not so a priori to say that a >>> >> metrics approach is not now really feasible for some disciplines. >>> > >>> > Nothing a priori about it: A posteriori, every discipline so far >>> tested >>> > has shown positive correlations between its journal citation >>> counts and its >>> > RAE rankings, including several Humanities disciplines. >>> > >>> > The advantage of having one last profligate panel-based RAE in >>> parallel >>> > with the metric one in 2008 is that not a stone will be left >>> unturned. >>> > If there prove to be any disciplines having small or non-existent >>> > correlations with metrics, they can and should be evaluated >>> otherwise. >>> > But let us not assume, a priori, that there will be any such >>> > disciplines. >>> > >>> >> I emphasize that my point is not a philosophical one, but strictly >>> >> whether as yet a worked out scheme for handling all Humanities >>> >> disciplines rightly is in place, or capable of being mounted without >>> >> some significant further developments, or even thought out >>> adequately. >>> > >>> > It depends entirely on the size of the metric correlations with the >>> > present RAE rankings. Some disciplines may need some supplementary >>> > forms >>> > of (non-metric) evaluation if their correlations are too weak. >>> That is >>> > an >>> > empirical question. Meanwhile, the metrics will also be growing in >>> > power >>> > and diversity. >>> > >>> >> That's not an antagonistic question, simply someone asking for the >>> >> basis for the evangelistic stance of Stevan and some others. >>> > >>> > I evangelize for OA self-archiving of research and merely advocate >>> > further development, testing and use of metrics in research >>> performance >>> > assessment, in all disciplines, until/unless evidence appears that >>> > there >>> > are exceptions. So far, the objections I know of are all only in the >>> > form of a priori preconceptions and habits, not objective data. >>> > >>> > Stevan Harnad >>> > >>> >> > Charles Oppenheim has authorised me to post this on his behalf: >>> >> > >>> >> > "Research I have done indicates that the same correlations >>> >> > between >>> >> > RAE scores and citation counts already noted in the sciences >>> >> > and >>> >> > social sciences apply just as strongly (sometimes more >>> strongly) >>> >> > in the humanities! But you are right, Richard, that >>> metrics are >>> >> > PERCEIVED to be inappropriate for the humanities and a lot of >>> >> > educating is needed on this topic." >>> > >>> >>> >>> >>> L. W. Hurtado, Professor of New Testament Language, Literature & >>> Theology >>> Director of Postgraduate Studies >>> School of Divinity, New College >>> University of Edinburgh >>> Mound Place >>> Edinburgh, UK. EH1 2LX >>> Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 >>> >> > > > > L. W. Hurtado, Professor of New Testament Language, Literature & Theology > Director of Postgraduate Studies > School of Divinity, New College > University of Edinburgh > Mound Place > Edinburgh, UK. EH1 2LX > Office Phone: (0)131 650 8920. FAX: (0)131 650 7952 -- Charles Finley, Executive Director Project Open Source | Open Access Knowledge Media Design Institute University of Toronto charles.finley at utoronto.ca Phone: 416.978.3778 http://open.utoronto.ca From harnad at ECS.SOTON.AC.UK Wed Sep 20 07:27:05 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Wed, 20 Sep 2006 12:27:05 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <200609201016.k8KAFvt0003809@dragon.reading.ac.uk> Message-ID: On Wed, 20 Sep 2006, Andrew A. Adams wrote: > What will be the impact amongst UK academics of making their funding (partly) > dependent on their citation rates? I think you may be misunderstanding the finding: UK academics' funding is *already* "dependent" on citation rates, inasmuch as RAE rankings are *already* highly correlated with citation rates -- and this, despite the fact that citation rates are not explicitly counted by RAE panels currently, because RAE specifically forbids it! > It is the "law of unintended consequence" or, alternately, the lack of > consideration about "game playing" that makes the current RAE such a bad > method of deciding funding allocation. Without prejudice as to the reliability or validity of the current RAE's rankings, such as they are, what makes the current RAE's grotesquely onerous submission and panel-review process such a bad method is that it wastes so much of UK researchers' time in submitting and reviewing research that has already been submitted to and reviewed by peer-reviewed journals -- instead of leaving UK researchers that time to do research! Metrics -- already highly correlated with the RAE rankings in all fields tested to date -- will remedy this. > if you tell academics the rules of a > game, then they will generally be quite good at playing that game to win. > This skews the activity of academics away from their proper business unless > you can show that the way to win the game is to do the "correct" things > anyway. The time/money-wasting RAE process has already skewed academics away from their proper business (research and teaching) toward preparing their RAE returns. If what you mean is that metrics will breed abuses, you are right, but it will also breed powerful ways of detecting and deterring those abuses. Open Access metrics are accessible and analysable, openly, and computationally, by all. If someone tampers with the data (excessive self-citation or collaborator circle-citation, or excessive text overlap, or salami-slicing, or robotic download-padding, or tampering with text), that is all detectable, name-and-shameable, and penalizable -- which will all act as a deterrent, especially after the first few culprits are exposed. In contrast, the RAE's needless and profligate paranoia about getting (and licensing!) the exact photocopies of journal articles -- lest authors submit false or doctored texts -- was absolutely absurd, and again, the remedy would have been to consult official databases for confirmation if in doubt about authorship or authenticity, and relying on metrics instead of re-review. (And the ones with the biggest interest in policing against malfeasance are the authors and their own institutions, for they are the ones who stand to lose the most from being named-and-shamed, not the RAE or HEFCE.) > Now, citation counts are derived from worldwide academia, but remember that > various other countries are also considering RAE-style measurements and some > are looking at metrics, while others seem to be mostly following the UK > example so we can assume they may well move to a similar metric-based system > sometime in the future. Agreed. (But it is not clear what your point here is: Heaven forfend that the rest of the world should ape, instead, the bad old panel-RAE!) > It's all very well to claim that without citation-metric-based funding > decisions, that citation metrics mirror existing measurements. But what > evidence is there that this is a robust relationship under the assumption > that we move to a citation-metric measurement and away from the current system? I couldn't follow quite what this meant, except if it was that currently, with panel review merely incidentally correlated with metrics, there is no direct incentive for metric abuse, whereas with direct reliance on metrics, metric abuse is likely to be enhanced. You are quite right about that, but the reply is as above: Metric abuse will also be far more detectable, and punishable, which should provide the necessary counter-incentive. Ceterum censeo: Metrics does not just mean citation counts -- and it definitely does not mean just prior-funding counts! > Note: this discussion may well be veering too far off-topic from OA policy > issues. Slightly. But Open Access to text is inexorably linked to Open Access to the text metrics. Stevan Harnad http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html From eugene.garfield at THOMSON.COM Wed Sep 20 12:47:07 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Wed, 20 Sep 2006 12:47:07 -0400 Subject: Interpeting the medical literature book reivew Message-ID: Select Journal or Resource JAMA & Archives Home JAMA Archives of Dermatology Facial Plastic Surgery Family Medicine (1992-2000) General Psychiatry Internal Medicine Neurology Ophthalmology Otolaryngology-Head & Neck Surgery Pediatrics & Adolescent Medicine Surgery Student JAMA (1998-2004) JAMA & Archives CME Calendar of Events JAMA CareerNet For The Media Users' Guides to the Medical Literature Peer Review Congress Go Institution: ISI | Sign In as Individual Vol. 296 No. 11, September 20, 2006 Featured Link * E-mail Alerts Book and Media Reviews Article Options * Extract * PDF * Send to a Friend * Similar articles in this journal Literature Track * Add to File Drawer * Download to Citation Manager * Articles in PubMed by *Satya-Murti S * Contact me when this article is cited Medical Literature Interpreting the Medical Literature by Stephen H. Gehlbach, 5th ed, 293 pp, with illus, paper, $34.95, ISBN 0-07-143789-4, New York, NY, McGraw-Hill Medical Publishing, 2006. JAMA. 2006;296:1410-1411. Interpreting the Medical Literature guides us in trying to comprehend scholarly medical publications. Journal articles have enormous potential to inform but also to obfuscate. Gehlbach has long been teaching us how to detect bias, confounding, and flawed analysis.1 The past editions of his book were meant "not to daunt tentative and curious readers but to escort them."2 The new fifth edition augments the value of its predecessors by adding examples from recently published journal articles and replacing some earlier discussions. In fact, there are fewer pages, a trend-bucking move for new editions of long-established works. During the past decade, readers have learned the principles of evidence-based medicine (EBM) thanks to the valuable teachings of Sackett,3 Guyatt and Rennie,4 and Jenicek.5 These authors have variously emphasized evidence collection, article evaluation, and logical fallacies. The superb detail of their books, however, could daunt the willing but tentative initiate, for whom Gehlbach's would instead be an ideal starting point. Among the partitions of scientific medical articles, the methods section plays a vital role. Analysis of generated data and interpretive discussion of the findings are equally critical aspects. Familiarity with the underpinning concepts, although often assumed, is not the same as a firm understanding of the fundamentals. In this area Gehlbach's book excels over other currently available texts dealing with clinical epidemiology or EBM.3 -5 Gehlbach begins with chapters detailing study designs: prospective, retrospective, and experimental. He stresses the merits of each depending on the type of inquiry one launches, availability of subjects, funds, patience, and political will. He proceeds to a painless evaluation of statistical methods and their shortcomings. Sections detailing reliability, validity, correlation, diagnostic test performance, and evaluations of treatment and harm follow. Reviews, meta-analyses, reflections, and advice conclude the work. The volume's strength is the inclusion and discussion of abundant examples from published works that have appeared in major journals, from the 1950s through 2005. Postmenopausal hormone therapy, clinical metrics to predict postmenopausal osteoporosis, sepsis in children, benefits of exercise, risks from cyclooxygenase 2 inhibitors, the cost of universal meningococcal vaccines, and many more are foci of discussion. We learn in elegant and clear language the merits and flaws of each study design-eg, prospective, experimental, retrospective case-control-subject and control selection, data analysis, and authors' interpretations of findings. Praise is prominent, and criticism, where due, is piquant. Gehlbach's forte is his ability to dissect effectively major published articles dealing with common topics. He distills the original data, analyzes strengths and faults, and reduces the authors' message to universally understandable denominators. For instance, intake of -tocopherol but not -carotene reduces angina, reaching a significant P value; however, the low magnitude of this effect, less than 2 in 1000 cases per year, is of arguable clinical significance. Olestra intake, contrary to initial impressions, seems no more conducive to gastrointestinal adverse effects than dietary triglycerides when study participants are ingeniously blinded. Numerous similar, important research findings come under explicative scrutiny. Cardiac deaths are lower among pet owners, calling for "perhaps prescriptions for beagles rather than beta blockers." What is the reason for this observed reduction in mortality? Is it the pets themselves or the personality predisposing to pet ownership? Such confounders ("the epidemiologist's eternal triangle") are explored with humor. The mantric aura around null hypotheses is dispelled using a judicial analogy (assume "innocence until guilt is proven" when considering intervention) and illustrative examples. Greenhalgh's How to Read a Paper is another source that approaches the art of reading and interpretation in a similar vein.6 Gehlbach's teaching and writing are skillful. When I find a topic particularly ponderous as covered in other standard works on clinical epidemiology, I often reach for Gehlbach's clarification. Thus, his book leaves one hankering for discussions of certain topics that are missing. These include Bayesian analysis, noninferiority study designs,7 the STARD (Standards for Reporting of Diagnostic Accuracy) initiative,8 and the controversial journal impact factor.9 -10 (He does discuss CONSORT [Consolidated Standards for Reporting of Trials],11 the other standard for improving the quality of reports of randomized controlled trials.) Perhaps Gehlbach might tackle these subjects in future editions. Interpreting the Medical Literature is about understanding the methodology and logic of scientific inquiry. It teaches interpretational skills needed for understanding medical articles. It has a long and deserving tradition of clarity in discussing research designs. Its inimitable merit is the inclusion and analysis of illustrative examples from major published articles. It is light on jargon, sparing in statistical complexity, and solid in emphasis of the fundamentals. Anyone with trepidations about the complexities of epidemiology and EBM must reach for this gentle book for an auspicious start. Financial Disclosures: None reported. Saty Satya-Murti, MD, Reviewer Robert J. Dole Medical Center Wichita, Kan University of Kansas Medical Center Kansas City smurti at cox.net REFERENCES 1. Gehlbach SH. Interpreting the Medical Literature. 4th ed. New York, NY: McGraw-Hill Medical Publishing Division; 2002. 2. Satya-Murti S, reviewer. Review of: Gehlbach SH. Interpreting the Medical Literature. 4th ed. JAMA. 2002;288:3169. FREE FULL TEXT 3. Sackett DL. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd ed. New York, NY: Churchill Livingstone; 2000. 4. Guyatt G, Rennie D. Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice. Chicago, Ill: AMA Press; 2002. 5. Jenicek M. Foundations of Evidence-Based Medicine. New York, NY: Parthenon Publishing Group; 2002. 6. Greenhalgh T. How to Read a Paper: The Basics of Evidence-based Medicine. 3rd ed. Malden, Mass: BMJ Books; 2006. http://bmj.bmjjournals.com/collections/read.shtml. Accessed June 15, 2006. 7. Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ. Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA. 2006;295:1152-1160. FREE FULL TEXT 8. Rennie D. Improving reports of studies of diagnostic tests: the STARD initiative. JAMA. 2003;289:89-90. FREE FULL TEXT 9. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295:90-93. FREE FULL TEXT 10. The PLoS Medicine Editors. The impact factor game: it is time to find a way to assess the scientific literature. PLoS Med. doi:10.1371/journal.pmed.0030291. Accessed June 17, 2006. 2006;3:e291. FULL TEXT | PUBMED 11. CONSORT. The CONSORT statement. http://www.consort-statement.org. Accessibility verified June 27, 2006. Book and Media Reviews Section Editor: Harriet S. Meyer, MD, Contributing Editor, JAMA. HOME | CURRENT ISSUE | PAST ISSUES | COLLECTIONS | CME | CAREERNET | CONTACT US | HELP CONDITIONS OF USE | PRIVACY POLICY (c) 2006 American Medical Association. All Rights Reserved. When responding, please attach my original message __________________________________________________ Eugene Garfield, PhD. email: garfield at codex.cis.upenn.edu home page: www.eugenegarfield.org Tel: 215-243-2205 Fax 215-387-1266 Chairman Emeritus, ISI www.isinet.com 3501 Market Street, Philadelphia, PA 19104-3302 President, The Scientist LLC. www.the-scientist.com 400 Market Street, Suite 1250, Philadelphia, PA 19106-2501 Past President, American Society for Information Science and Technology (ASIS&T) www.asist.org -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 1034 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 73 bytes Desc: image002.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.gif Type: image/gif Size: 73 bytes Desc: image003.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.gif Type: image/gif Size: 73 bytes Desc: image004.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.gif Type: image/gif Size: 1817 bytes Desc: image005.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.gif Type: image/gif Size: 177 bytes Desc: image006.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.gif Type: image/gif Size: 73 bytes Desc: image007.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image008.gif Type: image/gif Size: 73 bytes Desc: image008.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image009.gif Type: image/gif Size: 198 bytes Desc: image009.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image010.gif Type: image/gif Size: 884 bytes Desc: image010.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image011.gif Type: image/gif Size: 73 bytes Desc: image011.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image012.gif Type: image/gif Size: 99 bytes Desc: image012.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image013.gif Type: image/gif Size: 73 bytes Desc: image013.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image014.gif Type: image/gif Size: 170 bytes Desc: image014.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image015.gif Type: image/gif Size: 152 bytes Desc: image015.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image016.gif Type: image/gif Size: 157 bytes Desc: image016.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image017.gif Type: image/gif Size: 92 bytes Desc: image017.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image018.gif Type: image/gif Size: 146 bytes Desc: image018.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image019.gif Type: image/gif Size: 146 bytes Desc: image019.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image020.gif Type: image/gif Size: 99 bytes Desc: image020.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image021.gif Type: image/gif Size: 73 bytes Desc: image021.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image022.gif Type: image/gif Size: 73 bytes Desc: image022.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image023.gif Type: image/gif Size: 73 bytes Desc: image023.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image024.gif Type: image/gif Size: 73 bytes Desc: image024.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image025.gif Type: image/gif Size: 261 bytes Desc: image025.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image026.gif Type: image/gif Size: 43 bytes Desc: image026.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image027.gif Type: image/gif Size: 73 bytes Desc: image027.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image028.gif Type: image/gif Size: 73 bytes Desc: image028.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image029.gif Type: image/gif Size: 84 bytes Desc: image029.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image030.gif Type: image/gif Size: 131 bytes Desc: image030.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image031.gif Type: image/gif Size: 73 bytes Desc: image031.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image032.gif Type: image/gif Size: 73 bytes Desc: image032.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image033.gif Type: image/gif Size: 73 bytes Desc: image033.gif URL: From eugene.garfield at THOMSON.COM Wed Sep 20 13:23:14 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Wed, 20 Sep 2006 13:23:14 -0400 Subject: September issue of Libri on line Message-ID: http://www.librijournal.org/2006-3toc.html go to url for full contents page. Libri VOLUME 56, NUMBER 3, SEPTEMBER 2006 International Journal of Libraries and Information Services Vol 56 (2006), No 3, pages 133-199 ISSN 0024-2667 Table of Contents Winner of LIBRI Best Student Paper Award 2006 Combining Quantitative Methods and Grounded Theory for Researching E-Reverse Auctions ANDREA L?SCH Abstract. Even though many authors claim that e-reverse auctions (e-RAs) are detrimental to the effective building and management of buyer-supplier relationships (Emiliani and Stec 2004), not much is known about how specific characteristics of e-RAs may contribute to such negative effects on buyer-seller relations (Jap 2003). This study sets out not only to provide a first investigation of context, participants' information behaviour, and buyer-supplier relationships in e-RAs, but also to illustrate new methods for theory building in the e-RA and information systems domain. Following a grounded-theory approach, a comprehensive online questionnaire was developed (L?sch and Lambert 2006) based on the critical review of the literature and the results of a preceding exploratory study (L?sch 2005). Usable responses were received from 89 buyers and 54 suppliers, including both users and non-users of e-RAs. The data were analyzed using a novel approach to quantitative analysis based on suggestions by Glaser (1994). The re sults indicate that e-RAs have fewer negative effects on buyer-supplier relationships than currently assumed. They also show how context and the participants' information behaviour correlate with buyer-supplier relationships, thus providing first suggestions for a better management of e-RAs. The paper also thus provides a first illustration of how quantitative methodology might be use fully applied to information systems research, an area which is dominated by the use of qualitative methodology. The Effect of Open Access on Citation Impact: A Comparison Study Based on Web Citation Analysis YANJUN ZHANG Abstract. The academic impact advantage of Open Access (OA) is a prominent topic of debate in the library and publishing communities. Web citations have been proposed as comparable to, even replacements for, bibliographic citations in assessing the academic impact of journals. In our study, we compare Web citations to articles in an OA journal, the Journal of Computer-Mediated Communication (JCMC), and a traditional access journal, New Media & Society (NMS), in the communication discipline. Web citation counts for JCMC are significantly higher than those for NMS. Furthermore, JCMC receives significantly higher Web citations from the formal scholarly publications posted on the Web than NMS does. The types of Web citations for journal articles were also examined. In the Web context, the impact of a journal can be assessed using more than one type of source: citations from scholarly articles, teaching materials and non-authoritative documents. The OA journal has higher percentages of citations from the third type, which suggests that, in addition to the research community, the impact advantage of open access is also detectable among ordinary users participating in Web-based academic communication. Moreover, our study also proves that the OA journal has impact advantage in developing countries. Compared with NMS, JCMC has more Web citations from developing countries. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eugene.garfield at THOMSON.COM Wed Sep 20 13:32:22 2006 From: eugene.garfield at THOMSON.COM (Eugene Garfield) Date: Wed, 20 Sep 2006 13:32:22 -0400 Subject: FW: article on interdisciplinary peer panels fyi (also see otherpapers in this special issue on interdisciplinarity Message-ID: Title: Citation ranking versus peer evaluation of senior faculty research performance: A case study of Kurdish scholarship Author(s): Meho LI , Sonnenwald DH Source: JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 51 (2): 123-138 JAN 15 2000 Document Type: Article Language: English Cited References: 73 Times Cited: 7 Abstract: The purpose of this study is to analyze the relationship between citation ranking and peer evaluation in assessing senior faculty research performance. Other studies typically derive their peer evaluation data directly from referees, often in the form of ranking, This study uses two additional sources of peer evaluation data: citation content analysis and book review content analysis. Two main questions are investigated: (a) To what degree does citation ranking correlate with data from citation content analysis, book reviews, and peer ranking? (b) Is citation ranking a valid evaluative indicator of research performance of senior faculty members? Citation data, book reviews, and peer ranking were compiled and examined for faculty members specializing in Kurdish studies. Analysis shows that normalized citation ranking and citation content analysis data yield identical ranking results. Analysis also shows that normalized citation ranking and citation content analysis, book reviews, and peer ranking perform similarly (i.e., are highly correlated) for high-ranked and low-ranked senior scholars, Additional evaluation methods and measures that take into account the context and content of research appear to be needed to effectively evaluate senior scholars whose performance ranks relatively in the middle. Citation content analysis data did appear to give some specific and important insights into the quality of research of these middle performers, however, further analysis and research is needed to validate this finding. This study shows that citation ranking can provide a valid indicator for comparative evaluation of senior faculty research performance. KeyWords Plus: INFORMATION-SCIENCE RESEARCH; BOOK SELECTION; BASIC RESEARCH; INDICATORS; PSYCHOLOGY; LIBRARY; ART Addresses: Meho LI (reprint author), Univ N Carolina, Sch Lib & Informat Sci, CB 3360,100 Manning Hall, Chapel Hill, NC 27599 USA Univ N Carolina, Sch Lib & Informat Sci, Chapel Hill, NC 27599 USA Publisher: JOHN WILEY & SONS INC, 605 THIRD AVE, NEW YORK, NY 10158-0012 USA Subject Category: COMPUTER SCIENCE, INFORMATION SYSTEMS; INFORMATION SCIENCE & LIBRARY SCIENCE IDS Number: 270AM ISSN: 0002-8231 -----Original Message----- From: Lokman I. Meho [mailto:meho at indiana.edu] Sent: Tuesday, September 19, 2006 6:10 PM To: meho at indiana.edu Subject: RE: article on interdisciplinary peer panels fyi (also see otherpapers in this special issue on interdisciplinarity Attached is: Meho, Lokman I., and Diane H. Sonnenwald. "Citation Ranking Versus Peer Evaluation of Senior Faculty Research Performance: A Case Study of Kurdish Scholarship." Journal of the American Society for Information Science, 51, no. 2 (January 15, 2000): 123-138. Lokman I. Meho, Ph.D. Assistant Professor School of Library and Information Science Indiana University 1320 East 10th Street, LI 011 Bloomington, IN 47405-3907 Tel: (812) 856-2323 Fax: (812) 855-6166 E-mail: meho at indiana.edu http://www.slis.indiana.edu/faculty/meho/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 931 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 345 bytes Desc: image002.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: meho-sonnenwald.pdf Type: application/pdf Size: 171031 bytes Desc: meho-sonnenwald.pdf URL: From garfield at CODEX.CIS.UPENN.EDU Mon Sep 25 15:58:35 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Mon, 25 Sep 2006 15:58:35 -0400 Subject: Gregor M, Schneider O. "The world is watching: Rankings of Czech and Slovak economics departments" FINANCE A UVER-CZECH JOURNAL OF ECONOMICS AND FINANCE 55 (11-12): 518-530 2005 Message-ID: An HTML attachment was scrubbed... URL: From garfield at CODEX.CIS.UPENN.EDU Mon Sep 25 16:02:58 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Mon, 25 Sep 2006 16:02:58 -0400 Subject: Martin Gregor, Schneider O. "Rankings of Czech and Slovak economics departments" FINANCE A UVER-CZECH JOURNAL OF ECONOMICS AND FINANCE 55 (11-12): 518-530 2005 Message-ID: E-mail Addresses: Martin Gregor : gregor at fsv.cuni.cz O. Schneider : schneider at fsv.cuni.cz The authors have kindly made the full text available at : http://ies.fsv.cuni.cz/index.php? module=publication&action=publication&id_publication=1552&lng=en_GB Scroll down and click on Downloadable Article at the bottom of the page Or, go to : http://ies.fsv.cuni.cz/storage/publication/1552_gregor-os.pdf Title: The world is watching: Rankings of Czech and Slovak economics departments Author(s): Gregor M, Schneider O Source: FINANCE A UVER-CZECH JOURNAL OF ECONOMICS AND FINANCE 55 (11-12): 518-530 2005 Document Type: Article Language: English Cited References: 18 Times Cited: 1 Abstract: This paper discusses the methodology of quantitative measures of research output. The authors illustrate various approaches to the contentious issue of to how to treat co-authored papers, how to best affiliate migrating authors, and how to quantify the quality of economic periodicals. The authors also summarize the main findings from the three papers published in this issue, which are devoted to the quantification of publishing activity of Czech and Slovak economists, and which reveal a high concentration of research activity in only a few institutions in the Czech and Slovak republics and the rather convoluted standards of promotion in Czech universities. Addresses: Gregor M (reprint author), Charles Univ, Fac Social Sci, Inst Ecosyst Studies, Prague, Czech Republic Charles Univ, Fac Social Sci, Inst Ecosyst Studies, Prague, Czech Republic E-mail Addresses: gregor at fsv.cuni.cz, schneider at fsv.cuni.cz Publisher: FACULTY SOCIAL SCIENCES CHARLES UNIV, C/O EDITORS OFFICE, VINOHRADSKA 49, PRAGUE 2 120 74, CZECH REPUBLIC Subject Category: BUSINESS, FINANCE IDS Number: 996PJ ISSN: 0015-1920 CITED REFERENCES : BAUWENS L NEW METHOD RANK U RE : 1998 BRAUNINGER M Reputation and relevance of economics journals KYKLOS 56 : 175 2003 CIAIAN P CZECH J EC FINANCE 55 : 547 2005 CONROY ME The productivity of economics departments in the US: Publications in the Core journals JOURNAL OF ECONOMIC LITERATURE 33 : 1966 1995 COUPE T J EUROPEAN EC ASS 1 : 1309 2003 DOLADO JJ SPANISH EC REV 5 : 85 1990 DUSANSKY R Rankings of US economics departments JOURNAL OF ECONOMIC PERSPECTIVES 12 : 157 1998 FARIA JR RES OUTPUT ACAD EC B : 2000 GRAVES PE ECONOMICS DEPARTMENTAL RANKINGS - RESEARCH INCENTIVES, CONSTRAINTS, AND EFFICIENCY AMERICAN ECONOMIC REVIEW 72 : 1131 1982 GUIMARAES P PORTUGUESE EC J : 3 2002 HIX S POLITICAL STUDIES : 293 2004 KALAITZIDAKIS P J EUROPEAN EC ASS 1 : 1346 2003 KING I NZ EC PAPERS 36 : 97 2002 LUBRANO M J EUROPEAN EC ASS 1 : 1367 2003 MACHACEK M CZECH J EC FINANCE 55 : 564 2005 NEARY JP J EUROPEAN EC ASS 1 : 1239 2003 SINHA D Rankings of Australian economics departments, 1988-2000 ECONOMIC RECORD 78 : 136 2002 TURNOVEC F CZECH J EC FINANCE 55 : 531 2005 From michele.catanzaro at TIN.IT Tue Sep 26 10:40:35 2006 From: michele.catanzaro at TIN.IT (michele.catanzaro@tin.it) Date: Tue, 26 Sep 2006 15:40:35 +0100 Subject: network visualization linux Message-ID: Hallo, I would be most grateful if somebody of you could tell me if there is any program for network visualiztion I can istall in a linux environment (something like a pajek for Linux). Indeed, I need a quite simple program, for small to medium networks. I tried Osprey, but it asks an input with a formar I can't reproduce. Thank you in advance, Michele From loet at LEYDESDORFF.NET Tue Sep 26 14:20:33 2006 From: loet at LEYDESDORFF.NET (Loet Leydesdorff) Date: Tue, 26 Sep 2006 20:20:33 +0200 Subject: The Knowledge-Based Economy: Modeled, Measured, Simulated Message-ID: The Knowledge-Based Economy: Modeled, Measured, Simulated; 385 pp., US$ 18.95 ?Challenging, theoretically rich yet anchored in detailed empirical analysis, Loet Leydesdorff?s exploration of the dynamics of the knowledge-economy is a major contribution to the field. Drawing on his expertise in science and technology studies, systems theory, and his internationally respected work on the ?triple helix?, the book provides a radically new modelling and simulation of knowledge systems, capturing the articulation of structure, communication, and agency therein. This work will be of immense interest to both theorists of the knowledge-economy and practitioners in science policy.? Andrew Webster Science & Technology Studies, University of York, UK _____ ?This book is a ground-breaking collection of theory and techniques to help understand the internal dynamics of the modern knowledge-based economy, including issues such as stability, anticipation, and interactions amongst components. The combination of theory, measurement, and modelling gives the necessary power with which to address the complexity of modern networked social systems. Each on its own would partly illuminate an innovation system, but the combination sheds a far brighter light.? Mike Thelwall Information Science, University of Wolverhampton, UK _____ ?The sociologist Niklas Luhmann is considered one of the few social scientists possibly able to explain a decisive event once it has happened. In this book, Loet Leydesdorff answers the challenge to take Luhmann?s analysis one step further by introducing anticipation into the theory. This book provides a fascinating exploration of the use of recursion and incursion to model social processes.? Dirk Baecker Sociology, Universit?t Witten/Herdecke, Germany _____ Loet Leydesdorff Amsterdam School of Communications Research (ASCoR) Kloveniersburgwal 48, 1012 CX Amsterdam Tel.: +31-20- 525 6598; fax: +31-20- 525 3681 loet at leydesdorff.net ; http://www.leydesdorff.net/ NEW: The Knowledge-Based Economy: Modeled, Measured, Simulated. The Self-Organization of the Knowledge-Based Society; The Challenge of Scientometrics -------------- next part -------------- An HTML attachment was scrubbed... URL: From garfield at CODEX.CIS.UPENN.EDU Wed Sep 27 14:07:52 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Wed, 27 Sep 2006 14:07:52 -0400 Subject: Picoli S, Mendes RS, Malacarne LC, Lenzi EK "Scaling behavior in the dynamics of citations to scientific journals" EUROPHYSICS LETTERS 75 (4): 673-679 AUG 2006 Message-ID: full text available at : http://www.edpsciences.org/articles/epl/pdf/2006/16/epl9502.pdf Title: Scaling behavior in the dynamics of citations to scientific journals Author(s): Picoli S, Mendes RS, Malacarne LC, Lenzi EK Source: EUROPHYSICS LETTERS 75 (4): 673-679 AUG 2006 Document Type: Article Language: English Cited References: 37 Times Cited: 0 Abstract: We analyze a database comprising the impact factor (citations per recent items published) of scientific journals for a 13-year period (1992-2004). We find that i) the distribution of impact factors follows asymptotic power law behavior, ii) the distribution of annual logarithmic growth rates has an exponential form, and iii) the width of this distribution decays with the impact factor as a power law with exponent beta similar or equal to 0.22. The results ii) and iii) are surprising similar to those observed in the growth dynamics of organizations with complex internal structure suggesting the existence of common mechanisms underlying the dynamics of these systems. We propose a general model for such systems, an extension of the simplest model for firm growth, and compare their predictions with our empirical results. KeyWords Plus: GROWTH DYNAMICS; SCIENCE; LAW; NETWORKS Addresses: Picoli S (reprint author), Univ Estadual Maringa, Dept Fis, Ave Colombo 5790, Maringa, Parana BR-87020900 Brazil Univ Estadual Maringa, Dept Fis, Maringa, Parana BR-87020900 Brazil Publisher: EDP SCIENCES S A, 17, AVE DU HOGGAR, PA COURTABOEUF, BP 112, F- 91944 LES ULIS CEDEX A, FRANCE IDS Number: 070EN ISSN: 0295-5075 CITED REFERENCES : *ASS COMM SCI LIB LIB ASS : 1968 AMARAL LAN Power law scaling for a system of interacting units with complex internal structure PHYSICAL REVIEW LETTERS 80 : 1385 1998 BARABASI AL Emergence of scaling in random networks SCIENCE 286 : 509 1999 COLE JR ORTEGA HYPOTHESIS SCIENCE 178 : 368 1972 DUNNE T J ECON 104 : 671 1989 FU DF The growth of business firms: Theoretical framework and empirical evidence PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA 102 : 18801 2005 GARFIELD E NEW FACTORS IN EVALUATION OF SCIENTIFIC LITERATURE THROUGH CITATION INDEXING AMERICAN DOCUMENTATION 14 : 195 1963 GARFIELD E CITATION ANALYSIS AS A TOOL IN JOURNAL EVALUATION - JOURNALS CAN BE RANKED BY FREQUENCY AND IMPACT OF CITATIONS FOR SCIENCE POLICY STUDIES SCIENCE 178 : 471 1972 GARFIELD E CITATION INDEXES FOR SCIENCE - NEW DIMENSION IN DOCUMENTATION THROUGH ASSOCIATION OF IDEAS SCIENCE 122 : 108 1955 GELLMANN M NONEXTENSIVE ENTROPY : 2004 GIBRAT R INEQALITES EC : 1931 GOFFMAN W BRADFORDS LAW AND LIBRARY ACQUISITIONS NATURE 226 : 922 1970 HUBERMAN BA Strong regularities in World Wide Web surfing SCIENCE 280 : 95 1998 LEE YK Universal features in the growth dynamics of complex organizations PHYSICAL REVIEW LETTERS 81 : 3275 1998 LOTKA AJ J WASHINGTON ACADEMY 16 : 317 1926 LYRA ML Generalized Zipf's law in proportional voting processes EUROPHYSICS LETTERS 62 : 131 2003 MANTEGNA RN SCALING BEHAVIOR IN THE DYNAMICS OF AN ECONOMIC INDEX NATURE 376 : 46 1995 NARANAN S POWER LAW RELATIONS IN SCIENCE BIBLIOGRAPHY - SELF-CONSISTENT INTERPRETATION JOURNAL OF DOCUMENTATION 27 : 83 1971 NARANAN S BRADFORDS LAW OF BIBLIOGRAPHY OF SCIENCE - AN INTERPRETATION NATURE 227 : 631 1970 OLAMI Z SELF-ORGANIZED CRITICALITY IN A CONTINUOUS, NONCONSERVATIVE CELLULAR AUTOMATON MODELING EARTHQUAKES PHYSICAL REVIEW LETTERS 68 : 1244 1992 PENG CK LONG-RANGE ANTICORRELATIONS AND NON-GAUSSIAN BEHAVIOR OF THE HEARTBEAT PHYSICAL REVIEW LETTERS 70 : 1343 1993 PICOLI S Statistical properties of the circulation of magazines and newspapers EUROPHYSICS LETTERS 72 : 865 2005 PLEROU V Similarities between the growth dynamics of university research and of competitive economic activities NATURE 400 : 433 1999 PRICE DJD GENERAL THEORY OF BIBLIOMETRIC AND OTHER CUMULATIVE ADVANTAGE PROCESSES JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 27 : 292 1976 PRICE DJD NETWORKS OF SCIENTIFIC PAPERS SCIENCE 149 : 510 1965 REDNER S How popular is your paper? An empirical study of the citation distribution EUROPEAN PHYSICAL JOURNAL B 4 : 131 1998 SEGLEN PO THE SKEWNESS OF SCIENCE JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE 43 : 628 1992 SOARES DJB Preferential attachment growth model and nonextensive statistical mechanics EUROPHYSICS LETTERS 70 : 70 2005 SORNETTE D CRITICAL PHENOMENA N : 2004 SORNETTE D Endogenous versus exogenous shocks in complex networks: An empirical test using book sale rankings PHYSICAL REVIEW LETTERS 93 : Art. No. 228701 2004 SORNETTE D Linear stochastic dynamics with nonlinear fractal properties PHYSICA A 250 : 295 1998 STANLEY MHR Scaling behaviour in the growth of companies NATURE 379 : 804 1996 THURNER S Nonextensive aspects of self-organized scale-free gas-like networks EUROPHYSICS LETTERS 72 : 197 2005 TSALLIS C Are citations of scientific papers a case of nonextensivity? EUROPEAN PHYSICAL JOURNAL B 13 : 777 2000 TSALLIS C POSSIBLE GENERALIZATION OF BOLTZMANN-GIBBS STATISTICS JOURNAL OF STATISTICAL PHYSICS 52 : 479 1988 VICSEK T FRACTAL GROWTH PHENO : 1992 WHITE DR PHYS REV E 73 : 2005 From garfield at CODEX.CIS.UPENN.EDU Wed Sep 27 14:25:33 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Wed, 27 Sep 2006 14:25:33 -0400 Subject: Foot MM "The study of books" ASLIB PROCEEDINGS 58 (1-2): 20-33 2006 Message-ID: Mirjam M. Foot : E-mail Addresses: m.foot at ucl.ac.uk Title: The study of books Author(s): Foot MM Source: ASLIB PROCEEDINGS 58 (1-2): 20-33 2006 Document Type: Article Language: English Cited References: 42 Times Cited: 0 Abstract: Purpose - This paper aims to show how the concept of "Bibliography" has changed since the late Accepted 21 September nineteenth century. It proposes discussing what "Bibliography" did and did not include in the various 2005 stages of its development; how the study of "Bibliography" moved from the UK to the USA; how it narrowed down from an originally much wider concept and how, under the influence of French historians over the past three decades, it has widened out again, reaching a better synthesis of the study of books as material objects with the study of the history of the book. Design/methodology/approach - A discussion and critical assessment of the writings of the major main stream bibliographers and book historians is presented. Findings - From an original (nineteenth century) emphasis on enumerative bibliography, the concept of "Bibliography" widened out (from the end of the nineteenth century) to include historical bibliography and the study of books as material objects; in the mid-twentieth century this wider approach narrowed down, as a consequence of much emphasis being placed on descriptive, analytical, critical and textual bibliography. Under influence of French book historians the emphasis has changed again and the value of a wider historical approach and greater inclusivity in subjects has brought the study of historical bibliography and that of the history of the book much closer together. Research limitations/implications - This research looks only at Western Europe and the USA. Practical implications - Practical implications of this study are: the widening-out of the subject to include all physical manifestations of the book; the dimension of creative reading; and the emphasis on the importance of artifactual evidence for correct establishment and interpretation of texts has had implications for preservation. Originality/value - This paper is a critical assessment of the literature, drawing the logical consequences of its findings. It presents an argument for the inclusion of all aspects of the book as a physical object, as well as for the importance of using all available evidence. Addresses: Foot MM (reprint author), Univ Coll London, Sch Lib Arch & Informat Studies, London, England Univ Coll London, Sch Lib Arch & Informat Studies, London, England E-mail Addresses: m.foot at ucl.ac.uk Publisher: EMERALD GROUP PUBLISHING LIMITED, 60/62 TOLLER LANE, BRADFORD BD8 9BY, W YORKSHIRE, ENGLAND Subject Category: COMPUTER SCIENCE, INFORMATION SYSTEMS; INFORMATION SCIENCE & LIBRARY SCIENCE IDS Number: 048IH ISSN: 0001-253X CITED REFERENCES: ADAMS RG LIB Q 7 : 317 1937 ALSTON RC BIBLIO COMPUTERS RES : 1990 BOWERS F BIBLIO MODERN LIB : 1966 BOWERS F LIBRARY 8 : 1 1953 BOWERS F PRINCIPLES BIBLIO DE : 1994 BROWN JD LIBRARY 4 : 144 1903 CARTER J 19 CENTURY ENGLISH B : 1952 COLE GW 1850 1935 : 165 1990 COLE GW PAPERS BIBLIO SOC AM 14 : 1 1920 COLE GW PAPERS BIBLIO SOC AM 10 : 119 1916 DARNTON R BOOKS SOC HIST : 3 1983 DAVISON P BOOK ENCOMPASSED : 1992 GREG WW LIBRARY 13 : 113 1932 GREG WW LIBRARY 11 : 241 1930 GREG WW NEOPHILOLOGUS 18 : 1933 GREG WW STUDIES RETOSPECT : 23 1945 GREG WW T BIBLIO SOC 12 : 39 1912 HELLINGA WG COPY PRINT NETHERLAN : 1962 MCGANN JM CRITIQUE MODERN TEXT : 1983 MCKENZIE DF BIBL SOC CENT LECT 1 : 1993 MCKENZIE DF BIBLIO SOC TEXTS : 1986 MCKENZIE DF BOOK ENCOMPASSED : 290 1992 MCKENZIE DF HIST BOOK BRIT 3 : 1999 MCKENZIE DF STUDIES BIBLIO 22 : 1 1969 MCKERROW RB INTRO BIBLIO LIT STU : 1994 NEEDHAM P BRADSHAW METHOD : 1988 PERRY R EMBODIED KNOWLEDGE + THE MATERIAL OBJECT THAT ONCE EXISTED IN A SOCIAL- CONTEXT RATHER THAN FOR INFORMATION THAT CAN BE ABSTRACTED AND REPRODUCED IN ANY FORM HARVARD LIBRARY BULLETIN 4 : 57 1993 POLLARD AW COMMENTARIES PHYS BO : 68 1960 POLLARD AW COMMUNICATION 0823 : 1932 POLLARD AW LIBRARY 13 : 255 1932 RAABE P BOOKS SOC HIST : 251 1983 SADLIER M STUDIES RETROSPECT : 146 1945 STODDARD RE PAPERS BIBLIO SOC AM 94 : 2000 TANSELLE GT F BOWERS 80 : 3 1985 TANSELLE GT HIST BOOKS FIELD STU : 1981 TANSELLE GT LIT ARTIFACTS : 1 1998 TANSELLE GT PRINCIPLES BIBLIO DE : 1994 TANSELLE GT STUD BIBLIOGR 27 : 55 1974 WILSON FP STUDIES RETROSPECT : 76 1945 WILSON JD P BRIT AC 30 : 1948 WILSON JD WHAT HAPPENS HAMLET : 1935 ZICH R RES COLLECTIONS INFO : 10 1990 From garfield at CODEX.CIS.UPENN.EDU Wed Sep 27 14:34:30 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Wed, 27 Sep 2006 14:34:30 -0400 Subject: Aphinyanaphongs Y, Statnikov A, Aliferis CF "A comparison of citation metrics to machine learning filters for the identification of high quality MEDLINE documents " Journal of the American Medical Informatics Association 13(4):446-455 July-August 2006. Message-ID: E-mail Addresses:C.F. Aliferis : constantin.aliferis at vanderbilt.edu Title: A comparison of citation metrics to machine learning filters for the identification of high quality MEDLINE documents Author(s): Aphinyanaphongs Y, Statnikov A, Aliferis CF Source: JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 13 (4): 446- 455 JUL-AUG 2006 Document Type: Article Language: English Cited References: 37 Times Cited: 0 Abstract: objective: The present study explores the discriminatory performance of existing and novel gold-standard-specific machine learning (GSS-ML) focused filter models (i.e., models built specifically for a retrieval task and a gold standard against which they ate evaluated) and compares their performance to citation count and impact factors, and non-specific machine learning (NS-ML) models (i.e., models built for a different task and/or different gold standard). Design: Three gold standard corpora were constructed using the SSOAB bibliography, the ACPJ-cited treatment articles, and the ACPJ-cited etiology articles. Citation counts and impact factors were obtained for each article. Support vector machine models were used to classify the articles using combinations of content, impact factors, and citation counts as predictors. Measurements: Discriminatory performance was estimated using the area under the receiver operating characteristic curve and n-fold cross-validation. Results: For all three gold standards and tasks, GSS-ML filters outperformed citation count, impact factors, and NS-ML filters. Combinations of content with impact factor or citation count produced no or negligible improvements to the GSS machine learning filters. Conclusions: These experiments provide evidence that when building information retrieval filters focused on a retrieval task and corresponding gold standard, the filter models have to be built specifically for this task and gold standard. Under those conditions, machine learning filters outperform standard citation metrics. Furthermore, citation counts and impact factors add marginal value to discriminatory performance. Previous research that claimed better performance of citation metrics than machine learning in one of the corpora examined here is attributed to using machine learning filters built for a different gold standard and task. KeyWords Plus: DETECTING CLINICALLY SOUND; OPTIMAL SEARCH STRATEGIES; TEXT CATEGORIZATION; RETRIEVAL Addresses: Aliferis CF (reprint author), Vanderbilt Univ, Dept Biomed Informat, Eskind Biomed Lib, Discovery Syst Lab, Room 412,2209 Garland Ave, Nashville, TN 37232 USA Vanderbilt Univ, Dept Biomed Informat, Eskind Biomed Lib, Discovery Syst Lab, Nashville, TN 37232 USA E-mail Addresses: constantin.aliferis at vanderbilt.edu Publisher: ELSEVIER SCIENCE INC, 360 PARK AVE SOUTH, NEW YORK, NY 10010- 1710 USA Subject Category: COMPUTER SCIENCE, INFORMATION SYSTEMS; COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS; INFORMATION SCIENCE & LIBRARY SCIENCE; MEDICAL INFORMATICS IDS Number: 064EU ISSN: 1067-5027 CITED REFERENCES : ACP J 131 : A15 1999 LIBSVM LIB SUPPORT V : 2005 PUBMED : 2005 ALIFERIS C P AMIA S WASH DC : 2003 ALIFERIS CF METMBS : 371 1908 APHINYANAPHONGS Y Text categorization models for high-quality article retrieval in internal medicine JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 12 : 207 2005 APHINYANAPHONGS Y MEDINFO : 2004 BAEZAYATES R MODERN INFORMATION R : 1999 BERNSTAM EV J AM MED INFORM ASS : 2005 DELONG ER COMPARING THE AREAS UNDER 2 OR MORE CORRELATED RECEIVER OPERATING CHARACTERISTIC CURVES - A NONPARAMETRIC APPROACH BIOMETRICS 44 : 837 1988 DUDA S AMIA S WASH D C : 2005 DUDOIT S 126 UC BERK DIV BIOS : 2003 DUMAIS S P ACM CIKM98 NOV : 1998 FAWCETT T HPL20034 : 2003 GARFIELD E CAN CITATION INDEXIN : 1965 GARFIELD E INT J CLIN HLTH PSYC 3 : 363 2003 GARFIELD E SCI PUBL POLICY 19 : 321 1992 GUYON I Gene selection for cancer classification using support vector machines MACHINE LEARNING 46 : 389 2002 HAND DJ A simple generalisation of the area under the ROC curve for multiple class classification problems MACHINE LEARNING 45 : 171 2001 HAYNES RB DEVELOPING OPTIMAL SEARCH STRATEGIES FOR DETECTING CLINICALLY SOUND STUDIES IN MEDLINE JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 1 : 447 1994 HSU CW PRACTICAL GUIDE SUPP : 2005 JENKINS M HLTH INFO LIB J 21 : 148 2004 JOACHIMS T LEARNING CLASSIFY TE : 2002 KLEINBERG P ACM SIAM S DISCR A : 1997 LEOPOLD E Text categorization with support vector machines. How to represent texts in input space ? MACHINE LEARNING 46 : 423 2002 PAGANO M PRINCIPLES BIOSTATIS : 2000 PAGE L PAGERANK CITATION RA : 1998 PORTER MF AN ALGORITHM FOR SUFFIX STRIPPING PROGRAM-AUTOMATED LIBRARY AND INFORMATION SYSTEMS 14 : 130 1980 PROVOST F ICML 98 15 INT C MAC : 1998 SALTON G TERM-WEIGHTING APPROACHES IN AUTOMATIC TEXT RETRIEVAL INFORMATION PROCESSING & MANAGEMENT 24 : 513 1988 SCHEFFER T ERROR ESTIMATION MOD : 1999 SUN A ICDM : 2001 TSAMARDINOS I AI STAT : 2003 VAPNIK V STAT LEARNING THEORY : 1998 WEISS S COMPUTER SYSTEMS LEA : 1991 WILCZYNSKI NL Optimal search strategies for detecting clinically sound prognostic studies in EMBASE: An analytic survey JOURNAL OF THE AMERICAN MEDICAL INFORMATICS ASSOCIATION 12 : 481 2005 YANG Y 22 ANN ACM C RES DEV : 1999 From garfield at CODEX.CIS.UPENN.EDU Wed Sep 27 14:51:01 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Wed, 27 Sep 2006 14:51:01 -0400 Subject: Dimitrova DV, Bugeja M. "Consider the source: Predictors of online citation permanence in communication journals " Portal-Libraries and the Academy 6(3): 269-283 July 2006 Message-ID: E-mail Addresses: danielad at iastate.edu, bugeja at iastate.edu Title: Consider the source: Predictors of online citation permanence in communication journals Author(s): Dimitrova DV, Bugeja M Source: PORTAL-LIBRARIES AND THE ACADEMY 6 (3): 269-283 JUL 2006 Document Type: Article Language: English Cited References: 22 Times Cited: 0 Abstract: This study focuses on six leading communication journals and their use of online citations in articles published between 2000 and 2003. The study uses content analysis to explore if there is a relationship between citation characteristics and their stability. The findings show that online citations in the .gov and .org domains are more likely to remain accessible over time. Year of publication and URL level also emerged as significant predictors of online citation permanence. More than 37 percent of the online citations have disappeared from the original source over a four-year period (2000-2003). The implications of these findings are discussed in the context of reliability and replicability of scholarship. Addresses: Dimitrova DV (reprint author), Iowa State Univ, Greenlee Sch Journalism & Commun, Ames, IA USA Iowa State Univ, Greenlee Sch Journalism & Commun, Ames, IA USA E-mail Addresses: danielad at iastate.edu, bugeja at iastate.edu Publisher: JOHNS HOPKINS UNIV PRESS, JOURNALS PUBLISHING DIVISION, 2715 NORTH CHARLES ST, BALTIMORE, MD 21218-4363 USA Subject Category: INFORMATION SCIENCE & LIBRARY SCIENCE IDS Number: 069AG ISSN: 1531-2542 CITED REFERENCES : BANKS MA BIOMEDICAL DIGITAL L 2 : 1 2005 BIZZELL P RHETORICAL TRADITION : 504 1990 BUGEJA INFORM RES 9 : 2004 BUGEJA M INSIDE HIGHER E 0422 : 2005 BUGEJA M IOWA J COMMUNICATION 37 : 77 2005 BURKE K RHETORIC MOTIVES : 1969 DAVIS PM The effect of the Web on undergraduate citation behavior 1996-1999 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 52 : 309 2001 DAVIS PM Effect of the web on undergraduate citation behavior: Guiding student scholarship in a networked age PORTAL-LIBRARIES AND THE ACADEMY 3 : 41 2003 DIMITROVA IN PRESS NEW MEDIA S GRAFTON A FOOTNOTE CURIOUS HIS : 1997 GROSS L IMAGE ETHICS DIGITAL : 19 2003 JENKINS PO COLL RES LIB NEWS 63 : 164 2002 KUSHKOWSKI JD Web citation by graduate students: A comparison of print and electronic theses PORTAL-LIBRARIES AND THE ACADEMY 5 : 259 2005 LEDERMAN D INSIDE HIGHER E 1013 : 2005 MARKWELL J "Link rot" limits the usefulness of web-based educational materials in biochemistry and molecular biology BIOCHEMISTRY AND MOLECULAR BIOLOGY EDUCATION 31 : 69 2003 MEEHAN B SLIS NETWORK : 10 2005 NYBERG S CITE CHECKING LIB RE RUMSEY M Runaway train: Problems of permanence, accessibility, and stability in the use of Web sources in law review citations LAW LIBRARY JOURNAL 94 : 27 2002 SELLITTO C The impact of impermanent web-located citations: A study of 123 scholarly conference publications JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 56 : 695 2005 SMITH A INFORM RES 9 : 4 2004 TAYLOR MK "Linkrot" and the usefulness of Web site bibliographies REFERENCE & USER SERVICES QUARTERLY 39 : 273 2000 TYLER DC Librarians and link rot: Comparative analysis with some methodological considerations PORTAL-LIBRARIES AND THE ACADEMY 3 : 615 2003 From garfield at CODEX.CIS.UPENN.EDU Wed Sep 27 14:58:31 2006 From: garfield at CODEX.CIS.UPENN.EDU (=?windows-1252?Q?Eugene_Garfield?=) Date: Wed, 27 Sep 2006 14:58:31 -0400 Subject: Abt HA "An anomalous journal impact factor" ASTRONOMISCHE NACHRICHTEN 327 (7): 737-738 2006 Message-ID: E-mail Addresses: Helmut Abt : abt at noao.edu Title: An anomalous journal impact factor Author(s): Abt HA Source: ASTRONOMISCHE NACHRICHTEN 327 (7): 737-738 2006 Document Type: Article Language: English Cited References: 2 Times Cited: 0 Abstract: The impact factor (average number of citations per paper) for the Astrophysical Journal Supplement Series jumped between 2003 and 2004 from 6.247 to 15.231, giving it the world's second highest impact factor for an astronomical journal in 2004. Was this change due to a computing error or to an unusual occurrence? It is shown that it was due to the extremely high citation rates (average of 160 citations per year) for 13 papers in the special issue devoted to the Wilkinson Microwave Anisotropy Probe. (c) 2006 WILEY-VCH Verlag GmbH &Co. KGaA, Weinheim. Addresses: Abt HA (reprint author), Kitt Peak Natl Observ, Box 26732, Tucson, AZ 85726 USA Kitt Peak Natl Observ, Tucson, AZ 85726 USA E-mail Addresses: abt at noao.edu Publisher: WILEY-V C H VERLAG GMBH, PO BOX 10 11 61, D-69451 WEINHEIM, GERMANY Subject Category: ASTRONOMY & ASTROPHYSICS IDS Number: 074ZX ISSN: 0004-6337 CITED REFERENCES : ABT HA B AAS 36 : 576 2004 LEQUEX J ASTRON ASTROPHYS 235 : E1 1990 -------------- next part -------------- A non-text attachment was scrubbed... Name: D:\MMistry\Desktop\Impact Factor.pdf Type: application/pdf Size: 371322 bytes Desc: not available URL: From harnad at ECS.SOTON.AC.UK Fri Sep 29 05:25:32 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Fri, 29 Sep 2006 10:25:32 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <080f01c6e3a4$37651170$0400000a@AMILO> Message-ID: On Fri, 29 Sep 2006, Sally Morris (Chief Executive) wrote: > I disagree with Stevan about the advisability of limiting the number of > papers which may be submitted. Similar rules were introduced a few years > ago in the US - the objective is to discourage authors from a 'never mind > the quality, feel the width' mentality which can (and arguably does) lead to > excessive publication, via salami-slicing and other (sometimes even less > desirable - see http://www.alpsp.org/events/2006/PET/default.htm) methods Sally (and no doubt many others) vastly under-rate the power of OA metrics here: (1) Yes salami-slicing is bad. (2) But metrics makes it easily detectable, and penalizable, by differential weighting. (3) Example: Researcher A receives a total of 100 citations for 10 papers, averaging 10 per paper; Researcher B receives a total of 100 citations for 4 papers, averaging 25 per paper. Easy to give the lower average a lower weight; as a sub-test, easy to check the citations for the top four papers too... (4) The main idea is to stop wasting time and money re-submitting the papers (to RAE) and re-reviewing them (by RAE panels) and let the metrics do the work instead. (5) Nor are citations count and averages and top-slicing near being the only metrics that can enter into the weighted equation: There are downloads, co-citations (what kind of research/researcher is it cited *with*), authority metrics (what kind of research/researcher is it cited *by*), endogamy/exogamy metrics (how incestuous are the citations, in the range: self-citations, co-author citations, mutual citation circles, within-specialty citations, interdisciplinarity), growth rate of citations (and downloads), latency and longevity scores, etc. (6) All of those metrics can be gathered and weighted, with the weights adjusted to the features of the field (some fields are rapid, narrow growth, some are slower, broader growth, some more endogamous, some more exogamous, etc.). (7) "Semantic" (in reality syntactic) content-based metrics can also measure the degree of textual overlap between papers, both multiple papers by the same author, and overlap with papers by other authors... (8) And that just scratches the surface of OA metric possibilities. My own understanding is that restricting RAE submissions to only 4 papers had been done partly to keep the work of the panel tractable (mooted now with metrics) and partly to discourage salami-slicing -- but I know of no evidence whether it *did* discourage salami-slicing (in either the UK or the US: does anyone have data?): After all, RAE is not the only fish in the sea, for the author. (Objective evidence on whether it had any effect, by the way, would have to be metric!) But in a metric RAE, salami-slicing would become its own enemy, just as self-citing and plagiarism would be. (And before you mention self-padded downloads, that's readily detectable and name-shameable too, not only by checking IPs but via triangulation with other metrics that are normally correlated with downloads, such as citations!) Stevan Harnad PS Whatever works in the UK, the US will eventually catch up too: Metrics will be the measure in both cases, validated, as needed, against peer evaluation, the specific needs of a field, and internal validation through triangulation. > ----- Original Message ----- > From: "Stevan Harnad" > > > > At the heart of this are not one, not two, not three, but *four* pieces > > of patent nonsense so absurd as to take one's breath away. Most of the > > nonsense is on RAE/HEFCE's end; one cannot blame the publishers for play > > along (especially as the gentleman's agreement holds some hope of > > forestalling OA a bit longer, or at least the role the RAE might have > > played in hastening OA's arrival): > > > > (1) The first piece of nonsense is the RAE's pedantic and > > dysfunctional insistence on laying their hands directly on the > > "originals," the publisher's version of each article per author, > > rather than sensibly settling for the author's peer-reviewed final > > drafts (postprints). > > > > (2) The second is the equally foolish notion that the RAE somehow > > needs special permission to do this, or, worse, might even have needed > > to *pay* for the right, but for this "gentleman's agreement"! (Of > > course the publishers are more than happy to play along with this > > self-imposed farce on RAE's part; but if no one had ever absurdly > > suggested in the first place that when an author sends a copy of his > > own paper to his own funder for evaluation, *he needs his publisher's > > permission*, none of this nonsense would ever even have come up!) > > > > (3) The idea of restricting submissions to only *four* papers > > was originally floated by RAE in part out of the hope that > > this limitation would act as a counterweight against salami-sliced > > publication. It didn't. And it's time to drop this absurd, arbitrary > > limit on what work can be submitted. > > > > (4) Of course the other reason the number was kept down to four was > > the even more dysfunctional feature of the RAE that is only now, > > at long last, being deservedly jettisoned (the submissions and panel > > reviews themselves!); yet one hand does not seem to be aware of what > > the other is doing: For once the unnecessary and time/money-wasting > > "peer-*re*-reviewing" that the RAE panels had been trying to > > do is at last abandoned in favour of metrics, there will be no > > need for either a 4-item cap or any compulsive attempt to get the > > "originals" to the panel. The authors' self-archived postprints > > in their own institutional OA IRs will suffice (and the only thing > > the RAE panels -- if there still *are* any RAE panels -- need do, > > if suspicious about any particular item, is a database search (say, > > in Web of Knowledge or Scopus or PubMed) to make sure that the item > > in question did indeed appear in the journal indicated, under the > > name of the author in question). > > > > What will moot all of this is of course the OA self-archiving mandates > > by RCUK and the UK universities themselves, which will fill the UK > > universities' IRs, which will in their turn -- with the help of the IRRA > > http://irra.eprints.org/ -- mediate the submission of both the postprints > > and the metrics to the RAE. Then this ludicrous side-show about the > > "licensing" of the all-important "originals" to the RAE, for "peer > > re-review" via the mediation of CrossRef and the publishers will at last > > be laid to rest, once and for all. > > > > RAE 2008 will be its last hurrah... > > > > Stevan Harnad > > > > ______________________________________________________________________ > > This email has been scanned by the MessageLabs Email Security System. > > For more information please visit http://www.messagelabs.com/email > > ______________________________________________________________________ > > From harnad at ECS.SOTON.AC.UK Fri Sep 29 09:51:50 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Fri, 29 Sep 2006 14:51:50 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: <7.0.1.0.2.20060929133606.022c2640@dsl.pipex.com> Message-ID: Prior Amsci Topic Threads: "UK Research Assessment Exercise (RAE) review" (Oct 2002) http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#2326 "Future UK RAEs to be Metrics-Based" (Mar 2006) http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/subject.html#5251 "Let 1000 RAE Metric Flowers Bloom: Avoid Matthew Effect as Self-Fulfilling Prophecy" (June 2006) http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/5418.html Re: http://www.universitiesuk.ac.uk/mediareleases/show.asp?MR=469 http://www.russellgroup.ac.uk/news.html http://education.guardian.co.uk/RAE/story/0,,1882755,00.html http://www.egovmonitor.com/node/7841 http://www.ucl.ac.uk/news/news-articles/0609/06092604 The UUK and the universities are spot-on in their criticism of the replacement of the old panel RAE by *one metric* (prior research funding). That would be *absurd* and extremely unfair, counterproductive and arbitrary. But that has next to nothing to do with replacing the RAE's tremendously wasteful panel-based exercise by *metricS* (plural), which include a rich array of objective performance indicators rather than just one self-fulfilling prophecy (prior funding). UUK and the universities are also quite right that the metrics need to be tested and validated, field by field (some already have been), and that the metric formula and the weights for each of the metrics have to be adjusted to each discipline. Whether, and if so how much, panel review will still be needed in some disciplines once the metric formula has been tested and optimised is an empirical question (but my own guess is: not much). Stevan Harnad http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html From dgoodman at PRINCETON.EDU Sat Sep 30 01:09:27 2006 From: dgoodman at PRINCETON.EDU (David Goodman) Date: Sat, 30 Sep 2006 01:09:27 -0400 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: Problem is , Stevan, that though you ought to be right it does not always work out that way. If X publishes a project chopped up into 4 short papers at about the same time, if I'm working in the field I'll cite them all. It is usual in the US for a tenure committee in a first rate place to limit the number of papers submitted, and there are two reasons: first, as you said, so they don't have to read them all--time is finite. second, tho, is that they really want to see the best work, and judge from that, not all the other papers that might have been published to give each grad student something for his CV. And it can be worse than salami slicing--it can be what I will call for want of a better metaphor salami squashing, which I define as saying the same thing is slightly differet ways for as many journals as you can--there is a limit, though, since eventually you run into the same referees. And there are side benefits: the fewer papers, the lower the publication costs, no matter who it is that pays. One of the reasons I like a system of OAJournals so much is that they for the first time provide a real incentive to publish in larger chunks. From a large grant, 5 papers a year at $3000 isn't much, but 20 would pay for another post-doc. David Goodman, Ph.D., M.L.S. previously: Bibliographer and Research Librarian Princeton University Library dgoodman at princeton.edu ----- Original Message ----- From: Stevan Harnad Date: Friday, September 29, 2006 5:33 am Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based To: SIGMETRICS at LISTSERV.UTK.EDU > Adminstrative info for SIGMETRICS (for example unsubscribe): > http://web.utk.edu/~gwhitney/sigmetrics.html > > On Fri, 29 Sep 2006, Sally Morris (Chief Executive) wrote: > > > I disagree with Stevan about the advisability of limiting the > number of > > papers which may be submitted. Similar rules were introduced a > few years > > ago in the US - the objective is to discourage authors from a > 'never mind > > the quality, feel the width' mentality which can (and arguably > does) lead to > > excessive publication, via salami-slicing and other (sometimes > even less > > desirable - see http://www.alpsp.org/events/2006/PET/default.htm) > methods > Sally (and no doubt many others) vastly under-rate the power of OA > metrics here: > > (1) Yes salami-slicing is bad. > > (2) But metrics makes it easily detectable, and penalizable, by > differential weighting. > > (3) Example: Researcher A receives a total of 100 citations for 10 > papers, averaging 10 per paper; Researcher B receives a total of 100 > citations for 4 papers, averaging 25 per paper. Easy to give the lower > average a lower weight; as a sub-test, easy to check the citations for > the top four papers too... > > (4) The main idea is to stop wasting time and money re-submitting the > papers (to RAE) and re-reviewing them (by RAE panels) and let the > metrics do the work instead. > > (5) Nor are citations count and averages and top-slicing near being > the only metrics that can enter into the weighted equation: There are > downloads, co-citations (what kind of research/researcher is it cited > *with*), authority metrics (what kind of research/researcher is it > cited*by*), endogamy/exogamy metrics (how incestuous are the > citations, in > the range: self-citations, co-author citations, mutual citation > circles,within-specialty citations, interdisciplinarity), growth > rate of citations > (and downloads), latency and longevity scores, etc. > > (6) All of those metrics can be gathered and weighted, with the > weightsadjusted to the features of the field (some fields are > rapid, narrow > growth, some are slower, broader growth, some more endogamous, some > moreexogamous, etc.). > > (7) "Semantic" (in reality syntactic) content-based metrics can also > measure the degree of textual overlap between papers, both multiple > papers by the same author, and overlap with papers by other authors... > > (8) And that just scratches the surface of OA metric possibilities. > > My own understanding is that restricting RAE submissions to only 4 > papers had been done partly to keep the work of the panel tractable > (mooted now with metrics) and partly to discourage salami-slicing -- > but I know of no evidence whether it *did* discourage salami-slicing > (in either the UK or the US: does anyone have data?): After all, RAE > is not the only fish in the sea, for the author. (Objective > evidence on > whether it had any effect, by the way, would have to be metric!) > > But in a metric RAE, salami-slicing would become its own enemy, > just as > self-citing and plagiarism would be. (And before you mention self- > paddeddownloads, that's readily detectable and name-shameable too, > not only > by checking IPs but via triangulation with other metrics that are > normallycorrelated with downloads, such as citations!) > > Stevan Harnad > > PS Whatever works in the UK, the US will eventually catch up too: > Metrics will be the measure in both cases, validated, as needed, > againstpeer evaluation, the specific needs of a field, and internal > validationthrough triangulation. > > > ----- Original Message ----- > > From: "Stevan Harnad" > > > > > > At the heart of this are not one, not two, not three, but > *four* pieces > > > of patent nonsense so absurd as to take one's breath away. Most > of the > > > nonsense is on RAE/HEFCE's end; one cannot blame the publishers > for play > > > along (especially as the gentleman's agreement holds some hope of > > > forestalling OA a bit longer, or at least the role the RAE > might have > > > played in hastening OA's arrival): > > > > > > (1) The first piece of nonsense is the RAE's pedantic and > > > dysfunctional insistence on laying their hands directly on the > > > "originals," the publisher's version of each article per > author,> > rather than sensibly settling for the author's peer- > reviewed final > > > drafts (postprints). > > > > > > (2) The second is the equally foolish notion that the RAE > somehow> > needs special permission to do this, or, worse, might > even have needed > > > to *pay* for the right, but for this "gentleman's > agreement"! (Of > > > course the publishers are more than happy to play along with > this> > self-imposed farce on RAE's part; but if no one had ever > absurdly> > suggested in the first place that when an author > sends a copy of his > > > own paper to his own funder for evaluation, *he needs his > publisher's> > permission*, none of this nonsense would ever > even have come up!) > > > > > > (3) The idea of restricting submissions to only *four* papers > > > was originally floated by RAE in part out of the hope that > > > this limitation would act as a counterweight against salami- > sliced> > publication. It didn't. And it's time to drop this > absurd, arbitrary > > > limit on what work can be submitted. > > > > > > (4) Of course the other reason the number was kept down to > four was > > > the even more dysfunctional feature of the RAE that is only > now,> > at long last, being deservedly jettisoned (the > submissions and panel > > > reviews themselves!); yet one hand does not seem to be aware > of what > > > the other is doing: For once the unnecessary and time/money- > wasting> > "peer-*re*-reviewing" that the RAE panels had been > trying to > > > do is at last abandoned in favour of metrics, there will be no > > > need for either a 4-item cap or any compulsive attempt to > get the > > > "originals" to the panel. The authors' self-archived postprints > > > in their own institutional OA IRs will suffice (and the only > thing> > the RAE panels -- if there still *are* any RAE panels -- > need do, > > > if suspicious about any particular item, is a database > search (say, > > > in Web of Knowledge or Scopus or PubMed) to make sure that > the item > > > in question did indeed appear in the journal indicated, > under the > > > name of the author in question). > > > > > > What will moot all of this is of course the OA self-archiving > mandates> > by RCUK and the UK universities themselves, which will > fill the UK > > > universities' IRs, which will in their turn -- with the help of > the IRRA > > > http://irra.eprints.org/ -- mediate the submission of both the > postprints> > and the metrics to the RAE. Then this ludicrous side- > show about the > > > "licensing" of the all-important "originals" to the RAE, for "peer > > > re-review" via the mediation of CrossRef and the publishers > will at last > > > be laid to rest, once and for all. > > > > > > RAE 2008 will be its last hurrah... > > > > > > Stevan Harnad > > > > > > > ______________________________________________________________________> > This email has been scanned by the MessageLabs Email Security System. > > > For more information please visit http://www.messagelabs.com/email > > > > ______________________________________________________________________> > > > From harnad at ECS.SOTON.AC.UK Sat Sep 30 02:07:49 2006 From: harnad at ECS.SOTON.AC.UK (Stevan Harnad) Date: Sat, 30 Sep 2006 07:07:49 +0100 Subject: Future UK RAEs to be Metrics-Based In-Reply-To: Message-ID: On Sat, 30 Sep 2006, David Goodman wrote: > If X publishes a project chopped up into 4 short papers at about the > same time, if I'm working in the field I'll cite them all. (0) I can only repeat: if we want to give higher weight to average citations than total citations, light-weight papers can be down-weighted. > It is usual in the US for a tenure committee... to > limit the number of papers submitted... > so they don't have to read them all... > they really want to see the best work (1) This is about UK national RAE, for departments, not US tenure review, for individuals. (2) This is about replacing RAE panel re-review by metrics. There is no earthly reason *metrics* should be restricted to four papers (or restricted in any way: the way to constrain metrics is via weightings). > the fewer papers, the lower the publication > costs, no matter who... pays... I like... > OAJournals [because] they for the first time provide a real > incentive to publish in larger chunks. From a large grant, 5 papers a year > at $3000 isn't much, but 20 would pay for another post-doc. (3) This is about OA self-archiving, not OA journals, and about RAE submission via IRs. Stevan Harnad > ----- Original Message ----- > From: Stevan Harnad > Date: Friday, September 29, 2006 5:33 am > Subject: Re: [SIGMETRICS] Future UK RAEs to be Metrics-Based > To: SIGMETRICS at LISTSERV.UTK.EDU > > > Adminstrative info for SIGMETRICS (for example unsubscribe): > > http://web.utk.edu/~gwhitney/sigmetrics.html > > > > On Fri, 29 Sep 2006, Sally Morris (Chief Executive) wrote: > > > > > I disagree with Stevan about the advisability of limiting the > > number of > > > papers which may be submitted. Similar rules were introduced a > > few years > > > ago in the US - the objective is to discourage authors from a > > 'never mind > > > the quality, feel the width' mentality which can (and arguably > > does) lead to > > > excessive publication, via salami-slicing and other (sometimes > > even less > > > desirable - see http://www.alpsp.org/events/2006/PET/default.htm) > > methods > > Sally (and no doubt many others) vastly under-rate the power of OA > > metrics here: > > > > (1) Yes salami-slicing is bad. > > > > (2) But metrics makes it easily detectable, and penalizable, by > > differential weighting. > > > > (3) Example: Researcher A receives a total of 100 citations for 10 > > papers, averaging 10 per paper; Researcher B receives a total of 100 > > citations for 4 papers, averaging 25 per paper. Easy to give the lower > > average a lower weight; as a sub-test, easy to check the citations for > > the top four papers too... > > > > (4) The main idea is to stop wasting time and money re-submitting the > > papers (to RAE) and re-reviewing them (by RAE panels) and let the > > metrics do the work instead. > > > > (5) Nor are citations count and averages and top-slicing near being > > the only metrics that can enter into the weighted equation: There are > > downloads, co-citations (what kind of research/researcher is it cited > > *with*), authority metrics (what kind of research/researcher is it > > cited*by*), endogamy/exogamy metrics (how incestuous are the > > citations, in > > the range: self-citations, co-author citations, mutual citation > > circles,within-specialty citations, interdisciplinarity), growth > > rate of citations > > (and downloads), latency and longevity scores, etc. > > > > (6) All of those metrics can be gathered and weighted, with the > > weights adjusted to the features of the field (some fields are > > rapid, narrow > > growth, some are slower, broader growth, some more endogamous, some > > more exogamous, etc.). > > > > (7) "Semantic" (in reality syntactic) content-based metrics can also > > measure the degree of textual overlap between papers, both multiple > > papers by the same author, and overlap with papers by other authors... > > > > (8) And that just scratches the surface of OA metric possibilities. > > > > My own understanding is that restricting RAE submissions to only 4 > > papers had been done partly to keep the work of the panel tractable > > (mooted now with metrics) and partly to discourage salami-slicing -- > > but I know of no evidence whether it *did* discourage salami-slicing > > (in either the UK or the US: does anyone have data?): After all, RAE > > is not the only fish in the sea, for the author. (Objective > > evidence on > > whether it had any effect, by the way, would have to be metric!) > > > > But in a metric RAE, salami-slicing would become its own enemy, > > just as > > self-citing and plagiarism would be. (And before you mention self- > > padded downloads, that's readily detectable and name-shameable too, > > not only > > by checking IPs but via triangulation with other metrics that are > > normally correlated with downloads, such as citations!) > > > > Stevan Harnad > > > > PS Whatever works in the UK, the US will eventually catch up too: > > Metrics will be the measure in both cases, validated, as needed, > > against peer evaluation, the specific needs of a field, and internal > > validation through triangulation. > > > > > ----- Original Message ----- > > > From: "Stevan Harnad" > > > > > > > > At the heart of this are not one, not two, not three, but > > *four* pieces > > > > of patent nonsense so absurd as to take one's breath away. Most > > of the > > > > nonsense is on RAE/HEFCE's end; one cannot blame the publishers > > for play > > > > along (especially as the gentleman's agreement holds some hope of > > > > forestalling OA a bit longer, or at least the role the RAE > > might have > > > > played in hastening OA's arrival): > > > > > > > > (1) The first piece of nonsense is the RAE's pedantic and > > > > dysfunctional insistence on laying their hands directly on the > > > > "originals," the publisher's version of each article per > > author,> > rather than sensibly settling for the author's peer- > > reviewed final > > > > drafts (postprints). > > > > > > > > (2) The second is the equally foolish notion that the RAE > > somehow> > needs special permission to do this, or, worse, might > > even have needed > > > > to *pay* for the right, but for this "gentleman's > > agreement"! (Of > > > > course the publishers are more than happy to play along with > > this> > self-imposed farce on RAE's part; but if no one had ever > > absurdly> > suggested in the first place that when an author > > sends a copy of his > > > > own paper to his own funder for evaluation, *he needs his > > publisher's> > permission*, none of this nonsense would ever > > even have come up!) > > > > > > > > (3) The idea of restricting submissions to only *four* papers > > > > was originally floated by RAE in part out of the hope that > > > > this limitation would act as a counterweight against salami- > > sliced> > publication. It didn't. And it's time to drop this > > absurd, arbitrary > > > > limit on what work can be submitted. > > > > > > > > (4) Of course the other reason the number was kept down to > > four was > > > > the even more dysfunctional feature of the RAE that is only > > now,> > at long last, being deservedly jettisoned (the > > submissions and panel > > > > reviews themselves!); yet one hand does not seem to be aware > > of what > > > > the other is doing: For once the unnecessary and time/money- > > wasting> > "peer-*re*-reviewing" that the RAE panels had been > > trying to > > > > do is at last abandoned in favour of metrics, there will be no > > > > need for either a 4-item cap or any compulsive attempt to > > get the > > > > "originals" to the panel. The authors' self-archived postprints > > > > in their own institutional OA IRs will suffice (and the only > > thing> > the RAE panels -- if there still *are* any RAE panels -- > > need do, > > > > if suspicious about any particular item, is a database > > search (say, > > > > in Web of Knowledge or Scopus or PubMed) to make sure that > > the item > > > > in question did indeed appear in the journal indicated, > > under the > > > > name of the author in question). > > > > > > > > What will moot all of this is of course the OA self-archiving > > mandates> > by RCUK and the UK universities themselves, which will > > fill the UK > > > > universities' IRs, which will in their turn -- with the help of > > the IRRA > > > > http://irra.eprints.org/ -- mediate the submission of both the > > postprints> > and the metrics to the RAE. Then this ludicrous side- > > show about the > > > > "licensing" of the all-important "originals" to the RAE, for "peer > > > > re-review" via the mediation of CrossRef and the publishers > > will at last > > > > be laid to rest, once and for all. > > > > > > > > RAE 2008 will be its last hurrah... > > > > > > > > Stevan Harnad > > > > > > > > > > ______________________________________________________________________> > This email has been scanned by the > MessageLabs Email Security System. > > > > For more information please visit http://www.messagelabs.com/email > > > > > > ______________________________________________________________________> > > > > > >